Why Docker is eating your disk space

I learned recently, in less than ideal circumstances, that Docker doesn't do any log rotation by default 😱

So if you're running your application in production with Docker and using Docker's default logging driver you might be awakened some night at 3 AM to fix a service outage when you can barely think straight. 😅

The default logging driver is json-file which caches the container logs as JSON. In practice, this means Docker is writing all the container stdout and stderr to a JSON file located in the host systems /var/lib/docker/containers folder.

You can find out the locations and sizes of these log files with docker inspect. You might need to run this with sudo.

du -h $(docker inspect --format='{{.LogPath}}' $(docker ps -qa))

If you need to delete logs of a container before continuing, you can use echo to overwrite them as empty. This might require stopping the containers first.

echo "" > $(docker inspect --format='{{.LogPath}}' container_name)

Changing the default logging driver

You can change the default logging driver by adding a configuration for it in /etc/docker/daemon.json. Docker supports multiple logging drivers but the simplest solution to our disk gorging logs problem is to use the local driver. The official Docker documentation explicitly refers to using it to prevent disk-exhaustion.

Let's create a basic config for local logging driver.

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
  }
}

We are defining the logging driver with log-driver and setting the maximum size of a log file before it is rolled. This means, when the file reaches 100 megabytes, a new file is created and the old one is archived. max-file is here set to "3", so at any point in time there will be only three log files stored. When the third file reaches 100 megabytes, a new file is created and the oldest log file is deleted.

Next, we need to reload the config for the Docker daemon. The Docker documentation suggests we can use SIGHUP signal to reload the configuration. Unfortunately, this doesn't work on all configuration options with logging driver being one of them.

This means we will have to restart the Docker daemon to reload the config. The command for restarting the daemon depends on the system it's running on. On Linux systems using [systemd] docker daemon can be restarted using the service command.

service docker restart

Now if we check the logging driver currently in use with info, it should print local

docker info --format '{{.LoggingDriver}}'

Now we have changed the default logging driver but if we have running containers this is not enough. The configuration change will only affect new containers. This means we will have to recreate all our existing containers. Depending on your setup you can do this using docker-compose or by running docker build

docker-compose up -d --force-recreate

Further reading

Docker logging drivers
Configuring local driver