Originally posted on Boatswain Blog.

We all need logs!

Sometimes working with Docker makes me feel like working with a black box especially when playing with the Docker image from by the community and it doesn't go the way as expected. In many cases, reading logs takes up a large portion of time for debugging.

This article is about setting up log rotation for Docker containers.

The default logging driver

We could configure different logging driver for containers and by default the stdout and stderr of the container are written in a json file located in /var/lib/docker/containers/[container-id]/[container-id]-json.log. If you leave it unattended, it could takes up a large amount of disk space as shown below.

Purge the log manually

In case this json log file takes up a significant amount of the disk, we could purge it using the following command.

truncate -s 0 <logfile>

We could setup a cronjob to purge these json log files regularly but for long term, it would be better to setup log rotation.

Setup the log rotation

Configure the default logging driver

This could be done by adding the following values in /etc/docker/daemon.json. Create this file if it doesn't exist.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "10"
  }
}

The json-file logging driver has a few more options and we could even change to other logging drivers such as syslog. For more information, please refer to the Docker Docs - Configure logging drivers.

Execute the following commands to reload the updated daemon.json. The new configuration will apply to all newly created container after restart.

$ systemctl daemon-reload

$ systemctl restart docker

Configure the logging driver for a container

The configuration could also be done on container level if you do not want to apply it globally.

The docker run command

We could specify the logging driver and options in the docker run command. For example:

$ docker run \
    --log-driver json-file \
    --log-opt max-size=10m \
    --log-opt max-file=10 \
    alpine echo hello world

Using docker-compose

The logging driver and options could also be configured using docker-compose. For example:

version: '3.2'
services:
  nginx:
    image: 'nginx:latest'
    ports:
      - '80:80'
    logging:
      driver: "json-file"
      options:
        max-size: "1k"
        max-file: "3"

Verify if the setup is working.

Summary

Although the default setting works fine, you never know when the container logs take up all the disk space and that could be avoided by the above few steps. Other than that, logs are important asset which is not only useful when something goes wrong but also contains a lot of hidden values. So never never let the logs go.

If you are looking for a log management SAAS solution, consider using Boatswain and we help you managing all the logs as well as monitoring your Docker servers. 💫

Insufficient facts always invite danger.
-- Spock @ Star Trek