Managing MySQL Server Logs: Rotate, Compress, Retain & Delete

MySQL Server generates several logs that can help you monitor the activities of the server. However, once these logs are enabled, they can grow in size and start taking up too much disk space. This is why it’s important to have an automated way of archiving and preserving MySQL log files for a certain duration, as well as deleting the old ones. In this blog post, we describe some best practices for setting up and managing MySQL error logs, general logs and slow query logs for your MySQL deployments.

Setting Up MySQL Server Logging

Let’s look at how to setup the following 3 types of logs:

Error Log

Logs all the problems encountered during starting, running, or stopping mysqld. This log can be enabled by having the following option in /etc/my.cnf file:

log_error=/var/log/mysql/mysqld.log

General Query Log

Logs established client connections and statements received from clients. This log can be enabled by having the following option in /etc/my.cnf file:

general_log=ON
general_log_file=/var/log/mysql/general.log

Slow Query Log

Logs queries that took more than long_query_time seconds to execute. This log can be enabled by the following option in /etc/my.cnf file:

slow_query_log=ON
slow_query_log_file=/var/log/mysql/mysql-slowquery.log

Setting Up Criteria For Log Rotation

As an example, let’s have some criteria for managing general MySQL query logs. We can come up with a suitable set of criteria for log management by asking the following questions:

Q: What is the maximum size that the log file can grow?

A: Let’s say it can grow up to 300 MB after which it needs to be rotated and compressed.

Q: What is the frequency that you want the log file to be rotated?

A: We can say that we want logs to be rotated on a daily basis.

Q: How many old log files you want to retain?

A: We would like to retain the last 30 log files.

Based on the above criteria, the overall disk space required for general query log management is about 1.2 GB. Assuming a 90% compression ratio - we will have 30 compressed log files of size 30 MB each and a live log file of about 300 MB.

Managing The Logs Using Linux logrotate Utility

logrotate is a Linux utility that helps with the efficient administration of log files and provides options for automatic rotation, compression, and removal of log files. The criteria established above can be configured for logrotate utility by creating a configuration file in the /etc/logrotate.d folder.

Let’s call this configuration file mysqlgeneral and the contents of the file will be:

/var/log/mysql/general.log{
        compress
        dateext
        maxsize 300M
        copytruncate
        maxage 365
        dateformat -%Y%m%d%s
        daily
        rotate 30
        notifempty
}

With the above options for logrotate, the general query logs get rotated either on a daily basis or when the log file size exceeds 300 MB. The old logs are compressed and 30 such files will be preserved. Log rotation will be skipped if the log file is empty due to the setting ‘notifempty’.

The ‘copytruncate’ option is to ensure that current log file is never deleted during rotation and only its contents get truncated. This is important since some applications expect that the log file is always available and it’s not possible to delete the log without stopping the application first.

Now that the log rotation configuration is set for the general query log, the logrotate utility has to be run so that the above configuration is executed. This is typically done through a cron job. We can set this to be running every hour by placing the logrotate script in /etc/cron.hourly directory:

#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

So, with a few simple steps, we have set up log rotation for MySQL general logs based on our criteria. The same approach can be followed for MySQL error logs and slow query logs as well. Check out these other posts to learn more about optimizing your MySQL deployments:

Blog