Severalnines Blog
The automation and management blog for open source databases

ClusterControl Tips & Tricks: wtmp Log Rotation Settings for Sudo User


Requires ClusterControl. Applies to all supported database clusters. Applies to all supported operating systems (RHEL/CentOS/Debian/Ubuntu).

ClusterControl requires a super-privileged SSH user to provision database nodes. If you are running as non-root user, the corresponding user must able to execute sudo commands with or without sudo password. Unfortunately, this could generate another issue where performing remote command with “sudo” requires an interactive session (tty). We will explain this in details in the next sections.

What’s up with sudo?

By default, most of the RHEL flavors have the following configured under /etc/sudoers:

Defaults requiretty

When an interactive session (tty) is required, each time the sudo user SSH into the box with -t flag (force pseudo-tty allocation), entries will be created in /var/log/wtmp for the creation and destruction of terminals, or the assignment and release of terminals. These logs only record interactive sessions. If you didn’t specify -t, you would see the following error:

sudo: sorry, you must have a tty to run sudo

The root user does not require an interactive session when running remote SSH command, the entries only appear in /var/log/secure or /var/log/auth.log depending on the system configuration. Different distributions have different defaults in this regards. SSH does not make a file into wtmp if it is a non-interactive session.

To check the content of wtmp, we use the following command:

$ last -f /var/log/wtmp
ec2-user pts/0        ip-10-0-0-79.ap- Wed Oct 28 11:16 - 11:16  (00:00)
ec2-user pts/0        ip-10-0-0-79.ap- Wed Oct 28 11:16 - 11:16  (00:00)
ec2-user pts/0        ip-10-0-0-79.ap- Wed Oct 28 11:16 - 11:16  (00:00)

On Debian/Ubuntu system, sudo user does not need to acquire tty as it defaults to have no “requiretty” configured. However, ClusterControl defaults to append -t flag if it detects the SSH user as a non-root user. Since ClusterControl performs all the monitoring and management tasks as this user, you may notice that /var/log/wtmp will grow rapidly, as shown in the following section.

Possible Symptoms

One symptom of a growing wtmp log is that the accounts-daemon uses an excessive amount of CPU. There is a known Linux bug in the accounts-daemon causing it to re-read the entire tmp log all of the time.

Log rotation for wtmp

Example: Take note of the following default configuration of wtmp in RHEL 7.1 inside /etc/logrotate.conf:

/var/log/wtmp {
    create 0664 root utmp
    minsize 1M
    rotate 1

By running the following commands on one of the database nodes managed by ClusterControl, we can see how fast /var/log/wtmp grows every minute:

[user@server ~]$ a=$(du -b /var/log/wtmp | cut -f1) && sleep 60 && b=$(du -b /var/log/wtmp | cut -f1) && c=$(expr $b - $a ) && echo $c

From the above result, ClusterControl causes the log file to grow 89 KB per minute, which equals to 128MB per day. If the mentioned logrotate configuration is used (monthly rotation), /var/log/wtmp alone may consume 3.97 GB of disk space! If the partition where this file resides (usually under “/” partition) is small (it’s common to have “/” partition smaller, especially if it’s a cloud instance), there is a potential risk you would fill up the disk space on that partition in less than one month. 


The workaround is to play with the log rotation of wtmp. This is applicable to all operating systems mentioned in the beginning of this post. For those who are affected by this, you have to change the log rotation behaviour so it does not grow more than expected. The following is what we recommend:

/var/log/wtmp {
     size 100M
     create 0664 root utmp
     rotate 3

The above settings specify that the maximum size of wtmp should be 100 MB and, and we should keep the 3 most recent (compressed) files and remove older ones.

Logrotate uses crontab (under /etc/cron.daily/logrotate) to work. It is not a daemon so no need to reload its configuration. When the crontab executes logrotate, it will use the new config file automatically.

Happy Clustering!

PS.: To get started with ClusterControl, click here!

Related Post

Automating MySQL Replication with ClusterControl 1.4.0 - what’s new

This blog post will go through new replication features in ClusterControl 1.4.0, including enhanced multi-master deployment, managing replication topology changes, automated failover and handling of replication errors.

Posted in:

Announcing ClusterControl 1.4 - the MySQL Replication & MongoDB Edition

Release of ClusterControl 1.4 with key new features related to MySQL Replication and MongoDB in particular, along with performance improvements and bug fixes.

How to use the ClusterControl Query Monitor for MySQL, MariaDB and Percona Server

Bad query performance is probably the most common problem that a database administrator has to deal with. This blog post shows you how to use ClusterControl to monitor queries on your MySQL, MariaDB or Percona Server.

Posted in:

Secure MongoDB and Protect Yourself from the Ransom Hack

Recently, several attackers were able to break into thousands of MongoDB systems, wipe the databases and leave a ransom note. This could have been prevented if those in charge would have followed some standard security procedures. This blog post describes how to protect yourself from MongoDB ransomware. What is it, why is it a problem and what can you do to protect yourself against it?