Receive Email Notifications from ClusterControl

As sysadmins and DBAs, we need to be notified whenever something critical happens to our database.  But would it not be nicer if we were informed upfront, and still have time to perform pre-emptive maintenance and retain high availability? Being informed about anomalies or anything that may degrade cluster health and performance is key. In this tips and tricks post, we will explain how you can set up email notifications in ClusterControl and stay up to date with your cluster state.

Email Notification Types in ClusterControl

First we will explain the two types of email notifications that ClusterControl can send. The normal notifications will be sent instantly, once an alert is triggered or an important event occurs. This instant mail type (deliver) is necessary if you wish to immediately receive critical or warning notifications that require swift action.

The other type is called digest, where ClusterControl will accumulate all notifications and then send them each day in a single email on a preset time. Informational and warning notifications, that do not need immediate action can best be sent via the digest email.

Then there is a third option: not to send a notification and ignore the message. This, obviously, should only be configured if you are absolutely certain you don’t wish to receive this type of notification.

Setting up Email Notifications per User

There are two methods for setting up email notifications in ClusterControl. The below is the first one, where you can set the email notifications on a user level. Go to Settings > Email Notifications.

Here you can select an existing user and load it’s current settings. You can change at what time digest emails are to be sent, and to prevent ClusterControl from sending too many emails, what the limit is for the non-digest emails. Be careful: if you set this too low, you will no longer receive notifications for the remainder of the day! Setting this to -1 sets this to unlimited. Per alarm/event category, the email notifications can be set to the notification type necessary.

Keep in mind that this setting is on a global level, so this accounts for all clusters.

Setting up Email Notifications per Cluster

On the cluster level, the notifications can be set for both users and additional email addresses. This interface can be found via Cluster > Settings > General Settings > Email Notifications.

Here you can select an existing user/email address and load it’s current settings. You can change at what time digest emails are to be sent, and to prevent ClusterControl from sending too many emails, what the limit is for the non-digest emails. Again here, if you set this too low, you will no longer receive notifications for the remainder of the day! Setting this to -1 sets this to unlimited. Per alarm/event category the email notifications can be set to the notification type necessary.

Keep in mind all settings are on a cluster specific level, so this only changes settings for the selected cluster.

Adding and Removing Email Addresses

Apart from defining the email notification settings, you can also add new email addresses by clicking on the plus-button. (+) This can be handy if you wish to send notifications to, for example, a distribution list inside your company.

Removing email addresses can be done, by selecting the email address that needs removal and click the minus-button. (-)

Configuring the Mail Server

To be able to send email, you need to tell ClusterControl how to send emails. There are two options: via sendmail or via an SMTP server.

When you make use of sendmail, the server where you have installed ClusterControl should have a local command line mail client installed. ClusterControl will send it’s email using the -r option to set the from-address. As sendmail may not deliver your email reliably, the recommended method of sending email would be via SMTP.

If you decide to use an SMTP server instead, you may need to authenticate against this server. Check with your hosting provider if this is required.

Once set in the first cluster, the mail server settings will be carried over to any new cluster created.

Sending a test email

In the Configure Mail Server interface, you can also send a test email. This will create a backend job, that will send an email to all configured recipients for this cluster under Email Notification Settings.

Troubleshooting

If your test email is not arriving and you have set your mail server settings to sendmail, you can check its workings from the ClusterControl host.

CMON log files

You can check your CMON logfiles and see if the email has been sent.

In /var/log/cmon_.log, you should see something similar to this:

2016-12-09 12:44:11 : (INFO) Executing email job.

If you see a log line like this, you may want to increase the daily message limit:

2016-12-09 12:44:47 : (WARNING) Refusing to send more than 10 messages daily to 'mailto://[email protected]'

As said earlier: if the message limit has been reached, you will no longer receive notifications.

A message about the -r option indicate your mail client does not support the from-header:

2016-12-09 12:44:17 : (WARNING) mail command doesn't support -r SENDER argument, retrying without that.

You can follow this support article to learn how which packages to install.

Sendmail log files

You can also check the local sendmail log files (/var/log/maillog) and see if your email gets delivered. A typical sendmail connection flow looks like the following:

Dec  9 17:36:41 localhost sendmail[24529]: uB9HafLM024529: [email protected], size=326, class=0, nrcpts=1, msgid=<584aeba9.9LBxfOatDgnTC+vm%[email protected]>, relay=root@localhost
Dec  9 17:36:41 localhost postfix/smtpd[24530]: connect from n1[127.0.0.1]
Dec  9 17:36:41 localhost postfix/smtpd[24530]: 2C0AF4094CF9: client=n1[127.0.0.1]
Dec  9 17:36:41 localhost postfix/cleanup[24533]: 2C0AF4094CF9: message-id=<584aeba9.9LBxfOatDgnTC+vm%[email protected]>
Dec  9 17:36:41 localhost sendmail[24529]: uB9HafLM024529: [email protected], [email protected] (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30326, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (Ok: queued as 2C0AF4094CF9)
Dec  9 17:36:41 localhost postfix/qmgr[1256]: 2C0AF4094CF9: from=, size=669, nrcpt=1 (queue active)
Dec  9 17:36:41 localhost postfix/smtpd[24530]: disconnect from n1[127.0.0.1]
Dec  9 17:36:41 localhost postfix/smtp[24534]: 2C0AF4094CF9: to=, relay=mail.yourcompany.com[94.142.240.10]:25, delay=0.38, delays=0.05/0.02/0.08/0.24, dsn=2.0.0, status=sent (250 OK id=1cFP69-0002Ns-Db)

If these entries are not to be found inside the log file, you can increase the loglevel of Sendmail.

Command line email

A final check would be to run the mail command and see if that arrives:

echo "test message" | mail -r [email protected] -s "test subject" [email protected]

If the message from the command line arrives, but the ClusterControl message does not, it may be related to not having set the from-email address in ClusterControl. ClusterControl will then send the email from the default user on the system. If the hostname is not properly set on the ClusterControl host to a fully qualified domain name, this may result in your email server not accepting any emails by an unqualified domain name, or non-existing user.

We hope these tips help you configure notifications in ClusterControl.