7. Administration

This section provides detailed information on how to administer ClusterControl.

7.1. Upgrading ClusterControl

There are several ways to upgrade ClusterControl to the latest version. However, we recommend users to perform Online Upgrade where the instuctions are mostly up-to-date. By default, the ClusterControl UI will notify users of the latest release available via the notification bar, as shown in the following screenshot:

_images/docs_notification_update.png

You can disable the notification at ClusterControl UI > Settings > General Settings > Version > uncheck “Check for updates”. We recommend our users to keep up-to-date with the latest release once it is available. Details of changes in each release can be found in the Change Logs.

7.1.1. Online Upgrade

This is the recommended way to upgrade ClusterControl. The following upgrade procedures require internet connection on ClusterControl node.

7.1.1.1. Redhat/CentOS

Even if you have previously installed from our tarball or .deb packages, it is possible to switch to yum repository and perform the upgrade.

  1. Setup the Severalnines Repository.
  2. This step is only applicable if you are upgrading from earlier than version 1.3.x and you have cluster_id parameter inside /etc/cmon.cnf (usually when you deployed ClusterControl using the deprecated Severalnines Configurator). Before performing the upgrade procedure as described in step #3, create a default /etc/cmon.cnf and move the current configuration into /etc/cmon.d directory. Starting from version 1.3.0, ClusterControl Controller (CMON) handles its configuration files differently, where it expects the /etc/cmon.cnf is the default configuration file containing minimal parameters. The minimal parameters are:
mysql_port=[controller mysql server's mysql port]
mysql_password=[cmon user password]
mysql_hostname=[controller hostname/IP address]
hostname=[controller hostname/IP address]

2a) Create a new CMON configuration directory:

$ mkdir /etc/cmon.d

2b) Check the value of cluster_id in the existing /etc/cmon.cnf:

$ grep cluster_id /etc/cmon.cnf
cluster_id=1

Attention

If the grep command returns nothing, you can skip the remaining sub-steps and proceed to step 3.

2c) Copy the existing cmon.cnf into the configuration directory. The destination must be in cmon_[cluster_id].cnf. For example, if you have cluster_id=1 as retrieved in step 2b, the destination file name is cmon_1.cnf:

$ cp /etc/cmon.cnf /etc/cmon.d/cmon_1.cnf

2d) Edit the existing /etc/cmon.cnf and remove all parameters except the 4 parameters mentioned in step 2, for example:

mysql_port=3306
mysql_password=cmon
mysql_hostname=192.168.1.13
hostname=192.168.1.13

2e) Verify the new hierarchy of CMON configuration files. It should be similar to below:

$ ls -1 /etc/cmon*
/etc/cmon.cnf

/etc/cmon.d:
cmon_1.cnf

Now, we can safely perform the package upgrade as described in the next steps.

Note

Your old configuration file now lives in /etc/cmon.d/cmon_[cluster_id].cnf.

  1. Clear yum cache so it will retrieve the latest repository list and perform the upgrade:
$ yum clean all
$ yum install clustercontrol clustercontrol-cmonapi clustercontrol-controller clustercontrol-nodejs
  1. If you are upgrading from version 1.3.0 or later, you can skip this step. Upgrade the CMON database for ClusterControl controller. When performing an upgrade from an older version, it is compulsory to apply the SQL modification files relative to the upgrade. For example, when upgrading from version 1.2.8 to version 1.3.2, apply all SQL modification files between those versions in sequential order:
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db.sql
$ mysql -f- h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.8-1.2.9.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.9-1.2.10.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.10-1.2.11.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.11-1.2.12.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.12-1.3.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.0-1.3.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.1-1.3.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_data.sql

Attention

ClusterControl 1.3.0 introduces automatic schema upgrade where it will check the CMON DB version upon startup after the upgrade. If the schema version is not as expected, it will perform the import automatically.

  1. Upgrade the dcps database for ClusterControl UI:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/html/clustercontrol/sql/dc-schema.sql
  1. Clear the ClusterControl UI cache:
$ rm -f /var/www/html/clustercontrol/app/tmp/cache/models/*
  1. Restart the CMON controller service:
$ service cmon restart

Upgrade is now complete. Verify the new version at ClusterControl UI > Settings > General Settings > Version or by using command cmon -v. You should re-login if your ClusterControl UI session is active.

7.1.1.2. Debian/Ubuntu

Even if you have previously installed from our tarball or .deb packages, it is possible to switch to apt repository and perform the upgrade.

  1. Setup the Severalnines Repository.
  2. This step is only applicable if you are upgrading from earlier than version 1.3.x and you have cluster_id parameter inside /etc/cmon.cnf (usually when you deployed ClusterControl using the deprecated Severalnines Configurator). Before performing the package upgrade as described in step #3, create a default /etc/cmon.cnf and move the current configuration into /etc/cmon.d directory. Starting from version 1.3.0, ClusterControl Controller (CMON) handles its configuration files differently, where it expects the /etc/cmon.cnf is the default configuration file containing minimal parameters. The minimal parameters are:
mysql_port=[controller mysql server's mysql port]
mysql_password=[cmon user password]
mysql_hostname=[controller hostname/IP address]
hostname=[controller hostname/IP address]

2a) Create a new CMON configuration directory:

$ mkdir /etc/cmon.d

2b) Check the value of cluster_id in the existing /etc/cmon.cnf:

$ grep cluster_id /etc/cmon.cnf
cluster_id=1

Attention

If the grep command returns nothing, you can skip the remaining sub-steps and proceed to step 3.

2c) Copy the existing cmon.cnf into the configuration directory. The destination must be in cmon_[cluster_id].cnf. For example, if you have cluster_id=1 as retrieved in step 2b, the destination file name is cmon_1.cnf:

$ cp /etc/cmon.cnf /etc/cmon.d/cmon_1.cnf

2d) Edit the existing /etc/cmon.cnf and remove all parameters except the 4 parameters mentioned in step 2, for example:

mysql_port=3306
mysql_password=cmon
mysql_hostname=192.168.1.13
hostname=192.168.1.13

2e) Verify the new hierarchy of CMON configuration files. It should be similar to below:

$ ls -1 /etc/cmon*
/etc/cmon.cnf

/etc/cmon.d:
cmon_1.cnf

Now, we can safely perform the package upgrade as described in the next steps.

Note

Your old configuration file now lives in /etc/cmon.d/cmon_[cluster_id].cnf.

  1. Update the repository list and perform the upgrade:
$ sudo apt-get update
$ sudo apt-get install clustercontrol clustercontrol-cmonapi clustercontrol-controller clustercontrol-nodejs
  1. If you are upgrading from version 1.3.0 or later, you can skip this step. Upgrade the CMON database for ClusterControl controller. When performing an upgrade from an older version, it is compulsory to apply the SQL modification files relative to the upgrade. For example, when upgrading from version 1.2.8 to version 1.3.2, apply all SQL modification files between those versions in sequential order:
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db.sql
$ mysql -f- h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.8-1.2.9.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.9-1.2.10.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.10-1.2.11.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.11-1.2.12.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.12-1.3.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.0-1.3.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.1-1.3.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_data.sql

Attention

ClusterControl 1.3.0 introduces automatic schema upgrade where it will check the CMON DB version upon startup after the upgrade. If the schema version is not as expected, it will perform the import automatically.

  1. Upgrade the dcps database for ClusterControl UI:
# For Ubuntu 14.04/Debian 8 or later, where wwwroot is /var/www/html:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/html/clustercontrol/sql/dc-schema.sql
# For Debian 7 and Ubuntu 12.04, where wwwroot is /var/www:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/clustercontrol/sql/dc-schema.sql
  1. Clear the ClusterControl UI cache:
# For Ubuntu 14.04/Debian 8 or later, where wwwroot is /var/www/html:
$ sudo rm -f /var/www/html/clustercontrol/app/tmp/cache/models/*
# For Debian 7 and Ubuntu 12.04, where wwwroot is /var/www:
$ sudo rm -f /var/www/clustercontrol/app/tmp/cache/models/*
  1. Restart the CMON controller service:
$ sudo service cmon restart

Upgrade is now complete. Verify the new version at ClusterControl UI > Settings > General Settings > Version or by using command cmon -v. You should re-login if your ClusterControl UI session is active.

7.1.2. Offline Upgrade

The following upgrade procedures can be performed without internet connection on ClusterControl node. You can get the ClusterControl packages from Severalnines download site.

7.1.2.1. Manual Upgrade

7.1.2.1.1. Redhat/CentOS
  1. Download the latest version of ClusterControl related RPM packages from Severalnines download site:
wget http://severalnines.com/downloads/cmon/clustercontrol-cmonapi-1.3.2-226-x86_64.rpm
wget http://severalnines.com/downloads/cmon/clustercontrol-controller-1.3.2-1391-x86_64.rpm
wget http://severalnines.com/downloads/cmon/clustercontrol-nodejs-1.3.2-73-x86_64.rpm
wget http://severalnines.com/downloads/cmon/clustercontrol-1.3.2-1910-x86_64.rpm
  1. This step is only applicable if you are upgrading to version 1.3.x and you have cluster_id parameter inside /etc/cmon.cnf (usually when you deployed ClusterControl using Severalnines Configurator). Before performing the package upgrade as described in step #3, create a default /etc/cmon.cnf and move the current configuration into /etc/cmon.d directory. Starting from version 1.3.0, ClusterControl Controller (CMON) handles its configuration files differently, where it expects the /etc/cmon.cnf is the default configuration file containing minimal parameters. The minimal parameters are:
mysql_port=[controller mysql server's mysql port]
mysql_password=[cmon user password]
mysql_hostname=[controller hostname/IP address]
hostname=[controller hostname/IP address]

2a) Create a new CMON configuration directory:

$ mkdir /etc/cmon.d

2b) Check the value of cluster_id in the existing /etc/cmon.cnf:

$ grep cluster_id /etc/cmon.cnf
cluster_id=1

Attention

If the grep command returns nothing, you may skip the remaining sub-steps and proceed to step 3.

2c) Copy the existing cmon.cnf into the configuration directory. The destination must be in cmon_[cluster_id].cnf. For example, if you have cluster_id=1 as retrieved in step 2b, the destination file name is cmon_1.cnf:

$ cp /etc/cmon.cnf /etc/cmon.d/cmon_1.cnf

2d) Edit the existing /etc/cmon.cnf and remove all paramaeters except the 4 parameters mentioned in step 2, for example:

mysql_port=3306
mysql_password=cmon
mysql_hostname=192.168.1.13
hostname=192.168.1.13

2e) Verify the new hierarchy of CMON configuration files. It should be similar to below:

$ ls -1 /etc/cmon*
/etc/cmon.cnf

/etc/cmon.d:
cmon_1.cnf

Now, we can safely perform the package upgrade as described in the next steps.

Note

Your old configuration file now lives in /etc/cmon.d/cmon_[cluster_id].cnf.

  1. Install via yum so dependencies are met:
$ yum localinstall clustercontrol-*
  1. If you are upgrading from version 1.3.0 or later, you can skip this step. Upgrade the CMON database for ClusterControl controller. When performing an upgrade from an older version, it is compulsory to apply the SQL modification files relative to the upgrade. For example, when upgrading from version 1.2.8 to version 1.3.2, apply all SQL modification files between those versions in sequential order:
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db.sql
$ mysql -f- h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.8-1.2.9.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.9-1.2.10.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.10-1.2.11.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.11-1.2.12.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.12-1.3.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.0-1.3.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.1-1.3.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_data.sql

Attention

ClusterControl 1.3.0 introduces automatic schema upgrade where it will check the CMON DB version upon startup after the upgrade. If the schema version is not as expected, it will perform the import automatically.

  1. Upgrade the dcps database for ClusterControl UI:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/html/clustercontrol/sql/dc-schema.sql
  1. Clear the ClusterControl UI cache:
$ rm -f /var/www/html/clustercontrol/app/tmp/cache/models/*
  1. Restart the CMON controller service:
$ service cmon restart

Upgrade is now complete. Verify the new version at ClusterControl UI > Settings > General Settings > Version. You should re-login if your ClusterControl UI session is active.

7.1.2.1.2. Debian/Ubuntu

Even if you have previously installed from our tarball or .deb packages, it is possible to switch to apt repository and perform the upgrade.

  1. Download the latest version of ClusterControl related DEB packages from Severalnines download site:
wget http://severalnines.com/downloads/cmon/clustercontrol-nodejs_1.3.2-73_x86_64.deb
wget http://severalnines.com/downloads/cmon/clustercontrol_1.3.2-1910_x86_64.deb
wget http://severalnines.com/downloads/cmon/clustercontrol-cmonapi_1.3.2-226_x86_64.deb
wget http://severalnines.com/downloads/cmon/clustercontrol-controller-1.3.2-1391-x86_64.deb
  1. This step is only applicable if you are upgrading to version 1.3.x and you have cluster_id parameter inside /etc/cmon.cnf (usually when you deployed ClusterControl using Severalnines Configurator). Before performing the package upgrade as described in step #3, create a default /etc/cmon.cnf and move the current configuration into /etc/cmon.d directory. Starting from version 1.3.0, ClusterControl Controller (CMON) handles its configuration files differently, where it expects the /etc/cmon.cnf is the default configuration file containing minimal parameters. The minimal parameters are:
mysql_port=[controller mysql server's mysql port]
mysql_password=[cmon user password]
mysql_hostname=[controller hostname/IP address]
hostname=[controller hostname/IP address]

2a) Create a new CMON configuration directory:

$ mkdir /etc/cmon.d

2b) Check the value of cluster_id in the existing /etc/cmon.cnf:

$ grep cluster_id /etc/cmon.cnf
cluster_id=1

Attention

If the grep command returns nothing, you may skip the remaining sub-steps and proceed to step 3.

2c) Copy the existing cmon.cnf into the configuration directory. The destination must be in cmon_[cluster_id].cnf. For example, if you have cluster_id=1 as retrieved in step 2b, the destination file name is cmon_1.cnf:

$ cp /etc/cmon.cnf /etc/cmon.d/cmon_1.cnf

2d) Edit the existing /etc/cmon.cnf and remove all paramaeters except the 4 parameters mentioned in step 2, for example:

mysql_port=3306
mysql_password=cmon
mysql_hostname=192.168.1.13
hostname=192.168.1.13

2e) Verify the new hierarchy of CMON configuration files. It should be similar to below:

$ ls -1 /etc/cmon*
/etc/cmon.cnf

/etc/cmon.d:
cmon_1.cnf

Now, we can safely perform the package upgrade as described in the next steps.

Note

Your old configuration file now lives in /etc/cmon.d/cmon_[cluster_id].cnf.

  1. Install via dpkg:
$ dpkg -i clustercontrol_1.3.2-1910_x86_64.deb clustercontrol-cmonapi_1.3.2-226_x86_64.deb clustercontrol-controller-1.3.2-1391-x86_64.deb clustercontrol-nodejs_1.3.2-73_x86_64.deb
  1. Upgrade the CMON database for ClusterControl controller. When performing an upgrade from an older version, it is compulsory to apply the SQL modification files relative to the upgrade. For example, when upgrading from version 1.2.8 to version 1.3.2, apply all SQL modification files between those versions in sequential order:
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db.sql
$ mysql -f- h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.8-1.2.9.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.9-1.2.10.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.10-1.2.11.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.11-1.2.12.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.12-1.3.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.0-1.3.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.1-1.3.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_data.sql

Attention

ClusterControl 1.3.0 introduces automatic schema upgrade where it will check the CMON DB version upon startup after the upgrade. If the schema version is not as expected, it will perform the import automatically.

  1. Upgrade the dcps database for ClusterControl UI:
# For Ubuntu 14.04/Debian 8 or later, where wwwroot is /var/www/html:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/html/clustercontrol/sql/dc-schema.sql
# For Debian 7 and Ubuntu 12.04, where wwwroot is /var/www:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/clustercontrol/sql/dc-schema.sql
  1. Clear the ClusterControl UI cache:
# For Ubuntu 14.04/Debian 8 or later, where wwwroot is /var/www/html:
$ sudo rm -f /var/www/html/clustercontrol/app/tmp/cache/models/*
# For Debian and Ubuntu 12.04, where wwwroot is /var/www:
$ sudo rm -f /var/www/clustercontrol/app/tmp/cache/models/*
  1. Restart the CMON controller service:
$ sudo service cmon restart

Upgrade is now complete. Verify the new version at ClusterControl UI > Settings > General Settings > Version. You should re-login if your ClusterControl UI session is active.

7.2. Backing Up ClusterControl

The backup tool in s9s_upgrade_cmon is deprecated. To backup ClusterControl manually, you can use your own method to copy or export following files:

7.2.1. ClusterControl CMON Controller

  • CMON main configuration file: /etc/cmon.cnf
  • CMON configuration directory and all its content: /etc/cmon.d/*
  • CMON cron file: /etc/cron.d/cmon
  • CMON init.d file: /etc/init.d/cmon
  • CMON logfile: /var/log/cmon.log or /var/log/cmon*
  • CMON helper scripts: /usr/bin/s9s_*
  • CMON database dump file:
mysqldump -ucmon -p[mysql_password] -h[mysql_hostname] -P[mysql_port] cmon > cmon_dump.sql

7.2.2. ClusterControl UI

  • ClusterControl upload directory: [wwwroot]/cmon*
  • ClusterControl CMONAPI: [wwwroot]/cmonapi*
  • ClusterControl UI: [wwwroot]/clustercontrol*
  • ClusterControl UI database dump file:
mysqldump -ucmon -p[mysql_password] -h[mysql_hostname] -P[mysql_port] dcps > dcps_dump.sql

Where, [wwwroot] is equal to the Apache document root and [mysql_password], [mysql_hostname], [mysql_port] are values defined in CMON configuration file.

7.3. Restoring ClusterControl

Manual restoration can be performed by reverting the backup action and copying everything back to its original location. Restoration may require you to re-grant the ‘cmon’ user since the backup will not import the grant table of it. Please review the CMON Database section on how to grant the ‘cmon’ user cmon.

7.4. Securing ClusterControl

7.4.1. Firewall and Security Group

If users used Severalnines Configurator to deploy a cluster, the deployment script disables firewalls by default to minimize the possibilities of failure during the cluster deployment. Once it is completed, it is important to secure the ClusterControl node and the database cluster. We recommend user to isolate their database infrastructure from the public Internet and just whitelist the known hosts or networks to connect to the database cluster.

ClusterControl requires ports used by the following services to be opened/enabled:

  • ICMP (echo reply/request)
  • SSH (default is 22)
  • HTTP (default is 80)
  • HTTPS (default is 443)
  • MySQL (default is 3306)
  • CMON RPC (default is 9500)
  • HAproxy statistic page (if HAproxy is installed on ClusterControl node - default is 9600)
  • MySQL load balance through HAproxy (if HAproxy is installed on ClusterControl node - default is 3307 or 33306)
  • MySQL load balance through MaxScale (if MaxScale is installed on ClusterControl node - default is 6033)
  • Streaming port for database backup through netcat (default is 9999)

7.4.2. SSH

SSH is very critical for ClusterControl. It must be possible to SSH from the ClusterControl server to the other nodes in the cluster without password, thus the database nodes must accept the SSH port configured in CMON configuration file. Following best practices are recommended:

  • Permit a very few people in the organization to access to the servers. The fewer the better.
  • Lock down SSH access so it is not possible to SSH into the nodes from any other server than the ClusterControl server.
  • Lock down the ClusterControl server so that it is not possible to SSH into it directly from the outside world.

7.4.3. File Permission

CMON configuration and log files contain sensitive information e.g mysql_password or sudo where it stores user’s password. Ensure CMON configuration file, e.g /etc/cmon.cnf and /etc/cmon.d/cmon_[clusterid].cnf (if exists) have permission 700 while CMON log files, e.g /var/log/cmon.log and /var/log/cmon_[clusterid].log has 740 and both are owned by root.

7.4.4. HTTPS

By default, the installation script installs and configures a self-signed certificate for ClusterControl UI. You can access it by pointing your browser to https://ClusterControl_host/clustercontrol. If you would like to use your own SSL certificate (e.g https://secure.domain.com/clustercontrol), just replace the key and certificate path inside Apache’s SSL configuration file and restart Apache daemon. Make sure the server’s hostname matches with the SSL domain name that you would like to use.

7.5. Running on Custom Port

ClusterControl is configurable to support non-default port for selected services:

7.5.1. SSH

ClusterControl requires same custom SSH port across all nodes in the cluster. Make sure you specified the custom port number in ssh_port option at CMON configuration file, for example:

ssh_port=55055

7.5.2. HTTP or HTTPS

Running HTTP or HTTPS on custom port will change the ClusterControl UI and the CMONAPI URL e.g http://ClusterControl_host:8080/clustercontrol and https://ClusterControl_host:4433/cmonapi. Thus, you may need to re-register the new CMONAPI URL for managed cluster at ClusterControl UI Cluster Registration page.

7.5.3. MySQL

If you are running MySQL for CMON database on different ports, several areas need to be updated:

Area File Example
CMON configuration file /etc/cmon.cnf or /etc/cmon.d/cmon_N.cnf mysql_port=[custom_port]
ClusterControl CMONAPI database setting wwwroot/cmonapi/config/database.php define('DB_PORT', '[custom_port]');
ClusterControl UI database setting wwwroot/clustercontrol/bootstrap.php define('DB_PORT', '[custom_port]');

Note

Where [wwwroot] is equal to the Apache document root and [custom_port] is the MySQL custom port.

7.5.4. HAProxy

By default, HAproxy statistic page will be configured to run on port 9600. To change to another port, change following line in /etc/haproxy/haproxy.cfg:

listen admin_page 0.0.0.0:[your custom port]

Save and restart the HAproxy service.

7.6. Housekeeping

ClusterControl monitoring data will be purged based on the value set at ClusterControl UI > Settings > General Settings > History (default is 7 days). Some users might find this value to be too low for auditing purposes. You can increase the value accordingly however, the longer collected data exist in CMON database, the bigger space it needs. It is recommended to lower the disk space threshold under ClusterControl UI > Settings > Thresholds > Disk Space Utilization so you will get early warning in case CMON database grows significantly.

If you intend to manually purge the monitoring data, you can truncate following tables (recommended to truncate based on the following order):

mysql> TRUNCATE TABLE mysql_advisor_history;
mysql> TRUNCATE TABLE mysql_statistics_tm;
mysql> TRUNCATE TABLE ram_stats_history;
mysql> TRUNCATE TABLE cpu_stats_history;
mysql> TRUNCATE TABLE disk_stats_history;
mysql> TRUNCATE TABLE net_stats_history;
mysql> TRUNCATE TABLE mysql_global_statistics_history;
mysql> TRUNCATE TABLE mysql_statistics_history;
mysql> TRUNCATE TABLE cmon_log_entries;
mysql> TRUNCATE TABLE collected_logs;

The CMON Controller process has internal log rotation scheduling where it will log up to 5 MB in size before archiving /var/log/cmon.log and /var/log/cmon_[cluster id].log. The archived log will be named as cmon.log.1 (or cmon_[cluster id].log.1) sequentially, with up to 9 archived log files (total of 10 log files rotation).

7.7. Migrating IP Address or Hostname

ClusterControl relies on proper IP address or hostname configuration. To migrate to a new set of IP address or hostname, please update the old IP address/hostname occurrences in following files:

  • CMON configuration file: /etc/cmon.cnf and /etc/cmon.d/cmon_N.cnf (hostname and mysql_hostname values)
  • ClusterControl CMONAPI configuration file: [wwwroot]/cmonapi/config/bootstrap.php
  • HAproxy configuration file (if installed): /etc/haproxy/haproxy.cfg

Note

This section does not cover IP address migration for your database nodes. The easiest solution would be to remove the database cluster from ClusterControl UI using Delete Cluster and add it again by using Import Existing Server/Cluster in the deploy dialog.

Next, revoke ‘cmon’ user privileges for old hosts on ClusterControl node and all managed database nodes:

REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'cmon'@'[old ClusterControl IP address or hostname]';

Then, grant cmon user with new IP address or hostname on ClusterControl node and all managed database nodes:

GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'[new ClusterControl IP address or hostname]' IDENTIFIED BY '[mysql password]' WITH GRANT OPTION;
FLUSH PRIVILEGES;

Or, instead of revoke and re-grant, you can just simply update the MySQL user table:

UPDATE mysql.user SET host='[new IP address]' WHERE host='[old IP address]';
FLUSH PRIVILEGES;

Restart CMON service to apply the changes:

service cmon restart

Examine the output of the CMON log file to verify the IP migration status. The CMON Controller should report errors and shut down if it can not connect to the specified database hosts or the CMON database. Once the CMON Controller is started, you can remove the old IP addresses/hostname from the managed host list at ClusterControl > Manage > Hosts.

7.8. Standby ClusterControl Server for High Availability

It is possible to have several ClusterControl servers to monitor a single cluster. This is useful if you have a multi-datacenter cluster and you may need to have ClusterControl on the remote site to monitor and manage the alive nodes if connection between them goes down. However, ClusterControl servers must be configured to be working in active/passive mode to avoid race conditions when digesting queries and recovering failed node or cluster.

In active mode, the ClusterControl node act as a primary controller, where it can perform automatic recovery and parsing MySQL slow log query for query monitoring. The secondary ClusterControl node however must have following things configured:

  • Cluster/Node auto recovery must be turned off.
  • Query sampling must be disabled.

7.8.1. Installing Standby Server

Steps in this section must be performed on the secondary ClusterControl server.

  1. Install ClusterControl as explained in the Getting Started page.
  2. Add the same cluster via ClusterControl > Add Existing Server/Cluster. Ensure you choose “Enable Node AutoRecovery: No” and “Enable Cluster AutoRecovery: No” in the dialog box. Click “Add Cluster” to start the job.
  3. Once the cluster is added, disable query sampling by go to ClusterControl > Settings > Query Monitoring > Sampling Time = -1.

Nothing should be performed on the primary side. The primary ClusterControl server shall perform automatic recovery in case of node or cluster failure.

7.8.2. Failover Method

If you want to make the standby server run in active mode, just do as follow (assume the primary ClusterControl is unreachable at the moment):

  • Cluster/Node auto recovery must be turned on. Click on both red power icons in the summary bar until they appear in green colour.
  • Enable query sampling. Go to ClusterControl > Settings > Query Monitor and change “Sampling Time” to other than “-1”.

That’s it. You should notice that the standby server has taken over the primary role.

7.9. Changing ‘cmon’ or ‘root’ Password

ClusterControl has a helper script to change MySQL root password of your database cluster and for cmon database user called s9s_change_passwd. It requires you to supply the old password so cmon user could access the database nodes and perform password update automatically. This tool is NOT intended for password reset.

On ClusterControl server, get s9s-admin tools from our Github repository:

git clone https://github.com/severalnines/s9s-admin.git

If you have already cloned s9s-admin, it’s important for you to update it first:

cd s9s-admin
git pull

To change password for the ‘cmon’ user:

cd s9s-admin/ccadmin
./s9s_change_passwd --cmon -i1 -p <current cmon password> -n <new cmon password>

To change password for the ‘root’ user:

cd s9s-admin/ccadmin
./s9s_change_passwd --root -i1 -p <cmon password> -o <old root password> -n <new root password>

Warning

The script only supports alpha-numeric characters. Special characters like “$!%?” will not work.

7.10. Uninstall

If ClusterControl is installed on a dedicated host (i.e., not co-located with your application), uninstalling ClusterControl is pretty straightforward. It is enough to bring down the ClusterControl node and revoke the cmon user privileges from the managed database nodes:

REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'cmon'@'[ClusterControl address or hostname]';

If ClusterControl is installed through Severalnines repository, use following command to uninstall via respective package manager:

yum remove -y clustercontrol clustercontrol-cmonapi clustercontrol-controller clustercontrol-nodejs # Redhat/CentOS
sudo apt-get remove -y clustercontrol clustercontrol-cmonapi clustercontrol-controller clustercontrol-nodejs # Debian/Ubuntu

Else, to uninstall ClusterControl Controller manually so you can to re-use the host for other purposes, kill the CMON process and remove all ClusterControl related files and databases:

killall -9 cmon
rm -rf /usr/sbin/cmon
rm -rf /usr/bin/cmon*
rm -rf /usr/bin/s9s_*
rm -rf /usr/local/cmon*
rm -rf /usr/share/cmon*
rm -rf /etc/init.d/cmon
rm -rf /etc/cron.d/cmon
rm -rf /var/log/cmon*
rm -rf /etc/cmon*
rm -rf [wwwroot]/cmon*
rm -rf [wwwroot]/clustercontrol*
rm -rf [wwwroot]/cc-*

For CMON and ClusterControl UI databases and privileges:

DROP SCHEMA cmon;
DROP SCHEMA dcps;
REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'cmon'@'[ClusterControl address or hostname]';
REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'cmon'@'127.0.0.1';

Note

Replace [wwwroot] with value defined in CMON configuration file.