7. Administration

This section provides detailed information on how to administer ClusterControl.

7.1. Upgrading ClusterControl

There are several ways to upgrade ClusterControl to the latest version. However, we recommend users to perform Online Upgrade where the instuctions are mostly up-to-date.

7.1.1. Online Upgrade

This is the recommended way to upgrade ClusterControl. The following upgrade procedures require internet connection on ClusterControl node.

7.1.1.1. Redhat/CentOS

  1. Setup the Severalnines Repository.
  2. This step is only applicable if you are upgrading from version older than 1.3.x and you have cluster_id parameter inside /etc/cmon.cnf (usually when you deployed ClusterControl using the deprecated Severalnines Configurator). Before performing the upgrade procedure as described in step #3, create a default /etc/cmon.cnf and move the current configuration into /etc/cmon.d directory. Starting from version 1.3.0, ClusterControl Controller (CMON) handles its configuration files differently, where it expects the /etc/cmon.cnf is the default configuration file containing minimal parameters. The minimal parameters are:
mysql_port={controller mysql server port}
mysql_password={cmon user password}
mysql_hostname={controller hostname/IP address}
hostname={controller hostname/IP address}

2a) Create a new CMON configuration directory:

$ mkdir /etc/cmon.d

2b) Check the value of cluster_id in the existing /etc/cmon.cnf:

$ grep cluster_id /etc/cmon.cnf
cluster_id=1

Attention

If the grep command returns nothing, you can skip the remaining sub-steps and proceed to step 3.

2c) Copy the existing cmon.cnf into the configuration directory. The destination must be in cmon_{cluster_id}.cnf. For example, if you have cluster_id=1 as retrieved in step 2b, the destination file name is cmon_1.cnf:

$ cp /etc/cmon.cnf /etc/cmon.d/cmon_1.cnf

2d) Edit the existing /etc/cmon.cnf and remove all parameters except the 4 parameters mentioned in step 2, for example:

mysql_port=3306
mysql_password=cmon
mysql_hostname=192.168.1.13
hostname=192.168.1.13

2e) Verify the new hierarchy of CMON configuration files. It should be similar to below:

$ ls -1 /etc/cmon*
/etc/cmon.cnf

/etc/cmon.d:
cmon_1.cnf

Now, we can safely perform the package upgrade as described in the next steps.

Note

Your old configuration file now lives in /etc/cmon.d/cmon_{cluster_id}.cnf.

  1. Clear yum cache so it will retrieve the latest repository list and perform the upgrade:
$ yum clean all
$ yum install clustercontrol clustercontrol-cmonapi clustercontrol-controller clustercontrol-ssh clustercontrol-notifications
  1. If you are upgrading from version 1.3.0 or later, you can skip this step. Upgrade the CMON database for ClusterControl controller. When performing an upgrade from an older version, it is compulsory to apply the SQL modification files relative to the upgrade. For example, when upgrading from version 1.2.8 to version 1.4.2, apply all SQL modification files between those versions in sequential order:
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db.sql
$ mysql -f- h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.8-1.2.9.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.9-1.2.10.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.10-1.2.11.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.11-1.2.12.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.12-1.3.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.0-1.3.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.1-1.3.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.2-1.4.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.4.0-1.4.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.4.1-1.4.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_data.sql

Attention

ClusterControl 1.3.0 introduces automatic schema upgrade where it will check the CMON DB version upon startup after the upgrade. If the schema version is not as expected, it will perform the import automatically.

  1. Upgrade the dcps database for ClusterControl UI:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/html/clustercontrol/sql/dc-schema.sql
  1. Clear the ClusterControl UI cache:
$ rm -f /var/www/html/clustercontrol/app/tmp/cache/models/*
  1. Restart the ClusterControl services:

For sysvinit and upstart:

$ service cmon restart
$ service cmon-ssh restart
$ service cmon-events restart

For systemd:

$ systemctl restart cmon
$ systemctl restart cmon-ssh
$ systemctl restart cmon-events

Upgrade is now complete. Verify the new version at ClusterControl UI > Settings > General Settings > Version or by using command cmon -v. You should re-login if your ClusterControl UI session is active.

7.1.1.2. Debian/Ubuntu

  1. Setup the Severalnines Repository.
  2. This step is only applicable if you are upgrading from version older than 1.3.x and you have cluster_id parameter inside /etc/cmon.cnf (usually when you deployed ClusterControl using the deprecated Severalnines Configurator). Before performing the package upgrade as described in step #3, create a default /etc/cmon.cnf and move the current configuration into /etc/cmon.d directory. Starting from version 1.3.0, ClusterControl Controller (CMON) handles its configuration files differently, where it expects the /etc/cmon.cnf is the default configuration file containing minimal parameters. The minimal parameters are:
mysql_port={controller mysql server port}
mysql_password={cmon user password}
mysql_hostname={controller hostname/IP address}
hostname={controller hostname/IP address}

2a) Create a new CMON configuration directory:

$ mkdir /etc/cmon.d

2b) Check the value of cluster_id in the existing /etc/cmon.cnf:

$ grep cluster_id /etc/cmon.cnf
cluster_id=1

Attention

If the grep command returns nothing, you can skip the remaining sub-steps and proceed to step 3.

2c) Copy the existing cmon.cnf into the configuration directory. The destination must be in cmon_{cluster_id}.cnf. For example, if you have cluster_id=1 as retrieved in step 2b, the destination file name is cmon_1.cnf:

$ cp /etc/cmon.cnf /etc/cmon.d/cmon_1.cnf

2d) Edit the existing /etc/cmon.cnf and remove all parameters except the 4 parameters mentioned in step 2, for example:

mysql_port=3306
mysql_password=cmon
mysql_hostname=192.168.1.13
hostname=192.168.1.13

2e) Verify the new hierarchy of CMON configuration files. It should be similar to below:

$ ls -1 /etc/cmon*
/etc/cmon.cnf

/etc/cmon.d:
cmon_1.cnf

Now, we can safely perform the package upgrade as described in the next steps.

Note

Your old configuration file now lives in /etc/cmon.d/cmon_{cluster_id}.cnf.

  1. Update the repository list and perform the upgrade:
$ sudo apt-get update
$ sudo apt-get install clustercontrol clustercontrol-cmonapi clustercontrol-controller clustercontrol-ssh clustercontrol-notifications
  1. If you are upgrading from version 1.3.0 or later, you can skip this step. Upgrade the CMON database for ClusterControl controller. When performing an upgrade from an older version, it is compulsory to apply the SQL modification files relative to the upgrade. For example, when upgrading from version 1.2.8 to version 1.4.2, apply all SQL modification files between those versions in sequential order:
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db.sql
$ mysql -f- h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.8-1.2.9.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.9-1.2.10.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.10-1.2.11.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.11-1.2.12.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.12-1.3.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.0-1.3.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.1-1.3.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.2-1.4.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.4.0-1.4.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.4.1-1.4.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_data.sql

Attention

ClusterControl 1.3.0 introduces automatic schema upgrade where it will check the CMON DB version upon startup after the upgrade. If the schema version is not as expected, it will perform the import automatically.

  1. Upgrade the dcps database for ClusterControl UI:
# For Ubuntu 14.04/Debian 8 or later, where wwwroot is /var/www/html:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/html/clustercontrol/sql/dc-schema.sql
# For Debian 7 and Ubuntu 12.04, where wwwroot is /var/www:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/clustercontrol/sql/dc-schema.sql
  1. Clear the ClusterControl UI cache:
# For Ubuntu 14.04/Debian 8 or later, where wwwroot is /var/www/html:
$ sudo rm -f /var/www/html/clustercontrol/app/tmp/cache/models/*
# For Debian 7 and Ubuntu 12.04, where wwwroot is /var/www:
$ sudo rm -f /var/www/clustercontrol/app/tmp/cache/models/*
  1. Restart the ClusterControl services:

For sysvinit and upstart:

$ service cmon restart
$ service cmon-ssh restart
$ service cmon-events restart

For systemd:

$ systemctl restart cmon
$ systemctl restart cmon-ssh
$ systemctl restart cmon-events

Upgrade is now complete. Verify the new version at ClusterControl UI > Settings > General Settings > Version or by using command cmon -v. You should re-login if your ClusterControl UI session is active.

7.1.2. Offline Upgrade

The following upgrade procedures can be performed without internet connection on ClusterControl node. You can get the ClusterControl packages from Severalnines download site.

7.1.2.1. Manual Upgrade

7.1.2.1.1. Redhat/CentOS
  1. Download the latest version of ClusterControl related RPM packages from Severalnines download site:
$ wget https://severalnines.com/downloads/cmon/clustercontrol-1.4.2-3505-x86_64.rpm
$ wget https://severalnines.com/downloads/cmon/clustercontrol-cmonapi-1.4.2-279-x86_64.rpm
$ wget https://severalnines.com/downloads/cmon/clustercontrol-controller-1.4.2-2013-x86_64.rpm
$ wget https://severalnines.com/downloads/cmon/clustercontrol-notifications-1.4.2-57-x86_64.rpm
$ wget https://severalnines.com/downloads/cmon/clustercontrol-ssh-1.4.2-25-x86_64.rpm
  1. This step is only applicable if you are upgrading from version older than 1.3.x and you have cluster_id parameter inside /etc/cmon.cnf (usually when you deployed ClusterControl using Severalnines Configurator). Before performing the package upgrade as described in step #3, create a default /etc/cmon.cnf and move the current configuration into /etc/cmon.d directory. Starting from version 1.3.0, ClusterControl Controller (CMON) handles its configuration files differently, where it expects the /etc/cmon.cnf is the default configuration file containing minimal parameters. The minimal parameters are:
mysql_port={controller mysql server port}
mysql_password={cmon user password}
mysql_hostname={controller hostname/IP address}
hostname={controller hostname/IP address}

2a) Create a new CMON configuration directory:

$ mkdir /etc/cmon.d

2b) Check the value of cluster_id in the existing /etc/cmon.cnf:

$ grep cluster_id /etc/cmon.cnf
cluster_id=1

Attention

If the grep command returns nothing, you may skip the remaining sub-steps and proceed to step 3.

2c) Copy the existing cmon.cnf into the configuration directory. The destination must be in cmon_{cluster_id}.cnf. For example, if you have cluster_id=1 as retrieved in step 2b, the destination file name is cmon_1.cnf:

$ cp /etc/cmon.cnf /etc/cmon.d/cmon_1.cnf

2d) Edit the existing /etc/cmon.cnf and remove all paramaeters except the 4 parameters mentioned in step 2, for example:

mysql_port=3306
mysql_password=cmon
mysql_hostname=192.168.1.13
hostname=192.168.1.13

2e) Verify the new hierarchy of CMON configuration files. It should be similar to below:

$ ls -1 /etc/cmon*
/etc/cmon.cnf

/etc/cmon.d:
cmon_1.cnf

Now, we can safely perform the package upgrade as described in the next steps.

Note

Your old configuration file now lives in /etc/cmon.d/cmon_{cluster_id}.cnf.

  1. Install via yum so dependencies are met:
$ yum localinstall clustercontrol-*
  1. If you are upgrading from version 1.3.0 or later, you can skip this step. Upgrade the CMON database for ClusterControl controller. When performing an upgrade from an older version, it is compulsory to apply the SQL modification files relative to the upgrade. For example, when upgrading from version 1.2.8 to version 1.4.2, apply all SQL modification files between those versions in sequential order:
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db.sql
$ mysql -f- h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.8-1.2.9.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.9-1.2.10.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.10-1.2.11.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.11-1.2.12.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.12-1.3.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.0-1.3.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.1-1.3.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.2-1.4.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.4.0-1.4.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.4.1-1.4.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_data.sql

Attention

ClusterControl 1.3.0 introduces automatic schema upgrade where it will check the CMON DB version upon startup after the upgrade. If the schema version is not as expected, it will perform the import automatically.

  1. Upgrade the dcps database for ClusterControl UI:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/html/clustercontrol/sql/dc-schema.sql
  1. Clear the ClusterControl UI cache:
$ rm -f /var/www/html/clustercontrol/app/tmp/cache/models/*
  1. Restart the ClusterControl services:

For sysvinit and upstart:

$ service cmon restart
$ service cmon-ssh restart
$ service cmon-events restart

For systemd:

$ systemctl restart cmon
$ systemctl restart cmon-ssh
$ systemctl restart cmon-events

Upgrade is now complete. Verify the new version at ClusterControl UI > Settings > General Settings > Version. You should re-login if your ClusterControl UI session is active.

7.1.2.1.2. Debian/Ubuntu
  1. Download the latest version of ClusterControl related DEB packages from Severalnines download site:
$ wget https://severalnines.com/downloads/cmon/clustercontrol_1.4.2-3505_x86_64.deb
$ wget https://severalnines.com/downloads/cmon/clustercontrol-cmonapi_1.4.2-279_x86_64.deb
$ wget https://severalnines.com/downloads/cmon/clustercontrol-controller-1.4.2-2013-x86_64.deb
$ wget https://severalnines.com/downloads/cmon/clustercontrol-notifications_1.4.2-57_x86_64.deb
$ wget https://severalnines.com/downloads/cmon/clustercontrol-ssh_1.4.2-25_x86_64.deb
  1. This step is only applicable if you are upgrading from version older than 1.3.x and you have cluster_id parameter inside /etc/cmon.cnf (usually when you deployed ClusterControl using Severalnines Configurator). Before performing the package upgrade as described in step #3, create a default /etc/cmon.cnf and move the current configuration into /etc/cmon.d directory. Starting from version 1.3.0, ClusterControl Controller (CMON) handles its configuration files differently, where it expects the /etc/cmon.cnf is the default configuration file containing minimal parameters. The minimal parameters are:
mysql_port={controller mysql server's mysql port}
mysql_password={cmon user password}
mysql_hostname={controller hostname/IP address}
hostname={controller hostname/IP address}

2a) Create a new CMON configuration directory:

$ mkdir /etc/cmon.d

2b) Check the value of cluster_id in the existing /etc/cmon.cnf:

$ grep cluster_id /etc/cmon.cnf
cluster_id=1

Attention

If the grep command returns nothing, you may skip the remaining sub-steps and proceed to step 3.

2c) Copy the existing cmon.cnf into the configuration directory. The destination must be in cmon_{cluster_id}.cnf. For example, if you have cluster_id=1 as retrieved in step 2b, the destination file name is cmon_1.cnf:

$ cp /etc/cmon.cnf /etc/cmon.d/cmon_1.cnf

2d) Edit the existing /etc/cmon.cnf and remove all paramaeters except the 4 parameters mentioned in step 2, for example:

mysql_port=3306
mysql_password=cmon
mysql_hostname=192.168.1.13
hostname=192.168.1.13

2e) Verify the new hierarchy of CMON configuration files. It should be similar to below:

$ ls -1 /etc/cmon*
/etc/cmon.cnf

/etc/cmon.d:
cmon_1.cnf

Now, we can safely perform the package upgrade as described in the next steps.

Note

Your old configuration file now lives in /etc/cmon.d/cmon_{cluster_id}.cnf.

  1. Install via dpkg:
$ dpkg -i clustercontrol_1.4.2-3505_x86_64.deb clustercontrol-cmonapi_1.4.2-279_x86_64.deb clustercontrol-controller-1.4.2-2013-x86_64.deb clustercontrol-notifications_1.4.2-57_x86_64.deb clustercontrol-ssh_1.4.2-25_x86_64.deb
  1. Upgrade the CMON database for ClusterControl controller. When performing an upgrade from an older version, it is compulsory to apply the SQL modification files relative to the upgrade. For example, when upgrading from version 1.2.8 to version 1.4.2, apply all SQL modification files between those versions in sequential order:
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db.sql
$ mysql -f- h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.8-1.2.9.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.9-1.2.10.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.10-1.2.11.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.11-1.2.12.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.2.12-1.3.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.0-1.3.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.1-1.3.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.3.2-1.4.0.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.4.0-1.4.1.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_db_mods-1.4.1-1.4.2.sql
$ mysql -f -h127.0.0.1 -ucmon -p cmon < /usr/share/cmon/cmon_data.sql

Attention

ClusterControl 1.3.0 introduces automatic schema upgrade where it will check the CMON DB version upon startup after the upgrade. If the schema version is not as expected, it will perform the import automatically.

  1. Upgrade the dcps database for ClusterControl UI:
# For Ubuntu 14.04/Debian 8 or later, where wwwroot is /var/www/html:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/html/clustercontrol/sql/dc-schema.sql
# For Debian 7 and Ubuntu 12.04, where wwwroot is /var/www:
$ mysql -f -h127.0.0.1 -ucmon -p dcps < /var/www/clustercontrol/sql/dc-schema.sql
  1. Clear the ClusterControl UI cache:
# For Ubuntu 14.04/Debian 8 or later, where wwwroot is /var/www/html:
$ sudo rm -f /var/www/html/clustercontrol/app/tmp/cache/models/*
# For Debian and Ubuntu 12.04, where wwwroot is /var/www:
$ sudo rm -f /var/www/clustercontrol/app/tmp/cache/models/*
  1. Restart the ClusterControl services:

For sysvinit and upstart:

$ service cmon restart
$ service cmon-ssh restart
$ service cmon-events restart

For systemd:

$ systemctl restart cmon
$ systemctl restart cmon-ssh
$ systemctl restart cmon-events

Upgrade is now complete. Verify the new version at ClusterControl UI > Settings > General Settings > Version. You should re-login if your ClusterControl UI session is active.

7.2. Backing Up ClusterControl

The backup tool in s9s_upgrade_cmon is deprecated. To backup ClusterControl manually, you can use your own method to copy or export following files:

7.2.1. ClusterControl CMON Controller

  • CMON binary: /usr/sbin/cmon
  • CMON SSH binary: /usr/sbin/cmon-ssh
  • CMON Events binary: /usr/sbin/cmon-events
  • CMON main configuration file: /etc/cmon.cnf
  • CMON configuration directory and all its content: /etc/cmon.d/*
  • CMON cron file: /etc/cron.d/cmon
  • CMON init.d file: /etc/init.d/cmon
  • CMON logfile: /var/log/cmon.log or /var/log/cmon*
  • CMON helper scripts: /usr/bin/s9s_*
  • CMON database dump file:
mysqldump -ucmon -p{mysql_password} -h{mysql_hostname} -P{mysql_port} cmon > cmon_dump.sql

7.2.2. ClusterControl UI

  • ClusterControl upload directory: {wwwroot}/cmon*
  • ClusterControl CMONAPI: {wwwroot}/cmonapi*
  • ClusterControl UI: {wwwroot}/clustercontrol*
  • ClusterControl UI database dump file:
mysqldump -ucmon -p{mysql_password} -h{mysql_hostname} -P{mysql_port} dcps > dcps_dump.sql

Where, {wwwroot} is equal to the Apache document root and {mysql_password}, {mysql_hostname}, {mysql_port} are values defined in CMON configuration file.

7.3. Restoring ClusterControl

Manual restoration can be performed by reverting the backup action and copying everything back to its original location. Restoration may require you to re-grant the ‘cmon’ user since the backup will not import the grant table of it. Please review the CMON Database section on how to grant the ‘cmon’ user cmon.

7.4. Securing ClusterControl

7.4.1. Firewall and Security Group

If users used Severalnines Configurator to deploy a cluster, the deployment script disables firewalls by default to minimize the possibilities of failure during the cluster deployment. Once it is completed, it is important to secure the ClusterControl node and the database cluster. We recommend user to isolate their database infrastructure from the public Internet and just whitelist the known hosts or networks to connect to the database cluster.

ClusterControl requires ports used by the following services to be opened/enabled:

  • ICMP (echo reply/request)
  • SSH (default is 22)
  • HTTP (default is 80)
  • HTTPS (default is 443)
  • MySQL (default is 3306)
  • CMON RPC (default is 9500)
  • CMON RPC-TLS (default is 9501)
  • CMON Events (default is 9510)
  • CMON SSH (default is 9511)
  • Streaming port for database backup through netcat (default is 9999)

7.4.2. SSH

SSH is very critical for ClusterControl. It must be possible to SSH from the ClusterControl server to the other nodes in the cluster without password, thus the database nodes must accept the SSH port configured in CMON configuration file. Following best practices are recommended:

  • Permit a very few people in the organization to access to the servers. The fewer the better.
  • Lock down SSH access so it is not possible to SSH into the nodes from any other server than the ClusterControl server.
  • Lock down the ClusterControl server so that it is not possible to SSH into it directly from the outside world.

7.4.3. File Permission

CMON configuration and log files contain sensitive information e.g mysql_password or sudo where it stores user’s password. Ensure CMON configuration file, e.g /etc/cmon.cnf and /etc/cmon.d/cmon_[clusterid].cnf (if exists) have permission 700 while CMON log files, e.g /var/log/cmon.log and /var/log/cmon_[clusterid].log has 740 and both are owned by root.

7.4.4. HTTPS

By default, the installation script installs and configures a self-signed certificate for ClusterControl UI. You can access it by pointing your browser to https://ClusterControl_host/clustercontrol. If you would like to use your own SSL certificate (e.g https://secure.domain.com/clustercontrol), just replace the key and certificate path inside Apache’s SSL configuration file and restart Apache daemon. Make sure the server’s hostname matches with the SSL domain name that you would like to use.

7.5. Running on Custom Port

ClusterControl is configurable to support non-default port for selected services:

7.5.1. SSH

ClusterControl requires same custom SSH port across all nodes in the cluster. Make sure you specified the custom port number in ssh_port option at CMON configuration file, for example:

ssh_port=55055

7.5.2. HTTP or HTTPS

Running HTTP or HTTPS on custom port will change the ClusterControl UI and the CMONAPI URL e.g http://ClusterControl_host:8080/clustercontrol and https://ClusterControl_host:4433/cmonapi. Thus, you may need to re-register the new CMONAPI URL for managed cluster at ClusterControl UI Cluster Registration page.

7.5.3. MySQL

If you are running MySQL for CMON database on different ports, several areas need to be updated:

Area File Example
CMON configuration file /etc/cmon.cnf or /etc/cmon.d/cmon_N.cnf mysql_port={custom_port}
ClusterControl CMONAPI database setting {wwwroot}/cmonapi/config/database.php define('DB_PORT', '{custom_port}');
ClusterControl UI database setting {wwwroot}/clustercontrol/bootstrap.php define('DB_PORT', '{custom_port}');

Note

Where {wwwroot} is the Apache document root and {custom_port} is the MySQL custom port.

7.5.4. HAProxy

By default, HAproxy statistic page will be configured to run on port 9600. To change to another port, change following line in /etc/haproxy/haproxy.cfg:

listen admin_page 0.0.0.0:{your custom port}

Save and restart the HAproxy service.

7.6. Housekeeping

ClusterControl monitoring data will be purged based on the value set at ClusterControl UI > Settings > General Settings > History (default is 7 days). Some users might find this value to be too low for auditing purposes. You can increase the value accordingly however, the longer collected data exist in CMON database, the bigger space it needs. It is recommended to lower the disk space threshold under ClusterControl UI > Settings > Thresholds > Disk Space Utilization so you will get early warning in case CMON database grows significantly.

If you intend to manually purge the monitoring data, you can truncate following tables (recommended to truncate based on the following order):

mysql> TRUNCATE TABLE mysql_advisor_history;
mysql> TRUNCATE TABLE mysql_statistics_tm;
mysql> TRUNCATE TABLE ram_stats_history;
mysql> TRUNCATE TABLE cpu_stats_history;
mysql> TRUNCATE TABLE disk_stats_history;
mysql> TRUNCATE TABLE net_stats_history;
mysql> TRUNCATE TABLE mysql_global_statistics_history;
mysql> TRUNCATE TABLE mysql_statistics_history;
mysql> TRUNCATE TABLE cmon_log_entries;
mysql> TRUNCATE TABLE collected_logs;

The CMON Controller process has internal log rotation scheduling where it will log up to 5 MB in size before archiving /var/log/cmon.log and /var/log/cmon_{cluster id}.log. The archived log will be named as cmon.log.1 (or cmon_{cluster id}.log.1) sequentially, with up to 9 archived log files (total of 10 log files rotation).

7.7. Migrating IP Address or Hostname

ClusterControl relies on proper IP address or hostname configuration. To migrate to a new set of IP address or hostname, please update the old IP address/hostname occurrences in the following files:

  • CMON configuration file: /etc/cmon.cnf and /etc/cmon.d/cmon_N.cnf (hostname and mysql_hostname values)
  • ClusterControl CMONAPI configuration file: {wwwroot}/cmonapi/config/bootstrap.php
  • HAproxy configuration file (if installed): /etc/haproxy/haproxy.cfg

Note

This section does not cover IP address migration of your database nodes. The easiest solution would be to remove the database cluster from ClusterControl UI using Delete Cluster and add it again by using Import Existing Server/Cluster in the deployment dialog.

Next, revoke ‘cmon’ user privileges for old hosts on ClusterControl node and all managed database nodes:

mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'cmon'@'{old ClusterControl IP address or hostname}';

Then, grant cmon user with new IP address or hostname on ClusterControl node and all managed database nodes:

mysql> GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'{new ClusterControl IP address or hostname}' IDENTIFIED BY '{mysql password}' WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES;

Or, instead of revoke and re-grant, you can just simply update the MySQL user table:

mysql> UPDATE mysql.user SET host='{new IP address}' WHERE host='{old IP address}';
mysql> FLUSH PRIVILEGES;

Restart CMON service to apply the changes:

$ service cmon restart

Examine the output of the CMON log file to verify the IP migration status. The CMON Controller should report errors and shut down if it can not connect to the specified database hosts or the CMON database. Once the CMON Controller is started, you can remove the old IP addresses/hostname from the managed host list at ClusterControl > Manage > Hosts.

7.8. Standby ClusterControl Server for High Availability

It is possible to have several ClusterControl servers to monitor a single cluster. This is useful if you have a multi-datacenter cluster and you may need to have ClusterControl on the remote site to monitor and manage the alive nodes if connection between them goes down. However, ClusterControl servers must be configured to be working in active/passive mode to avoid race conditions when digesting queries and recovering failed node or cluster.

In active mode, the ClusterControl node act as a primary controller, where it can perform automatic recovery and parsing MySQL slow log query for query monitoring. The secondary ClusterControl node however must have following things configured:

  • Cluster/Node auto recovery must be turned off.
  • Query sampling must be disabled.

7.8.1. Installing Standby Server

Steps in this section must be performed on the secondary ClusterControl server.

  1. Install ClusterControl as explained in the Getting Started page.
  2. Add the same cluster via ClusterControl > Add Existing Server/Cluster. Ensure you choose “Enable Node AutoRecovery: No” and “Enable Cluster AutoRecovery: No” in the dialog box. Click “Add Cluster” to start the job.
  3. Once the cluster is added, disable query sampling by go to ClusterControl > Settings > Query Monitoring > Sampling Time = -1.

Nothing should be performed on the primary side. The primary ClusterControl server shall perform automatic recovery in case of node or cluster failure.

7.8.2. Failover Method

If you want to make the standby server run in active mode, just do as follow (assume the primary ClusterControl is unreachable at the moment):

  • Cluster/Node auto recovery must be turned on. Click on both red power icons in the summary bar until they appear in green colour.
  • Enable query sampling. Go to ClusterControl > Settings > Query Monitor and change “Sampling Time” to other than “-1”.

That’s it. You should notice that the standby server has taken over the primary role.

7.9. Changing ‘cmon’ or ‘root’ Password

ClusterControl has a helper script to change MySQL root password of your database cluster and for cmon database user called s9s_change_passwd. It requires you to supply the old password so cmon user could access the database nodes and perform password update automatically. This tool is NOT intended for password reset.

On ClusterControl server, get s9s-admin tools from our Github repository:

$ git clone https://github.com/severalnines/s9s-admin.git

If you have already cloned s9s-admin, it’s important for you to update it first:

$ cd s9s-admin
$ git pull

To change password for the ‘cmon’ user:

$ cd s9s-admin/ccadmin
$ ./s9s_change_passwd --cmon -i1 -p <current cmon password> -n <new cmon password>

To change password for the ‘root’ user:

$ cd s9s-admin/ccadmin
$ ./s9s_change_passwd --root -i1 -p <cmon password> -o <old root password> -n <new root password>

Warning

The script only supports alpha-numeric characters. Special characters like “$!%?” will not work.

7.10. Uninstall

If ClusterControl is installed on a dedicated host (i.e., not co-located with your application), uninstalling ClusterControl is pretty straightforward. It is enough to bring down the ClusterControl node and revoke the cmon user privileges from the managed database nodes:

mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'cmon'@'[ClusterControl address or hostname]';

If ClusterControl is installed through Severalnines repository, use following command to uninstall via respective package manager:

$ yum remove -y clustercontrol clustercontrol-cmonapi clustercontrol-controller clustercontrol-ssh clustercontrol-notifications # Redhat/CentOS
$ sudo apt-get remove -y clustercontrol clustercontrol-cmonapi clustercontrol-controller clustercontrol-ssh clustercontrol-notifications # Debian/Ubuntu

Else, to uninstall ClusterControl Controller manually so you can to re-use the host for other purposes, kill the CMON process and remove all ClusterControl related files and databases:

$ killall -9 cmon
$ rm -rf /usr/sbin/cmon*
$ rm -rf /usr/bin/cmon*
$ rm -rf /usr/bin/s9s_*
$ rm -rf /usr/share/cmon*
$ rm -rf /etc/init.d/cmon
$ rm -rf /etc/cron.d/cmon
$ rm -rf /var/log/cmon*
$ rm -rf /etc/cmon*
$ rm -rf {wwwroot}/cmon*
$ rm -rf {wwwroot}/clustercontrol*
$ rm -rf {wwwroot}/cc-*

For CMON and ClusterControl UI databases and privileges:

mysql> DROP SCHEMA cmon;
mysql> DROP SCHEMA dcps;
mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'cmon'@'{ClusterControl address or hostname}';
mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'cmon'@'127.0.0.1';

Note

Replace {wwwroot} with value defined in CMON configuration file.