Starting from v1.2.12, ClusterControl consists of four components:
|ClusterControl controller (cmon)||clustercontrol-controller||The brain of ClusterControl. A backend service performing automation, management, monitoring and scheduling tasks. All the collected data will be stored directly inside CMON database|
|ClusterControl REST API ||clustercontrol-cmonapi||Interprets request and response data between ClusterControl UI and CMON database|
|ClusterControl UI||clustercontrol||A modern web user interface to visualize and manage the cluster. It interacts with CMON controller via remote procedure call (RPC) or REST API interface|
|ClusterControl NodeJS||clustercontrol-nodejs||This optional package is introduced in ClusterControl version 1.2.12 to provide an interface for notification services and integration with 3rd party tools|
6.1. ClusterControl Controller (CMON)¶
ClusterControl Controller (CMON) is the core backend process that performs all automation and management procedures. It is usually installed as
/usr/sbin/cmon. It comes with a collection of helper scripts in
/usr/bin directory (prefixed with s9s_) to complete specific tasks. However, some of the scripts have been deprecated due to the corresponding tasks are now being handled by the CMON core process.
CMON controller package is available at Severalnines download site. Redhat-based systems should download and install the RPM package while Debian-based systems should download and extract the DEB package. The package name is formatted as:
- RPM package (Redhat-based systems):
- DEB package (Debian-based systems):
A configuration file
/etc/cmon.cnf is required to initially setup the CMON Controller. It is possible to have several configuration files each for multiple clusters as described in the next sections.
6.1.1. Command Line Arguments¶
By default if you just run
cmon (without any arguments), cmon defaults to run in the background. ClusterControl Controller (cmon) supports several command line options as shown below:
- Print the help.
- Print the manual for configuration parameters. See Configuration Options for details.
- Prints out the version number and build info.
The path of the log file to be used.
- Run in foreground. Ctrl + C to exit.
- Running directory.
- Listen on RPC port. Default is 9500.
- Enable TLS on RPC port. Default is false.
- Bind Remote Procedure Call (RPC) to IP addresses (default is 127.0.0.1,::1). By default cmon binds to ‘127.0.0.1’ and ‘::1’. If another bind-address is needed, then it is possible to define the bind addresses in the file
/etc/default/cmon. The CMON init script translates those RPC_* ones into command line options.
- Try to upgrade the CMON schema (Supported from CMON version 1.2.12 and later).
- Sets the user name to access the CMON database.
- Uses the password to access teh CMON database.
- Access the CMON database on the given host.
- Sets the CMON database name.
- Additional RPC URL where backend sends events.
6.1.2. Configuration File¶
A single CMON Controller process is able to monitor one or more clusters. Each of the cluster requires one exclusive configuration file residing in the
/etc/cmon.d/ directory. For instance, the default CMON configuration file is located at
/etc/cmon.cnf, and commonly used to store the default (minimal) configuration for CMON process to run.
Example of the CMON main configuration file located at
mysql_port=3306 mysql_hostname=127.0.0.1 mysql_password=cm0nP4ss mysql_basedir=/usr hostname=10.0.0.196 logfile=/var/log/cmon.log rpc_key=390faeffb8166277a4f25336a69efa50915635a7
For the first cluster (cluster_id=1), the configuration options should be stored inside
/etc/cmon.d/cmon_1.cnf. For the second cluster, it would be
cluster_id=2 respectively, and so on. The following shows example content of CMON cluster’s configuration file located at
cluster_id=1 cmon_user=cmon created_by_job=1 db_stats_collection_interval=30 enable_query_monitor=1 galera_vendor=codership galera_version=3.x group_owner=1 host_stats_collection_interval=60 hostname=10.0.0.196 logfile=/var/log/cmon_1.log mode=controller monitored_mountpoints=/var/lib/mysql/ monitored_mysql_port=3306 monitored_mysql_root_password=7XU@Wy4nqL9 mysql_bindir=/usr/bin/ mysql_hostname=127.0.0.1 mysql_password=cm0nP4ss mysql_port=3306 mysql_server_addresses=10.0.0.99:3306,10.0.0.253:3306,10.0.0.181:3306 mysql_version=5.6 name='Galera Cluster' os=redhat osuser=root owner=1 pidfile=/var/run basedir=/usr repl_password=9hHRgQLSsZz3Vd4a repl_user=rpl_user rpc_key=3V0RaV6dE8KSyClE ssh_identity=/root/ashrafawskey.pem ssh_port=22 type=galera vendor=codership
An example of CMON configuration file hierarchy is as follows:
|Example cluster||Configuration file||Cluster identifier||Log file location|
|Cluster #1 (Galera)||/etc/cmon.d/cmon_1.cnf||cluster_id=1||logfile=/var/log/cmon_1.log|
|Cluster #2 (MySQL Cluster)||/etc/cmon.d/cmon_2.cnf||cluster_id=2||logfile=/var/log/cmon_2.log|
|Cluster #N (cluster type)||/etc/cmon.d/cmon_N.cnf||cluster_id=N||logfile=/var/log/cmon_N.log|
It’s highly recommendeded to separate CMON logging for each cluster to its own log file. In the above example, we can see that
logfile are two imporant configuration options to distinguish the cluster.
The CMON Controller will import the configuration options defined in each configuration file into the CMON database during process starts up. Once loaded, CMON then use all the loaded information to manage clusters based on the
6.1.3. Configuration Options¶
All of the options and values as described below must not contain any whitespace between them. Any changes to the CMON configuration file requires a CMON service restart before they are applied. The s
The configuration options can be divided into a number of types:
- Operating system
- Nodes (MySQL, MongoDB, PostgreSQL)
- Security & Encryption
Following is the list of common configuration options inside CMON Controller configuration file. You can also see them by using
--help-config parameter in the terminal:
$ cmon --help-config
- Cluster identifier. This will be used by CMON to indicate which cluster to provision. It must be unique, two clusters can not share the same ID.
- Cluster name. The cluster name configured under ClusterControl > DB cluster > Settings > General Settings > Cluster Name precedes this.
- Alias to
- Cluster type. Supported values are galera, mysql_single, mysqlcluster, mongodb, postgresql_single, replication, group_replication.
- Alias to
- The ID of the job created this cluster. This is usually automatically generated by ClusterControl.
- CMON role. Supported values are controller, dual, agent, hostonly.
- CMON controller mode (deprecated). Agents are no longer supported. 0 for agentful or 1 for agentless (default).
logfile=<path to log file>
- CMON log file location. This is where CMON logs its activity. The file will be automatically generated if it doesn’t exist. CMON will write to syslog by default.
pidfile=<path to PID directory>
- CMON process identifier file directory. Keep the default value is recommended.
- The MySQL hostname or IP address where CMON database resides. Using IP address is recommended.
- The MySQL password for user cmon to connect to CMON database. Alphanumeric values only.
- The MySQL port used by CMON to connecto to CMON database.
22.214.171.124. Operating system¶
- Operating system runs across the cluster, including ClusterControl host. ‘redhat’ for Redhat-based distributions (CentOS/Red Hat Enterprise Linux/Oracle Linux) or ‘debian’ for Debian-based distributions (Debian/Ubuntu).
- Operating system user that will be used by CMON to perform automation tasks like cluster recovery, backups and upgrades. This user must be able to perform super-user activities. Using root is recommended.
- Alias to
- Alias to
sudo="echo '<sudo password>' | sudo -S 2>/dev/null"
- The command used to obtain superuser permissions. If sudo user requires password, specify the sudo command with sudo password here. The sudo command must be trimmed by redirecting stderr to somewhere else. Therefore, it is compulsary to have
-S 2>/dev/nullappended in the sudo command.
sudo="echo 'My5ud0' | sudo -S 2>/dev/null"
- Alias to
- Hostname or IP address of the ClusterControl Controller (cmon) host.
wwwroot=<path to CMONAPI and ClusterControl UI>
- Path to CMONAPI and ClusterControl UI. If not set, it defaults to ‘/var/www/html’ for Redhat-based distributions or ‘/var/www’ for Debian-based distributions.
- Database vendor name. ClusterControl needs to know this in order to distinguish the vendor’s relevant naming convention especially for package name, daemon name, deployment steps, recovery procedures and lots more. Supported value at the moment is percona, codership, mariadb, mongodb, oracle.
ssh_identify=<path to SSH key or key pair>
- The SSH key or key pair file that will be used by CMON to connect managed nodes (including ClusterControl node) passwordlessly. If undefined, CMON will use the home directory of
os_userand look for
ssh_keypath=<path to SSH key or key pair>
- Alias to
identity_file=<path to SSH key or key pair>
- Alias to
- The SSH port used by CMON to connect to managed nodes. If undefined, CMON will use port 22.
- The SSH options used by CMON to connect to managed nodes. Details on SSH manual page.
- Setting for libssh: should it acquire a remote tty. Default is 1 (true).
- The SSH password used for connection to nodes.
- Network timeout value in seconds for SSH connections. Default is 30.
- Alias to
- Setting for libssh logging verbosity to stdout. Accepted values are 0 (NONE), 1 (WARN), 2 (INFO), 3 (DEBUG), 4 (TRACE).
monitored_mountpoints=<list of paths to be monitored>
- The MySQL/MongoDB/TokuMX/PostgreSQL data directory used by database nodes for disk performance in comma separated list.
monitored_nics=<list of NICs to be monitored>
- List of network interface card (NIC) to be monitored for network performance in comma separated list.
- Database statistic collections interval in seconds. The lowest value is 1. Default is 30 seconds.
- Host statistic collections interval in seconds. The lowest value is 1. Default is 30 seconds.
- Load balancer stats collection interval. Default is 30.
- How often database growth and table checks are performed in seconds. This translates to information_schema queries. Default is 10800 seconds (3 hours). 0 means disabled.
- Database log files collection interval. Default is 600.
- If a query takes longer than
db_long_query_time_alarmto execute, an alarm will be raised containing detailed information about blocked and long running transactions. Default is 10 seconds.
- Maximum number of database objects that ClusterControl will pull from monitored database nodes.
- Database statistic collections interval in seconds. Default is 5.
- This determine whether ClusterControl should enable MySQL time machine status and variable collections. The status time machine allows you to select status variable for a time range and compare the values at the start and end of that range from ClusterControl UI. Default is 0, meaning it is disabled.
- If undefined, CMON defaults to 0 (false) and will NOT perform automatic recovery if it detects cluster failure. Supported value is 1 (cluster recovery is enabled) or 0 (cluster recovery is disabled).
- If undefined, CMON default to 0 (false) and will NOT perform automatic recovery if it detects node failure. Supported value is 1 (node recovery is enabled) or 0 (node recovery is disabled).
- If undefined, CMON defaults to 0 (false) and will NOT perform automatic recovery if it detects node or cluster failure. Supported value is 0 (cluster and node recovery are disabled) or 1 (cluster and node recovery are enabled). This setting will internally set
enable_cluster_autorecoveryto the specified value.
- The netcat port used to stream backups. Default is 9999.
126.96.36.199. Nodes (MySQL)¶
- Comma separated list of MySQL hostnames or IP addresses (with or without port is supported). For MySQL Cluster, this should be the list of MySQL API node IP addresses.
- Exclusive for MySQL Cluster. Comma separated list of data node hostnames or IP addresses.
- Exclusive for MySQL Cluster. Comma separated list of management node hostnames or IP addresses.
- Exclusive for MySQL Cluster. NDB connection string for the cluster.
- Exclusive for MySQL Cluster. NDB binary for data node. Supported values are ndbd or ndbmtd.
- Exclusive for MySQL Cluster. Directory where configuration files (my.cnf/config.ini) of the cluster is stored.
- MySQL port for the managed cluster. ClusterControl all DB nodes are running on the same MySQL port.
- MySQL root password for the managed cluster. ClusterControl assumes all DB nodes are using the same root password. This is required when you want to scale your cluster by adding a new DB node or replication slave.
mysql_basedir=<MySQL base directory location>
- The MySQL base directory used by CMON to find MySQL client related binaries.
mysql_bindir=<MySQL binary directory location>
- The MySQL binary directory used by CMON to find MySQL client related binaries.
- The MySQL replication user.
- Password for
- Enable/Disable automatic management of the MySQL server
read_onlyvariable. Default is 1 (true), which means ClusterControl will set the
read_only=ONif the MySQL replication role is slave.
- The galera port to be used. Default is 4567.
- Comma separated list of MySQL slaves which should be used as potential master candidates. If this variable is set, only those hosts will be considered. This parameter takes precedence over
- Comma separated list of MySQL slaves which will never be considered a master candidate. You can use it to list slaves that are used for backups or analytical queries. If the hardware varies between slaves, you may want to put here the slaves which use slower hardware.
replication_failover_whitelisttakes precedence over this parameter if it is set.
- Default is 0. Skip the check process for additional missing transactions before promoting a slave to a master and just use the most advanced slave. Such process may result in a serious problem though - if an errant transaction is found, replication may be broken.
- Default is 1. ClusterControl will perform the MySQL master switch only once and will be aborted immediately if the switchover fails, unless the controller is restarted or you specify this variable to 0.
- Default is -1, which means that failover won’t happen if a master candidate is lagging. ClusterControl will wait indefinitely for it to apply all missing transactions from its relay logs. This is safe, but, if for some reason, the most up-to-date slave is lagging badly, failover may takes hours to complete. If set to 0, failover happens immediately, no matter if the master candidate is lagging or not.
188.8.131.52. Nodes (MongoDB)¶
- Comma separated list of MongoDB/TokuMX shard or replica IP addresses with port.
- Comma separated list of MongoDB/TokuMX arbiter IP addresses with port.
- Example: mongoarbiter_server_addresses=192.168.0.11:27019,192.168.0.12:27019,192.168.0.13:27019
- Comma separated list of MongoDB/TokuMX config server IP addresses with port.
- Comma separated list of MongoDB/TokuMX mongos IP addresses with port.
mongodb_basedir=<location MongoDB base directory>
- The MongoDB base directory used by CMON to find mongodb client related binaries.
- MongoDB admin/root username.
- Password for
- The database containing user information to use for authentication. Default is admin.
- The cluster’s nodes authenticating to each other using this key.
184.108.40.206. Nodes (PostgreSQL)¶
- The PostgreSQL node instances.
- Alias to
- The PostgreSQL admin user name. Default is postgres.
- Alias to
- The password used for PostgreSQL user.
- Alias to
220.127.116.11. Encryption and Security¶
- Path to SSL key, for SSL encryption between CMON process and the CMON database.
- Path to SSL certificate, for SSL encryption between CMON process and the CMON database.
- Path to SSL CA, for SSL encryption between CMON process and the CMON database.
- Path to SSL key, for SSL encryption between CMON process and managed MySQL Servers.
- Path to SSL cert, for SSL encryption between CMON process and managed MySQL Servers.
- Path to SSL CA, for SSL encrption between CMON and managed MySQL Servers.
- Path to storage location of SSL related files. This is required when you want to add new node in an encrypted Galera cluster.
- Unique secret token for authentication. To interact with individual cluster via CMON RPC interface (port 9500), one must use this key or else you would get ‘HTTP/1.1 403 Access denied’.
- ClusterControl UI needs this key stored as RPC API Token to communicate with CMON RPC interface. Each cluster should be configured with different
rpc_keyvalue. This value is automatically generated when new cluster/server is created or added into ClusterControl.
Starting from version 1.2.5, ClusterControl introduces an agentless mode of operation. There is now no need to install agents on the managed nodes. User only need to install the CMON controller package on the ClusterControl host, and make sure that passwordless SSH and the CMON database user GRANTs are properly set up on each of the managed hosts.
The agentless mode is the default and recommended type of setup. Starting from version 1.2.9, an agentful setup is no longer supported.
6.1.5. CMON database¶
The CMON Controller requires a MySQL database running on
mysql_hostname as defined in CMON configuration file. The database name and user is ‘cmon’ and is immutable.
The CMON database is the persistent store for all monitoring data collected from the managed nodes, as well as all ClusterControl meta data (e.g. what jobs there are in the queue, backup schedules, backup statuses, etc.). ClusterControl CMONAPI contains logic to query the CMON DB, e.g. for cluster statistics that is presented in the ClusterControl UI.
The CMON database dump files are shipped with the CMON Controller package and can be found under
/usr/share/cmon once it installed. When performing a manual upgrade from an older version, it is compulsory to apply the SQL modification files relative to the upgrade. For example, when upgrading from version 1.2.0 to version 1.2.5, apply all SQL modification files between those versions in sequential order:
Note that there is no 1.2.1 to 1.2.2 SQL modification file. That means there is no changes on the CMON database structure between those versions. The database upgrade procedure will not remove any of the existing data inside the CMON database. You can just use simple MySQL import command as follow:
mysql -f -ucmon -p[cmon_password] -h[mysql_hostname] -P[mysql_port] cmon < /usr/share/cmon/cmon_db.sql mysql -f -ucmon -p[cmon_password] -h[mysql_hostname] -P[mysql_port] cmon < /usr/share/cmon/cmon_data.sql
Replace the variables in square brackets with respective values defined in CMON configuration file.
MySQL user ‘cmon’ needs to have proper access to CMON DB by performing following grant:
Grant all privileges to ‘cmon’ at
hostname value (as defined in CMON configuration file) on ClusterControl host:
GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'[hostname]' IDENTIFIED BY '[mysql_password]' WITH GRANT OPTION;
Grant all privileges for ‘cmon’ at 127.0.0.1 on ClusterControl host:
GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'127.0.0.1' IDENTIFIED BY '[mysql_password]' WITH GRANT OPTION;
For each managed database server, on the managed database server, grant all privileges to cmon at controller’s
hostname value (as defined in CMON configuration file) on each of the managed database host:
GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'[hostname]' IDENTIFIED BY '[mysql_password]' WITH GRANT OPTION;
Don’t forget to run
FLUSH PRIVILEGES on each of the above statement so the grant will be kept after restart. If users deploy using the deployment package generated from the Severalnines Cluster Configurator and installer script, this should be configured correctly.
6.1.6. Database Client¶
For MySQL-based clusters, CMON Controller requires MySQL client to connect to CMON database. This package usually comes by default when installing MySQL server required by CMON database.
For MongoDB/TokuMX cluster, the CMON Controller requires to have both MySQL and MongoDB client packages installed and correctly defined in CMON configuration file on
For PostgreSQL, the CMON controller doesn’t require any PostgreSQL clients installed on the node. All PostgreSQL commands will be executed locally on the managed PostgreSQL node via SSH.
If users deploy using the deployment package generated from the Severalnines Cluster Configurator, this should be configured automatically.
6.2. ClusterControl REST API (CMONAPI)¶
The CMONAPI is a RESTful interface, and exposes all ClusterControl functionality as well as monitoring data stored in the CMON DB. Each CMONAPI connects to one CMON DB instance. Several instances of the ClusterControl UI can connect to one CMONAPI as long as they utilize the correct CMONAPI token and URL. The CMON token is automatically generated during installation and is stored inside
You can generate the CMONAPI token manually by using following command:
python -c 'import uuid; print uuid.uuid4()' | sha1sum | cut -f1 -d' '
By default, the CMONAPI is running on Apache and located under
/var/www/html/cmonapi (Redhat/CentOS/Ubuntu >14.04) or
/var/www/cmonapi (Debian/Ubuntu <14.04). The value is relative to
wwwroot value defined in CMON configuration file. The web server must support rule-based rewrite engine and able to follow symlinks.
The CMONAPI page can be accessed through following URL:
http|https://[ClusterControl IP address or hostname]/cmonapi
Both ClusterControl CMONAPI and UI must be running on the same version to avoid misinterpretation of request and response data. For instance, ClusterControl UI version 1.2.6 needs to connect to the CMONAPI version 1.2.6.
We are gradually in the process of migrating all functionalities in REST API to RPC interface. Kindly expect the REST API to be obselete in the near future.
6.3. ClusterControl UI¶
ClusterControl UI provides a modern web user interface to visualize the cluster and perform tasks like backup scheduling, configuration changes, adding nodes, rolling upgrades, etc. It requires a MySQL database called ‘dcps’, to store cluster information, users, roles and settings. It interacts with CMON controller via remote procedure call (RPC) or REST API interface.
You can install the ClusterControl UI independently on another server by running following command:
yum install clustercontrol # RHEL/CentOS sudo apt-get install clustercontrol # Debian/Ubuntu
Omit ‘sudo’ if you are running as root.
The ClusterControl UI can connect to multiple CMON Controller servers (if they have installed the CMONAPI) and provides a centralized view of the entire database infrastructure. Users just need to register the CMONAPI token and URL for a specific cluster on the Cluster Registrations page.
The ClusterControl UI will load the cluster in the database cluster list, similar to the screenshot below:
Similar to the CMONAPI, the ClusterControl UI is running on Apache and located under
/var/www/html/clustercontrol (Redhat/CentOS/Ubuntu >14.04) or
/var/www/clustercontrol (Debian <8/Ubuntu <14.04). The web server must support rule-based rewrite engine and must be able to follow symlinks.
ClusterControl UI page can be accessed through following URL:
http|https://[ClusterControl IP address or hostname]/clustercontrol
Please refer to User Guide for more details on the functionality available in the ClusterControl UI.
6.4. ClusterControl NodeJS¶
At the time of this writing, Severalnines contributes two NodeJS plugins available at NPM page.
This package works differently if compared to ClusterControl plugin interface, whereby ClusterControl executes the plugin script if only alarm is raised/closed. Alarm’s rules is hardcorded in ClusterControl which is not as dynamic as Advisors. Advisors extends the ClusterControl capability in health checks and notifications, built on top of ClusterControl Domain Specific Language (DSL). Each Advisors will have to be compiled and scheduled directly from ClusterControl’s Developer Studio. The list of scheduled Custom Advisors is available at ClusterControl > Performance > Advisors.
We have future plan to push alarms to NodeJS interface, so NodeJS can push them into a web socket, and all the subscribers (clients) will get those instantly.
|||We are gradually in the process of migrating all functionalities in REST API to RPC interface. Kindly expect the REST API to be obselete in the near future.|