How to deploy high availability CloudStack with MariaDB Galera Cluster

Apache CloudStack is a comprehensive orchestration solution designed to deploy Infrastructure as a Service (IaaS) clouds. As cloud infrastructures grow to encompass hundreds or even thousands of servers — often running various applications and services in production — ensuring high availability becomes essential. 

Similar to OpenStack, CloudStack’s management server can be set up in a multi-node configuration. However, because it relies on a MySQL database to maintain the state of its components, the database represents a potential single point of failure. The official CloudStack documentation suggests using MySQL replication with manual failover to address this risk. However, we believe there is a more effective approach.

This blog post demonstrates how to set up redundant CloudStack management servers using a MariaDB Galera Cluster on Rocky Linux 8 64-bit. The architecture includes two load balancer nodes in front of the management and database servers. 

This approach allows for leveraging the high availability features of MariaDB Galera Cluster (such as automatic failover, real-time data consistency across nodes, and no replication lag), while sidestepping its limitations by avoiding concurrent access to multiple nodes. The deployment involves a three-node Galera Cluster using ClusterControl.

Architecture & prerequisites

The architecture involves both the database servers and the CloudStack Management Servers, outlining how they are interconnected, configured for high availability, and optimized for performance and scalability as shown below:

Please note that this blog post does not include instructions for installing the hypervisor and storage hosts. Our setup includes 4 servers configured as follows:

• ClusterControl: ClusterControl host
• mgmn 1: CloudStack Management
• mgmn 2: CloudStack Management
• DB Node 1: Database Server 
• DB Node  2: Database Server, HAProxy, and Keepalived
• DB Node 3: Database Server,  HAProxy, and Keepalived (backup)

Our main steps will be:

1. Prepare 6 hosts
2. Install ClusterControl
3. Deploy 3 MariaDB Galera Cluster 11.4 
4. Configure Keepalived and HAProxy for database and CloudStack load balancing
5. Install CloudStack Management #1
6. Install CloudStack Management #2

Preparing your hosts

1. Add the following hosts definition in /etc/hosts of all nodes:

10.10.10.10	clustercontrol
10.10.10.20 	virtual-ip mgm.cloustack.local 
10.10.10.11 	mariadb1 lb1 haproxy1
10.10.10.12 	mariadb2 lb1 haproxy2
10.10.10.13    mariadb3
10.10.10.14 	mgm1.cloudstack.local mgmn1
10.10.10.15 	mgm2.cloudstack.local mgmn2

2. Install NTP daemon (Chrony):

Accurate time synchronization across your cloud servers is essential, making NTP configuration a critical step. Since NTP isn’t installed by default, we’ll proceed with its installation Chrony, and setup now. To install Chrony, follow these steps:

$ dnf -y install chrony
$ systemctl enable chrony
$ systemctl start chrony

3. Ensure each host is using a valid FQDN, for example on mgm1:

$ hostname --fqdn mgm1.cloudstrack.local

4. Disable Firewall & SELinux

To simplify the setup, we’ll disable the firewall to prevent it from blocking connections. This can be done by running the following two commands:

$ systemctl stop firewalld
$ systemctll disable firewalld

To set SELinux to permissive mode on the running system, execute the following command:

$ setenforce 0

To make this change persistent, update the /etc/selinux/config file to set SELinux to permissive mode, as shown in the example below:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

Installing ClusterControl

On the ClusterControl server, run the following commands:

$ wget https://severalnines.com/downloads/cmon/install-cc
$ chmod +x install-cc
$ ./install-cc

It will install the ClusterControl in the server. After the installation completes, open your web browser to https://{ClusterControl_host}/ and create the default admin user by specifying a username (username “admin” is reserved) and password on the welcome page.

Deploying MariaDB Galera Cluster

The deployment of Percona XtraDB Cluster via ClusterControl’s GUI is straightforward:

1. Go to the top right corner of the ClusterControl UI, and click Deploy a Cluster.

2. There will be a dialogue box with the option Create a database cluster or Import a database cluster — select ‘Create a Database Cluster’.

3. Enter the cluster details, i.e. name and tags

4. Enter the SSH configuration, including the SSH user and password

5. Enter the node configuration, including the admin/root password, server port and data directory if desired

6. Enter the node details, i.e. their IP addresses

7. The final page before ClusterControl deploys MariaDB Galera Cluster 11.4 is a preview page. Please review and verify that all options and values are correct.

ClusterControl will trigger a job to deploy MariaDB Galera Cluster 11.4.

CloudStack requires certain parameters to be adjusted under the [mysqld] section in /etc/my.cnf, as shown below:

innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'

Make the changes in ClusterControl by navigating to Manage -> Configuration, as shown below:

Installing HAProxy and Keepalived

HAProxy and Keepalived are commonly used together to provide high availability (HA) for databases in a cluster setup. HAProxy is a powerful load balancer and proxy server that can distribute incoming database traffic across multiple database nodes, ensuring that the traffic is balanced and that no single node becomes a bottleneck. It’s used for efficient load distribution and high availability by routing requests only to healthy database nodes. 

On the other hand, Keepalived provides failover and redundancy for the service itself. It manages IP failover between nodes, allowing a virtual IP Address to float between the nodes in the event of a failure. When combined, HAProxy and Keepalived work together to ensure that database services remain highly available and resilient, even in the event of server failures, by automatically redirecting traffic to the remaining healthy nodes.

The deployment of HAProxy in ClusterControl can be achieved as follows:

1. Go to the cluster menu in MariaDB Galera Cluster 11.4, then select Action -> Add new -> Load Balancer.

2. A dialog box will appear for creating or importing a balancer. Select the ‘Create Balancer’ option.

3. ClusterControl supports various load balancers, such as HAProxy, ProxySQL, and MaxScale. Select the HAProxy.

4. Select the IP address of the existing node, then continue.

5. Define the admin HAProxy password and any parameters if you want to customize them.

6. Include all the database server nodes in the load balancer.

7. A preview page will appear before ClusterControl begins the installation. Wait until it finishes.

Repeat the installation steps to create another HAProxy service.

After successfully deploying HAProxy, we now need to configure Keepalived with a Virtual IP address, so that the CloudStack Management Server can later connect to the database using the Virtual IP address.

In the same menu as HAProxy, select the Keepalived service instead.

1. Select the Keepalived service

2. Select the load balancer type and choose the existing HAProxy service.

3. Enter the Virtual IP address and the network interface.

4. The final page is a preview page before proceeding with the deployment of Keepalived through ClusterControl.

In addition to configuring load balancing for the database cluster, we also need to set it up for the application, as we are using the same HAProxy.We also need to add the load balancing definition for CloudStack. According to the documentation, we need to load balance port 8080 and 8025. To allow session stickiness, we will use source load balancing algorithm, where the same source address will be forwarded to the same management server unless it fails. On lb1 and lb2, open /etc/haproxy/haproxy.cfg and add the following lines:

frontend cloudstack_ui_8080
	bind *:8080 
	mode http 
	option httpchk OPTIONS /client 
	option forwardfor 
	option httplog 
	balance source 
	server mgmn1.cloudstack.local 10.10.10.14:8080 maxconn 32 check inter 5000 
	server mgm2.cloudstack.local 10.10.10.15:8080 maxconn 32 check inter 5000 

frontend cloudstack_systemvm_8250 
	bind *:8250 
	mode tcp 
	balance source 
	server mgmn1.cloudstack.local 10.10.10.14:8250 maxconn 32 check 
	server mgmn2.cloudstack.local 10.10.10.15:8250 maxconn 32 check

 

Installing CloudStack Management Server #1

** The following steps should be performed on mgmn1 (IP address: 10.10.10.14).

1. Add the CloudStack repository, create /etc/yum.repos.d/cloudstack.repo and insert the following information.

[cloudstack]
name=cloudstack
baseurl=http://download.cloudstack.org/centos/$releasever/4.20/
enabled=1
gpgcheck=0

2. Install the CloudStack Management server:

# dnf -y install cloudstack-management

3. Create a root user with a wildcard host for the CloudStack database setup. On mariadb1 (IP Address: 10.10.10.11), run the following statement:

# mysql -u root -p

And execute the following statements:

MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO root@'%' IDENTIFIED BY 'password' WITH GRANT OPTION;
MariaDB [(none)]> FLUSH PRIVILEGES;

4. On mgmn1, run the following command to configure the CloudStack databases:

# cloudstack-setup-databases
cloudstack:<pwd-cloudstack-user>@10.10.10.20
--deploy-as=root:<pwd-root-user>

You will see process logs similar to the ones below:

Mysql user name:cloudstack                                                      
[ OK ]
Mysql user password:******                                                      
[ OK ]
Mysql server ip:localhost                                                       
[ OK ]
Mysql server port:3306                                                          
[ OK ]
Mysql root user name:root                                                       
[ OK ]
Mysql root user password:******                                                 
[ OK ]
Checking Cloud database files ...                                               
[ OK ]
Checking local machine hostname ...                                             
[ OK ]
Checking SELinux setup ...                                                      
[ OK ]
Detected IP address as 10.10.10.20, will use as cluster management server node IP
[ OK ]
Preparing /etc/cloudstack/management/db.properties                              
[ OK ]
Applying /usr/share/cloudstack-management/setup/create-database.sql             
[ OK ]
Applying /usr/share/cloudstack-management/setup/create-schema.sql               
[ OK ]
Applying /usr/share/cloudstack-management/setup/create-database-premium.sql     
[ OK ]
Applying /usr/share/cloudstack-management/setup/create-schema-premium.sql       
[ OK ]
Applying /usr/share/cloudstack-management/setup/server-setup.sql                
[ OK ]
Applying /usr/share/cloudstack-management/setup/templates.sql                   
[ OK ]
Processing encryption ...                                                       
[ OK ]
Finalizing setup ...                                                            
[ OK ]

5. Setup the CloudStack management application:

# cloudstack-setup-management

Wait until the following message appears:

Starting to configure CloudStack Management Server:
Configure CloudStack Management Server ...[OK]
CloudStack Management Server setup is Done!
Please ensure ports 8080, 8250, 8443, and 9090 are opened and not firewalled for the management server and not in use by other processes on this host.

6. We will use the same server for CloudStack Agent, install the NFS server to store the System Template Setup.

# dnf -y install nfs-utils

We now need to set up NFS to provide access to two separate shares. This configuration is managed through the /etc/exports file. Make sure the file contains the following entries:

/export/secondary 
*(rw,async,no_root_squash,no_subtree_check)
/export/primary 
*(rw,async,no_root_squash,no_subtree_check)

And then create the directories:

# mkdir -p /export/primary
# mkdir /export/secondary

NFSv4 requires that the domain settings are consistent across all clients. In our setup, the domain is set to “local,” so make sure the domain configuration in /etc/idmapd.conf is uncommented and defined as shown below:

Domain = local

Start the services:

# systemctl enable rpcbind
# systemctl enable nfs-server
# systemctl start rpcbind
# systemctl start nfs-server

7. Install the System Templates

CloudStack relies on several system virtual machines to deliver key functionalities, including console access for instances, networking services, and storage management.

We need to download the systemVM template and upload it to the secondary storage. Since we’re working directly on the NFS server, we’ll use the local path /export/secondary. If you were on a different system, you would first mount the secondary storage to a temporary directory and use that mount point instead of /export/secondary.

/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /export/secondary -u http://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.0-x86_64-kvm.qcow2.bz2 -h kvm -F

8. CloudStack Agent installation

After we have configured the System Templates in the shared storage, we need to install the KVM agent:

# dnf -y install cloudstack-agent

9. KVM Configuration

KVM consists of two components that need to be configured: libvirt and QEMU.

We need to edit the QEMU configuration by uncommenting the following parameter in the file /etc/libvirt/qemu.conf 

vnc_listen=0.0.0.0

CloudStack relies on libvirt to manage instances, so it’s essential to ensure that libvirt is properly configured. Since libvirt is a dependency of cloud-agent, it should already be installed.

To enable live migration, libvirt must be configured to accept unsecured TCP connections. Additionally, we need to disable libvirt‘s use of Multicast DNS advertising. Both of these settings can be adjusted in the /etc/libvirt/libvirtd.conf file.

conf 

listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
auth_tcp = "none"
mdns_adv = 0

Enabling “listen_tcp” in libvirtd.conf alone isn’t sufficient—we also need to adjust additional parameters and update the /etc/sysconfig/libvirtd file.

#LIBVIRTD_ARGS="--listen"

We’ll need to apply socket masking (Redhat/Rocky Linux 8):

# systemctl mask libvirtd.socket libvirtd-ro.socket 
libvirtd-admin.socket libvirtd-tls.socket 
libvirtd-tcp.socket

The last thing to do is restart libvirt

# systemctl restart libvirtd

10. Open the CloudStack management UI at virtual IP, http://10.10.10.14:8080/client/ with default user ‘admin’ and password ‘password’. Configure your CloudStack environment by following the deployment wizard and let CloudStack build the infrastructure:

The installation of the first management server is now complete. We’ll now proceed with the second management server.

Installing CloudStack Management Server #2

** The following steps should be performed on mgmn2 (IP Address: 10.10.10.15)

1. Add the CloudStack repository, create /etc/yum.repos.d/cloudstack.repo and insert the following information.

[cloudstack]
name=cloudstack
baseurl=http://download.cloudstack.org/centos/$releasever/4.20/
enabled=1
gpgcheck=0

2. Install the CloudStack Management server: 

$ dnf -y install cloudstack-management

3. Run the following command to setup the CloudStack database (note the absence of –deploy-as argument):

$ cloudstack-setup-databases
cloud:[email protected]:3307

4. Setup the CloudStack management application:

$ cloudstack-setup-management

** Allow some time for the CloudStack application to bootstrap on each startup. You can monitor the process at /var/log/cloudstack/management/catalina.out. At this point, this management server will automatically discover the other management server and form a cluster. 

Both management servers are load balanced and accessible via virtual IP, 10.10.10.20.

Lastly, change the management host IP address on every agent host at /etc/cloudstack/agent/agent.properties to the virtual IP address similar to below:

host=10.10.10.20

Restart the cloudstack agent service to apply the change:

$ service cloudstack-agent restart

Verifying your setup

1. Check the HAProxy statistics by logging into the HAProxy admin page at lb1 host port 9600. The default username/password is admin/admin. You should see the status of nodes from the HAProxy point of view. Our Galera cluster is in master-standby mode, while the CloudStack management servers are load balanced:

2. Check and observe the traffic on your database cluster from the ClusterControl overview page at https://10.10.10.10/clustercontrol :

Wrapping up

Ensuring high availability for the CloudStack Management Servers is critical, especially in production environments. This setup guarantees service continuity even in the event of failures. For instance, if a database node becomes unavailable, the system can still operate using the remaining healthy nodes. Similarly, if one Management Server goes down, other Management Servers in the cluster can continue handling user traffic and API requests without interruption. Implementing high availability minimizes downtime, enhances reliability, and ensures a seamless experience for end users and administrators alike.

To see how ClusterControl can help you maintain high availability for your CloudStack Management Servers through the deployment and management of HA Galera Cluster, try it free for 30 days. In the meantime, follow us on LinkedIn and X, or subscribe to our monthly newsletter (to the right) to stay up-to-date with all things database ops.

References:

https://docs.cloudstack.apache.org/projects/archived-cloudstack-installation/en/latest/management-server/index.html 

https://docs.cloudstack.apache.org/en/4.20.0.0/adminguide/reliability.html#management-server-load-balancing

https://docs.cloudstack.apache.org/en/4.20.0.0/installguide/management-server/index.html#install-the-management-server-on-the-first-host