Clustering MySQL Backend in OpenStack

OpenStack uses a database to store data for all OpenStack services such as compute nodes available, running instances, state and so on. The DB takes part in every user request, from listing all instances to creating new VMs, so it is central to the operation of OpenStack. In a high availability environment, the database should be clustered so as to avoid a single point of failure (SPOF). 

In this post, we will setup up high availability clustering for the MySQL backend in OpenStack using Galera Cluster and ClusterControl. You will learn how to set up a multi-master MySQL setup, fronted by a redundant HAProxy load-balancer. 

Our end goal is to achieve resilience to database failures, and allow OpenStack components to communicate with the database nodes via one Virtual IP address.

We will need five hosts, three to run MySQL-wsrep with Galera Cluster, and two for ClusterControl/HAProxy/Keepalived. 

Please note that deploying MySQL Galera Cluster using the following way will remove any existing OpenStack dependencies, for example mysql-libs in the designated host. So we advise you to not co-locate it with your OpenStack controller. 

1. Use the Galera Configurator to generate a deployment package for Galera, and run it from the ClusterControl host. The deployment package will install the Galera Cluster as well as ClusterControl. For instructions, read this blog post on how to automatically deploy Galera. 

2. Next, we will install HAProxy. You can set up one instance on the ClusterControl host, and a second one on another host. Make sure that the ClusterControl host can do passwordless SSH to the new host where you will install the second HAProxy instance.

Install HAproxy on the ClusterControl host (172.16.0.31) using Manage Load Balancer. You can monitor the status of the installation in your ClusterControl UI under Jobs.

Repeat the same thing for the second HAProxy instance on host 172.16.0.32 with a different LB name:
 

3. Install Keepalived with virtual IP address 172.16.0.30 by using Manage Load Balancer:

At this point, the load balanced MySQL Galera Cluster is listening on port 33306 and accessible through virtual IP address, 172.16.0.30. You can also see from the database cluster summary bar that the HAproxy and Keepalived instances have been added and are running:

4. Create database schema for nova, keystone and glance as well as respective database users and privileges by using Manage Schema and Users:

Setup completed. You can now change following SQL connection flag in OpenStack component’s configuration file to the Galera cluster:

Nova – /etc/nova/nova.conf:

--sql_connection=mysql://nova:novapassword@172.16.0.30:33306/nova

Keystone – /etc/keystone/keystone.conf:

[sql]
connection = mysql://keystone:keystonepassword@172.16.0.30:33306/keystone

Glance – /etc/glance/glance-registry.conf:

sql_connection = mysql://glance:glancepassword@172.16.0.30:33306/glance

If this is a new installation, you need to initialize and sync the component’s database to the most recent version by using following command:

Nova:

$ nova-manage db sync

Keystone:

$ keystone-manage db_sync

Glance:

$ glance-manage db_sync

If you are migrating from an existing OpenStack database, you might need to stop the OpenStack related services for each component and migrate the data manually. Then change the connection string so the components start using Galera instead.