blog
ClusterControl’s Vault Integration Adds Dynamic Data Security Protections
The security is paramount nowadays, especially for the organizations that deal with the sensitive data – financial, health, personal data. In those cases you are obliged to comply with numerous security and data protection standards like HIPAA, PCI DSS or similar. With the most recent release, ClusterControl makes it even easier to meet the strict standards. ClusterControl 1.8.0 comes with the support for integration with HashiCorp Vault. Let’s take a look at the improvements it brings.
ClusterControl integration with HashiCorp Vault
ClusterControl is a management platform therefore it requires full access to the systems it manages – databases. It has to store passwords for management users. Currently it’s done in the cluster configuration files – all the required access details for the cluster are stored in one of the configuration files located in /etc/cmon.d/cmon_X.cnf. Here is the example for the PostgreSQL cluster:
postgresql_password=pass
postgresql_server_addresses=10.0.0.105:5432,10.0.0.106:5432
postgresql_user=myuser
repl_password=!ROos}Shf9Z,6fWdF2Iq83V0wXom!P03
repl_user=cmon_replication
rpc_key=yx9i2r8olpou9lgiAs you can see, whoever can access this file will be able to access the PostgreSQL nodes. It also stores rpc_key that can be used to manage the cluster using ClusterControl. Of course, those files are accessible only to the root user and one can argue that if someone has gained access to the root account, you are pretty much in a bad spot anyway, but ensuring that there are no passwords in the configuration file is still a good practice.
root@vagrant:~# ls -alh /etc/cmon.d/
total 28K
drwx------ 2 root root 4.0K Sep 23 10:02 .
drwxr-xr-x 103 root root 4.0K Sep 23 13:34 ..
-rw------- 1 root root 668 Sep 23 13:34 cmon_1.cnf
-rw------- 1 root root 695 Sep 23 13:34 cmon_2.cnf
-rw------- 1 root root 600 Sep 23 13:34 cmon_3.cnf
-rw------- 1 root root 585 Sep 23 13:34 cmon_4.cnf
-rw-r--r-- 1 root root 81 Sep 1 2016 README.txtHere enters HashiCorp Vault – it is a product dedicated to storing sensitive data, just like the passwords we want to protect. It made sense to use it to integrate with ClusterControl. As of now, in ClusterControl 1.8.0, the only way to integrate is to use a pre-deployed Vault. Vault can be deployed in a number of ways, including clustered deployment. Covering this is out of the scope of this blog, we will still show you how to test the integration by deploying a standalone node of HashiCorp Vault and configure it with ClusterControl. Let’s take a look at the steps required to accomplish this.
Deploying HashiCorp Vault
The deployment process for HashiCorp Vault is pretty straightforward. At first, we have to create directory for the data:
mkdir -p /root/vault/data
cd /root/vaultThen we should download and decompress the Vault:
wget https://releases.hashicorp.com/vault/1.5.3/vault_1.5.3_linux_amd64.zip
unzip vault_1.5.3_linux_amd64.zipNext, we should create SSL certificate:
# create an SSL certificate
if [ ! -f cert.crt ]; then
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout cert.key -out cert.crt
fiThen we should create Valut’s configuration file:
# create a Vault configuration
cat - > vault.hcl <Once this is done, you can start the Vault. It will have to be initialized and later unsealed before ClusterControl will be able to actually use it.
# Start the vault server:
$ ./vault server -config=vault.hclStarting from then the UI at https://IP:8200/ui/ is available and you can use it for management actions regarding the Vault. It is not necessary to use it though. We have to initialize the Vault now:
root@vagrant:~/vault# VAULT_SKIP_VERIFY=1 ./vault operator init -key-shares=1 -key-threshold=1
Unseal Key 1: HfovvS/3HdBEN8XwnrnNKd65eB7I1j8WHSbvUhjG4Xk=
Initial Root Token: s.f1kl9XUBuFjdyV048QIgHoJh
Vault initialized with 1 key shares and a key threshold of 1. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 1 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 1 key to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.You can see here unseal key and root token. Unseal key is required in order to unseal the Vault, which we will do in the next step. Root token will be used by ClusterControl to connect to the Vault.
Let’s unseal the Vault:
root@vagrant:~/vault# VAULT_SKIP_VERIFY=1 ./vault operator unseal HfovvS/3HdBEN8XwnrnNKd65eB7I1j8WHSbvUhjG4Xk=
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.5.3
Cluster Name vault-cluster-2bf2e433
Cluster ID 350fd80a-67c8-bda0-6d6a-e77354dacfa2
HA Enabled falseNow we have to setup ClusterControl to use the Vault to store passwords. This can be accomplished by creating following entries in the /etc/cmon.cnf:
vault_token=s.f1kl9XUBuFjdyV048QIgHoJh
vault_auto_migrate=trueVault_token should contain a root token for the Vault and vault_auto_migrate=true means that ClusterControl will automatically migrate data from configuration files into the Vault. This is a good moment to make a copy of your existing configuration files located in /etc/cmon.d/ as you will need them should you decide to revert this process and do not use Vault at some point in the future.
Once you have the configuration set up, you have to restart cmon process to migrate data into the Vault:
service cmon restartThis is it, ClusterControl, from now on, will use the data in the Vault to access the backend databases. We can easily check what exactly is stored there:
root@vagrant:~/vault# export VAULT_TOKEN=s.f1kl9XUBuFjdyV048QIgHoJh
root@vagrant:~/vault# ./vault secrets list
Path Type Accessor Description
---- ---- -------- -----------
clustercontrol/ kv kv_939a83de Created by ClusterControl
cubbyhole/ cubbyhole cubbyhole_42b40ca0 per-token private secret storage
identity/ identity identity_594e8bfc identity store
sys/ system system_9509de58 system endpoints used for control, policy and debuggingNow, knowing the path, we can query the data:
root@vagrant:~/vault# ./vault kv list clustercontrol/
Keys
----
clustercontrol/
root@vagrant:~/vault# ./vault kv list clustercontrol/clustercontrol
Keys
----
cluster_0
cluster_1
cluster_2
cluster_3
cluster_4Let’s retrieve data for cluster_4:
root@vagrant:~/vault# ./vault kv get clustercontrol/clustercontrol/cluster_4
====== Metadata ======
Key Value
--- -----
created_time 2020-09-24T10:51:27.242621166Z
deletion_time n/a
destroyed false
version 5
============ Data ============
Key Value
--- -----
backup_user_password Y3rocu-7wWGSJ-56iMK7cd8@yK+70AzH
mysql_password pass
postgresql_password pass
repl_password !ROos}Shf9Z,6fWdF2Iq83V0wXom!P03
rpc_key yx9i2r8olpou9lgiAs you can see, the passwords are stored in the Vault. You can access the data using the UI as well, after authenticating with root token:
Please keep in mind that ClusterControl, once it is configured to work with Vault, will require Vault to be available and unsealed before ClusterControl can work properly. If it won’t be available, ClusterControl won’t work.
Should you want to revert this process, you should follow those steps:
- Get all the keys and values from the Vault for all of the clusters.
- Stop cmon process
- Remove Vault configuration from /etc/cmon.cnf
- Add the keys and values you extracted from the Vault into respective configuration files in /etc/cmon.d. You can use the backup copy of the configuration from before Vault has been used but please keep in mind that some additional passwords might have been added in the meantime – double check that your old configuration files contain all the data that is stored in the Vault.
We hope this short introduction to ClusterControl integration with HashiCorp Vault will be useful. If you decided to give it a try, let us know how it worked for you.