ClusterControl makes MySQL enterprise-grade at Black Hills Corp
Responsible for 1500 databases spread across multiple data centers, the Black Hills Corp team needed help in setting a new standard for their MySQL databases. Choosing Galera clusters was the first step, operationalizing them was the next — ClusterControl helped them go from hours to minutes.
Background
Black Hills Corporation is a customer-focused, growth-oriented utility US company with a tradition of improving life with energy and a vision to be the energy partner of choice.
Based in Rapid City, South Dakota, the company serves 1.2 million natural gas and electric utility customers in eight US states: Arkansas, Colorado, Iowa, Kansas, Montana, Nebraska, South Dakota, and Wyoming. The company also generates wholesale electricity and produces natural gas, oil and coal
Challenge
Black Hills Corporation, like many other large companies, manages large infrastructures of Microsoft SQL Server and Oracle. The database infrastructure consists of over 50TB of data across 1500 databases and is managed by a team of DBAs. Many systems considered non-business-critical, however, run on MySQL.
This included a number of web properties, as well as internal applications. The databases were traditionally not under any specific SLA, and little attention was given to the management of these. However, running disaster recovery tests two times a year, these non-critical systems could be down for at least 12 hours each time. The extended downtime was becoming a bit of an issue, something that the IT team wanted to avoid.
I nearly destroyed the cluster trying to resolve some of the early problems we encountered.
Jordan Marshall, DBA Manager
The DBA team, who up to this point had not been managing the MySQL databases, also wanted to assume control of the open-source technology while bringing them up to a higher standard of stability utilizing advanced clustering technology. After much research, they decided on Galera Cluster as the technology of choice for this upgrade from the master-master replication which had been deployed prior.
While experts in Microsoft SQL Server and Oracle ExaData, no one on the team was an expert in MySQL. After spending time online researching, coupled with the database documentation, they were able to create, what they believed was, a production-ready cluster. Then the next step was to understand how to manage the system to ensure smooth operations.
This meant visibility into database performance, as well as detection of anomalies. The system would be distributed across different data centers, and it was important that any connectivity issues or failures did not jeopardize the integrity of the database.
In case of failures, either of a single node or of an entire data center, the team required a way to automatically failover applications to available nodes. Eventually, they took to the web to find a software solution capable of managing a large and complex cluster to achieve the following:
- A production-ready Galera Cluster
- Ability to create and restore backups
- Automatic failover across multiple data centers
- An effective monitoring solution
The DBA team found ClusterControl online and began experimenting with it. Since they had already deployed a Galera cluster, the first step was to import it into ClusterControl. It immediately started throwing some errors with trying to perform certain management tasks on the cluster. Turns out a username and password were mismatched between nodes, something that was overlooked by the DBA team. This prompted the team to set aside the work they had already done and use the deployment functionality in ClusterControl to build a new cluster.
Solution
It took their DBA ten minutes to deploy a production-ready Galera cluster with no errors when previously it had taken hours to do. “Your software solved the lack of understanding with MySQL,” said Jordan Marshall, Manager of Database Administration for Black Hills Corporation. “It’s a nice handy tool that makes me feel confident that we can run this in a production environment.”
ClusterControl checked all the boxes of their requirements. It provided multi-data center support, advanced automation, backup, failover functionality, and unified monitoring across its data centers. To sell their management on the spend, Jordan provided an estimate of the labor cost for the manual setup as well as some custom scripting and tooling.
It's phenomenal software…I'm not usually impressed with vendors or the software we buy, because usually it's over-promised and under-delivered.
Jordan Marshall, DBA Manager
The cost of ClusterControl was less, and that was before you factored in the cost of maintenance and upkeep of the databases. While cost is a factor, the time savings of ClusterControl’s automation may have been even more important as time is a precious commodity for the DBA team.
Outcome
Due to the success of the migration of their MySQL replication stacks to Galera clusters and the confidence they feel toward ClusterControl, MySQL has moved up the list of their preferred database technologies to the number two spot, displacing Oracle. The company plans on exploring the idea of expanding their use of open-source as a cost-effective solution for their business applications in an effort to reduce overall project costs while still utilizing ClusterControl and its enterprise-grade features to ensure performance and uptime.
Summary
Setting a new standard
The Black Hills DBA team wanted to assume control over the MySQL databases underlying non-critical apps and bring them up to a greater standard of stability using clustered technology.
Filling the MySQL skills gap
The team was unable to transfer their SQL Server and Oracle ExaData ops skillsets to MySQL, so they looked for a solution that could help manage multi-data center, clustered deployments.
Demonstrating value for money
After establishing perfect problem/solution fit, they were able to demonstrate ClusterControl’s true value by calculating the cost to manually set up and build out custom scripts and tooling.
Ready to automate your database?
Sign up now and you’ll be running your database in just minutes.