Driving productivity and reducing costs with database automation
Solving high availability challenges in a MySQL master-slave setup.
Background
Technicolor SA, formerly Thomson SA and Thomson Multimedia is a global provider of solutions for the creation, management, post-production, delivery, and access to video, for the Communication, Media, and Entertainment industries. The company ranks among the worldleading suppliers of digital content delivery services and home access devices, including set-top boxes and broadband gateways. Clients include studios, broadcasters, cinema/television, production, and network operators (telcos, broadband, satellite, and cable operators).
As the world leader in xDSL gateways and multiple-play applications, Technicolor develops a wide range of xDSL products. Technicolor’s gateways are distributed mostly through operators and service providers, such as Telstra, Bouygues Telecom, Telmex, Portugal Telecom, Teliasonera, BT, FT, KPN, and Telefonica as well as through distributors.
Challenge
Technicolor maintains a test facility that simulates an ISP environment to assist Telecom operators worldwide in the integration of their technologies into networks, The environment is highly dependent on centralized Authentication, Authorization, and Accounting (AAA) protocols that enable the connection of a number of devices to a backend network. FreeRADIUS is used as the AAA platform since it is based on open industry standards and is widely used in the ISP world. Because of its central role in the ISP infrastructure, continuous availability and data integrity are important requirements.
With high availability and data integrity being such important requirements, Technicolor’s technical team decided to use a MySQL Cluster backend.
MySQL Cluster is a powerful product and addressed the availability and data integrity problems. At the same time, it introduced another set of issues – deployment became increasingly difficult, while the software became more complex. The technical staff would have to spend a significant amount of time to understand how the cluster operated, and how it should be configured to suit their needs. Advanced skills would be required to deploy and manage the database cluster, thus the need to find a solution with low management overhead.
Technicolor used a replicated MySQL master-slave set-up to provide high availability as well as scalability to the FreeRADIUS platform. However, MySQL Replication had its limitations:
- Fail-over was not automatic and had to be performed by highly skilled staff.
- Due to hardware problems, software bugs, or the use of nondeterministic functions slave servers could easily end up with different data from the master server.
- When a master crashed, it often caused corruption of the binary log. Once it came back up, the slave server would not be able to continue from the last binary log position, which would break replication.
- In a RADIUS case, when a device is authenticated, a record is inserted in the master server. Subsequent accounting requests could fail on the slave, as it did not yet have that new record.
These serious issues caused downtime of the AAA platform and thus affected the overall ISP infrastructure.
Solution
Clustered environments involve more resources to administer; therefore, management capabilities are a critical component of a database cluster.
Technicolor’s technical team used the Severalnines Configurator for MySQL Cluster to generate a deployment package for their on-premise MySQL Cluster installation. The deployment package completely automated the deployment of the database cluster and
installed ClusterControl.
Thanks to Severalnines and its ClusterControl, we decreased the time to deploy MySQL Cluster from a week to about 2 hours. The automation of database management tasks allowed us to improve DBA productivity and lower our costs.
Werner Van Driessche, System Administrator at Technicolor
Outcome
Once the infrastructure was in place, it was important to manage and monitor the database with as little overhead as possible. Service interruptions would impact the whole network and require continuous monitoring of the database as well as an alert system for potential problems.
ClusterControl could provide a single, consolidated view of the database’s health as well as of its underlying platform (OS, hardware, network). In addition, it helped to accurately isolate the source of problems. Management was simplified and backups could be scheduled from a
web user interface. As part of the end-to-end network testing activities, staff outside of the Database Team needed an easy way to restart nodes or restart the whole cluster, which they could do via a web interface. Freed from these routine tasks, the DBA could focus on
more important jobs such as architecture design and performance tuning, or work on other projects.
As an additional benefit, ClusterControl could also monitor the FreeRADIUS nodes and restart them if they went down.
Summary
Supplement existing ops team with Virtual DBA
ClusterControl freed its technical team to focus on more value-added tasks such as architecture design and performance tuning. They were able to put a management infrastructure in place to automate basic database administration.
Quickly build a highly available database
Technicolor was able to build a highly available database cluster in a very short time, eliminate the risk of downtime, and roll out upgrades, patches, and new database features at the simple push of a button.
End-to-end network testing activities
ClusterControl introduced self-service by allowing the technical team to provide capabilities that could be safely controlled by end-users.
Ready to automate your database?
Sign up now and you’ll be running your database in just minutes.