How to Support Your DevOps Team With Their Database Needs

DevOps, by a simple definition, is a person who has to perform an operational duty “the developer’s way”. It’s like a blending of two worlds – software development and system or database administration – for rapid development to improve the service delivery to the end-user. There are multiple areas that directly influence the efficiency of the DevOps team when working with databases, and we are going to touch those areas to help you prepare your DevOps team.

DevOps Preferred Database Vendor & Version

The first thing is to know what kind of database system (DBMS), vendor and versions that they need. Do they need to perform transaction processing (OLTP), analytics processing (OLAP) or both? The workload type is the important aspect to be determined at first which directly impacts the architectural design of a system. Depending on the budget and restriction, the open-source DBMS is probably the best option to go due to less cost of ownership and without locking-in to a specific vendor or brand.

As your application grows, with the right database vendor and version, most probably you won’t need to move or migrate your data to another database system/vendor to support your application demands. Some database management systems have built-in support for high availability and scalability, which allows you to scale your database without introducing any interruption to the database service at all. 

Once you already are comfortable with the chosen DBMS, it might be too late and costly for you to migrate to another DBMS, and sometimes it just is not worth it to relearn everything. Finding the right DBMS at the early stage is going to be beneficial in the long run. Spend a bit of time researching the best DBMS that suits your needs and requirements, and make sure it is worth every penny.

DevOps Database Management Tools

Database management tasks can vary depending on your environment, policy and requirement. Some of the common management tasks for databases are:

• Backup management (full, partial, differential, incremental backups, storage, compression, encryption)
• Restore management (restoration, storage)
• Configuration management (configuration changes, runtime, configuration file)
• Database software management (upgrade, downgrade, repository, permission, ownership)
• Service control (start, stop, restart, bootstrap, rolling restart)
• User/schema management (database object management – users, schemas, tables, views, privileges)
• Security management (certificates, keys, encryptions, access control list, auditing)
• Cluster/replication management (topology changes, cluster configuration, scaling)

Management tools are important to shorten the amount of time a DevOps team needs to spend on configuration and maintenance activities. Backup and restore are the most important thing (but the most hassle) in the database, since data loss is a big no in the database world. A good backup regime involves the configuration of many backup variables like backup method, scheduling, destination, retention period, encryption key and compression level. Those variables must be perfectly tuned to keep you away from having another problem to the database servers. You probably want to check out this blog post to give some ideas on how to design a good backup strategy for your database. 

A good management tool shall offer a toolset to manage all the components that we want to control. For on-premises infrastructures, Chef, Puppet, Ansible, and SaltStack are all popular configuration management tools designed to install and manage software on existing servers. The popular term, Infrastructure as Code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. IaC helps DevOps in minimizing human errors and streamlining the management and deployment process. However, this requires a higher learning curve and scripting knowledge to write the definition files for the management tasks for the whole infrastructure. Using other specialized management tools for the database like ClusterControl, MySQL Workbench or SQLyog will shorten the learning curve but some of those tools require an additional cost on support and subscription.

If you have more financial budgets and require far less control on the machine, you can opt for DBaaS cloud offerings like Amazon RDS, Google Cloud SQL and Azure SQL Database. Almost all of cloud DBaaS offerings in the market provide its own management toolset (or automation) for backup, configuration management, upgrades, provisioning and scaling.

DevOps Database Monitoring Tools

The DevOps team should be given a proactive monitoring tool to monitor the database. A proactive monitoring tool doesn’t just provide deep visibility into the key performance indicator (KPI) of your database and applications, it should be able to detect any anomaly, perform automatic recovery, advise on the over-tuned or under-tuned parameters as well as predictive analytics. This will help the team to understand whether the application and database are in good shape at a glance, and also provide a base justification for capacity planning while being assistive to guide the DevOps to find the root cause or bottleneck of an issue.

For alert and notification, DevOps must always get notified of the state of the database cluster. A great monitoring tool does not only notify the user via email, but it should also allow integration with other collaboration tools like incident management systems (PagerDuty, OpsGenie, ServiceNow), messaging tools (Telegram, Whatsapp, Signal, WeChat), team communication tools (Slack, Microsoft Teams, Discord) as well as webhooks integration for any other external integrations. For instance, all database events triggered by ClusterControl can be forwarded to other messaging tools like a Telegram channel. Consider the following database cluster and events:

The triggered events will be pushed to the configured Telegram channel. Here is an example of what you would get in the channel if something went wrong with the cluster (depending on the events that you’ve subscribed):

Not all DevOps members are trained specialized professionals, and having a good proactive monitoring system can really help them to have a clear understanding of what is going on with the database, eventually avoiding disruptions and unplanned downtime.

DevOps Database Automation Tools

Managing the database infrastructure lifecycle involves many risky and repetitive tasks that require expertise and experience. Human beings have feelings. They have bad days, personal problems, pressure for results and many types of distractions. It’s common to forget a step or misfire a destructive command, especially on a daily repetitive task. A well-defined configuration creates a stable process and the machine will never miss a single step.

Repetitive tasks are common in multi-tier architecture setup. Every tier has its own set of management, deployment and scaling tasks that have to be performed in its own way. For example, changing a cluster-wide database parameter must be followed up with a database rolling restart (restart one node at a time) to load the new changes without service disruption and downtime. These repetitive tasks should be automated to reduce the risk of human error but at the same time, must be able to handle unexpected issues correctly.

Repetitive tasks like deployment, configuration management, backup, restore and software upgrade can be automated with infrastructure provisioning tools like Terraform, Heat (OpenStack) or CloudFormation (AWS) together with configuration management tools like Ansible, Chef, Salt or Puppet. However, there are always missing parts and pieces that need to be covered by a collection of custom scripts or commands like failover, resyncing, recovery, scaling and many more. Rundeck, an open-source runbook automation tool can be used to manage all the custom scripts, which can bring us closer to achieving full automation.

A fully automated database infrastructure requires all important components to work in-sync together like monitoring, alerting, notification, management, scaling, security and deployment. ClusterControl is a pretty advanced automation tool to deploy, manage, monitor and scale your MySQL, MariaDB, PostgreSQL and MongoDB servers. It supports the handling of complex topologies with all kinds of database clustering and replication technologies offered by the supported DBMS. ClusterControl has all the necessary tools to replace specialized DBAs to maintain your database infrastructure. We believe that existing sysadmins or DevOps teams alongside ClusterControl would be enough to handle most of the operational burden of your database infrastructure.

DevOps Database Professional Training

Investing in professional database training is an investment in solution management, and ultimately reduces costs associated with database emergency and problem resolutions. Like any IT operations area, database management skills are based on experience, expertise and knowledge. The more people in the DevOps team have those skills, the quicker they can diagnose issues, troubleshoot and implement solutions. 

There are a variety of options available from online training to instructor-led on-site classes at your facility commonly provided by the database principal, vendors or any certified training providers. There are also live virtual classes, where you can attend the training just like a normal class, interacting directly with the instructor and the rest of the participants without even leaving your workstation. 

The training courses commonly focus on database fundamentals, database configuration knowledge, reference architecture, experience sharing, tuning, backup and recovery techniques that are geared to support your application needs help and guarantee that your data is always available for both your business and your customers. Training participants will commonly end up with a clear understanding of what is right or wrong, do and don’ts, understand the misconceptions and best practices after they have completed the training.

Most organizations and businesses consider training as an expense when it is actually an investment. The knowledge gain during the training is beneficial for a long-term run. Just spending a couple of days for the training session, in return, the skills, knowledge and confidence boost for the DevOps team to work on supporting the critical systems can be useful for years, even decades.

Conclusion

DevOps is the backbone of IT operation and must be supported with all the necessary tools to improve the productivity, efficiency and effectiveness for your business continuation.