Severalnines Blog
The automation and management blog for open source databases

Become a MongoDB DBA: How to scale reads

Severalnines
October 10, 2016

In previous posts of our “Become a MongoDB DBA” series, we covered Deployment, Configuration, Monitoring (part 1), Monitoring (part 2), backup and restore. From this blog post onwards, we shift our focus to the scaling aspects of MongoDB.

One of the cornerstones of MongoDB is that it is built with high availability and scaling in mind. Scaling can be done either vertically (bigger hardware) or horizontally (more nodes). Horizontal scaling is what MongoDB is good at, and it is not much more than spreading the workload to multiple machines. In effect, we’re making use of multiple low-cost commodity hardware boxes, rather than upgrading to a more expensive high performance server.

MongoDB offers both read- and write scaling, and we will uncover the differences of these two strategies for you. Whether to choose read- or write scaling all depends on the workload of your application: if your application tends to read more often than it writes data you will probably want to make use of the read scaling capabilities of MongoDB. Today we will cover MongoDB read scaling.

Read scaling considerations

With read scaling, we will scale out our read capacity. If you have used MongoDB before, or have followed this blog series, you may be aware that actually all reads end up on the primary by default. Regardless if your replicaSet contains nine nodes, your read requests still go to the primary. Why was this done deliberately?

In principle, there are a few considerations to make before you start reading from a secondary node directly. First of all: the replication is asynchronous, so not all secondaries will give the same results if you read the same data at the same point in time. Secondly: if you distribute read requests to all secondaries and use up too much of their capacity, if one of them becomes unavailable, the other secondaries may not be able to cope with the extra workload. Thirdly: on sharded clusters you should never bypass the shard router, as data may be out-of-date or data may have been moved to another shard. If you do use the shard router and set the read preference correctly, it may still return incorrect data due to incomplete or terminated chunk migrations.

As you have seen these are serious considerations you should make before scaling out your read queries on MongoDB. In general, unless your primary is not able to cope with the read workload it is receiving, we would advise against reading from secondaries. The price you pay for inconsistency is relatively high, compared to the benefits of offloading work from the master.

The main issue here seems to be the eventual consistency of MongoDB, so your application needs to be able to work around that. Also if you would have an application that is not bothered by stale data, analytics for instance, you could benefit greatly from using the secondaries.

Reading from a secondary

There are two things that are necessary to make reading from a secondary possible: tell the MongoDB client driver that you actually wish to read from a secondary (if possible) and tell the MongoDB secondary server that it is okay to read from this node.

Setting read preference

For the driver, all you have to do is set the read preference. When reading data you simply set the read preference to read from a secondary. Let’s go over each and every read preference and explain what it does:

primary Always read from the primary (default)
primaryPreferred Always read from the primary, read from secondary if the primary is unavailable
secondary Always read from a secondary
secondaryPreferred Always read from a secondary, read from the primary if no secondary is available
nearest Always read from the node with the lowest network latency

It is clear the default mode is the least preferred if you wish to scale out reads. PrimaryPreferred is not much better, as it will pick 99.999% of the time the primary. Still if the primary becomes unavailable you will have a fallback for read requests.

Secondary should work fine for scaling reads, but as you leave out the primary the reads will never have a fallback if no secondary is available. SecondaryPreferred is slightly better, but the reads will hit almost all of the time the secondaries, which still causes an uneven spread of reads. Also if no secondaries are available, in most cases there will be no longer a cluster and the primary will demote itself to a secondary. Only when an arbiter is part of the cluster, the secondaryPreferred mode makes sense.

Nearest should always pick the node with the lowest network latency. Even though this sounds great from an application perspective, this will not guarantee you get an even spread in read operations. But it will work very well in multi-regions where latency is high, and delays are noticeable. In such cases, reading from the nearest node means your application will be able to serve out data with the minimum latency.

Filtering nodes with tags

In MongoDB you can tag nodes in a replicaSet. This allows you to make groupings of nodes and use them for many purposes, including filtering them when reading from secondary nodes.

An example of a replicaSet with tagging can be:

{
    "_id" : "myrs",
    "version" : 2,
    "members" : [
             {
                     "_id" : 0,
                     "host" : "host1:27017",
                     "tags" : {
                             "dc": "1",
                             "rack": "e3"
                     }
             }, {
                     "_id" : 1,
                     "host" : "host2:27017",
                     "tags" : {
                             "dc": "1",
                             "rack": "b2"
                     }
             }, {
                     "_id" : 0,
                     "host" : "host3:27017",
                     "tags" : {
                             "dc": "2",
                             "rack": "q1"
                     }
             }
    ]
}

This tagging allows us to limit our secondary to exist, for instance, in our first datacenter:

db.getMongo().setReadPref(‘secondaryPreferred’, [ { "dc": "1" } ] )

Naturally the tags can be used with all read preference modes, except Primary.

Enabling secondary queries

Apart from setting the read preference in the client driver, there is another limitation. By default MongoDB disables reads from a secondary server side, unless you specifically tell the host to allow read operations. Changing this is relatively easy, all you have to do is connect to the secondary and run this command:

myset:SECONDARY> rs.slaveOk()

This will enable reads on this secondary for all incoming connections. You can also run this command in your application, but that would then imply your application is aware it could encounter a server that did not enable secondary reads.

Reading from a secondary in a shard

It is also possible to read from a secondary node in MongoDB sharded clusters. The MongoDB shard router (mongos) will obey the read preference set in the request and forward the request to a secondary in the shard(s). This also means you will have to enable reads from a secondary on all hosts in the sharded environment.

And as said earlier: an issue that may arise with reading from secondaries on a sharded environment, is that it might be possible to receive incorrect data from a secondary. Due to the migration of data between shards, data may be in transit from one shard to another. Reading from a secondary may then return incomplete data.

Adding more secondary nodes

Adding more secondary nodes to a MongoDB replicaSet would imply more overhead for replication. However unlike MySQL, syncing the oplog on secondaries is not only limited to the primary node. MongoDB can also sync the oplog from other secondaries, as long as they are up to date with the primary. This means oplog servicing is also possible from other secondaries, and we thus automatically have “intermediate” primaries. This means theoretically that if you add more secondaries, the performance impact will be limited. Keep in mind that this also means the data trickles down to the other nodes a bit slower, as the oplog entries have to pass at least two nodes now.

Conclusion

We have described what the impact is on MongoDB when reading from its secondaries, and what caveats to be aware of. If you don’t necessarily need to scale your reads, it is better not to perform this pre-emptive optimization. However if you think your setup would benefit from offloading the primary, better work around the issues described in this post.

Related Post

Planets9s - Eurofunk replaces Oracle with feature-rich Severalnines ClusterControl

Severalnines
December 02, 2016

Planets9s is a weekly communication on all the latest Severalnines resources and technologies around automation and management of open source databases such as MySQL, MariaDB, PostgreSQL & MongoDB.

Posted in: DB Ops Devops Planets9s

How to build a stable MySQL Replication environment

Severalnines
November 29, 2016

During this webinar, participants will learn how to build a production-ready environment with MySQL replication. From OS and DB configuration checklists to schema changes and disaster recovery, they’ll have the information needed to plan and prepare the best setup for their requirements.

Posted in: DB Ops Events

Planets9s - Top 9 Tips for MySQL Replication, MongoDB Sharding & NinesControl

Severalnines
November 25, 2016

Planets9s is a weekly communication on all the latest Severalnines resources and technologies around automation and management of open source databases such as MySQL, MariaDB, PostgreSQL & MongoDB.

Posted in: DB Ops Devops Planets9s

Become a MongoDB DBA: Sharding ins- and outs - part 2

Art van Scheppingen
November 23, 2016

In our previous post, we showed you how to shard your MongoDB databases and covered the theory behind it. But you are not done yet. How do you monitor the shards to make sure they are performing, and that data is distributed evenly between your shards? And how do you consistently backup your data across shards? This is our ninth post in the “Become a MongoDB DBA” blog series.