blog
Transaction Considerations for MongoDB in Production
A MongoDB transaction is a group of throughput operations (reads and writes ) that require all these operations to succeed or fail together. For instance consider transferring money from account A to account B. If the write operation of updating account A fails, then consequently there will be no update for account B. Like ways if you manage to update account A but the transfer fails to be credited to account B, then the operation on account A will be reverted and the whole transaction will fail.
Transactions are generally designed this way to handle data inconsistency problems by keeping track of all the updates it makes along the way and if any of them fails, the connection breaks and the database rolls back by undoing all the changes that might have happened as a result of the transaction.
Production Considerations for Transactions
Transactions are powerful operations in MongoDB which go an extra mile in keeping track of the activities happening during the transaction. The downside of transaction operations is increased latency which lags other users from accessing the database until the transaction is complete by locking the involved resources. As this may be an appropriate way of coordinating updates for linked data, in the end it leads to unfriendly user experience.
In the production environment we have highlighted some of the considerations one should undertake when involving transactions and they include:
- Feature Availability and Compatibility
- Transaction Runtime Limit
- Limit Size of the Oplog
- WiredTiger Cache
- Security before the Transactions
- Shard Configuration Restriction
- Sharded Cluster and Arbiters
- Primary Secondary Arbiter (PSA) Configuration
- Locks acquisition
- Pending DDL Operations and Transactions
- In-Progress Transactions and Write Conflicts
- In-Progress Transactions and Stale Reads
- In-Progress Transactions and Chunk Migration
- Outside Reads During Transaction Commit
- Drivers
These considerations are applicable regardless if running a replica set or a sharded cluster.
Feature Availability and Compatibility
MongoDB supports multi-document transactions on replica sets from version 4.0. Multi-document transactions are multi-statements that can be executed sequentially without affecting each other. For instance, you can create a user and update another user in a single transaction.
Distributed transactions were introduced in MongoDB version 4.2 to maintain an identical syntax to the transactions introduced in version 4.0. Full consistency and atomicity is maintained in that if a commit fails on one shard, all transactions in participant shards will be aborted.
Your drivers should be updated for the MongoDB 4.2 in order to use transactions in MongoDB 4.2 deployments.
The minimum feature compatibility version for using transactions with a replica set is MongoDB version 4.0 whereas for a sharded cluster is 4.2.
For checking the featureCompatibilityVersion use the command
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) on the connected member.
Transaction Runtime Limit
This is a defined time in seconds within which a transaction should have come to completion. If a transaction exceeds this time it will be considered to have expired hence aborted by a periodic cleanup process. By default the value is set to be less than one minute but can be modified using the transactionLifetimeLimitSeconds parameter. The minimum value for this parameter is 1 second. To change the value for example to 50 seconds
db.adminCommand( { setParameter: 1, transactionLifetimeLimitSeconds: 50 } )Or during the startup of the mongod
mongod --setParameter "transactionLifetimeLimitSeconds=50"When setting the parameter for a sharded cluster, the parameter must be modified for all shards of a replica set member.
Limit size of the Oplog
MongoDB 4.0 creates a single operational log entry at commit time when a transaction contains any write operation. This is to say, the oplog will contain all write operations within the transaction and not that each write operation will have its own oplog. In that case, the oplog entry must be within the 16MB BSON document size.
In MongoDB version 4.2 multiple oplogs are created to accommodate all write operations within a transaction hence removing the total size limit for a transaction. However, each oplog entry must be within the 16MB BSON document size limit.
WiredTiger Cache
As mentioned before, transactions affect the performance of the database by locking resources involved. Among the resources is the WiredTiger cache that receives more pressure from the transactions. In order to avoid some of the storage cache pressure:
- All abandoned transactions should be aborted so that they can free some space in the cache.
- If an error occurs during an individual operation in the transaction, the transaction will remain in the cache until some directive operation is applied. The best thing will be to abandon the transaction and retry it.
- You can set the transactionLifetimeLimitSeconds parameter such that expired transactions will be aborted automatically.
Security before the Transactions
Operations in aborted transactions are still audited if running with auditing. However you need to consider that there will be no event to indicate that the transaction aborted.
When running with access control, one must have privileges for the operations in the transaction that is read or readWrite.
Shard Configuration Restriction
If the writeConcerMajorityJournalDefault is set to false or a shard with a voting member using the in-memory storage engine, transaction cannot be run on such a cluster.
Sharded Cluster and Arbiters
If any transaction operation reads from or writes to a shard containing an arbiter, the transaction will error and abort even after spanning through multiple shards.
Primary Secondary Arbiter (PSA) Configuration
In this case we consider if one wanted to reduce cache pressure by disabling the read concern majority.
If there is a shard with disabled read concern majority, read concern snapshot cannot be used for a transaction otherwise the transaction will throw an error and abort thereafter.
This is because readConcern level snapshot cannot be supported in a sharded cluster with enableMajorityReadConcern set to false.
If any transaction involves a write operation to a shard with read concern majority disabled, the transaction will error and abort even if the transaction spanned through multiple shards.
For a replica set, there is a possibility to specify the read concern to local, majority or even snapshot and transactions can still work. However, if there is a plan to transition this to a shard then you will need to avoid using the snapshot option.
If you may want to check the status of the read concern majority, run db.serverStatus() on the mongod instances and then check the storageEngine.supportsCommittedReads field. If the value is false, the the read concern majority is disabled
Locks Acquisition
When a transaction begins it will request for locks required for the involved operations. If the request waits beyond some set maxTransactionLockRequestTimeoutMillis ( by default is 5 milliseconds ) parameter, the transaction aborts. When you create or drop a collection and later issue a transaction whose operation will involve this collection, it is important to issue a create or drop operation with write concern majority to ensure that the transaction can acquire the required locks.
If you increase this value, this can help remove transaction aborts on momentary concurrent lock acquisitions such as fast-running metadata operations. Nonetheless, with this value being high, aborts may be delayed even for transaction operations that are deadlocked.
To set this value use the command
db.adminCommand( { setParameter: 1, maxTransactionLockRequestTimeoutMillis: 20 } )You can also enable or disable creation of a collection or an index inside a transaction using the shouldMultiDocTxnCreateCollectionAndIndexes.
To set the parameter during startup, either specify the parameter in the configuration file or run the command:
mongod --setParameter shouldMultiDocTxnCreateCollectionAndIndexes=falseIf you wish to modify the parameter during runtime:
db.adminCommand( { setParameter: 1, shouldMultiDocTxnCreateCollectionAndIndexes: true } )Consider setting the parameter for all shards if using a sharded cluster.
Pending DDL Operations and Transactions
If a transaction is in progress, DDL operation issued before it will wait since they will not obtain the locks when they are accessing the same database or collection.
An example of a DDL operation is createIndex(). If a transaction is in progress, this operation will have to wait until its completion. However, this operation does not affect transactions on other collections in the database.
The same case applies for a database that involves some database lock for example the collMod has to wait for the transaction to complete.
In-Progress Transactions and Write Conflicts
If a write operation is issued outside a transaction in progress but happens to conflict with the transaction by modifying an involved document, then the transaction will be aborted. This can only be resolved by the transaction obtaining a lock to modify the document and therefore the write operation outside the transaction will have to wait for the transaction to complete.
In-Progress Transactions and Stale Reads
Stale reads may happen because transaction operations use snapshots before the writes. For instance, a transaction can be in progress and then a write operation outside deletes the document the transaction modifies, the document will still be available within the transaction. A workaround for this is to use db.collection.findOneAndUpdate().
In-Progress Transactions and Chunk Migration
An error will occur if a chunk migration interleaves with a transaction and the migration comes into completion before the transaction takes a lock on the collection. The transaction will also abort in this case and one has to retry. Some of the errors that will be encountered in this case are:
- an error from cluster data placement change … migration commit in progress for
- Cannot find shardId the chunk belonged to at cluster time …
Outside Reads During Transaction Commit
In this case we consider a read operation on the same document that is to be modified by the transaction. If the transaction is to write against multiple shards:
- Outside reads using read concerns other than snapshot or linearizable will not wait for the writes of the transaction to be applied but read the before-transaction version of the documents that are in existence.
- If the reads involve snapshot or linearizable read concerns, the operations have to wait for the transaction to be complete and the document changes visible.
Drivers
Use updated drivers for MongoDB version 4.2 if you would like to use transactions on MongoDB 4.2. If some shard members have drivers for MongoDB 4.0 instead of 4.3, the transaction will fail and log some errors such as:
- 251 cannot continue txnId -1 for session … with txnId 1
- 50940 cannot commit with no participants
Conclusion
Transactions in MongoDB are important when data integrity and consistency need to be held. However, transactions come with some reduced database performance degradation more especially increasing latency of throughput operation. If not well managed/configured, transactions may affect the overall user experience of applications.
