Decoding the MongoDB Error Logs

Sometimes decoding MongoDB error logs can be tricky and can consume big chunks of your valuable time. In this article, we will learn how to examine the MongoDB error logs by dissecting each part of the log messages.

Common Format for MongoDB Log Lines

Here is the log line pattern for version 3.0 and above…

   [] 

Log line pattern for previous versions of MongoDB only included:

 [] 

Let’s look at each tag.

Timestamps

Timestamp field of log message stores the exact time when a log message was inserted in the log file. There are 4 types of timestamps supported by MongoDB. The default format is: iso8601-local. You can change it using –timeStampFormat parameter.

Severity

The following table describes the meaning of all possible severity levels.

Component

After version 3.0, log messages now include “component” to provide a functional categorization of the messages. This allows you to easily narrow down your search by looking at the specific components.

Context

Context part of the error message generally contains the thread or connection id. Other values can be initandlisten. This part is surrounded by square brackets. Log messages of any new connection to MongoDB will have context value as initandlisten, for all other log messages, it will be either thread id or connection id. For e.g

2018-05-29T19:06:29.731+0000 [initandlisten] connection accepted from 127.0.0.1:27017 #1000 (13 connections now open)
2018-05-29T19:06:35.770+0000 [conn1000] end connection 127.0.0.1:27017 (12 connections now open)

Message

Contains the actual log message.

Log File Location

The default location on the server is: /var/log/mongodb/mongodb.log

If log file is not present at this location then you can check in the MongoDB config file. You can find MongoDB config file at either of these two locations.

/etc/mongod.conf or /yourMongoDBpath/mongod.conf

Once you open the config file, search for logpath option in it. logpath option tells MongoDB where to log all the messages.

Analyzing a Simple Log Message

Here is an example of a typical MongoDB error message…

2014-11-03T18:28:32.450-0500 I NETWORK [initandlisten] waiting for connections on port 27017

Timestamp: 2014-11-03T18:28:32.450-0500
Severity: I
Component: NETWORK
Context: [initandlisten]
Message: waiting for connections on port 27017

The most important part of this error is the message portion. In most of the cases, you can figure out the error by looking at this field. Sometimes if the message is not clear to you, then you can go for the component part. For this message, Component’s value is Network which means the log message is related to a network issue.

If you are not able to resolve your error, you can check the severity of the log message which says this message is for informational purpose. Further, you can also check out other parts of the log message like timestamp or context to find the complete root cause.

Decoding Common Error Log Messages

1. Message:

   2018-05-10T21:19:46.942 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.

   Resolution: Create admin user in authentication database

2. Message:

   2018-05-10T21:19:46.942 E COMMAND  [initandlisten] ** ERROR: getMore command failed. Cursor not found

   Resolution: Remove the timeout from the cursor or increase the cursor batch size.

3. Message:

   2018-05-10T21:19:46.942 E INDEX  [initandlisten] ** ERROR:E11000 duplicate key error index: test.collection.$a.b_1 dup key: { : null }

   Resolution: Violation of unique key constraint. Try to insert document with different key.

4. Message:

   2018-05-10T21:19:46.942 E NETWORK  [initandlisten] ** ERROR:Timed out connecting to localhost:27017.

   Resolution: Latency between the driver and the server is too great, the driver may give up. You can change setting by adding connectionTimeout option in connection string.

5. Message:

   2018-05-10T21:19:46.942 E WRITE  [initandlisten] ** ERROR: A write operation resulted in an error. E11000 duplicate key error index: test.people.$_id_ dup key: { : 0 }

   Resolution: Remove duplication of _id field value from conflicting documents.

6. Message:

   2018-05-10T21:19:46.942 E NETWORK  [initandlisten] ** ERROR: No connection could be made because the target machine actively refused it 127.0.0.1:27017 at System.Net.Sockets.Socket.EndConnect

   Resolution: Either server is not running on port 27017 or try to restart the server with correct host and port.

Log Management Tools

MongoDB 3.0 has updated its logging features to provide better insights for all database activities. There are many log management tools available in the market like MongoDB Ops Manager, log entries, mtools etc.

Conclusion

Logging is as important as Replication or Sharding for good and proper database management. For better database management, one should be able to decode the logs easily to rectify the exceptions/errors quickly. I hope that after reading this tutorial, you will feel more comfortable while analyzing complex MongoDB logs.