Severalnines Blog
The automation and management blog for open source databases

Decoding the MongoDB Error Logs

Akash Kathiriya
Posted in:

Sometimes decoding MongoDB error logs can be tricky and can consume big chunks of your valuable time. In this article, we will learn how to examine the MongoDB error logs by dissecting each part of the log messages.

Common Format for MongoDB Log Lines

Here is the log line pattern for version 3.0 and above...

<timestamp> <severity> <component> [<context>] <message>

Log line pattern for previous versions of MongoDB only included:

<timestamp> [<context>] <message>

Let’s look at each tag.

Timestamps

Timestamp field of log message stores the exact time when a log message was inserted in the log file. There are 4 types of timestamps supported by MongoDB. The default format is: iso8601-local. You can change it using --timeStampFormat parameter.

Timestamp Format Name Example
iso8601-local 1969-12-31T19:00:00.000+0500
iso8601-utc 1970-01-01T00:00:00.000Z
ctime Wed Dec 31 19:00:00.000
ctime-no-ms Wed Dec 31 19:00:00

Severity

The following table describes the meaning of all possible severity levels.

Severity Level Description
F Fatal- The database error has caused the database to no longer be accessible
E Error - Database errors which will stop DB execution.
W Warning - Database messages which explains potentially harmful behaviour of DB.
I Informational - Messages just for information purpose like ‘A new connection accepted’.
D Debug - Mostly useful for debugging the DB errors

Component

After version 3.0, log messages now include “component” to provide a functional categorization of the messages. This allows you to easily narrow down your search by looking at the specific components.

Component Error Description
Access Related to access control
Command Related to database commands
Control Related to control activities
FTDC Related to diagnostic data collection activities
Geo Related to parsing geospatial shapes
Index Related to indexing operations
Network Related to network activities
Query Related to queries
REPL Related to replica sets
REPL_HB Related to replica sets heartbeats
Rollback Related to rollback db operations
Sharding Related to sharding
Storage Related to storage activities
Journal Related to journal activities
Write Related to db write operations

Context

Context part of the error message generally contains the thread or connection id. Other values can be initandlisten. This part is surrounded by square brackets. Log messages of any new connection to MongoDB will have context value as initandlisten, for all other log messages, it will be either thread id or connection id. For e.g

2018-05-29T19:06:29.731+0000 [initandlisten] connection accepted from 127.0.0.1:27017 #1000 (13 connections now open)
2018-05-29T19:06:35.770+0000 [conn1000] end connection 127.0.0.1:27017 (12 connections now open)

Message

Contains the actual log message.

Log File Location

The default location on the server is: /var/log/mongodb/mongodb.log

If log file is not present at this location then you can check in the MongoDB config file. You can find MongoDB config file at either of these two locations.

/etc/mongod.conf or /yourMongoDBpath/mongod.conf

Once you open the config file, search for logpath option in it. logpath option tells MongoDB where to log all the messages.

Analyzing a Simple Log Message

Here is an example of a typical MongoDB error message...

2014-11-03T18:28:32.450-0500 I NETWORK [initandlisten] waiting for connections on port 27017

Timestamp: 2014-11-03T18:28:32.450-0500
Severity: I
Component: NETWORK
Context: [initandlisten]
Message: waiting for connections on port 27017

The most important part of this error is the message portion. In most of the cases, you can figure out the error by looking at this field. Sometimes if the message is not clear to you, then you can go for the component part. For this message, Component’s value is Network which means the log message is related to a network issue.

If you are not able to resolve your error, you can check the severity of the log message which says this message is for informational purpose. Further, you can also check out other parts of the log message like timestamp or context to find the complete root cause.

ClusterControl
Single Console for Your Entire Database Infrastructure
Find out what else is new in ClusterControl

Decoding Common Error Log Messages

  1. Message:

    2018-05-10T21:19:46.942 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.

    Resolution: Create admin user in authentication database

  2. Message:

    2018-05-10T21:19:46.942 E COMMAND  [initandlisten] ** ERROR: getMore command failed. Cursor not found

    Resolution: Remove the timeout from the cursor or increase the cursor batch size.

  3. Message:

    2018-05-10T21:19:46.942 E INDEX  [initandlisten] ** ERROR:E11000 duplicate key error index: test.collection.$a.b_1 dup key: { : null }

    Resolution: Violation of unique key constraint. Try to insert document with different key.

  4. Message:

    2018-05-10T21:19:46.942 E NETWORK  [initandlisten] ** ERROR:Timed out connecting to localhost:27017.

    Resolution: Latency between the driver and the server is too great, the driver may give up. You can change setting by adding connectionTimeout option in connection string.

  5. Message:

    2018-05-10T21:19:46.942 E WRITE  [initandlisten] ** ERROR: A write operation resulted in an error. E11000 duplicate key error index: test.people.$_id_ dup key: { : 0 }

    Resolution: Remove duplication of _id field value from conflicting documents.

  6. Message:

    2018-05-10T21:19:46.942 E NETWORK  [initandlisten] ** ERROR: No connection could be made because the target machine actively refused it 127.0.0.1:27017 at System.Net.Sockets.Socket.EndConnect

    Resolution: Either server is not running on port 27017 or try to restart the server with correct host and port.

Log Management Tools

MongoDB 3.0 has updated its logging features to provide better insights for all database activities. There are many log management tools available in the market like MongoDB Ops Manager, log entries, mtools etc.

Conclusion

Logging is as important as Replication or Sharding for good and proper database management. For better database management, one should be able to decode the logs easily to rectify the exceptions/errors quickly. I hope that after reading this tutorial, you will feel more comfortable while analyzing complex MongoDB logs.