Severalnines Blog
The automation and management blog for open source databases

MongoDB Tutorial - On-premises Cluster Management and Monitoring of MongoDB Replica Set

Severalnines
Posted in:

Replica Sets in MongoDB are very useful. They provide multiple copies of data, automated failover and read scalability. A Replica Set can consist of up to 12 nodes, with only one primary node (or master node) able to accept writes. In case of primary node failure, a new primary is auto-elected.

It is advisable to have an odd number of nodes in a Replica Set, so as to avoid vote locking when a new primary node is being elected. Replica Sets require a majority of the remaining nodes present to elect a primary. If you have e.g. 2 nodes in a Replica Set, then one option is to add an arbiter. An arbiter is a mongod instance that is part of a Replica Set, but does not hold any data. Because of the minimal resource requirements, it can be colocated with an application server or the ClusterControl server. The arbiter should not be colocated with any of the members of the Replica Set.

In this post, we will show you how to install and configure a Replica Set, and then manage it using ClusterControl. It is similar to a previous post on how to manage and monitor a Sharded Cluster. Our hosts are running Debian Squeeze 64bit.

Note that you can also deploy a Replica Set using our MongoDB Configurator (not longr available), this will automate the whole process.

Install MongoDB

**Following steps should be performed on mongo1, mongo2 and mongo3.

  1. Import the 10gen public GPG Key:

    $ apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
  2. Create a /etc/apt/sources.list.d/10gen.list using following command:

    echo ‘deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen’ | tee /etc/apt/sources.list.d/10gen.list
  3. Install MongoDB using package manager:

    $ apt-get update && apt-get install mongodb-10gen
  4. Configure MongoDB for replication by adding following lines into /etc/mongodb.conf. Take note that our replica set name is defined as rs0:

    # /etc/mongodb.conf
    port=27017
    dbpath=/var/lib/mongodb
    logpath=/var/log/mongodb/mongodb.log
    logappend=true
    replSet=rs0
    rest=true
    pidfilepath=/var/run/mongodb/mongod.pid
  5. Restart mongodb service to apply changes:

    $ service mongodb restart
Severalnines
 
Become a MongoDB DBA - Bringing MongoDB to Production
Learn about what you need to know to deploy, monitor, manage and scale MongoDB

Configure Replica Set

  1. Initiate replica set by login into mongo in mongo1:

    $ mongo

    And run following command:

    > rs.initiate()
  2. Add mongo2 into the replica set:

    > rs.add("mongo2");
  3. We need at least 3 members in a replica set in order to complete an election if 1 member goes down. Arbiter is a member of a replica set that exists solely to vote in elections. Arbiters do not replicate data. We are going to setup mongo3 as arbiter:

    > rs.addArb("mongo3");
  4. Check replica set status:

    > rs.status()

    You can also use the MongoDB REST interface which available at http://ip_address:28017/_replSet to check your replica set status similar to screenshot below:

Import Data into Replica Set

  1. Import some test data into replica set which available at http://media.mongodb.org/zips.json:

    $ wget http://media.mongodb.org/zips.json
    $ mongoimport --host 'rs0/mongo1,mongo2,mongo3' --db mydb --collection zip --file zips.json
  2. Check the imported data by connecting to mongo console and run following commands:

    > use mydb
    > db.zip.find()

Install ClusterControl Server - the automatic way

Here is the recommended way to get ClusterControl on top of your existing MongoDB replica set. We have built a collection of scripts available under Severalnines download page which will automate the bootstrapping process. You may refer to this knowledge base article for further details.

Install ClusterControl Server - the manual way

We will need a separate server to run ClusterControl, as illustrated below:

**Following steps should be performed on ClusterControl host.

  1. It is highly recommended to enable passwordless SSH with key authentication between ClusterControl and agents. Generate a RSA key and copy it to all nodes:

    $ ssh-keygen -t rsa # just press enter for all prompts
    $ ssh-copy-id -i ~/.ssh/id_rsa 192.168.197.41
    $ ssh-copy-id -i ~/.ssh/id_rsa 192.168.197.42
    $ ssh-copy-id -i ~/.ssh/id_rsa 192.168.197.43
    $ ssh-copy-id -i ~/.ssh/id_rsa 192.168.197.40
  2. Install Apache, PHP, MySQL and other required components:

    $ apt-get install apache2 php5-common php5-mysql php5-gd mysql-server mysql-client sudo mailutils -y
  3. Download ClusterControl for MongoDB from Severalnines download site:

    $ wget http://www.severalnines.com/downloads/cmon/cmon-1.2.4-64bit-glibc23-mongodb.tar.gz
  4. Extract ClusterControl into /usr/local directory and create symlink to CMON path:

    $ tar -xzf cmon-1.2.4-64bit-glibc23-mongodb.tar.gz -C /usr/local
    $ ln -s /usr/local/cmon-1.2.4-64bit-glibc23-mongodb /usr/local/cmon
  5. Copy the CMON binary, init and cron files into respective locations:

    $ cp /usr/local/cmon/bin/* /usr/bin
    $ cp /usr/local/cmon/sbin/* /usr/sbin
    $ cp /usr/local/cmon/etc/cron.d/cmon /etc/cron.d
    $ cp /usr/local/cmon/etc/init.d/cmon /etc/init.d
  6. Copy the CMON web apps:

    $ cp -rf /usr/local/cmon/www/* /var/www/
  7. Disable bind-address and name resolve in MySQL. This will allow us to use IP address only when granting database user. Add following line into /etc/mysql/my.cnf under [mysqld] directive:

    skip-name-resolve

    And comment following line:

    #bind-address
  8. Restart MySQL to apply changes:

    $ service mysql restart
  9. Create CMON database and import the schema for CMON:

    $ mysql -e “CREATE DATABASE cmon”
    $ mysql < /usr/local/cmon/sql/cmon_db.sql
    $ mysql < /usr/local/cmon/sql/cmon_data.sql
  10. Enter MySQL console and allow CMON database users:

    > GRANT ALL ON *.* TO 'cmon'@'192.168.197.120' IDENTIFIED BY 'cmonP4ss' WITH GRANT OPTION;
    > GRANT ALL ON *.* TO 'cmon'@'127.0.0.1' IDENTIFIED BY 'cmonP4ss' WITH GRANT OPTION;
    > GRANT SUPER,INSERT,SELECT,UPDATE,DELETE ON *.* TO 'cmon'@'192.168.197.121' IDENTIFIED BY 'cmonP4ss';
    > GRANT SUPER,INSERT,SELECT,UPDATE,DELETE ON *.* TO 'cmon'@'192.168.197.122' IDENTIFIED BY 'cmonP4ss';
    > GRANT SUPER,INSERT,SELECT,UPDATE,DELETE ON *.* TO 'cmon'@'192.168.197.123' IDENTIFIED BY 'cmonP4ss';
  11. Configure MySQL root password:

    $ mysqladmin -u root password ‘MyP4ss’
    $ mysqladmin -h127.0.0.1 -u root password ‘MyP4ss’
  12. Create CMON configuration file at /etc/cmon.cnf:

    $ vim /etc/cmon.cnf

    And add following lines:

    # /etc/cmon.cnf - cmon config file
    # id and name of cluster that this cmon agent is monitoring.
    # Must be unique for each monitored cluster, like server-id in mysql
    cluster_id=1
    name=default_repl_1
    mode=controller
    type=mongodb
     
    ##port of mysql server holding cmon database
    mysql_port=3306
     
    ##hostname/ip of mysql server holding cmon database
    mysql_hostname=192.168.197.120
     
    ##password for 'cmon' user on  the 'mysql_hostname'
    mysql_password=cmonP4ss
    local_mysql_port=3306
    local_mysql_password=cmonP4ss
     
    ##hostname/ip of the server of this cmon instance
    hostname=192.168.197.120
    cmon_core_dir=/root/s9s
     
    ##osuser - the user owning the cmon_core_dir above
    osuser=root
    os=debian
     
    ##logfile is default to syslog.
    logfile=/var/log/cmon.log
     
    ##location of cmon.pid file. The pidfile is written in /tmp/ by default
    pidfile=/var/run/
    nodaemon=0
    monitored_mountpoints=/var/lib/mongodb
    mongodb_server_addresses=192.168.197.121:271017,192.168.197.122:27017
    mongoarbiter_server_addresses=192.168.197.123:27017
    mongodb_basedir=/usr
     
    mysql_basedir=/usr
    db_stats_collection_interval=10
    host_stats_collection_interval=60
    ssh_opts=-nqtt

Install ClusterControl Agent

ClusterControl agents must reside in all MongoDB nodes. The agents are responsible for the following:

  • Collecting host stats (disk/network/CPU/RAM)
  • Reading and parsing log files

** Following steps should be performed on mongo1, mongo2 and mongo3.

  1. Download ClusterControl for MongoDB from Severalnines download site:

    $ wget http://www.severalnines.com/downloads/cmon/cmon-1.2.4-64bit-glibc23-mongodb.tar.gz
  2. Extract ClusterControl into /usr/local directory and create symlink to CMON path:

    $ tar -xzf cmon-1.2.4-64bit-glibc23-mongodb.tar.gz -C /usr/local
    $ ln -s /usr/local/cmon-1.2.4-64bit-glibc23-mongodb /usr/local/cmon
  3. Copy the CMON binary and init files into locations:

    $ cp /usr/local/cmon/bin/* /usr/bin
    $ cp /usr/local/cmon/sbin/* /usr/sbin
    $ cp /usr/local/cmon/etc/init.d/cmon /etc/init.d
  4. Create CMON configuration file at /etc/cmon.cnf and add following lines:

    # /etc/cmon.cnf - CMON config file
    ## id and name of cluster that this cmon agent is monitoring.
    ## Must be unique for each monitored cluster, like server-id in mysql
    cluster_id=1
    name=default_repl_1
    mode=agent
    type=mongodb
     
    # MySQL for CMON
    ## Port of mysql server holding cmon database
    mysql_port=3306
    ## Hostname/ip of mysql server holding cmon database
    mysql_hostname=192.168.197.120
     
    ## Password for 'cmon' user on  the 'mysql_hostname'
    mysql_password=cmonP4ss
    local_mysql_port=3306
    local_mysql_password=cmonP4ss
     
    ## Hostname/IP of the server of this cmon instance
    hostname=192.168.197.121
     
    ## osuser - the user owning the cmon_core_dir above
    osuser=root
     
    ## logfile is default to syslog
    logfile=/var/log/cmon.log
     
    ## location of cmon.pid file. The pidfile is written in /tmp/ by default
    pidfile=/var/run/
    nodaemon=0
     
    # MongoDB database path
    monitored_mountpoints=/var/lib/mongodb
  5. Repeat above steps for mongo2 and mongo3. Make sure to change the value of “hostname” to the IP address of respective nodes.

Start the Service

** Following steps should be performed on mongo1, mongo2 and mongo3 followed by ClusterControl.

Enable the CMON service on boot and start the service in agent hosts, followed by controller host:

$ update-rc.d cmon start 99 2 3 4 5 . stop 99 0 1 6 .
$ service cmon start

Configure ClusterControl UI

** Following steps should be performed on ClusterControl host.

  1. To install the new ClusterControl UI, SSH into the ClusterControl host, download the ClusterControl installation script, change script permissions and execute it:

    $ wget http://www.severalnines.com/downloads/cmon/setup-cc-ui.sh
    $ chmod +x setup-cc-ui.sh
    $ ./setup-cc-ui.sh
  2. To finalize the UI installation, open web browser and go to this URL, http://ClusterControl_IP_address/install and you should see “Install ClusterControl UI and API” page.

    **Please note the ClusterControl API Access Token, ClusterControl API URL, your login email and login password. We will use these later on the cluster registration page.

  3. After the installation, click “Click here to automatically register your cluster now!” and you will redirected to cmonapi page similar to screenshot below. Click “Login Now”.

  4. After that, you need to login using the email address you specified in the installation page with respective password. (default is “admin”). You should see the “Cluster Registrations” page similar to screenshot below. Enter the ClusterControl API token and URL:

  5. You will be redirected to ClusterControl UI located at https://ClusterControl_IP_address/clustercontrol, your MongoDB Cluster is listed on this page. Click on it to view your cluster:

You’re done! You are now able to manage your MongoDB Replica Set using ClusterControl.