MongoDB - Replication

MongoDB - Replication

Replication is the process of synchronizing data across multiple servers, creating a distributed system. This process provides redundancy and increases data availability by maintaining multiple copies of data on different database servers. Replication is a critical feature for building resilient applications, protecting a database from the loss of a single server, and enabling recovery from hardware failures or service interruptions.

Why Use Replication?

• Data Redundancy & Safety: Ensures multiple copies of your data exist, safeguarding against data loss on a single node.
• High Availability: If one server fails, the system can automatically failover to another, ensuring your application remains online (24/7 availability).
• Disaster Recovery: A replica member can be located in a separate data center to protect against regional outages.
• Zero-Downtime Maintenance: Administrative tasks like backups, index rebuilds, and compaction can be performed on secondary nodes without impacting the primary.
• Read Scalability: Distribute read operations across secondary nodes to increase read throughput for your application.
• Application Transparency: The replica set is managed by the database, and drivers can automatically detect the primary, making failover transparent to the application.

How Replication Works in MongoDB

MongoDB achieves replication using a replica set, which is a group of mongod instances that maintain the same data set. In a replica set, one node is designated as the primary node. The primary receives all write operations. All other instances are secondary nodes, which replicate the primary’s operation log (oplog) and apply the operations to their own data sets. A replica set can have only one primary node at any given time.

• A replica set typically consists of three or more data-bearing nodes.
• One node is the primary, and the remaining nodes are secondaries.
• All data write operations are sent to the primary and then replicated to the secondaries.
• In case of a primary node failure (automatic failover), an election is held among the remaining secondaries to elect a new primary.
• When a failed node recovers, it rejoins the replica set as a secondary node and catches up with the current data.

The diagram below illustrates a typical replica set topology. The client application, through the MongoDB driver, directs all write operations to the primary. The primary then replicates these changes to the secondary nodes. Read operations can be configured to go to the primary (for strong consistency) or to secondaries (for read scaling).

          +-----------------+
          | Client          |
          | Application     |
          +--------+--------+
                   |
      (Writes & Reads) | (Reads - Optional)
                   | /
         +---------v----------+      +------------------+      +------------------+
         |   Primary Node     |------>|  Secondary Node  |<-----|  Secondary Node  |
         |  (Accepts Writes)  |      |  (Replicates)    |      |  (Replicates)    |
         +--------------------+<----->+------------------+<----->+------------------+
                    (Heartbeats & Election)

Modern Replica Set Setup

While you can start mongod instances with command-line flags, the modern and recommended approach is to use a configuration file. For local development and testing, Docker Compose is an excellent tool for spinning up a replica set with ease.

Method 1: Using a Configuration File

1. Create three data directories, e.g., data/rs1, data/rs2, data/rs3.

2. Create a configuration file (mongod.conf) for each node. Here is a sample for the first node:

mongod-rs1.conf

storage:
  dbPath: data/rs1
net:
  port: 27017
  bindIp: localhost
replication:
  replSetName: "rs0"

3. Start each mongod instance using its respective configuration file:

mongod --config mongod-rs1.conf
mongod --config mongod-rs2.conf # (with port 27018, dbPath data/rs2)
mongod --config mongod-rs3.conf # (with port 27019, dbPath data/rs3)

4. Connect to one of the instances using the modern MongoDB Shell (mongosh):

mongosh --port 27017

5. From the mongosh prompt, initiate the replica set:

// Create a configuration object
config = {
    _id: "rs0",
    members: [
        { _id: 0, host: "localhost:27017" },
        { _id: 1, host: "localhost:27018" },
        { _id: 2, host: "localhost:27019" }
    ]
}

// Initiate the replica set with the configuration
rs.initiate(config)

6. You can check the status of your replica set with rs.status() and the configuration with rs.conf().

Method 2: Using Docker Compose (Recommended for Development)

Create a docker-compose.yml file:

version: '3.8'
services:
  mongo1:
    image: mongo
    ports:
      - 27017:27017
    command: ["--replSet", "rs0", "--bind_ip_all"]
  mongo2:
    image: mongo
    command: ["--replSet", "rs0", "--bind_ip_all"]
  mongo3:
    image: mongo
    command: ["--replSet", "rs0", "--bind_ip_all"]

mongosetup:
    image: mongo
    depends_on:
      - mongo1
      - mongo2
      - mongo3
    volumes:
      - ./setup-rs.js:/scripts/setup-rs.js
    entrypoint: [ "mongosh", "--host", "mongo1:27017", "/scripts/setup-rs.js" ]

Create a setup-rs.js file in the same directory:

rs.initiate(
  {
    _id: "rs0",
    members: [
      { _id: 0, host: "mongo1:27017" },
      { _id: 1, host: "mongo2:27017" },
      { _id: 2, host: "mongo3:27017" }
    ]
  }
)

Run docker-compose up to start and automatically configure your replica set.

Adding and Removing Members

To dynamically manage your replica set, connect to the primary node and use the rs.add() and rs.remove() commands.

Example: To add a new member running on localhost:27020:

rs.add("localhost:27020")

To remove a member:

rs.remove("localhost:27020")

Note: You must be connected to the primary to run these commands. You can verify this with the db.hello().isWritablePrimary command, which will return true if you are on the primary.