MongoDB ReplicaSet on Docker Environment

I was playing around with MongoDB ReplicaSet yesterday in a Docker environment. The purpose is to test the integration with a backend service that connects to MongoDB. It was a very frustrating process especially when done in a local environment (localhost) that I came to realization that it is the host name resolution that mess up with the setup.

Objective

My original objective was to just test the MongoDB ReplicaSet for production use. I didn’t intend to use it locally. I already have a stand alone MongoDB container locally and it works good whether I use it within the Docker environment or even outside Docker.

We are trying to use MongoDB multi-document transactions and because it requires a ReplicaSet, we have to test it ourselves first.

Docker Compose

The concept is simple. We just need to create at least 2 Docker containers using the official Docker image for MongoDB, isolate them inside a Docker network, then configure the replication. To automate the process, we will also create another container whose sole purpose is to connect to the primary server, initialize the replica set and insert initial data (seeding). For the purpose of testing, we will also expose each MongoDB servers via host’s port mapping.

version: '3.2'
services:
  mongo-primary:
    image: mongo:4.1.13-bionic
    command: --replSet rs0 --bind_ip_all
    ports:
      - "27017:27017"
    networks:
      - mongo-cluster
  mongo-secondary:
    image: mongo:4.1.13-bionic
    command: --replSet rs0 --bind_ip_all
    ports:
      - "27018:27017"
    networks:
      - mongo-cluster
    depends_on:
      - mongo-primary
  mongo-replicator:
    build: ./mongo-replicator
    networks:
      - mongo-cluster
    depends_on:
      - mongo-primary
      - mongo-secondary
networks:
  mongo-cluster:
    driver: bridge

ReplicaSet and Seeding

The mongo-replicator container is simply a container that runs commands that initialize the replica set and seed data. It does this by connecting to the primary server and execute the command from the JS files. Below are the example commands.

replicate.js

rs.initiate( {
  _id : "rs0",
  members: [
    { _id: 0, host: "mongo-primary:27017" },
    { _id: 1, host: "mongo-secondary:27017" },
  ]
});

seed.js

const today = new Date();

db.issue.insertMany([
  {
    "title": "Wehner and Sons Unbranded Wooden Hat Avon challenge",
    "description": "Wehner and Sons Unbranded Wooden Hat Avon challenge programming function Manager deposit Handmade Digitized",
    "created_at": today,
    "updated_at": today
  },
  {
    "title": "Luettgen, Kassulke and Welch Incredible Plastic Mouse interactive capacitor",
    "description": "Luettgen, Kassulke and Welch Incredible Plastic Mouse interactive capacitor quantifying Supervisor Bike Customer",
    "created_at": today,
    "updated_at": today
  }
}

setup.sh

#!/usr/bin/env sh

if [ -f /replicated.txt ]; then
  echo "Mongo is already set up"
else
  echo "Setting up mongo replication and seeding initial data..."
  # Wait for few seconds until the mongo server is up
  sleep 50
  mongo mongo-primary:27017 replicate.js
  echo "Replication done..."
  # Wait for few seconds until replication takes effect
  sleep 50
  mongo mongo-primary:27017/project_manager seed.js
  echo "Seeding done..."
  touch /replicated.txt
fi

And the Dockerfile for mongo-replicator

FROM mongo:4.1.13-bionic
ADD ./replicate.js /replicate.js
ADD ./seed.js /seed.js
ADD ./setup.sh /setup.sh
CMD ["/setup.sh"]

The replicator should only run during the creation of the containers. To ensure that it doesn’t re-initialize and re-seed our MongoDB servers, we’ve created a simple script setup.sh that handles the condition that it will only execute once.

Localhost usage

So I was excited to use the replica set locally. I fired up my IDE and updated the DATABASE_URL environment variable so that it will use the replica set like this:

mongodb://localhost:27017,localhost:27018/project_manager?replicaSet=rs0

However, I get an error saying that it couldn’t locate mongo-secondary. It shouldn’t find that hostname since it is defined inside the Docker containers. I thought exposing the ports are enough.

It turns out that the client will receive the replica set configuration from the server and resolve the hostnames given by MongoDB and ignore what I’ve initially provided. It seems that the given connection string is only used for initializing the connection but the updated replica set configuration is downloaded by the client and takes over the previous config.

It makes sense since we can add more servers to the replica set without updating our connection string. Still, there is no luck in using it in localhost environment. I suspect it should work if my app is inside Docker too and is configured on the same cluster as defined in Docker Compose. In production, I assume this would work as usual, no sweat!

Therefore, I gave up the idea of using it locally as a shared MongoDB server outside of its Docker environment. What I did is just use the primary server.

DATABASE_URL=mongodb://localhost:27017/project_manager

The replica set is will working on the background. All is good.

For our full Docker integration though, it works smooth. Here is an example docker-compose.yml file.

version: '3.2'
services:
  mongo-primary:
    image: mongo:4.1.13-bionic
    command: --replSet rs0 --bind_ip_all
    networks:
      - mongo-cluster
  mongo-secondary:
    image: mongo:4.1.13-bionic
    command: --replSet rs0 --bind_ip_all
    networks:
      - mongo-cluster
    depends_on:
      - mongo-primary
  mongo-seeder:
    build: ./backend/mongo-seeder
    networks:
      - mongo-cluster
    depends_on:
      - mongo-primary
      - mongo-secondary
  backend_api:
    build: backend/api
    expose:
      - "3000"
    ports:
      - "8450:3000"
    environment:
      DATABASE_TYPE: mongodb
      DATABASE_URL: mongodb://mongo-primary,mongo-secondary/project_manager?replicaSet=rs0
    depends_on:
      - mongo-primary
      - mongo-secondary
      - mongo-seeder
    networks:
      - service-local
      - mongo-cluster
  frontend_apm_react:
    build: frontend/apm-react
    expose:
      - "8401"
    ports:
      - "8401:8401"
    environment:
      PORT: 8401
      PUBLIC_URL: http://localhost:8401
      APM_API_URL: http://localhost:8450
    networks:
      - service-local
networks:
  service-local:
    driver: bridge
  mongo-cluster:
    driver: bridge

Logging in to each Mongo nodes

To login to the primary node, just use the mongo client. All operations will work the same with a stand alone mongo server.

rs0:PRIMARY> show dbs

To login to the secondary nodes, just use the same mongo client, but in order to start querying the database, you must set the replicaSet slave status to OK.

rs0:SECONDARY> rs.slaveOk()

After executing the command above, you should be able to query on the secondary/slave node.

Wrapping up

In conclusion, we are successful in setting up a full Docker environment for our application (MongoDB replica set, REST API and React App). However, if we attempt to use the replica set outside the Docker environment, ie: via host’s port mapping, we encounter hostname resolution issues.

As a workaround, we will just use the stand alone connection string instead of the replica set connection string.

I have uploaded the example in Github for future reference. However, it doesn’t include the client app.

That’s it!