How to use Python motor with a docker compose MongoDB replica set

I’ve found that docker-compose provides a clean environment for developing applications, and I wanted to try out change streams to create a horizontally scaleable Python app.

A quick search provided a selection of docker-compose files. All of them created a cluster, but when I tried to connect, Python hung! This is what I got working.

The Python package that provides async access to MongoDB is motor. The documentation for creating the connection refers me to the underlying PyMongo documentation for connecting to a cluster:

MongoClient('localhost', replicaset='foo')
MongoClient('mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=foo')

Turning to the MongoDB replica set configuration, we can see that the cluster is initialised with a script looking like this:

rs.initiate({
  "_id": "rs0",
  "members": [{
    "_id": 0, "host": "host1:27017"
  },{
    "_id": 1,"host": "host2:27017"
  },{
    "_id": 2,"host": "host3:27017"
  }]
})'

The problem I found running this with docker compose is how to specify the hosts. If the internal container address is “host1:27017” etc., but the external is localhost:27017, motor will discover the internal representation and try to connect to the other servers in the cluster as “host2:27017” etc. Unfortunately “host2” doesn’t resolve to an IP address outside of the container :(

Obviously the solution is to provide some known IP address, which we can achieve with the following “docker-compose.yml” file:

version: "3"

services:

  mongo0:
    image: mongo
    networks:
      mongo_network:
        ipv4_address: 172.16.238.10
    ports:
      - 27017:27017
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "rs0" ]

  mongo1:
    image: mongo
    networks:
      mongo_network:
        ipv4_address: 172.16.238.11
    ports:
      - 27018:27017
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "rs0" ]

  mongo2:
    image: mongo
    networks:
      mongo_network:
        ipv4_address: 172.16.238.12
    ports:
      - 27019:27017
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "rs0" ]

networks:
  mongo_network:
    ipam:
      driver: default
      config:
        - subnet: "172.16.238.0/24"

So now we can bring up the instances:

$ docker-compose up

Initialise the cluster (here I’m using a locally installed mongo client):

$ mongo --port 27017 --eval 'rs.initiate({"_id":"rs0","members":[{"_id":0,"host":"172.16.238.10:27017"},{"_id":1,"host":"172.16.238.11:27017"},{"_id":2,"host":"172.16.238.12:27017"}]})'

Now it’s time to test!

First I’ll create a “source.py” to insert some records:

import asyncio
import motor.motor_asyncio


async def main_async():
    client = motor.motor_asyncio.AsyncIOMotorClient('172.16.238.10', replicaset='rs0')
    db = client.test_database
    count = 0
    while count < 100:
        document = {'name': 'rob', 'count': count}
        result = await db.test_collection.insert_one(document)
        print('result %s' % repr(result.inserted_id))
        count += 1
        await asyncio.sleep(5)


asyncio.run(main_async())

Next I’ll make a “watcher.py” to monitor the inserts:

import asyncio
import motor.motor_asyncio


async def main_async():
    client = motor.motor_asyncio.AsyncIOMotorClient('172.16.238.10', replicaset='rs0')
    db = client.test_database
    async for change in db.test_collection.watch():
        print(change)


asyncio.run(main_async())

And it works :)

{'_id': {'_data': '825D2D74F1000000012B022C0100296E5A1004F3EEB8D7BE374F9994D7D43FC6E2343546645F696400645D2D74F1C0CFC68EB3C8DB950004'}, 'operationType': 'insert', 'clusterTime': Timestamp(1563260145, 1), 'fullDocument': {'_id': ObjectId('5d2d74f1c0cfc68eb3c8db95'), 'name': 'rob', 'count': 0}, 'ns': {'db': 'test_database', 'coll': 'test_collection'}, 'documentKey': {'_id': ObjectId('5d2d74f1c0cfc68eb3c8db95')}}
{'_id': {'_data': '825D2D74F6000000012B022C0100296E5A1004F3EEB8D7BE374F9994D7D43FC6E2343546645F696400645D2D74F6C0CFC68EB3C8DB960004'}, 'operationType': 'insert', 'clusterTime': Timestamp(1563260150, 1), 'fullDocument': {'_id': ObjectId('5d2d74f6c0cfc68eb3c8db96'), 'name': 'rob', 'count': 1}, 'ns': {'db': 'test_database', 'coll': 'test_collection'}, 'documentKey': {'_id': ObjectId('5d2d74f6c0cfc68eb3c8db96')}}
{'_id': {'_data': '825D2D74FB000000012B022C0100296E5A1004F3EEB8D7BE374F9994D7D43FC6E2343546645F696400645D2D74FBC0CFC68EB3C8DB970004'}, 'operationType': 'insert', 'clusterTime': Timestamp(1563260155, 1), 'fullDocument': {'_id': ObjectId('5d2d74fbc0cfc68eb3c8db97'), 'name': 'rob', 'count': 2}, 'ns': {'db': 'test_database', 'coll': 'test_collection'}, 'documentKey': {'_id': ObjectId('5d2d74fbc0cfc68eb3c8db97')}}

Hope this saves you some time.