How to use Python motor with a docker compose MongoDB replica set

I’ve found that docker-compose provides a clean environment for developing applications, and I wanted to try out change streams to create a horizontally scaleable Python app.

A quick search provided a selection of docker-compose files. All of them created a cluster, but when I tried to connect, Python hung! This is what I got working.

The Python package that provides async access to MongoDB is motor. The documentation for creating the connection refers me to the underlying PyMongo documentation for connecting to a cluster:

MongoClient('localhost', replicaset='foo')
MongoClient('mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=foo')

Turning to the MongoDB replica set configuration, we can see that the cluster is initialised with a script looking like this:

rs.initiate({
"_id": "rs0",
"members": [{
"_id": 0, "host": "host1:27017"
},{
"_id": 1,"host": "host2:27017"
},{
"_id": 2,"host": "host3:27017"
}]
})'

The problem I found running this with docker compose is how to specify the hosts. If the internal container address is “host1:27017” etc., but the external is localhost:27017, motor will discover the internal representation and try to connect to the other servers in the cluster as “host2:27017” etc. Unfortunately “host2” doesn’t resolve to an IP address outside of the container :(

Obviously the solution is to provide some known IP address, which we can achieve with the following “docker-compose.yml” file:

version: "3"

services
:

mongo0:
image: mongo
networks:
mongo_network:
ipv4_address: 172.16.238.10
ports:
- 27017:27017
entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "rs0" ]

mongo1:
image: mongo
networks:
mongo_network:
ipv4_address: 172.16.238.11
ports:
- 27018:27017
entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "rs0" ]

mongo2:
image: mongo
networks:
mongo_network:
ipv4_address: 172.16.238.12
ports:
- 27019:27017
entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "rs0" ]

networks:
mongo_network:
ipam:
driver: default
config:
- subnet: "172.16.238.0/24"

So now we can bring up the instances:

$ docker-compose up

Initialise the cluster (here I’m using a locally installed mongo client):

$ mongo --port 27017 --eval 'rs.initiate({"_id":"rs0","members":[{"_id":0,"host":"172.16.238.10:27017"},{"_id":1,"host":"172.16.238.11:27017"},{"_id":2,"host":"172.16.238.12:27017"}]})'

Now it’s time to test!

First I’ll create a “source.py” to insert some records:

import asyncio
import motor.motor_asyncio


async def main_async():
client = motor.motor_asyncio.AsyncIOMotorClient('172.16.238.10', replicaset='rs0')
db = client.test_database
count = 0
while count < 100:
document = {'name': 'rob', 'count': count}
result = await db.test_collection.insert_one(document)
print('result %s' % repr(result.inserted_id))
count += 1
await asyncio.sleep(5)


asyncio.run(main_async())

Next I’ll make a “watcher.py” to monitor the inserts:

import asyncio
import motor.motor_asyncio


async def main_async():
client = motor.motor_asyncio.AsyncIOMotorClient('172.16.238.10', replicaset='rs0')
db = client.test_database
async for change in db.test_collection.watch():
print(change)


asyncio.run(main_async())

And it works :)

{'_id': {'_data': '825D2D74F1000000012B022C0100296E5A1004F3EEB8D7BE374F9994D7D43FC6E2343546645F696400645D2D74F1C0CFC68EB3C8DB950004'}, 'operationType': 'insert', 'clusterTime': Timestamp(1563260145, 1), 'fullDocument': {'_id': ObjectId('5d2d74f1c0cfc68eb3c8db95'), 'name': 'rob', 'count': 0}, 'ns': {'db': 'test_database', 'coll': 'test_collection'}, 'documentKey': {'_id': ObjectId('5d2d74f1c0cfc68eb3c8db95')}}
{'_id': {'_data': '825D2D74F6000000012B022C0100296E5A1004F3EEB8D7BE374F9994D7D43FC6E2343546645F696400645D2D74F6C0CFC68EB3C8DB960004'}, 'operationType': 'insert', 'clusterTime': Timestamp(1563260150, 1), 'fullDocument': {'_id': ObjectId('5d2d74f6c0cfc68eb3c8db96'), 'name': 'rob', 'count': 1}, 'ns': {'db': 'test_database', 'coll': 'test_collection'}, 'documentKey': {'_id': ObjectId('5d2d74f6c0cfc68eb3c8db96')}}
{'_id': {'_data': '825D2D74FB000000012B022C0100296E5A1004F3EEB8D7BE374F9994D7D43FC6E2343546645F696400645D2D74FBC0CFC68EB3C8DB970004'}, 'operationType': 'insert', 'clusterTime': Timestamp(1563260155, 1), 'fullDocument': {'_id': ObjectId('5d2d74fbc0cfc68eb3c8db97'), 'name': 'rob', 'count': 2}, 'ns': {'db': 'test_database', 'coll': 'test_collection'}, 'documentKey': {'_id': ObjectId('5d2d74fbc0cfc68eb3c8db97')}}

Hope this saves you some time.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store