MongoDB Replica Set Upgrade

MongoDB software suppose to be straight forward but it does not mean you would make a mistake and destroy the cluster.

This article give you a well documented procedure on upgrading a 3 nodes MongoDB replica set running on Centos/Redhat 7.

Peform database backup

It is important to backup the database before any upgrade!

Personally, I prefer exporting the database as backup.

Schedule down time (optional)

Although hot upgrade is being performed here, but like any other software upgrades, there is a risk on upgrade failure and roll back might be needed. 

Therefore, please scheduling down time to give yourself some breathing room.

Verification before upgrades

FeatureCompatibilityVersion

• Login to the MongoDB databsae with mongo shell

mongo "mongodb://host1:27017,host2:27017,host3:27017/admin?replicaSet=myrelicaset"

• Check the FeatureCompatibilityVersion value

db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

It should return a result that like the following

"featureCompatibilityVersion": "3.4"

• If the value is not previous version (3.4 here), set the setFeatureCompatibilityVersion to the previous version with the follwing command

db.adminCommand( { setFeatureCompatibilityVersion: “3.4” } )

Replication status between nodes

• Using mongo shell and login to the MongoDB databsae replica set
• Run the follwing command:

rs.printSlaveReplicationInfo()

Ensure that no replica set member is in ROLLBACK or RECOVERING state.

MongoDB Replica status

• Using mongo shel and loginto the MongoDB databsae
• Run the follwing command:

rs.status()

Make sure you have two healthy SECONDARY nodes

Force a particular node as primary node (optional)

If you want to set a particular node as the primary node, you can setup node prioriy with the following commnad:

x=rs.config()
x.members[0].priority=1000
rs.reconfig(x)

Note: The above set the first node (members[0]) as primary

Check the result

rs.status()

MongoDB replica set upgrade

Stop the application (Optional)

Stopping the applicaiton using the database.

Upgrade the 1st SECONDARY node

• Copy or download the MongoDB 3.6 rpms to the node
• SSH to the secondary node
• Update /etc/mongod.conf file and make sure you have

  net:
  bindIP: 0.0.0.0

  Note: Starting on MongoDB 3.6, bindIP is default to localhost if it is not defined.
• Stop the mongod node

  sudo systemctl stop mongod

• Install mongodb 3.6 software.yum install -y mongodb-enterprise-server-3.6.4-1.el7.x86_64.rpm yum install -y mongodb-enterprise-mongos-3.6.4-1.el7.x86_64.rpm yum install -y mongodb-enterprise-tools-3.6.4-1.el7.x86_64.rpm yum install -y mongodb-enterprise-shell-3.6.4-1.el7.x86_64.rpm
• Start the mongod node

  sudo systemctl start mongod

• Check if the node join the cluster
  • Using Mongoshell to login to the replica set
  • Run 
  rs.status()
  • Make sure the node has join the replica set
  • Note: For whatever reason, I need to restart mongod a 2nd time to make the node join the replica set.

Upgrade the 2nd SECONDARY node

Repeat the above steps to upgrade the 2nd SECONDARY node

Upgrade PRIMARY node

Step down the primary

Before doing upgrade, you need to step down the primary node to a secondary node

• login to replica set with Mongo shell
• Run the follwoing command

rs.stepDown()

• check the replica status and make sure the primary node has been step down as secondary node

rs.status()

Upgrade the last node

• Repeat the secondary node upgrade procedure
• You are done