Restoring the database dump of an older version of mongo to a new version of mongo

Question

Currently, I have an older version of mongo, i.e 2.6 running on my system. I already have my site in production and have a lot of client data. I am planning an upgrade to mongo 3.2.

So, my question is whether mongorestore of mongo v3.2 work with data dump of v2.6? Or, is it known to create problems?

Any answers will be invaluable! Thanks

Solution 1

I asked this same question on the official MongoDB mailing list. They said not to upgrade more than 1 major version at a time. (Major versions being: 2.2, 2.4, 2.6, 3.0, 3.2, 3.4)

I didn't want to follow the normal upgrade process of installing every version Just to launch mongod and then shut it down. That feels to me like it would leave cruft behind and I like to have my infrastructure building scripted and version controlled. So, I decided to launch new EC2 instances with the latest Ubuntu (since my Mongo v2.4 servers were also 2 LTS versions behind) and the latest MongoDB. I used docker images of intermediate versions of MongoDB to do the data upgrades.

https://gist.github.com/RichardBronosky/2d04c7c2e9a5bea67cd9760a35415a3f#file-uat_mongodb_upgrade_from_prod-sh

The bulk of the solution is this:

# mongo.conf is using the default dbPath: /var/lib/mongodb
# this path is for temporary use by the mongo docker container
mkdir -p /data/db/dump
# see: https://hub.docker.com/_/mongo/ (search for /data/db)
# see: https://github.com/docker-library/mongo/blob/30d09dbd6343d3cbd1bbea2d6afde49f5d9a9295/3.4/Dockerfile#L59
cd /data/db
mongodump -h prodmongo.int

# Get major versions from https://hub.docker.com/r/library/mongo/tags/
step=0
for major_version in 2.6.12 3.0.14 3.2.11 3.4.1; do
    sudo docker stop some-mongo || true
    sudo docker rm   some-mongo || true
    sudo docker run --name some-mongo -v /data/db:/data/db -d mongo:$major_version
    false; while [[ $? > 0 ]]; do
        sleep 0.5
        sudo docker exec -it some-mongo mongo --eval 'printjson((new Mongo()).getDBNames())'
    done
    if (( $step == 0 )); then
        sudo docker exec -it some-mongo mongorestore /data/db/dump
    fi
    ((step += 1))
done

# Finish up with docker
sudo rm -rf /data/db/dump/*
sudo docker exec -it some-mongo bash -c 'cd /data/db; mongodump'
sudo docker stop some-mongo
sudo docker rm   some-mongo

# Load upgraded data into latest version of MongoDB (WiredTiger storage engine will be used)
mongorestore /data/db/dump
sudo rm -rf /data

Solution 2

As you have data from mongo 2.6, index field limitation is already fulfilled. Mongo 3.2 will restore this backup without any issue.

The other way you can upgrade your db (if you have replica set) is to replace one 2.6 member with 3.2 and wait for sync, then other one... This will give you business continuity :-)

Solution 3

Bruno's answer worked for me up to version 3.4, but then there was a problem each update required setting a feature compatibility version, then at 4.2 you can't start mongo if you're still using MMAPv1, you need to switch to WiredTiger.

So, thank you very much Bruno, you got me most of the way there.

Below is my updated script, hope it helps someone out!

#!/bin/bash

# Log commands to stdout
set -o xtrace
# Exit on error
set -o errexit
# Exit on use of unset variables
set -o nounset
# Exit and report on pipe failure
set -o pipefail

# mongo.conf is using the default dbPath: /var/lib/mongodb
# this path is for temporary use by the mongo docker container
mkdir -p /data/db/dump
# see: https://hub.docker.com/_/mongo/ (search for /data/db)
# see: https://github.com/docker-library/mongo/blob/30d09dbd6343d3cbd1bbea2d6afde49f5d9a9295/3.4/Dockerfile#L59
cd /data/db
# our docker host holding the database
mongodump -h $PROD_MONGO_IP

# Get major versions from https://hub.docker.com/r/library/mongo/tags/
step=0
prev_real_major=""
for major_version in 2.6.12 3.0.15 3.2.21 3.4.24 3.6.23 4.0.27 4.2.17 4.4.10 5.0.3; do
    real_major=`echo $major_version | cut -f1,2 -d "."`
    sudo docker stop some-mongo || true
    sudo docker rm some-mongo || true
    docker run --rm -d --name some-mongo -v /data/db:/data/db mongo:$major_version
    set +o errexit
    false; while [[ $? > 0 ]]; do
        sleep 0.5
        docker exec -it some-mongo mongo --eval 'printjson((new Mongo()).getDBNames())'
        if [[ $real_major > 3.3 ]]; then
            docker exec -it some-mongo mongo --eval "db.adminCommand( { setFeatureCompatibilityVersion: \"$real_major\" } )"
        fi
    done
    set -o errexit
    if (( $step == 0 )); then
        docker exec -it some-mongo mongorestore /data/db/dump
    fi
    # upgrade to WiredTiger
    if [[ $real_major == 4.0 ]]; then
        # delete the database dump from earlier
        rm -rf /data/db/dump/
        # dump the database again
        docker exec -w /data/db -it some-mongo mongodump
        # stop the existing mongo container
        docker stop some-mongo
        # delete everything in /data/db except /data/db/dump
        find /data/db -mindepth 1 ! -regex '^/data/db/dump\(/.*\)?' -delete
        # run the 4.0 mongo container again
        docker run --rm -d --name some-mongo -v /data/db:/data/db mongo:$major_version
        # restore the database, which automatically makes it wiretiger.
        docker exec -it some-mongo mongorestore /data/db/dump
    fi
    ((step += 1))
done

# Finish up with docker
sudo rm -rf /data/db/dump/*
docker exec -it some-mongo bash -c 'cd /data/db; mongodump'
docker stop some-mongo

# Commented these out because I did them manually, you decide what you want to do here
# Load upgraded data into latest version of MongoDB (WiredTiger storage engine will be used)
# mongorestore /data/db/dump
# sudo rm -rf /data

Comments

[Bruno Bronosky]

Yeah, I asked this question on the official MongoDB mailing list and they said not to upgrade more than 1 major version at a time. (Major versions being: 2.2, 2.4, 2.6, 3.0, 3.2, 3.4) groups.google.com/d/msg/mongodb-user/cbXeM283iHM/y0QbW1h4BQA‌​J

[Bruno Bronosky]

It always freaks me out when my answer gets upvotes and no comments. Can my code really be that clear? It is bash after all.

[Dmitry Shevkoplyas]

Commenting only for Bruno's piece of mind ;) Thank you, Bruno! You've saved my day (or two:)! It didn't fly "out of the box" because I coulnd't get "mongodump -h <host>" remotely, also had to mount new separate 30GB EBS on Amazon due to luck of required free space on "/" drive (and accordingly change paths here and there). Also during 1st step (when you do mongorestore) got an error: going into namespace [admin.system.users] assertion: 17415 Cannot restore users with schema version 1 to a system with server version 2.5.4 or greater", workaround was to remove /data/db/dump/admin folder.

[Shlomi Uziel]

Just adding a clarification for the sake of future reference: The above link states that only adjacent major version are compatible with each other for upgrade purposes, when they're part of the same replica set. However, it clearly suggests a path of directly restoring a dump of 2.4 into a 3.2, with the sole caveat of the authentication schema. So to answer the original question - this should work in practice.