MongoDB rs.initiate error: replSetInitiate quorum check failed because not all proposed set members responded affirmatively

Question

I have to start my own replica set with internal authentication enabled using X.509 certificates, but I failed. Any advice is welcome.

MongoDB 3.2 x64 on Debian 8.2 x64.

It is a problem from the MongoDB University course "M310: MongoDB Security".

A tutor:

You can bring up your server members with authentication options then rs.initiate, then create your user.

I have this folder structure:

~
`-- shared
    `-- certs
        |-- ca.pem
        |-- client.pem
        `-- server.pem

I created this bash script that sets up my replica set:

#!/bin/bash

course="M310"
exercise="HW-1.3"
workingDir="$HOME/${course}-${exercise}"
dbDir="$workingDir/db"
logName="mongodb.log"

ports=(31130 31131 31132)
replSetName="rs1"

host=`hostname -f`
initiateStr="rs.initiate({
                 _id: '$replSetName',
                 version: 1,
                 members: [
                  { _id: 0, host: '$host:${ports[0]}' },
                  { _id: 1, host: '$host:${ports[1]}' },
                  { _id: 2, host: '$host:${ports[2]}' }
                 ]
                })"

# create working folder
mkdir -p "$workingDir/"{r0,r1,r2}

# launch mongod's
for ((i=0; i < ${#ports[@]}; i++))
do
    mongod --dbpath "$workingDir/r$i" --logpath "$workingDir/r$i/$logName" --port ${ports[$i]} --replSet $replSetName --fork --sslMode requireSSL --sslPEMKeyFile ~/shared/certs/server.pem --sslCAFile ~/shared/certs/ca.pem --auth
    sleep 3
done

echo "Initiate replSet"
mongo --port ${ports[0]} --ssl --sslPEMKeyFile ~/shared/certs/client.pem --sslCAFile ~/shared/certs/ca.pem --host $host --eval "$initiateStr"

The last command returns this error:

MongoDB shell version: 3.2.11
connecting to: database.m310.mongodb.university:31130/test
{
        "ok" : 0,
        "errmsg" : "replSetInitiate quorum check failed because not all proposed set members responded affirmatively: database.m310.mongodb.university:31131 failed with not authorized on admin to execute command { replSetHeartbeat: \"rs1\", pv: 1, v: 1, from: \"database.m310.mongodb.university:31130\", fromId: 0, checkEmpty: true }, database.m310.mongodb.university:31132 failed with not authorized on admin to execute command { replSetHeartbeat: \"rs1\", pv: 1, v: 1, from: \"database.m310.mongodb.university:31130\", fromId: 0, checkEmpty: true }",
        "code" : 74
}

This is mongodb.log file of r0 instance:

2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] MongoDB starting : pid=6091 port=31130 dbpath=/home/enabokov/M310-HW-1.3/r0 64-bit host=database.m310.mongodb.university
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] db version v3.2.11
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] git version: 009580ad490190ba33d1c6253ebd8d91808923e4
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1k 8 Jan 2015
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] modules: none
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] build environment:
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten]     distmod: debian71
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten]     distarch: x86_64
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] options: { net: { port: 31130, ssl: { CAFile: "/home/enabokov/shared/certs/ca.pem", PEMKeyFile: "/home/enabokov/shared/certs/server.pem", mode: "requireSSL" } }, processManagement: { fork: true }, replication: { replSet: "rs1" }, security: { authorization: "enabled" }, storage: { dbPath: "/home/enabokov/M310-HW-1.3/r0" }, systemLog: { destination: "file", path: "/home/enabokov/M310-HW-1.3/r0/mongodb.log" } }
2017-01-18T15:48:08.536+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] 
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] 
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 7948 processes, 65536 files. Number of processes should be at least 32768 : 0.5 times number of files.
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] 
2017-01-18T15:48:10.833+0300 I REPL     [initandlisten] Did not find local voted for document at startup;  NoMatchingDocument: Did not find replica set lastVote document in local.replset.election
2017-01-18T15:48:10.833+0300 I REPL     [initandlisten] Did not find local replica set configuration document at startup;  NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2017-01-18T15:48:10.834+0300 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/home/enabokov/M310-HW-1.3/r0/diagnostic.data'
2017-01-18T15:48:10.834+0300 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-01-18T15:48:11.188+0300 I NETWORK  [initandlisten] waiting for connections on port 31130 ssl
2017-01-18T15:48:21.914+0300 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:56026 #1 (1 connection now open)
2017-01-18T15:48:21.922+0300 I ACCESS   [conn1] note: no users configured in admin.system.users, allowing localhost access
2017-01-18T15:48:21.927+0300 I REPL     [conn1] replSetInitiate admin command received from client
2017-01-18T15:48:21.947+0300 I REPL     [conn1] replSetInitiate config object with 3 members parses ok
2017-01-18T15:48:21.947+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to database.m310.mongodb.university:31131
2017-01-18T15:48:21.948+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to database.m310.mongodb.university:31132
2017-01-18T15:48:21.965+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to database.m310.mongodb.university:31131
2017-01-18T15:48:21.966+0300 W REPL     [ReplicationExecutor] Got error (Unauthorized: not authorized on admin to execute command { replSetHeartbeat: "rs1", pv: 1, v: 1, from: "database.m310.mongodb.university:31130", fromId: 0, checkEmpty: true }) response on heartbeat request to database.m310.mongodb.university:31131; { ok: 1.0, hbmsg: "" }
2017-01-18T15:48:21.966+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to database.m310.mongodb.university:31132
2017-01-18T15:48:21.967+0300 W REPL     [ReplicationExecutor] Got error (Unauthorized: not authorized on admin to execute command { replSetHeartbeat: "rs1", pv: 1, v: 1, from: "database.m310.mongodb.university:31130", fromId: 0, checkEmpty: true }) response on heartbeat request to database.m310.mongodb.university:31132; { ok: 1.0, hbmsg: "" }
2017-01-18T15:48:21.967+0300 E REPL     [conn1] replSetInitiate failed; NodeNotFound: replSetInitiate quorum check failed because not all proposed set members responded affirmatively: database.m310.mongodb.university:31131 failed with not authorized on admin to execute command { replSetHeartbeat: "rs1", pv: 1, v: 1, from: "database.m310.mongodb.university:31130", fromId: 0, checkEmpty: true }, database.m310.mongodb.university:31132 failed with not authorized on admin to execute command { replSetHeartbeat: "rs1", pv: 1, v: 1, from: "database.m310.mongodb.university:31130", fromId: 0, checkEmpty: true }
2017-01-18T15:48:21.969+0300 I NETWORK  [conn1] end connection 127.0.0.1:56026 (0 connections now open)

Without --auth parameter my replica set initiates successfully, but I need x509 authenticaion turned on for the problem purpose.

Solution

Your mongo node can not get the heartbeat (replSetHeartbeat) of the ones he is trying to connect to, in order to configure the replica set. This is because you have activated access control in the cluster.

When you activate mongo access control, you need to configure the internal authentication between the nodes of your replica set.

There are two different ways to do this:

• Using a Keyfile
• Using x509 auth (as you need)

You should follow the official documentation I referenced above.