Unable to start MongoDB 3.0.2 service on CentOS 7

Question

We are setting up a MongoDB server for the production environment on Amazon EC2 instance, but could not able to start the service. I've followed this documentation for setup. Here are the steps, I've taken for setting up the server:

Added following to /etc/yum.repos.d/mongodb-org-3.0.repo

[mongodb-org-3.0]
name=MongoDB Repository
baseurl=http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/
gpgcheck=0
enabled=1

And installed MongoDB 3.0.2 using sudo yum install -y mongodb-org-3.0.2

Created three partitions for data, journal & log:

sudo mkdir /mongo
sudo mkdir /mongo/data
sudo mkdir /mongo/log
sudo mkdir /mongo/journal

Created file system for three separate partitions:

sudo mkfs.ext4 /dev/xvdb
sudo mkfs.ext4 /dev/xvdc
sudo mkfs.ext4 /dev/xvdd

Created entry in fstab for reboot:

echo '/dev/xvdb /mongo/data ext4 defaults,auto,noatime,noexec 0 0
/dev/xvdc /mongo/journal ext4 defaults,auto,noatime,noexec 0 0
/dev/xvdd /mongo/log ext4 defaults,auto,noatime,noexec 0 0' | sudo tee -a /etc/fstab

And mounted the partitions:

sudo mount /mongo/data
sudo mount /mongo/journal
sudo mount /mongo/log

Given the permissions and created link

sudo chown mongod:mongod /mongo/data /mongo/journal /mongo/log
sudo ln -s /mongo/journal /mongo/data/journal

Configured ulimit & read ahead settings as given in the documentation link above. Verified permissions and partitions:

[deployer@prod-mongo ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      8.0G  1.3G  6.8G  16% /
devtmpfs        3.6G     0  3.6G   0% /dev
tmpfs           3.5G     0  3.5G   0% /dev/shm
tmpfs           3.5G   57M  3.4G   2% /run
tmpfs           3.5G     0  3.5G   0% /sys/fs/cgroup
/dev/xvdc       7.8G   36M  7.3G   1% /mongo/journal
/dev/xvdb       150G   51M  149G   1% /mongo/data
/dev/xvdd       3.9G   16M  3.6G   1% /mongo/log

Permissions:

[deployer@prod-mongo ~]$ ll /
total 32
lrwxrwxrwx.   1 root   root      7 Sep 29  2014 bin -> usr/bin
dr-xr-xr-x.   4 root   root   4096 Sep 29  2014 boot
drwxr-xr-x.  17 root   root   2860 May 11 12:11 dev
lrwxrwxrwx.   1 root   root      7 Sep 29  2014 lib -> usr/lib
lrwxrwxrwx.   1 root   root      9 Sep 29  2014 lib64 -> usr/lib64
drwxr-xr-x.   2 root   root      6 Jun 10  2014 mnt
drwxr-xr-x.   5 mongod mongod   41 May 11 05:06 mongo
drwxr-xr-x.  21 root   root    660 May 11 12:47 run
lrwxrwxrwx.   1 root   root      8 Sep 29  2014 sbin -> usr/sbin

Inside /mongo

[deployer@prod-mongo ~]$ ll /mongo/
total 12
drwxr-xr-x. 3 mongod mongod 4096 May 11 07:33 data
drwxr-xr-x. 3 mongod mongod 4096 May 11 07:31 journal
drwxr-xr-x. 3 mongod mongod 4096 May 11 08:58 log

After changing the configurations inside /etc/mongodb.conf

logpath=/mongo/log/mongod.log
dbpath=/mongo/data

and when I'm doing: sudo service mongod start, I'm getting this error:

Starting mongod (via systemctl):  Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
                                                           [FAILED]

Further logging:

[deployer@prod-mongo ~]$ sudo systemctl status mongod.service

mongod.service - SYSV: Mongo is a scalable, document-oriented database.
   Loaded: loaded (/etc/rc.d/init.d/mongod)
   Active: failed (Result: exit-code) since Tue 2015-05-12 04:42:10 UTC; 42s ago
  Process: 22881 ExecStart=/etc/rc.d/init.d/mongod start (code=exited, status=1/FAILURE)

May 11 04:42:10 ip-xx-xx-xx-xx.local runuser[22887]: pam_unix(runuser:session): session opened for user mongod by (uid=0)
May 11 04:42:10 ip-xx-xx-xx-xx.localdomain runuser[22887]: pam_unix(runuser:session): session closed for user mongod
May 11 04:42:10 ip-xx-xx-xx-xx.local mongod[22881]: Starting mongod: [FAILED]
May 11 04:42:10 ip-xx-xx-xx-xx.local systemd[1]: mongod.service: control process exited, code=exited status=1
May 11 04:42:10 ip-xx-xx-xx-xx.local systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..
May 11 04:42:10 ip-xx-xx-xx-xx.local systemd[1]: Unit mongod.service entered failed state.

I've followed various articles and blog posts and StackExchange answers but didn't get any solution. Am I missing something?

Update: If I'm directly running the mongodb service from the normal user something like this: sudo mongod --logpath ~/mongod.log --dbpath ~/mongodata, then this service is starting properly.

We tried changing the path of the pid file to another directory, that didn't help either.

Solution 1

In case, anyone encountered the same issue with MongoDB startup, here is the thread of comments https://jira.mongodb.org/browse/SERVER-18439. This is scheduled to be fixed in 3.1.

Solution 2

I'm guessing you're running a flavour of Linux that uses SELinux (RHEL or CentOS 7, perhaps?)

If so, the issue is that you don't have a permissive policy on your /mongo/ directory that permits access to daemons (like the mongod service.)

From Wikipedia:

SELinux can potentially control which activities a system allows each user, process and daemon, with very precise specifications. However, it is mostly used to confine daemons[citation needed] like database engines or web servers that have more clearly defined data access and activity rights. This limits potential harm from a confined daemon that becomes compromised. Ordinary user-processes often run in the unconfined domain, not restricted by SELinux but still restricted by the classic Linux access rights

To check whether this is the issue, try this at the shell:

sudo setenforce 0

This should disable SELinux policies and allow the service to run.

For a more permanent solution, see https://wiki.centos.org/HowTos/SELinux

Solution 3

I ran into this problem and actually found a solution for me.

In short, mongodb 3.2 uses the user 'mongod' while older versions use 'mongodb'. Some of the files and directories were owned by 'mongodb' (the older user). Once I chmod'd them to the 'mongod' user, I was able to use systemctl to control the mongod process.

More specifically, it was the "/var/log/mongodb/*" files that had the wrong user ownership.

root@<HOST>:# ls -alh /var/log/mongodb
total 664K
drwxr-xr-x   2 mongod  mongod  4.0K Oct 27 12:08 .
drwxr-xr-x. 22 root    root    4.0K Oct 27 11:51 ..
-rw-r--r--   1 mongodb mongodb 3.8K Oct 27 11:48 mongod.log
-rw-r--r--   1 mongodb mongodb  19K Apr 14  2016 mongod.log.2016-04-14T18-29-34
-rw-r--r--   1 mongodb mongodb 2.8K Apr 14  2016 mongod.log.2016-04-14T18-30-13
-rw-r--r--   1 mongodb mongodb  12K Apr 14  2016 mongod.log.2016-04-14T22-27-27
-rw-r--r--   1 mongodb mongodb  11K Apr 14  2016 mongod.log.2016-04-14T22-29-12
-rw-r--r--   1 mongodb mongodb 5.6K Apr 18  2016 mongod.log-20160418.gz
-rw-r--r--   1 mongodb mongodb    0 Apr 18  2016 mongod.log.2016-09-09T17-33-48
-rw-r--r--   1 mongodb mongodb 3.6K Sep  9 11:34 mongod.log.2016-09-09T17-34-52
-rw-r--r--   1 mongodb mongodb  23K Sep  9 11:49 mongod.log.2016-09-09T17-49-49
-rw-r--r--   1 mongodb mongodb 5.0K Sep  9 11:55 mongod.log.2016-09-09T17-55-15
-rw-r--r--   1 mongodb mongodb 5.0K Sep  9 12:02 mongod.log.2016-09-09T18-02-26
-rw-r--r--   1 mongodb mongodb 5.0K Sep  9 12:13 mongod.log.2016-09-09T18-13-17
-rw-r--r--   1 mongodb mongodb 5.0K Sep  9 12:25 mongod.log.2016-09-09T18-25-01
-rw-r--r--   1 mongodb mongodb 5.2K Sep  9 12:47 mongod.log.2016-09-09T18-47-54
-rw-r--r--   1 mongodb mongodb 5.0K Sep  9 12:52 mongod.log.2016-09-09T18-52-16
-rw-r--r--   1 mongodb mongodb 5.0K Sep  9 12:54 mongod.log.2016-09-09T18-54-49
-rw-r--r--   1 mongodb mongodb 5.0K Sep  9 13:01 mongod.log.2016-09-09T19-01-22
-rw-r--r--   1 mongodb mongodb 3.0K Sep  9 13:03 mongod.log.2016-09-09T19-03-21
-rw-r--r--   1 mongodb mongodb 215K Sep  9 14:25 mongod.log.2016-09-09T20-25-59
-rw-r--r--   1 mongodb mongodb 281K Sep 10 03:42 mongod.log-20160910
-rw-r--r--   1 mongodb mongodb    0 Sep 10 03:42 mongod.log.2016-10-27T17-42-42
-rw-r-----   1 mongod  mongod     0 Sep 29 22:03 mongod.log.rpmnew

Notice the owner of the directory is 'mongod' (the new user) while the log files are all owned by 'mongodb' (the old user).