/etc/security/limits.conf soft nproc limit appears to be ignored
Solution 1
Check the file /etc/security/limits.d/90-nproc.conf as this is likely overriding your settings. I wrote about this exact same issue last year http://scott.cm/max-processes-1024-limits-conf/
Solution 2
You could try to change the ulimit value at the beginning of the script, which is launching the mongodb database.
A child process inherits the resource limits from the calling process.
Related videos on Youtube
John M
Updated on September 18, 2022Comments
-
John M over 1 year
My MongoDB database was running into problems under load, with the following errors spamming the logs:
[initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable [initandlisten] can't create new thread, closing connection
I've come to the conclusion that I need to raise the "ulimit -u" or "Max processes" setting which were at 1024, and the usage could have been exceeding that given the web frontends launched (not sure how to check this).
I edited /etc/security/limits.conf to add the last two lines (the first two were already there):
* soft nofile 350000 * hard nofile 350000 * soft nproc 30000 * hard nproc 30000
Then I rebooted the system (BTW should I have done that, or should a mongod service restart be enough?)
After reboot, reviewing the process limits for mongod process it seems the soft limit has been ignored:
$ cat /proc/2207/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 1024 30000 processes Max open files 350000 350000 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 273757 273757 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us $ whoami mongod $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 273757 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 350000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
I expected that "Max processes" both hard and soft limits will be at 30000 as per the /etc/security/limits.conf file, but only the hard one is.
What am I doing wrong?
I'm running Amazon Linux on AWS EC2.
bash-4.1$ cat /etc/*-release Amazon Linux AMI release 2012.09
-
dmourati over 10 yearsLook in /etc/security/conf.d/? Sometimes there is a file there that overrides what is set in the main limits.conf.
-
John M over 10 yearsdmourati, thanks, you were right (same as Scott Mcinyre).
-
-
John M over 10 yearsI'd like to change the global value. It worked for 'nofile' and for the hard limit on 'nproc', but not for the soft limit on 'nproc'.
-
John M over 10 yearsThanks, that appears to have been it. Can you also tell me if just restarting the "mongod" process should be enough for it to read the new settings?
-
Scott Mcintyre over 10 yearsYeah once you change the limit all you will have to do is restart the mongod process and it will read the new limits.
-
John M over 10 yearsThanks, that worked as well, both limits are now at 30000. Thanks a lot for your help, Scott!
-
Alastair Irvine over 7 yearsInstead of editing a system file, I recommend creating a file in
/etc/security/limits.d/
that starts with99_local
(to show that you created it) and override anything you want there.