su: /bin/bash: Resource temporarily unavailable

Question

Unable to switch user to postgres.

Ulimit settings for postgres user is set with reasonable limits. We are not hitting max.

No errors in /var/log/messages.

Error:

BETA -bash-4.2# sudo su - postgres
su: /bin/bash: Resource temporarily unavailable

Settings:

BETA -bash-4.2# ps -auxww | grep -i postgr | wc -l
503
BETA -bash-4.2# lsof | grep -i postgr | wc -l
35225
BETA -bash-4.2# 

Ulimit for postgres process.

BETA -bash-4.2# cat /proc/26230/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             256580               256580               processes 
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       256580               256580               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

Netstat

BETA -bash-4.2# netstat -plan | grep -i post | grep ESTABLISHED | wc -l
496
BETA -bash-4.2# 

Ulimit settings

BETA -bash-4.2# cat /etc/security/limits.d/postgres_limits.conf 
# Limits settings for postgres

postgres soft nofile 4096
postgres hard nofile 4096

postgres soft nproc 400
postgres hard nproc 400

After I restart postgres, I was able to get in.

Resource util for postgres user.

BETA -bash-4.2# netstat -plan | grep -i post | grep ESTABLISHED | wc -l
1
BETA -bash-4.2# 

BETA -bash-4.2# lsof | grep -i postgr | wc -l
309
BETA -bash-4.2# ps -auxww | grep -i postgr | wc -l
8
BETA -bash-4.2#  

Solution

Just restarting postgres is not a long term solution, you will hit the limit again, unless you have physical resource constraints on the server such as memory. During the issue the number of processes opened(nproc) by postgres user was 503 and the estimated number of open files(nofile) was 35225 and yet your postgres_limits.conf shows that you have set nproc to only 400 and nofile to only 4096. Based on your data, you will need to increase those two parameters.