Increasing FcgidMaxProcesses does not help to solve "mod_fcgid: can't apply process slot" error

Question

I have a dedicated apache server with high resources.

• RAM - 64GB
• Cores - 20
• with CentOS 6.7 x64

and

Image - FCGI statistics with phpinfo

I run Magento application with MySQL as database engine which is also deployed on a separate highly resourced dedicated server.

Website was working fine since a month, but suddenly it has started throwing error though it is not under load/traffic-

Image - Server Load statistics with HTOP

mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper with 503 service temporary unavaiable error on browser.

So, I followed below steps to get rid of it(but none of them actually worked) :

1. I increased value of "FcgidMaxProcesses" in fcgid.conf and restarted apache, but it didn't make any difference. Error are still there.

2. I checked file ownership and permission of

   755 apache apache /var/run/mod_fcgid/sock
   644 root root /var/run/mod_fcgid/fcgid_shm

3. Rebooted server, expecting existing FCGI processes to get flushed and process count to start with 0 so process count will not hit FcgidMaxProcesses for some days

Below is my fcgid.conf file.

I know timeout values are way too high but I guess it has nothing to do with this error as page load time is around 4-5 seconds.

LoadModule fcgid_module modules/mod_fcgid.so

<IfModule !mod_fastcgi.c>
    AddHandler fcgid-script fcg fcgi fpl
</IfModule>

FcgidIPCDir /var/run/mod_fcgid/sock
FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm

FcgidProcessLifeTime 120
FcgidIdleTimeout 30
FcgidIdleScanInterval 5
FcgidMaxRequestsPerProcess 300
FcgidMaxProcesses 1200
FcgidIOTimeout 18000
FcgidBusyTimeout 18000

size of /usr/bin/php-cgi process is on an average 70MB

My question is why increasing value of FcgidMaxProcesses doesn't make any difference? Is there any other configuration file that needs to be altered?

I have parallel plesk installed. Digging /etc folder, I came across another services which may, I guess, affect fcgi/FastCGI operations:

1. php-fpm
2. sw-engine-fpm

Is Fcgid dependent on any such service to limit its processes? How do I detect that?

Please make me aware of the reality.

Thanks in advance.

Solution 1

Issue is resolved now.

And problem was not related to FCGID or any other Apache module configuration.

The reason behind sudden deteriorating behaviour of server was attack Yes, that's right! Though it was not DDoS, but continuous requests of attacker with different query string patterns(SQL injection) and page load time of the targeted page - excluding query string parameter - is around 3 seconds resulted in hit for threshold limit of FCGID process.

I came to know about it when I enabled mod_status which directly highlighted such attempts to breach.

I blocked the IP of that user and within a minute all operations back to normal.

Solution 2

ok about FcgidMaxProcesses, Nginx+FPM and possible attacks...

but remember that if you use Fcgid, you also need to check

FcgidMaxProcessesPerClass xxx

that means that any user on the server (so per user) has a max number of processes available and when you hit that limit a new slot can't be applied...

Comments

[Munjpara]

Thanks Oleg for your suggestion and regarding my question, referring to number of link related to my error, they suggest to increase FCGIDMAXPROCESSES value but in my case it does not solve the solve as same error is still getting registered in log. So my question is why is that so?

[Munjpara]

Moreover, my server is not, at all, under any load. Currently load average: 0.68, 0.97, 1.00 and still it is throwing the same error. I know it is fastcgi + Apache issue, but as I want to fine tune fcgi configuration at the first place and then look for an alternative, where should I start from as increasing maxprocesses in fcgid.conf is of no help in my case.

[Oleg Neumyvakin]

it's not a server performance issue, it's an issue in communication of apache and apache module. And as I've already said - it's can't be fixed with apache.

[Munjpara]

Oleg, but my server is not overloaded, so I thought it could be fixed with some changes in configurable variables.

[Oleg Neumyvakin]

I've understand it, the issue not in server performance but in intercommunications in apache and fcgid.

[Munjpara]

I am facing problem since last 4-5 days, till then it was working perfectly. Do you have any idea/clue what might have cause such an unexpected behaviour?

[Oleg Neumyvakin]

No any idea, it also may be bad PHP code or PHP settings, actually there are may be anything.

[cr0ybot]

THANK YOU for pointing me in this direction. I'm not sure why it didn't click that there was a single IP address showing up over and over again in the log files with the errors. Banning the IP resolved the issue.