mod_fcgid: read data timeout errors

Question

I moved to an unmanaged server that uses fcgid (before I was using mod_php), and in the error logs I see tons of such errors:

[Mon Apr 23 21:17:12 2012] [warn] [client 66.249.68.233] mod_fcgid: read data timeout in 31 seconds [Mon Apr 23 21:17:12 2012] [error] [client 66.249.68.233] Premature end of script headers: index.php

[Mon Apr 23 17:59:51 2012] [warn] [client 74.117.180.58] mod_fcgid: read data timeout in 31 seconds [Mon Apr 23 17:59:51 2012] [warn] [client 74.117.180.58] (110)Connection timed out: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function

There seem to be more of these when the load is higher (2-3) during backups, and I even managed to replicate this during a load of 3 when tar / mysqldump was running during a backup (the user sees a 500 error message after 30 seconds). Could the server be overloaded? This question seems to be related PHP + Fcgid hangs if download interrupted but not the same.

This is a top notch server, and I am surprised this would be too much. Here are some specs: 6-7 Drupal sites with Webmin

• Intel® Core™ i7-2600 Quadcore incl. Hyper-Threading Technology
• RAM 16 GB DDR3 RAM
• Hard disks2 x 3 TB SATA 6 Gb/s HDD 7200 rpm (Software-RAID 1)
• NIC1 Gbit OnBoard connected at 100 Mbit

Solution 1

These errors mean, that the scripts were running longer than 31 seconds and thus they were terminated, as your fcgid.conf says so. The standard timeout is 40 seconds btw.

You can easily check this behaviour by writing a test.php:

<?php sleep(32); ?>

This should give you an error 500 and put this error in your logs.

You have two possibilities to solve this:

1. Re-Make your index.php (or the application behind) and solve potential loop-issues (where the script runs forever and gets terminated after 31 seconds).

2. Set the timeouts higher. This has to be done for every vhost (don't forget SSL!), as this setting gets changed every time another vhost loads and will remain until the spawned process dies.
   Easiest way would be to edit /etc/apache2/mods-available/fcgid.conf. This is what we are using:

   IdleTimeout 3600
ProcessLifeTime 7200
IPCConnectTimeout 8
IPCCommTimeout 600
BusyTimeout 300

Edit: Oh, the second error is related to overly long query strings in URLs. To allow longer query strings, also edit fcgid.conf and insert

MaxRequestLen 15728640

Don't forget to restart apache to kill all running processes, so they get the new configs.

Solution 2

mysqldump by default will write lock the database while it runs so the data isn't altered during the backup, which can cause corruption. Drupal writes to the database on every request, so requests will hang while mysqldump is running, and eventually timeout.

If you're using InnoDB (or can convert to it), then you can use Percona XtraBackup to do hot backups. Short of that, don't pipe mysqldump to tar or gzip; dump the .sql file and then run tar/gzip on it after mysqldump has completed (and released the lock).

Also be aware that in certain versions of fcgid, there's a bug that causes it to only apply settings in VirtualHost blocks, and to apply those in the last-read block globally.

Comments

[giorgio79]

Thanks, I checked and MaxRequestLen is set to 1073741824 Also, if I would need to increase IPCCommTimeout to 600 for my app that would be a hell of a lot of time. This should be max 5-10 secs no? :)

[giorgio79]

Much appreciated will checkout XtraBackup.

[SebiF]

Of course, you should only increase slightly and have a look whether it fixes your problem. Normal values are between 20 and 40 (s). We use the 600 s for a Magento installation, which has to do extremely load-expensive operations. However, you should have an extra eye on it, if these timeouts occur while you are not mysqldumping.