Avoiding CGI timeout

Question

I have a function, that gets a large XML file, then parses it, and then uses the details, to get some more xml files, that are again parsed ( A webshop).

The problem is, that the processing takes about 15-20mins, but the script timeout's before that, and sends me a 500 error message.

Is there a way around this?

I read on a post here, that writing a log message every so often helps avoid this problem, but how do I do that? If I try to echo, write or return something, it just returns the value and ends my function.

Here are the error messages:

[Wed Oct 13 10:34:41 2010] [warn] [client 213.175.111.200] Timeout waiting for output from CGI script /home/virtual/site14/fst/var/www/interpreters/php-script
[Wed Oct 13 10:34:41 2010] [error] [client 213.175.111.200] Premature end of script headers: php-script
[Wed Oct 13 10:36:41 2010] [warn] [client 213.175.111.200] Timeout waiting for output from CGI script /home/virtual/site14/fst/var/www/interpreters/php-script

Thanks!

Solution

For those looking for an answer, there are a couple of things you could find, that seem to work:

1. Setting the TimeOut value on your Web server to 0 or something quite big. I ended up setting it to 0, since I needed my script to run , like, forever.
2. If 1 is not possible due to using shared hosting, Your most likely option is to return some log information periodically. I do have to warn you not to make this code into a function, since it is not possible to keep returning a function continuously. You can make a part of the code a function, but in the end, the Good old echo function will do it for you
3. You can try some Comet approaches, but they are fairly complex, and most probably wont solve every problem.

P.S. I somewhat got around my problem using javascript redirects, which were a pain in the a*s, so I do not recommend anyone doing it the same way.

Comments

[Ted Avery]

Thanks for following up on your original question. It's good for the Googlers like me trying to solve problems. For your solution #1, where exactly did you modify that TimeOut value?

[Janis Peisenieks]

Can be done both in config files as well as .htaccess .Look here: httpd.apache.org/docs/2.0/mod/core.html#timeout

[Alexis Wilke]

It looks like there is a specific request timeout module mods-available/reqtimeout.conf that can define timeouts with much greater granularity although probably not worth the trouble in our situations... (I'd even recommend to turn that one off!)