Persistent/keepalive HTTP with the PHP Curl library?
Solution 1
cURL PHP documentation (curl_setopt) says:
CURLOPT_FORBID_REUSE
-TRUE
to force the connection to explicitly close when it has finished processing, and not be pooled for reuse.
So:
- Yes, actually it should re-use connections by default, as long as you re-use the cURL handle.
- by default, cURL handles persistent connections by itself; should you need some special headers, check CURLOPT_HTTPHEADER
- the server may send a keep-alive timeout (with default Apache install, it is 15 seconds or 100 requests, whichever comes first) - but cURL will just open another connection when that happens.
Solution 2
Curl sends the keep-alive header by default, but:
- create a context using
curl_init()
without any parameters. - store the context in a scope where it will survive (not a local var)
- use
CURLOPT_URL
option to pass the url to the context - execute the request using
curl_exec()
- don't close the connection with
curl_close()
very basic example:
function get($url) {
global $context;
curl_setopt($context, CURLOPT_URL, $url);
return curl_exec($context);
}
$context = curl_init();
//multiple calls to get() here
curl_close($context);
Solution 3
On the server you are accessing keep-alive must be enabled and maximum keep-alive requests should be reasonable. In the case of Apache, refer to the apache docs.
You have to be re-using the same cURL context.
-
When configuring the cURL context, enable keep-alive with timeout in the header:
curl_setopt($curlHandle, CURLOPT_HTTPHEADER, array( 'Connection: Keep-Alive', 'Keep-Alive: 300' ));
Solution 4
If you don't care about the response from the request, you can do them asynchronously, but you run the risk of overloading your SOLR index. I doubt it though, SOLR is pretty damn quick.
Frank Farmer
Currently professionally working with the LAMP stack -- as well as with postgres, varnish, and nginx.
Updated on July 08, 2022Comments
-
Frank Farmer almost 2 years
I'm using a simple PHP library to add documents to a SOLR index, via HTTP.
There are 3 servers involved, currently:
- The PHP box running the indexing job
- A database box holding the data being indexed
- The solr box.
At 80 documents/sec (out of 1 million docs), I'm noticing an unusually high interrupt rate on the network interfaces on the PHP and solr boxes (2000/sec; what's more, the graphs are nearly identical -- when the interrupt rate on the PHP box spikes, it also spikes on the Solr box), but much less so on the database box (300/sec). I imagine this is simply because I open and reuse a single connection to the database server, but every single Solr request is currently opening a new HTTP connection via cURL, thanks to the way the Solr client library is written.
So, my question is:
- Can cURL be made to open a keepalive session?
- What does it take to reuse a connection? -- is it as simple as reusing the cURL handle resource?
- Do I need to set any special cURL options? (e.g. force HTTP 1.1?)
- Are there any gotchas with cURL keepalive connections? This script runs for hours at a time; will I be able to use a single connection, or will I need to periodically reconnect?
-
Frank Farmer almost 15 yearsThat's certainly interesting, but it doesn't address connection re-use at all. In fact, it would only make my connection overhead issues worse.
-
Oleg Barshay over 14 yearsFrank, I just re-tested my code and it looks to be on by default. Couldn't hurt to set it explicitly though.
-
renevanderark almost 10 yearsBrilliant! I was this close to posting my first stackoverflow question. This solution worked for our middleware provided we added the request header 'Connection: close'.
-
zeflex almost 9 years@OlegBarshay do you know if we need to remove
curl_close($curlHandle);
in order to keep alive the conn. ? -
Grain about 7 years@zeflex yes, you have to remove it, if you call
curl_close
the connection will be closed -
Malus Jan over 6 yearsYou also need to set cookie before second call, something like:
curl_setopt($context, CURLOPT_COOKIE, 'name=value');
for example for my request iscurl_setopt($context, CURLOPT_COOKIE, 'PHPSESSID=bl392rgi8q664l7faat33hfta4');
-
AaA over 2 yearsWill curl_exec create a new connection and set the handle again if server drop the connection, even though keep alive is still active?