How to disable keepalive in curl

Question

I'm trying to figure out how to consistently turn off keepalive across various client machines which are issuing HTTP requests via curl.

This is my target server:

- Ubuntu 18.04.2 LTS
- 4.15.0-47-generic
- HA-Proxy version 1.8.19-1ppa1~bionic 2019/02/12

This is client 1 where from I'm issuing curl (vanilla installation):

- Ubuntu 16.04.3 LTS
- 4.4.0-62-generic
- curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

This is client 2 where from I'm issuing curl (vanilla installation):

- Ubuntu 18.04 LTS
- 4.15.0-20-generic
- curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.0g zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3

To turn off keepalive I've tried using -H "Connection: close", --no-keepalive and --keepalive-time 1 and only the first option seems to work but only from client 1.

On client 1 (Ubuntu 16) the connection is not left open but from client 2 (Ubuntu 18) the connection is left open until it times out. I confirm that either by looking at the target server's watch -n 0.1 "netstat -na | fgrep CLIENT_IP_ADDRESS" or by using -vvv on both clients, which on client 1 is always * Closing connection 0 and on client 2 is always * Connection #0 to host www.example.com left intact.

The difference between client 1 and 2 are obviously the Ubuntu version and curl version. What is it that causes the the connection to be closed in client 1 but not in client 2?

I've also found out that if I change my target server to another machine running an old distro with an old apache httpd, whether I send out Connection: close or not doesn't make any difference and keepalive is always used from either of my 2 clients. So I suppose that the target server configuration also plays some role in that.

Edit: To complicate things even further, if from client 1 and 2 I issue requests to the target server via ab then the connection is killed instantly. That is expected because ab uses HTTP/1.0 and not HTTP/1.1. But then, if I target the old distro with apache then in both cases the connection remains open. Which means that, indeed, the receiving end plays a role as well.

Edit 2: I grabbed all /proc/sys/net/ipv4/ settings of both clients via:

for file in /proc/sys/net/ipv4/*
do
  echo "$file $(cat $file)"
done

and these can be found here:

• U16: https://pastebin.com/GcuuqhvA
• U18: https://pastebin.com/a2QWReFZ
• diff: https://pastebin.com/25Ccyz9h

Solution 1

The --no-keepalive option is only useful for TCP keepalive packets, as mentioned in this archived thread on the curl website: https://curl.haxx.se/mail/archive-2013-04/0037.html

Sounds like you'll need to disable HTTP keepalive packets specifically, which would be done on your servers using keepalive_timeout 0; as mentioned in this stackoverflow thread https://stackoverflow.com/questions/24924237/linux-curlnginx-cant-request-with-no-keepalive.

Sorry if the formatting is off, this is my first post :)

Hope that helps!

Solution 2

Per https://everything.curl.dev/usingcurl/persist

curl will always try to keep connections alive and reuse existing connections as far as it can.

Therefore, you cannot use it using the tools as they are.

As @pl-nowlan mentioned in their answer, the only way to do that would be on the server side.

Comments

[Admin]

Is there any tcp parameters changes between these 2 clients such as net.ipv4.tcp_syn_retries ?

[Admin]

In both clients cat /proc/sys/net/ipv4/tcp_syn_retries returns 6.

[Admin]

@asktyagi: I've added all /proc/sys/net/ipv4/ settings in the question.

[FooBee]

please read the question. This option has been mentioned.

[cherouvim]

Thanks for pointing me to the man curl. As I mention in the question I've tried that. Both client 1 and 2 leave the connection open when using --no-keepalive. I've found out that by using -H "Connection: close" at least kills the connection on client 1.

[Naveed Abbas]

Your formatting is fine... Welcome to the site :)