How to access http server from bash script with existing tcp connection?

Question

In a bash shell scipt, I use several command line tools (wget, curl, httpie) to test my http server.

When using e.g. curl to invoke a GET request, I see that a tcp connection is opened to my server and closed immediatly after the http communication has been completed.

$ curl http://10.5.1.1/favicon.ico -o /dev/null 

For better testing the keep-alive behaviour of my server, I want to keep the tcp connection open over multiple http request / response cycles.

I am able to perform poor http requests directly from bash like this:

exec 3<>/dev/tcp/10.5.1.1/80
printf "GET /\r\n\r\n" >&3
while IFS= read -r -u3 -t2 line || [[ -n "$line" ]]; do echo "$line"; done
exec 3>&-
exec 3<&-

But because my testing communication is much more complex (incl. authentication et cetera), I don't want to code this completely myself in bash or other scripting language but want to use existing tools instead. The idea is now to first open a tcp socket in bash and then forcing some command line tools to use this connection instead of open their own sockets. Maybe like this;

exec 3<>/dev/tcp/10.5.1.1/80
curl http://10.5.1.1/message.txt --use-existing-socket-fd=3
# ... do some other testing stuff ...
wget http://10.5.1.1/message.txt --use-existing-socket-fd=3
# ... do some other testing stuff ...
http http://10.5.1.1/message.txt --use-existing-socket-fd=3
exec 3>&-
exec 3<&-

Might this be possible using the above mentioned tools or with other tools?

Solution

The tool you are looking for is socat. Since you are testing one single web server, you can ask it to establish a permanent connection to that server (as long as it doesn't choose to close it -- adjust your timeouts accordingly) and once this is done, you can use the connection as a tunnel. Below are two ways to accomplish this.

Querying through a Unix socket

curl has a --unix-socket option that allows you to send HTTP requests and receive HTTP replies through a Unix socket (thank you, thrig, for your enlightening comment)

You would use it like this:

socat TCP:10.5.1.1:80 UNIX-LISTEN:/tmp/curl.sock,fork

Then, on another terminal:

curl --unix-socket /tmp/curl.sock http://10.5.1.1/message1.txt
curl --unix-socket /tmp/curl.sock http://10.5.1.1/message2.txt
...

Querying through a pseudo-HTTP-proxy

You can also make your tunnel available as a pseudo-proxy through which wget, curl,... would connect. This solution has the advantage to not be limited to curl.

This time, socat listens to a local TCP port (say 3128):

socat TCP:10.5.1.1:80 TCP-LISTEN:3128,fork,reuseaddr

Then, on another terminal:

export http_proxy='http://localhost:3128'
curl http://10.5.1.1/message.txt
wget http://10.5.1.1/message.txt
....

Note that since the HTTP client is using a proxy, the HTTP request will be slightly altered and this may not be desirable.

Of course, none of those two solutions are intended to be used with multiple HTTP servers since this is your web-server at the end of the channel that receives all the requests.

Comments

[thrig]

curl (of sufficient version) can use --abstract-unix-socket or --unix-socket though those after some fiddling around appear less practical than just using TCP

[xhienne]

Thanks a lot @thrig, I didn't know those options and I have updated my answer accordingly (only with --unix-socket though, since my curl is not fresh enough to support the --abstract-unix-socket option).

[thrig]

When I looked at tcpdump the socket option was sending SYN packets for each command, unlike when TCP is used, so they may be different.

[xhienne]

@thrig I see only one single SYN from client to server. Did you call socat the way I call it in my answer?

[thrig]

Ahh, I had the UNIX-LISTEN before the TCP, opposite of what you have

[Joe]

Pseudo-HTTP-proxy approach: The request will be altered from GET /message.txt to GET http://10.5.1.1/message.txt and this o.a. let Digest Access Authentication fail (because of altered URI). I'd now changed my server to strip that http://10.5.1.1 from the URI and now it also works with Digest Access Authentication :-)

[xhienne]

@Joe Didn't think about Authentication. Glad to learn that you found a way to circumvent that.