How can I use openssl to get results from HTTP GET requests?

Question

I need to use openssl to perform some HTTP GET requests in a shell script. The line I'm using to do this right now is shown below. This is parsing the content of an XML response of the following formats.

<Result>success</Result>

<Result>failure</Result>

echo -e "GET /test HTTP/1.1\r\nHost:$(hostname)\r\n\r\n" | openssl 2>&1 s_client -quiet -connect server-url:443 | grep -o -P --color '(?<=Result\>).*(?=\</Result)'

This works and returns the string 'success' or 'failure' accordingly. The problem I'm facing is that the openssl command does not terminate after doing the GET request but instead sits there waiting on more input. I believe this is due to the implicit -ign_eof which prevents automatic termination caused by the -quiet option. I've tried using the -no_ign_eof option but that causes the openssl command to terminate before the GET request has received a response so I can't get the content of the response if I use that.

How can I modify this command so I can pass the GET request through stdin (required as I want to put this in a loop) but have the openssl command terminate after each request?

Solution

What you really should be doing is using a tool designed for fetching Web resources, such as curl, wget, or libwww-perl's GET command. If nothing is available, you should have your system administrator install something appropriate.

With that out of the way...

The openssl command does not terminate because the web server didn't close the connection.

Remember that by default HTTP keeps connections open after each request as a performance optimization. Once one request finishes, another request can be sent over the same connection, rather than closing and reopening a new connection.

If you want to instruct the server to close the connection instead, you can send the Connection: close HTTP header.

Comments

[conorgriffin]

As mentioned in the comments on the question, unfortunately it is proving difficult to get open source tools installed on this AIX system and openssl seems to be the only tool available to me to do this right now. I've tried your suggestion and it works. Thanks

[Michael Hampton]

As a technical matter it's very easy to install the tools. I expect your roadblocks are entirely political in nature.

[conorgriffin]

yes they are, IBM even supply a list of open source tools on their website that includes curl www-03.ibm.com/systems/power/software/aix/linux/toolbox/…

[Michael Hampton]

Remember, and remind anyone who is in a position to listen, that workarounds like this ultimately cost money.

[Naveed Abbas]

It might work, but new-ish http servers sometimes can't handle 1.0.

[kasperd]

@kubanczyk I have yet to come across an http server that cannot handle HTTP/1.0. A much more likely problem would be an http server that requires a Host header and a client that only sends the Host header when using HTTP/1.1. But ultimately the reason using HTTP/1.0 would help here is that it has different default behavior regarding closing the connection compared to HTTP/1.1. Adding a Connection header to the request would address that.

[Michael Hampton]

The problem here is that # is a special character. Nothing after the # is intended to be sent to the server. It is interpreted client side only. It does not make sense to attempt to send it to the server.

[mforsetti]

please check RFC3986, you'll find that fragments (portion of URI that's after the #) is dereferenced solely by the useragent (e.g. browser). second, try adding Connection: keep-alive and Accept: text/html;q=0.8 on your openssl s_client request, you'll find the server replies with 301.