Using curl to request URL that redirects to relative URL with anchor tag

Question

I'm using curl to request a URL that redirects to a different URL using a Location: line like:

Location:/path/to/resource#name

As I understand it, that line in the redirect response is invalid per the HTTP specifications, so the overall curl call understandably fails (in this case, with a 400 response code). However, requesting the URL with wget or a web browser successfully renders the page (I assume through heuristics that fill in the absolute path or remove the anchor tag before the redirect).

Is there anything I can do to make curl do the same (do what is necessary to successfully follow the redirect, even if it is "officially" malformed)?

Edit: Some more details. The final response code is 400 (not 404 or something else). When I do a HEAD request (with curl -I -L), I get a 302 Found (with Location: /Error) that redirects to a 500 Server Error. However, if I do a regular request (without the -I option but with the -L option), I get a http_code (in curl's --write-out) of 400. So it seems like HEAD requests in this case function differently than standard GETs.

Solution 1

I discovered this particular issue occurs with curl 7.15.5 but works fine with curl 7.21.0. So there must have been a bug fix or feature implemented in between that solves it. If anyone knows exactly what patch or change addressed this, that would be appreciated!

Solution 2

As I understand it, curl doesn't follow Location headers by default.

You can enable such behaviour by using the -L or --location switch. Like so:

tom@slappy:~▶ curl -I -L http://shell/redirect.php     
HTTP/1.1 302 Found
Date: Tue, 17 Jan 2012 00:13:54 GMT
Server: Apache/2.2.17 (Ubuntu)
X-Powered-By: PHP/5.3.5-1ubuntu7.2
Location: /target.html#someAnchor
Content-Type: text/html

HTTP/1.1 200 OK
Date: Tue, 17 Jan 2012 00:13:54 GMT
Server: Apache/2.2.17 (Ubuntu)
Last-Modified: Tue, 17 Jan 2012 00:10:37 GMT
Accept-Ranges: bytes
Content-Length: 7
Content-Type: text/html

Note: -I is only used to show the headers rather than page content. I tested this using curl 7.21.6 running on Ubuntu.

Comments

[AbdelKh]

Sorry for the vagueness, I am using -L to follow redirects. The problem is that the malformed Location: line makes the curl call fail, while wget and browsers succeed. Your example does seem to work though, strange...

[Tom Hudson]

Can you post the complete headers for the request and response? (Just the whole thing with -I really) for comparison?

[AbdelKh]

Edited question to add more details.

[elmo]

Yes, this behavior was recently fixed in curl - daniel.haxx.se/blog/2011/12/31/top-3-curl-bugs-in-2011