HttpClient throws TruncatedChunkException accessing large chunked resource

Question

[using httpcore 4.1.4, httpclient 4.2.5, Oracle JDK 1.7.0_25]

I'm trying to 'proxy' a connection to a third party web service on behalf of a webapp's javascript (AJAX) code and it seems to fail on large chunked responses, erroring part way through a chunk by sending multiple RSTs and throwing a org.apache.http.TruncatedChunkException.

So I'm wondering:

1. why is http client trying to drop the connection?
2. is it doing something sensible? (ie. is the server possibly at fault) or is there something buggy going on here?

My basic approach is to copy everything from a servlet's request object to an apache components httpclient request and execute. More specfically, I:

1. create an apache commons httpclient DefaultHttpClient object,
2. copy all request headers across to a new request object,
3. set (/override) the host header on the new request with the host/port I'm proxying to,
4. copy all HTTP parameters to the new request,
5. copy any entity body across to the new request,
6. execute the request,
7. copy response headers to my servlet's response headers, and
8. copy any entity body as a stream across to the servlet's output stream.

The bit that is causing me issues is the last one. It seems to fail half way through a chunk and I get the following stacktrace:

org.apache.http.TruncatedChunkException: Truncated chunk ( expected size: 7752; actual size: 4077)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138)
at <mypackage>.<MyServlet>.service(<MyServlet>.java:XXX)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.resteasy.plugins.server.servlet.FilterDispatcher.doFilter(FilterDispatcher.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:724)

I've snooped it with Wireshark and get a process something like this:

source  dest    info
client  server  [SYN] seq=0
server  client  [SYN, ACK] seq=0 ack=1
client  server  [ACK] seq=1 ack=1
client  server  GET /url?param=value... HTTP/1.1
server  client  [ACK] seq=1 ack=221
server  client  [TCP segment of a reassembled PDU]
client  server  [ACK] seq=221 ack=4345
client  server  [FIN, ACK] seq=221 ack=4345
server  client  [TCP segment of a reassembled PDU]
client  server  [RST] seq=221
server  client  Continuation or non-HTTP traffic
client  server  [RST] seq=221

In my limited understanding, FIN means 'I'm done sending', which IMO is fair enough since the client headers are already sent. However RST/reset seems to just attempt to drop the connection.

The HTTP headers for client are:

GET /some/path?params=values HTTP/1.1
connection: Keep-Alive
host: target.host.com
accept: */*
user-agent: Wget/1.14 (linux-gnu)

And for the server:

HTTP/1.1 200 OK
Date: Mon, 16 Sep 2013 03:59:37 GMT
Server: Apache-Coyote/1.1
Content-Disposition: inline; filename=geoserver-GetFeature.text
Content-Type: text/xml; subtype=gml/2.1.2
Vary: Accept-Encoding
Connection: close
Transfer-Encoding: chunked

btw, this question: [restlet ]TruncatedChunkException: looks similar, but doesn't seem to have any helpful info.

Update: I've tried with a non-chunked site (/. :-) ) and it fails similarly with a:

org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body

Solution

OK, I found out what I'd done. I'd cleaned up my connection too early - basically the boilerplate in my connection method had a

finally
{
    client.getConnectionManager().shutdown();
}

but the method returned the stream object, so reading wasn't completed when the shutdown occurred.

Comments

[Amit Sadafule]

Then how did you release the connection?

[Phasmal]

Sorry, this was so long ago I don't remember! I suspect I would have added an explicit close after waiting on the result.