org.eclipse.jetty.io.EofException: Early EOF thrown while uploading large file

Question

While I am uploading large file(about 50 MB), I am getting org.eclipse.jetty.io.EofException: Early EOF excception. My jetty server version is "9.2.9.v20150224".Below is stack trace

org.eclipse.jetty.io.EofException: Early EOF
at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:505) ~[jetty-server-9.2.9.v20150224.jar:9.2.9.v20150224]
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:124) ~[jetty-server-9.2.9.v20150224.jar:9.2.9.v20150224]
at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:142) ~[httpcore-4.4.1.jar:4.4.1]
at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:96) ~[httpcore-4.4.1.jar:4.4.1]
at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:112) ~[httpclient-4.5.jar:4.5]
at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:117) ~[httpcore-4.4.1.jar:4.4.1]
at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:265) ~[httpcore-4.4.1.jar:4.4.1]
at org.apache.http.impl.conn.AbstractClientConnAdapter.sendRequestEntity(AbstractClientConnAdapter.java:231) ~[httpclient-4.5.jar:4.5]
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:237) ~[httpcore-4.4.1.jar:4.4.1]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122) ~[httpcore-4.4.1.jar:4.4.1]
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) ~[httpclient-4.5.jar:4.5]
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) ~[httpclient-4.5.jar:4.5]
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) ~[httpclient-4.5.jar:4.5]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117) ~[httpclient-4.5.jar:4.5]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.jar:4.5]
at org.apache.http.client.HttpClient$execute$2.call(Unknown Source) ~[na:na]

org.eclipse.jetty.io.EofException
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:129)
at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:676)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:471)
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:763)
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:135)
at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:237)
at java_io_Flushable$flush.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at SendResponseFilter.writeResponse(SendResponse.groovy:128)
at SendResponseFilter$writeResponse.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
at SendResponseFilter.writeResponse(SendResponse.groovy:102)
at SendResponseFilter.run(SendResponse.groovy:59)
at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112)
at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:197)
at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:161)
at com.netflix.zuul.FilterProcessor.postRoute(FilterProcessor.java:92)
at com.netflix.zuul.ZuulRunner.postRoute(ZuulRunner.java:87)
at com.netflix.zuul.http.ZuulServlet.postRoute(ZuulServlet.java:107)
at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:88)
at realdoc.spider.SpiderServlet.service(SpiderServlet.java:66)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at com.netflix.zuul.context.ContextLifecycleFilter.doFilter(ContextLifecycleFilter.java:40)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:259)
at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:222)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:172)
... 55 more

If anybody have any idea on this issue, please help me out.

Solution 1

These stacktraces just show the results of a connection termination, but not the reason why it terminated.

The java.io.IOException: Broken pipe means the connection was terminated, unexpectedly at the OS level.

The org.eclipse.jetty.io.EofException during org.eclipse.jetty.io.ChannelEndPoint.flush() means the response body content was unable to finish flushing to the network as the connection was terminated.

The org.eclipse.jetty.io.EofException: Early EOF during org.eclipse.jetty.server.HttpInput.read() means that while actively reading the Request (unclear if its body content or headers from stacktrace) from the Connection (a java.nio.channels.SocketChannel in this case), it terminated prematurely, and the complete HTTP Request was not received (headers and body content).

There might be other errors in the logs from either your client or the server that could indicate what is going on.
You could test that request with other tooling (like curl) to see if its your client, or the network, or Jetty. You could also use a tool like wireshark to capture the network traffic and see where the connection termination originated (client or server)

Solution 2

What I have found in the issue is that,in http header "Content-Length" was coming as something like xxx, because of sloppy internet wifi, before entire content was read(less than xxx), stream was closed and above exception is thrown.

Another issue is timeout issue.Before entire stream is read (as its a large file), due to timeout, stream is prematurely closed and exception thrown as above.

There can be other reasons also however I have faced those issues.