RESTEasy Exception: RESTEASY003770: Response is committed, can't handle exception

Question

I'm getting a strange exception from my RESTEasy server. This happens when a particularly large response is being returned to the client.

The exception occurs after my code has returned the result object to RESTEasy, so it's all inside the RESTEasy layer. The server is TomCat.

Small responses are fine, but large ones are triggering the error. This happens if I return JSON or XML, and is triggered by size of the response, not the content in the objects I'm returning.

I've searched around, but haven't found anything helpful yet, and am pretty much lost at this point....

org.jboss.resteasy.spi.UnhandledException: RESTEASY003770: Response is committed, can't handle exception
        at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:174)
        at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:478)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:422)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209)
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1502)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1458)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:339)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:418)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:406)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:97)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletResponseWrapper$DeferredOutputStream.write(HttpServletResponseWrapper.java:46)
        at org.jboss.resteasy.util.CommitHeaderOutputStream.write(CommitHeaderOutputStream.java:71)
        at org.jboss.resteasy.util.DelegatingOutputStream.write(DelegatingOutputStream.java:48)
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2003)
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.close(UTF8JsonGenerator.java:1049)
        at org.jboss.resteasy.plugins.providers.jackson.ResteasyJackson2Provider.writeTo(ResteasyJackson2Provider.java:209)
        at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.writeTo(AbstractWriterInterceptorContext.java:131)
        at org.jboss.resteasy.core.interception.ServerWriterInterceptorContext.writeTo(ServerWriterInterceptorContext.java:60)
        at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:120)
        at org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingInterceptor.aroundWriteTo(GZIPEncodingInterceptor.java:100)
        at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:124)
        at org.jboss.resteasy.plugins.interceptors.encoding.ServerContentEncodingAnnotationFilter.aroundWriteTo(ServerContentEncodingAnnotationFilter.java:60)
        at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:124)
        at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:98)
        at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:473)
        ... 27 more
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:124)
        at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
        at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:172)
        at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:139)
        at org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:244)
        at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:189)
        at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:41)
        at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:320)
        at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:93)
        at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
        at org.apache.coyote.Response.doWrite(Response.java:501)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388)
        ... 47 more

Update: I actually had this happen again! Scott's answer below covered the first case, but the second time was due to inserting a null value for a response header. RESTEasy accepted the null header value at the time I inserted it, but then blew up when it tried to serialize the response after exiting from my code.

Solution 1

Ok, having been through this now a couple of times - I've personally seen this behaviour when there is an issue with the front end reverse proxy if you are using one. For example, I've seen issues when using nginx where the system was deployed and run with different users. A permission issue on temporary directories means you might not see the issue until the response hits a certain size. For example (it's one line, the line breaks are for visibility):

2017-04-20T17:46:40.03987 [nginx] 2017/04/20 13:46:40 [crit] 64537#0: 
  *14195 open() "/usr/local/var/run/nginx/proxy_temp/6/43/0000000436" 
  failed (13: Permission denied) while reading upstream, client: 10.0.0.1, 
  server: foo.bar.com, request: "GET /this/awesome/endpoint HTTP/1.1", 
  upstream: "http://127.0.0.1:9999/this/awesome/endpoint", host: "foo.bar.com"

Solution 2

Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe

It seems client is closing the connection before response is send completely to it ,can you increase client timeout and then check

Comments

[Roy Wood]

Ah-- that's exactly the issue. Thanks!

[Adam]

same situation for me but the reverse proxy was killing connections due to a short time-out. Increasing the time-out solved the problem.