How to troubleshoot an Apache Server reverse proxy dropped connection

We are suddenly seeing an intermittent, but very frequent, error while POSTing files to a Tomcat servlet over https via an Apache Server reverse proxy. The error only seems to occur while POSTing files of 20 MB or more (videos). We have not seen the problem with smaller files of 2 to 5 MB (JPEGs).

The error occurs on two of the five servers we've tried it on.

On the server side of the connection, i.e., a Tomcat servlet built using Jersey, we get:

java.net.SocketException: Connection reset

The Apache Server on the same machine, acting as a reverse proxy, gives this error message:

[Thu May 15 17:08:58 2014] [error] proxy: pass request body failed to 127.0.0.1:8080 (localhost) from 192.168.16.xx ()

Setting Apache Server's logging level to debug and reproducing the problem produces no additional information – we still get that same error message without any associated messages.

A little less often, we don't get an exception on the Tomcat side, but checking the number of bytes transferred against the Content-Length header reveals that not everything made it through. The error in Apache Server in this second scenario is the same as in the first "proxy: pass request body failed…"

The Apache Server version is 2.2.15.29 on one server and 2.2.15.30 on the other, running under CentOS 6.2 in all cases. The reverse proxy rules are set up like so:

<IfModule mod_proxy.c>
        ProxyRequests Off

    # Case Manager Tomcat web service
        ProxyPass /casemanager http://localhost:8080/casemanager
        ProxyPassReverse /casemanager http://localhost:8080/casemanager

    # Matcher images directories
        ProxyPass /matcher-images http://x.x.x.108:80/matcher-images
</IfModule>

Note that the proxy pass-through is just going over http, not https.

We are using self-signed certificates for the SSL configuration.
OpenSSL version is 1.0.1e-fips on one of the failing servers and 1.0.0-fips on the other.

On the Tomcat side, we are running 7.0.26 and using Jersey 1.8.

I doubt that it matters, but the browser connection from which the POST originates is either Firefox 27 or Chrome 34.

In one case, our servlet was recently updated, although the code for handling files uploads has not recently changed. On the other server experiencing the problem, we are running a servlet build from months ago. In fact, on that second machine, we aren't aware of any code or configuration changes in the last few months — it had been sitting idle from mid-February until today.

What should I do next to troubleshoot this issue? Where should I be looking?

— Update —

Further testing shows that the connection is sometimes dropped even if I bypass Apache Server and POST directly to Tomcat. So it doesn't appear to be a proxy issue at all.

— Further Updates —

We are intermittently seeing problems copy the same large files via scp. It looks like the underlying issue is the firewall between our office/development subnet and the production/staging network. The error message when an scp copy fails is "Broken pipe."

Here is the stack trace from the servlet when the connection drops:

2014-05-16 13:20:44,566 [http-bio-8080-exec-7] ERROR com.objectvideo.wx.casemanager.service.resources.QueryFileService [null] – Failed to upload file(s).
javax.ws.rs.WebApplicationException: java.net.SocketException: Connection reset
at com.objectvideo.wx.casemanager.service.resources.QueryFileService.uploadRawFile(QueryFileService.java:342)
at com.objectvideo.wx.casemanager.service.resources.QueryFileService.uploadFile(QueryFileService.java:607)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:532)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:563)
at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118)
at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:326)
at org.apache.coyote.Request.doRead(Request.java:422)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:431)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:167)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1719)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1696)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1671)
at org.apache.commons.io.FileUtils.copyInputStreamToFile(FileUtils.java:1444)
at com.objectvideo.wx.casemanager.service.resources.QueryFileService.writeTempFile(QueryFileService.java:535)
at com.objectvideo.wx.casemanager.service.resources.QueryFileService.uploadRawFile(QueryFileService.java:320)
… 35 more