Apache Camel: Exhausted after delivery attempt -> how debug

Currently, I am using Camel as an orchestration engine (it receives a HTTP POST request which results into the Camel code sending HTTP POST/PUT/GET requests to some external servers, until at the end Camel sends (hopefully) a 200 OK back to the initiator of the 1st HTTP POST request).

load tool <-> Camel code <-> external server(s)

I am using Camel 2.9.2.

In order to check the performance, I am using httperf to send a lot of HTTP POST messages to our Camel implementation. When httperf is configured to send requests at a rate of 100 requests per second, I see the following error a lot:

ERROR Failed delivery for (MessageId: ID-wpm-seal-59304-133965736935
6-0-3482128 on ExchangeId: ID-wpm-seal-59304-1339657369356-0-3482126).
Exhausted after delivery attempt: 1 caught: org.apache.camel.
component.http.HttpOperationFailedException: HTTP operation failed
invoking http://anExternalServer.com:7000/M2M/resources/m2mSubscription
?subscriptionId=32477000000 with statusCode: 500

I wonder how to debug to see what is causing this error …

And also what it means:
1) does it mean that Camel could not further handle the request ?
2) does it mean that Camel tried to send a HTTP message to an external server, but this server did not reply/accept the request (maybe it could not accept the request due to overload) ?
-> this looked the most logical to me, but a wireshark trace showed that Camel did not try to setup a TCP connection to that external server … ?!
3) or …

Maybe I hit a limitation of number of TCP connections ? Or threads ?

Here is the full error dump:

DefaultErrorHandler ERROR Failed delivery for (MessageId: ID-wpm-seal-59304-133965736935
6-0-3482128 on ExchangeId: ID-wpm-seal-59304-1339657369356-0-3482126). Exhausted after delivery attempt: 1 caught: org.apache.camel.
component.http.HttpOperationFailedException: HTTP operation failed invoking http://anExternalServer.com:7000/M2M/resources/m2mSubscription
?subscriptionId=32477000000 with statusCode: 500
org.apache.camel.component.http.HttpOperationFailedException: HTTP operation failed invoking http://anExternalServer.com:7000/M2M/resource
s/m2mSubscription?subscriptionId=32477000000 with statusCode: 500
at org.apache.camel.component.http.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:229)[camel-http-2.9.2
.jar:2.9.2]
at org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:157)[camel-http-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper
.java:61)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:115)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:285)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)[camel-core-2.9.2.jar:2.9.2
]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:333)[camel-core-2.9.2.j
ar:2.9.2]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:223)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.interceptor.StreamCachingInterceptor.process(StreamCachingInterceptor.java:52)[camel-core-2.9.
2.jar:2.9.2]
at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:304)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:50)[camel-c
ore-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)[camel-core-2.9.2.jar:2.9.2
]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:115)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:285)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)[camel-core-2.9.2.jar:2.9.2
]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:333)[camel-core-2.9.2.j
ar:2.9.2]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:223)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:304)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:50)[camel-c
ore-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)[camel-core-2.9.2.jar:2.9.2
]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)[camel-core-2.9.2.jar:2.9.2]
at org.apache.camel.component.http.CamelServlet.service(CamelServlet.java:94)[camel-http-2.9.2.jar:2.9.2]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)[servlet-api.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)[catalina.jar:7.0.25]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)[catalina.jar:7.0.25]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)[catalina.jar:7.0.25]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)[catalina.jar:7.0.25]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)[catalina.jar:7.0.25]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)[catalina.jar:7.0.25]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)[catalina.jar:7.0.25]
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)[catalina.jar:7.0.25]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)[catalina.jar:7.0.25]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)[catalina.jar:7.0.25]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)[tomcat-coyote.jar:7.0.25]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)[tomcat-coyote.jar:7.0.25]
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)[tomcat-coyote.jar:7.0.25]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)[:1.6.0_22]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)[:1.6.0_22]
at java.lang.Thread.run(Thread.java:679)[:1.6.0_22]