We have a Tomcat 7.0.21 server running on OpenJDK Server VM (20.0-b11 mixed mode). The webapp runs well for usually several hours or days after startup – typical CPU load is 1-2%.
At some point, the tomcat process starts to consume 100% CPU. At some time later it will consume 200%, then 300% etc (there are 4 processors).
I ran this command to identify which threads are causing the problem:
ps H -eo user,pid,ppid,tid,time,%cpu,cmd --sort=%cpu
…and then got a thread dump from the JVM
kill -QUIT <processId>
Predictably with the CPU at 200%, there are 2 offending threads. Their stack traces are the same:
"http-bio-80-exec-19" daemon prio=10 tid=0x08dcfc00 nid=0x192e runnable [0x442fe000]
java.lang.Thread.State: RUNNABLE
at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:339)
at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:353)
at org.apache.jsp.returnBubble_jsp._jspService(returnBubble_jsp.java:343)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:389)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:333)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:185)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:151)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:269)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:300)
- locked <0x6f39c2b0> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
The jsp file that it refers to (returnBubble_jsp.java:343) is not really doing anything special – it's called frequently by every user. If we look at line 343 of the generated java:
out.write("</td></tr>\n");
So I'm fairly confident it's not my JSP that's causing it! Note that the line number does change in other offending threads, but it always a similar out.write method.
What is this thread doing? And how do I prevent it from happening?
Best Answer
try disabling the feedFetcher task, as it looks to be another thread which might be causing livelock problems;