How to correlate the Apache mod_jk log with the Tomcat instance log for troubleshooting

apache-2.2jbosslog-filesmod-jk

In my Apache mod_jk log I have entries in the format like:

[Wed Oct 26 10:59:43 2011] [3732:2460] [error] jk_ajp_common.c (1618): (myJbossServer) Tomcat is down or network problems. Part of the response has already been sent to the client

RedHat's Troubleshooting Guide says,

Note that with these messages the [11159:3086420192] portion of the
message serves as an identifier for the connection/request in
question. Therefore tracing back from the point of the error in logs
can help clarify the activity around the connection/request that lead
to the error.

How do I do this? I cannot find the identifier in my JBoss logs. I don't see any identifiers in the JBoss logs, errors, yes, tracking numbers, no. Where do I need to be looking to track this identifier down? I'm not really looking for a solution to the problem, only how to use the identifier and check between Apache mod_jk and Jboss.

EDIT: I am asking where can I find the identifier [3732:2460] in the myJBossServer logs.

Best Answer

Actually it's in front of you, the failed worker is the one inside brackets, called myJbossServer.

Check you workers.properties for that worker and you will find it's IP or hostname.

Around that time hopefully you'll get hints from your JBoss / operating system (sar get's very handy if you were experiencing problems with CPU / Memory / ecc BTW).

Actualy you made me curious about the numbers between square brackets. I tried searching the documentation but wasn't able to find what it was. Checking the source code though brought this:

int jk_log(jk_logger_t *l,
           const char *file, int line, const char *funcname, int level,
           const char *fmt, ...)
{

...

        if (line) { /* line==0 only used for request log item */
            /* Log [pid:threadid] for all levels except REQUEST.
             * This information helps to correlate lines from different logs.
             * Performance is no issue, because with production log levels
             * we only call it often, if we have a lot of errors
             */
            rc = snprintf(buf + used, usable_size - used,
                          "[%" JK_PID_T_FMT ":%" JK_PTHREAD_T_FMT "] ", getpid(), jk_gettid());

Looking at the RedHat document you pasted, that PID:TOD identification can help to match the same mod_jk.log you're seeing, not to cross reference the mod_jk.log with the JBoss one. If you want to crossreference the two logs then you should find some other way, setting a custom header in Apache / mod_jk and printing that in your JBoss log is the first way i thought right now. Or at least you can check the requests around the failure time.

Related Topic