Linux – Filter the output of a log file

filterlinuxlog-files

Question
I've a log-File which display all console-logs of my website in the range of 10.Oct to 1.Nov, start with the logs of 10.Oct.
I need all logs from the range of 25.Oct until today.

The problem is that there is a huge exception. Every day I get an IOException over 21 lines and that for 6030 users, so there are about 126'600 lines of code which I won't see; per day.

So I need to filter the output to be able to read the logs normal again.

What I already tried: 

less catalina.out | grep -v "java.io.IOException: Server returned HTTP response code: 401 for URL"

The log file is called catalina.out. But it don't my code don't hide the Exception, and I'm not able to scroll through the output.
So what i'm doing wrong? How should the code be?

INFO about me
I'm NOT familiar with Linux. I work in a support group, but today i'm alone and a website make some trouble. So I need to fix it by myself.
I google it now by 45 minutes but don't get it and the problem hurrys… So please be patient with me…

INFO's about the problem
Here all the facts and informations I know about my problem:
LOG-File Name: catalina.out
The whole IOException which I want to hide and get repeated 6030 times per day:

java.io.IOException: Server returned HTTP response code: 401 for URL: http://secure.intern.webpage/userpicture/u117054.jpg
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
        at java.net.URL.openStream(URL.java:1037)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.downloadUrl(PersonCleanupNightlyJob.java:270)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.downloadPic(PersonCleanupNightlyJob.java:245)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.doCleanup(PersonCleanupNightlyJob.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:64)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

The only thing which change in every exception is the picture-name. Everything else is always the same.

Best Answer

OK, genuinely interesting problem, given that it's got to function at the end of a pipe. There may be easier ways to do this, but I found this one worked (and it uses the sysadmin's Swiss Army Chainsaw, perl):

tail -f catalina.out | perl -n -e '{ if (/^java.io.IOException: Server returned HTTP response code: 401/) { $ignore=1;} elsif ($ignore>=1 and $ignore<20) {$ignore++;} else {$ignore=0; print $_;}  }'

Basically, it steps through STDIN one line at a time, looking for the search string ("^java.io.IOException: Server returned HTTP response code: 401"). Until it's found, it prints each line as it comes in; once it's found, it starts counting to 20, once for each new line in, printing nothing while it counts; once the count gets to 20, it resets the count to zero and resumes printing each new line in.

Edit: No problem. Start with tail -100000f catalina.out | ..., or indeed a larger number yet if the logfile has more lines than that. If you don't want to see the new stuff as it's added, try 

perl -n -e '{ if (/^java.io.IOException: Server returned HTTP response code: 401/) { $ignore=1;} elsif ($ignore>=1 and $ignore<20) {$ignore++;} else {$ignore=0; print $_;}  }' < catalina.out.

It's designed for use in a pipeline, so you can feed it whatever you want stripped, and you can send the output onto whatever needs it. It's the UNIX way!

Related Topic