Filter the output of a log file

Question

Question
I've a log-File which display all console-logs of my website in the range of 10.Oct to 1.Nov, start with the logs of 10.Oct.
I need all logs from the range of 25.Oct until today.

The problem is that there is a huge exception. Every day I get an IOException over 21 lines and that for 6030 users, so there are about 126'600 lines of code which I won't see; per day.

So I need to filter the output to be able to read the logs normal again.

What I already tried:

less catalina.out | grep -v "java.io.IOException: Server returned HTTP response code: 401 for URL"  

The log file is called catalina.out. But it don't my code don't hide the Exception, and I'm not able to scroll through the output.
So what i'm doing wrong? How should the code be?

---

INFO about me
I'm NOT familiar with Linux. I work in a support group, but today i'm alone and a website make some trouble. So I need to fix it by myself.
I google it now by 45 minutes but don't get it and the problem hurrys... So please be patient with me...

---

INFO's about the problem
Here all the facts and informations I know about my problem:
LOG-File Name: catalina.out
The whole IOException which I want to hide and get repeated 6030 times per day:

java.io.IOException: Server returned HTTP response code: 401 for URL: http://secure.intern.webpage/userpicture/u117054.jpg
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
        at java.net.URL.openStream(URL.java:1037)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.downloadUrl(PersonCleanupNightlyJob.java:270)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.downloadPic(PersonCleanupNightlyJob.java:245)
        at ch.sbb.wzuapp.business.batch.PersonCleanupNightlyJob.doCleanup(PersonCleanupNightlyJob.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:64)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

The only thing which change in every exception is the picture-name. Everything else is always the same.

Solution 1

OK, genuinely interesting problem, given that it's got to function at the end of a pipe. There may be easier ways to do this, but I found this one worked (and it uses the sysadmin's Swiss Army Chainsaw, perl):

tail -f catalina.out | perl -n -e '{ if (/^java.io.IOException: Server returned HTTP response code: 401/) { $ignore=1;} elsif ($ignore>=1 and $ignore<20) {$ignore++;} else {$ignore=0; print $_;}  }'

Basically, it steps through STDIN one line at a time, looking for the search string ("^java.io.IOException: Server returned HTTP response code: 401"). Until it's found, it prints each line as it comes in; once it's found, it starts counting to 20, once for each new line in, printing nothing while it counts; once the count gets to 20, it resets the count to zero and resumes printing each new line in.

Edit: No problem. Start with tail -100000f catalina.out | ..., or indeed a larger number yet if the logfile has more lines than that. If you don't want to see the new stuff as it's added, try

perl -n -e '{ if (/^java.io.IOException: Server returned HTTP response code: 401/) { $ignore=1;} elsif ($ignore>=1 and $ignore<20) {$ignore++;} else {$ignore=0; print $_;}  }' < catalina.out.

It's designed for use in a pipeline, so you can feed it whatever you want stripped, and you can send the output onto whatever needs it. It's the UNIX way!

Solution 2

grep knows to take files as arguments. Please check the man. You could use egrep, instead of grep -e (which is the same):

# egrep -v '^java.io|^\ {8}at' catalina.out | less

Solution 3

If you want to totally ignore exceptions you may want to try this :

cat catalina.out | grep -v -e '^java.io.IOExc' -e '       at ' | less

(replace the 7 spaces by a tab [Ctrl+V then Tab] if that doesn't work)

If need more removal, we'll start perl-grepping.

Comments

[mveroone]

Why not ignoring all lines that foillow and begin with ` at ` instead of counting to 20 ? if the java call stack is different, your code breaks

[MadHatter]

Because the problem as specified by the OP was to ignore 21 lines which started with a given string. If (s)he'd specified it differently, I'd've coded it differently, and that would be valid in that case. I tend not to override the spec without good reason (ie, the spec is dumb); in this case, there's no reason to assume that the next line might not also begin with " at", and be very important.

[Michael Schmidt]

I ran it but it donesn't work. And I can't add a tab... the output is the same like the output with only less catalina.out. Do you need more information...?

[Michael Schmidt]

When I run your code, don't read the whole log file. I just open it and get all NEW added lines. But I need the lines of the past 5 days, not these which are just added.

[Michael Schmidt]

+1. It's a huge code. I think with your answer, I learned most :)

[Michael Schmidt]

+1 With your code, I found the (for me) simplest way to solve my issue.

[Michael Schmidt]

+1 & accept. Works perfectly.

[Michael Schmidt]

The working version of your code is cat catalina.out | grep -v "java.io.IOException" | grep -v "at" | less. Maybe you could edit your answer.

[Michael Schmidt]

Please add | less to your code, so I'll be able to scroll and it is a working answer for me

[mveroone]

Using -einsterad of grep twice works the same. Plus, I wouldn't remove all lines containing "at" you might miss a lot of them.

[Michael Schmidt]

Other question. With grep "2013-10-25" I only get all the entrys of this day. How can I add this to your code? so using the opposite of -v ?

[Mihai]

Since -v is applied for all grep command, you should pipe it into another grep. If you want to extend to more days: ...| egrep '2013-10-2[5-9]' | less.

[Michael Schmidt]

Just realized that your answer is the only correct. Only your code ignores ONLY the io-exceptions. Thank you