onsdag 2. mars 2011

Logs - use groovy+grep to find trends in your logs

Sometimes it might be useful to see the real state of your application for instance in a test phase of a project by getting some statistics on exceptions that are thrown.

The problem: We have logs, and we look through them whenever something goes wrong. And we usually know that something goes wrong, when someone says something or when everything goes to hell. But we never really use them for anything else. I don't.

I've sometimes grepped my way through the logs to find patterns of exceptions, so I decided that this might be interesting as a tool. The premise for the script I wrote is that the log files were going to be rolled daily, so the files (hopefully) won't be too big. I read the whole file into memory in the script (something you can tweak if you need to use it on large files). You also have to have groovy(I'm on 1.6 I think) available and be on a unix machine :)

(Anyway, you needn't really do this in groovy, the principal is easy to get and you can do this in other types of scripts. Grep on the other hand is probably nice to have as it is very quick.)

I tested it on files over 10mb, and it runs quite fast.

Here's what you get:

-How many occurances of exceptions in general are in the log.
-How many unique exceptions are in the log.
-A file with the list of each of the unique exceptions, how many there are of each of them and at what line(s) in the log file you can find them.

The print out is written to a csv-file in the directory you are standing. If you run it every day, like with a cron tab, you can probably combine the files to see some interesting trends.

If you want the same sort of statistics, go to my github-space and check it out.
Copyright © >> /dev/null
Blogger Theme by BloggerThemes Design by Diovo.com