Logs - use groovy+grep to find trends in your logs

Sometimes it might be useful to see the real state of your application for instance in a test phase of a project by getting some statistics on exceptions that are thrown.

The problem: We have logs, and we look through them whenever something goes wrong. And we usually know that something goes wrong, when someone says something or when everything goes to hell. But we never really use them for anything else. I don't.

I've sometimes grepped my way through the logs to find patterns of exceptions, so I decided that this might be interesting as a tool. The premise for the script I wrote is that the log files were going to be rolled daily, so the files (hopefully) won't be too big. I read the whole file into memory in the script (something you can tweak if you need to use it on large files). You also have to have groovy(I'm on 1.6 I think) available and be on a unix machine :)

(Anyway, you needn't really do this in groovy, the principal is easy to get and you can do this in other types of scripts. Grep on the other hand is probably nice to have as it is very quick.)

I tested it on files over 10mb, and it runs quite fast.

Here's what you get:

-How many occurances of exceptions in general are in the log.
-How many unique exceptions are in the log.
-A file with the list of each of the unique exceptions, how many there are of each of them and at what line(s) in the log file you can find them.

The print out is written to a csv-file in the directory you are standing. If you run it every day, like with a cron tab, you can probably combine the files to see some interesting trends.

If you want the same sort of statistics, go to my github-space and check it out.
https://github.com/esschul/GroovyGrepExceptions

play framework, showing gravatars

Gravatar enables you to show a picture of a person, as long as you have their email-address, and they are registered at gravatar. But you have to do some work :)

So this post is about showing gravatars in play and how we did it when we had a lot of them.
So our stack for this example is jquery and the play! framework v1.1 (using grails templating).

So first the hashing method for getting the gravatar hash. We kept it on the model, but you can also put it in your controller if you have usage beyond the domain.

public static String gravatarhash(String gravatarId){
    if(gravatarId != null)
        return Codec.hexMD5(gravatarId.toLowerCase().trim());
     return null;
}

You call it from the html-file like this;

${models.Person.gravatarhash(person?.email).raw()}

where person in this case is a person with an email attribute populated from a controller.

So that's the hash that gravatar needs. If you have an email registered in gravatar and hash it using the method previosly specified, you ought to get a nice picture out with an img-tag

<img src="http://www.gravatar.com/avatar/${models.Person.gravatarhash(person.email).raw()}"/>

(If there is no email registered at gravatar, it will use a defaulted image. A default you can tweak, just check out their website).

So first off, if you have hundreds of gravatars; you don't put your gravatar url in a img-tag. The general browser will look at that as a resource it has to get on page load. Even with http caching, you don't want to load more resources than the user needs.

So in our case we had a button, and on the push of that button we want to show some pictures. We don't want it to take a long time to get in to the page initially, so what do we do? We load the images with javascript, and don't do it before the user pushes the button.

So we did something like this;

<a href="javascript" onclick="$('people').html('#{list items:people, as:'person' }<img src= quot;http://www.gravatar.com/avatar/${models.Person.gravatarhash(person.email).raw()}quot; alt=quot;${person.name}quot;/>#{/list}');$('people').toggle();">show me a hundred pictures!</a>

<div id="people" style="display:none"></div>

So now go out and add some pictures to your website!

play framework, adding ical

So here's my really quick guide to making ical available in your playapp. This is what you need;

Add ical4j's jars. Or just add the dependency to your pom.xml if you use that
The repo is;

<repository>
      <id>modularity-releases</id>
      <name>Modularity Releases Repository</name>
      <url>http://m2.modularity.net.au/releases</url>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </repository>

the dependency:

<dependency>
      <groupId>net.fortuna.ical4j</groupId>
      <artifactId>ical4j</artifactId>
      <version>1.0-rc1</version>
    </dependency>

Then install it with play mvn:up if you use the maven plugin. if not, then just download the jar and put it in the lib-directory.
Then you need to write some code to create a calendar instance. here's what I wrote for now;

public static Calendar createCalendar(Event event) throws SocketException {
        TimeZoneRegistry registry = TimeZoneRegistryFactory.getInstance().createRegistry();
        TimeZone timezone = registry.getTimeZone("Europe/Oslo");
        VTimeZone tz = timezone.getVTimeZone();
        // Create the event
        VEvent meeting = new VEvent(new DateTime(event.date), new DateTime(event.date), "javaBin: " + event.title);
        // Add timezone info..
        meeting.getProperties().add(tz.getTimeZoneId());
        UidGenerator ug = new UidGenerator(event.title);
        Uid uid = ug.generateUid();
        meeting.getProperties().add(uid);

        // Create a calendar
        net.fortuna.ical4j.model.Calendar icsCalendar = new net.fortuna.ical4j.model.Calendar();
        icsCalendar.getProperties().add(new ProdId("-//Events Calendar//iCal4j 1.0//EN"));
        icsCalendar.getProperties().add(CalScale.GREGORIAN);
        icsCalendar.getProperties().add(Version.VERSION_2_0);
        icsCalendar.getComponents().add(meeting);
        return icsCalendar;
}

Where as you can see the calendar is based on an event, and it's title.
Don't worry about any of the different types as "VEvent", as they are part of ical4j.

Then you need to render the file to the user, right?

Well, first go to iconfinder.net and find a nice icon for your ical-link.
In your html, write a link like this;

<a href="@{Application.ical(event?.id)}"><img src="/public/images/ical.png" alt="calendar"/></a>

then you write the rendering code in your controller;

public static void ical(Long id){
        try {
            Event event = Event.findById(id);
            Calendar calendar = ICalUtil.createCalendar(event);
            ByteArrayOutputStream bos = new ByteArrayOutputStream();
            CalendarOutputter outputter = new CalendarOutputter();
            outputter.output(calendar, bos);
            response.setHeader("Content-Type", "application/ics");
            InputStream is = new ByteArrayInputStream(bos.toByteArray());
            renderBinary(is,"javaBin.ics");
            bos.close();
            is.close();
        } catch (IOException e) {
            Logger.error("Io feil ved ical", e);
        } catch (ValidationException e) {
            Logger.error("Feil ved generering av ical", e);
        }
    }

Out of it you get an ical-calendar-item that people can use and enjoy ;) That's it.

The problem is all inside your head they say to me

No activity in this blog, since I attended JZ'10. I crashed and burned on the blogging marathon, and after writing two pieces, taking notes at a whole lot of the sessions, I said to my self that it wasn't worth it. So I quit.

The blog is not dead though. I just had to think it through; I don't know if I want to regurgiate what other people say without me actually getting something out of it. This sort of blogging is totally useless to me, so I just won't do it.

That's my excuse for not doing what I said I would do. 

Right now I'm working on a cool project for the community and I will write a little piece on that soon.

I will cover the november meeting as well (as long as it's interesting).

Session 1 : Emergent design

(btw, sorry about the delay. in contrast to my work, I don't always deliver on time! apologies aside .. here is the first one...)

Holding the lecture: Neil Ford

Link to his bio

So he has a page on this topic;

Evolutionary architecture and emergent design : investigating architecture and design

Link til this article

Ok, let's start off by saying, I was looking forward to this one. Not only being the first of the sessions, but also with Neil Ford, whom I've never seen, but only heard good things about. That being said, I did enjoy it. Sort of.

It was basic. Is that ok to say? It should be. I'll sum it up in the end.

The tagline "There are things we do not know we don't know" was the red line through the talk.

In bullet form I'd say he says this is the problems;

• software is changable, not like hardware. it's soft. this is the cause of the problems that might emerge.
• software we produce has two parts that can be changed, but some stuff is harder to change than other. architecture is harder to change than design choices.
• don't build lots of layers for extension that can be built on later. sometimes you create a generic obfuscation and sometimes you add complexity to something that might be simple.

There are ways to get out of the problems by:

• postpone decisions as long as you can. if you wait until you really have to, you can make the right decision easier. it's all about what you know. 
• fix stuff right away. waiting will only make it worse. you know it. I know it.
• try to take metrics from the complexity and usage of your code. the combination of these factors might give you an indication for what you ought to fix.
• write tests when writing code. your code will get a structure that might save you from making mistakes.
• make use of patterns, and try to find places where you've repeated yourself. 
• time spent early on design costs money and is prone for errors

There was more, but all in all, this is what I got out of the session. My memory might be playing tricks with me as well, since it's been a whole day since I saw it. But all in all, I found the session a bit basic, and I feel I've heard this in some form or another over the last years repeatedly.

JavaZone 2010 - introduction day #1

Welcome welcome, small and big to the event of the year! Forget about christmas, birthdays, midsummers eve and your grandfathers 80th aniversary; this is the one that counts.

About 2000 people are now on their way to Oslo Spectrum. There will be music, coding, bullshit and sessions. Over the years javaZone had numerous crazy stage shows performed by some insane people; Pain solution, Cirkus cirkör, Rammsund etc. It's always some cool show. I'll post some pictures after we're done.
(note: this year they haven't invited insane people, but rather asked companies to form garage bands, and they will be the entertainment. I'm skeptic, but I will keep an open mind.)

JavaZone first released a trailer to this years event. Someone ripped it and put it on youtube, and must have been viewed by more than 1 million ppl. Here's a link;

Java-4-ever trailer

then they followed it by a music video

Lady Java music video

Pretty professional! I think the split between ".net and java"-part is funny, even though I think the community is past this "conflict" by now.

I've planned to attend the following sessions the first day:

• Seven cost-efficient ways to reduce your project overruns (or maybe I will try "Emergent design", all depends how full the room will get)
• JRuby: Now With More J!
• How we blew our shot at beating Spotify, spending two metric truckloads of cash doing it
• Flying with Griffon
• How to Defend Against the OWASP Top Ten Web Security Threats
• 97 Things Every Programmer Should Know

Check out the agenda for more info. I probably won't be able to attend all of this, since I have other stuff to do here at the conference.

After the sessions there is club zone, and I'll try to cover that (we'll see if I bother).

In addition to being an attendant at the conference I'm also part of the crew organizing the event.
You can come and talk to me at the javaBin stand from 14:10-15:40 (on wednesday).

Well that's enough intro, I guess; Let the sessions begin!

JavaZone 2010 coming up. 8-9th of September.

So after a couple of months in the cave, the grizzly awakens with renewed strength. Me being the grizzly.

The main java event in Norway of the year is approaching, and I'm going to be there. Not only be there, but also try to blog about every single session I attend. A tour de force in blogging as far as I'm concerned. You can expect all my posts to be sort of incomplete and unsatisfying on some level :)

Check back here on the 8th for the first post!