Mark J. Wielaard

With the recent releases of Systemtap 0.9.9 and IcedTea7 1.11 some pieces of the systemtap whole system observability picture come together. On top of the “meta-probes” in hotspot (for garbage collection, monitors, object allocation, threads, etc.) we now also see the java language level tracing coming to life.

Always wondered what really was going on when your “simple” Hello world java program was running…

$ stap -e \
'probe hotspot.method_entry {log(thread_indent(1) . "=> " . class . "." . method . sig)}
probe hotspot.method_return {log(thread_indent(-1) . "<= " . class . "." . method . sig)}' \
-c 'java -XX:+ExtendedDTraceProbes Hello'
[...]
   0 java(26933):=> Hello.main([Ljava/lang/String;)V
   9 java(26933): => java/lang/ClassLoader.checkPackageAccess(Ljava/lang/Class;Ljava/security/ProtectionDomain;)V
  16 java(26933):  => java/lang/System.getSecurityManager()Ljava/lang/SecurityManager;
  23 java(26933):  <= java/lang/System.getSecurityManager()Ljava/lang/SecurityManager;
  30 java(26933):  => java/util/HashSet.add(Ljava/lang/Object;)Z
  37 java(26933):   => java/util/HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
  43 java(26933):    => java/lang/Object.hashCode()I
  50 java(26933):    <= java/lang/Object.hashCode()I
  56 java(26933):    => java/util/HashMap.hash(I)I
  63 java(26933):    <= java/util/HashMap.hash(I)I
  69 java(26933):    => java/util/HashMap.indexFor(II)I
  76 java(26933):    <= java/util/HashMap.indexFor(II)I
  83 java(26933):   <= java/util/HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
  89 java(26933):  <= java/util/HashSet.add(Ljava/lang/Object;)Z
  96 java(26933): <= java/lang/ClassLoader.checkPackageAccess(Ljava/lang/Class;Ljava/security/ProtectionDomain;)V
[...]
2611 java(26933):     => java/io/BufferedOutputStream.flush()V
2617 java(26933):      => java/io/BufferedOutputStream.flushBuffer()V
2624 java(26933):       <= java/io/BufferedOutputStream.flushBuffer()V
2630 java(26933):   <= java/io/BufferedOutputStream.flush()V
2636 java(26933):  <= java/io/PrintStream.newLine()V
2643 java(26933): <= java/io/PrintStream.println(Ljava/lang/String;)V
2649 java(26933):<= Hello.main([Ljava/lang/String;)V

The needed changes have also been added to IcedTea6 so expect them in a GNU/Linux distro near you soon.

Just did an aptitude dist-upgrade to get planet.classpath.org and the main IcedTea backup server from Debian Etch to Debian Lenny. Pretty smooth.

Needed to adjust some apache2 NameVirtualHost settings as outlined in the release notes. And sadly, had to pin mercurial to 0.9.5, since newer versions don’t work nicely with the forest extension and serving hg through http with hgwebdir. But that was it. Everything else went without any hiccups.

But if you use one of those servers and do notice something wrong, please let me know.

Another nice feature for Systemtap 0.9 was added by Josh Stone. Systemtap can collect data from any variable in scope at a probe point using the DWARF debug info. You can even dereference pointers, access struct members, array elements, etc. This is very powerful when collecting data during a trace and the systemtap runtime makes sure all data access is safe. But there were two issues making this less powerful than it could be.

First to the keep the tracing language simple systemtap only supports basic types (integers and strings), associative arrays or aggregates in stap scripts. This means that you could not easily pass program data around to an helper function to manipulate or format. Second sometimes programs “hide” the real type of a variable, or use a void * pointer that gets cast to the right type later on. You could work around this in the past by using embedded C and guru mode, but that wasn’t very nice, and made your script potentially unsafe.

So to make sure you can do this safely Josh added a @cast construct. This allows you to pass around a pointer to program data and interpret it as if it was any type described in the DWARF debuginfo for the program. All accesses are of course still checked for safety by the runtime.

A nice example of this feature in action is the following simple stap script to print the number of incoming connections for an executable by port number. We want to probe the kernel and get the inet_sock from the inet_csk_accept function when it returns successfully. Although this function handles inet sockets (it is part of inet_connection_sock.c), it passes around sock pointers. It can do this since an inet_sock struct starts with a sock pointer, later it will cast this to a full featured inet_sock pointer. So we do the same in our script:

global ports;

probe kernel.function("inet_csk_accept").return
{
  sock = $return
  if (sock != 0)
    {
      port = @cast(sock, "inet_sock")->num;
      ports[execname(), port]++;
    }
}

probe timer.s(30), end
{
  printf("Connections on ports: %s\n", ctime(gettimeofday_s()));
  foreach ([exec, port] in ports-)
    printf("%12s %4d: %4d\n", exec, port, ports[exec, port]);
  delete ports;
}

$ stap ports.stp
Connections on ports: Sat Feb 28 22:49:10 2009
       httpd   80:  172
       spamd  783:   30
        exim   25:   27
     portmap  111:   11
  imap-login  993:    8
      ypserv  818:    7
        sshd   22:    2

There are some more exciting network tracing examples in the Systemtap Examples collection.

We recently released Systemtap 0.9 and one of the nice new features included is the user space markers that Stan Cox has been working on. They were designed so that they should be compatible with dtrace static user space markers, so you can immediately take advantage of them if your program already has those included. It even comes with a little dtrace python script wrapper that automagically does the right thing during the build. A nice example of that is postgresql, which has a set of markers to observe transactions, database locks, etc. All you have to do is recompile postgresql with –enable-dtrace and tada, out roll systemtap enabled markers that you can use for getting some high level events from your database. Like for example (postgresql-transactions.stp) how many and how long transactions take:

$ stap -x `pgrep -n postgres` postgresql-transactions.stp
committed transactions:
transaction id: time
            34: 1593213ns
            36: 2817146ns
            37: 1463901ns
            38: 1427854ns

aborted transactions: 4

We are trying to get some of these static markers activated in packages compiled for Fedora as SystemtapStaticProbes F11 Feature, so you can use them out of the box.

But you can also add your own markers to existing code. Daniel Tralamazza has been experimenting with a small glibc patch to add markers around various pthread mutexes, for doing userland synchronization primitives analysis. Then you can easily get things like the top 10 most shared locks.

Sarah has been publishing sets of pictures from the Fosdem Free Java Meeting.

IcedTea 1.4 got released this week. And while it is full of new exciting stuff, you really should check out the XRender support by Clemens Eisserer. Especially if you often use java through remote X. It just flies! Trying it out is easy as soon as the new IcedTea hits a distro near you:

java -Dsun.java2d.xrender=True my.fancy.gui.HelloWorld

Also check out the JGears2 benchmark to compare your results.

So this speeds up the rendering pipeline to X enormously. Now the next step will be optimizing or rewriting the actual Render backend (pisces at this time) which seems to be the next bottleneck, at least for anti-aliased operations.

Clemens will give a talk about his work at Fosdem. Hope to see you all there.

Sometimes people ask me what Red Hat actually pays me for. Aren’t you working on something java related? Although my manager has been very generous and allows me to spend some (but not too much!) time on helping out the free java efforts, my main job is in the engineering tools group (gcc, gdb/Archer [formerly Frysk], binutils, elfutils, oprofile, etc.). Currently I am hacking on Systemtap which has been a lot of (low-level) fun.

LWN just published my article “A Systemtap update” which gives a high level overview of the project through some small examples (all work out of the box on Fedora 10 of course).  It is tucked away on the LWN kernel page, but I tried to show how Systemtap moved beyond the kernel and now provides complete system observability. At the end of the article I point out some of the other work that is being doing around debugging and tracing (elfutils dwarf framework, gcc vta branch, froggy/archer, the user-space breakpoint support layer) to show it is all connected to provide a better debugging and tracing environment for GNU/Linux.

Looking for a handy reference of all the talks in the Free Java developer room at Fosdem 2009 in Brussels, Saturday 7 and Sunday 8 February? Look no further! PDF Poster and ODG Poster.

Our libre-java meeting at Fosdem is approaching quickly. February 7 and 8 in Brussels, Belgium. If you want to give a talk, do a demo or have some general presentation/hack-session in our room, please act quickly so we can still schedule it (the deadline is end of this week!).

This is what will be at our disposal:

• the room “AW1.120” with a capacity of 74 seats (in the building “AW”)
  • on Saturday 2009-02-07 from 12:00 to 18:00
  • on Sunday 2009-02-08 from 09:00 to 17:00
• a video projector with VGA cable
• Internet connectivity (wifi A and B only, no wired)

This is what we need from you:

• Talk proposals and abstracts for activities.
  Please submit these BEFORE Saturday January 10!
  Either by emailing the Fosdem meeting committee at:
  fosdem2009@developer.classpath.org
  Or adding your proposal to the wiki:
  http://wiki.debian.org/Java/DevJam/2009/Fosdem

Also please add you name to the wiki even if you don’t want to present something so we know roughly how many people to expect.

planet.classpath.org moved servers and if done correctly nobody will notice (except for the new server having a totally sweet favicon ). But if you do happen to notice anything odd with the planet after the move, then please do yell and scream.