Mark J. Wielaard

Did an interview for FOSDEM about SystemTap. It discusses a wide range of topics. About when I got involved with Free Software, working for Red Hat, how FOSDEM helped the libre Java community, getting Fedora more observable by adding static markers into programs, the history of observation tools (tracers, profilers, debuggers) on GNU/Linux, comparisons to other tools like DTrace, GUI frontends, Eclipse integration, the future of SystemTap and of course why you should come to FOSDEM.

Really looking forward to Fosdem next month. This year I will be giving a talk What is my system doing – Full System Observability with SystemTap during one of the main tracks. There will be some demos of the new systemtap java and python tracing support that I blogged about earlier.

Michael Meeks published a though provoking essay called Some thoughts on Copyright Assignment. It is a must read when working with Free Software projects that request legal assignment of contributions to a corporation.

It contains sane recommendations both for individual contributors, project leaders and corporations seeking assignments.

Just checked in support for getting java backtraces from hotspot through systemtap. This is the next step in making not just the kernel and native programs observable, but also runtime based languages with SystemTap. And it is pretty powerful. It allows you to answer the question “How did I get here?” in combination with any of the SystemTap tapsets for VM level tracing, Java method entry/exit tracing, Native methods (JNI) tracing, or any other basic SystemTap function, statement or process probe that points into your java process. Just add one of the print_jstack() variants to your code (possibly including native frame and method signatures) and suddenly it is really easy to see what triggered a particular code path:

$ stap -e 'probe hotspot.jni.CallObjectMethod { log (probestr); print_jstack() }' -c 'java ModalTest'

CallObjectMethod(env=0x7f9d1000a1b8,obj=0x7f9d73efcaf8,methodid=0x7f9de4088b70)
sun/font/FontManager.getFont2D
sun/java2d/SunGraphics2D.checkFontInfo
sun/java2d/SunGraphics2D.getFontInfo
sun/java2d/pipe/GlyphListPipe.drawString
sun/java2d/SunGraphics2D.drawString
sun/swing/SwingUtilities2.drawString
sun/swing/SwingUtilities2.drawStringUnderlineCharAt
javax/swing/plaf/basic/BasicGraphicsUtils.drawStringUnderlineCharAt
sun/awt/X11/XButtonPeer.paintText
sun/awt/X11/XButtonPeer.paint
sun/awt/X11/XButtonPeer.paint
sun/awt/X11/XRepaintArea.paintComponent
sun/awt/RepaintArea.paint
sun/awt/X11/XComponentPeer.handleEvent
java/awt/Component.dispatchEventImpl
java/awt/Component.dispatchEvent
java/awt/EventQueue.dispatchEvent
java/awt/EventDispatchThread.pumpOneEventForFilters
java/awt/EventDispatchThread.pumpEventsForFilter
java/awt/EventDispatchThread.pumpEventsForFilter
java/awt/Dialog$1.run
java/awt/event/InvocationEvent.dispatch
java/awt/EventQueue.dispatchEvent
java/awt/EventDispatchThread.pumpOneEventForFilters
java/awt/EventDispatchThread.pumpEventsForFilter
java/awt/EventDispatchThread.pumpEventsForHierarchy
java/awt/EventDispatchThread.pumpEvents
java/awt/EventDispatchThread.pumpEvents
java/awt/EventDispatchThread.run

You currently need both IcedTea6 from mercurial and Systemtap from git to play with it. But both projects are planning new releases soon.

At FudCon, David Malcolm, Jon VanAlten, Will Cohen and I sat down, had some fun and made tracing python methods through systemtap possible:

0 python(20122): => search_function in Lib/encodings/__init__.py:71
15 python(20122):  => normalize_encoding in Lib/encodings/__init__.py:49
37 python(20122):  <= normalize_encoding
170 python(20122):  => <module> in Lib/encodings/utf_8.py:8
193 python(20122):   => IncrementalEncoder in Lib/encodings/utf_8.py:18
206 python(20122):   <= IncrementalEncoder
251 python(20122):   => IncrementalDecoder in Lib/encodings/utf_8.py:22
264 python(20122):   <= IncrementalDecoder
310 python(20122):   => StreamWriter in Lib/encodings/utf_8.py:25
323 python(20122):   <= StreamWriter
340 python(20122):   => StreamReader in Lib/encodings/utf_8.py:28
353 python(20122):   <= StreamReader
367 python(20122):  <= <module>
391 python(20122):  => getregentry in Lib/encodings/utf_8.py:33
410 python(20122):   => __new__ in Lib/codecs.py:77
429 python(20122):   <= __new__
440 python(20122):  <= getregentry
462 python(20122): <= search_function


The coolest part is that it works through the existing patch to python for adding dtrace support. Some small tweaks to the autoconf detection was needed, but the rest was used as is.

If you want to learn how to add static user space probes to your program/package please see Will’s excellent guide. Adding User Space Probing to an Application: A simple example adding markers to a user-space application with SystemTap.

Currently stuck at heathrow airport… But I will make it to Canada somehow for FudCon. Hopefully with some nice systemtap jstack java hotspot backtrace examples.

They finally actually started to work. Weeee!

After my somewhat cranky post about the JDK 7 M5 binaries being distributed under not very free and social terms. I was really happy to see Mark Reinhold’s announcement of the JDK 7 M5 + Jigsaw binary packages:

These binaries are licensed under the GPL (v2) with the Classpath Exception, plus the Assembly Exception.

Thank you Mark! Thank you Sun! You did make my day.

Mark Reinhold announced JDK 7 Milestone 5 which is derived from OpenJDK7. Lots of people contributed to that. But sadly Sun decided they weren’t going to release this as Free Software. I wanted to research this JDK7 M5 and compare it to the work we did on OpenJDK and IcedTea, but apparently I am not allowed to…

Sun might have the right to do this given they only use contributions granted to them under their SCA terms. But the following clauses from the JDK7 license seem somewhat unfair given that we all contributed parts to OpenJDK.

• “Licensed Software is “Confidential Information”. Licensee may not disclose or use Confidential Information”
• “Licensee may not modify or create derivative works of the Licensed Software, or reverse engineer, disassemble or decompile binary portions of the Licensed Software, or otherwise attempt to derive the source code from such portions.”
• “Licensee shall have no right to use the Licensed Software for productive or commercial use.”
• “LICENSEE DUTIES Licensee agrees to evaluate and test the Software for use in Licensee’s software environment and provide feedback to Sun in a manner reasonably requested by Sun. Any and all test results, error data, reports or other information, feedback or materials made or provided by Licensee relating to Software (collectively, “Feedback”) are the exclusive property of Sun and Licensee here by assigns all Feedback to Sun at no cost to Sun.”
• “This Agreement will commence on the date on which Licensee receives Licensed Software (the “Effective Date”) and will expire ninety (90) days from the Effective Date, unless terminated earlier as provided herein.”
• “Upon termination or expiration of this Agreement, Licensee will immediately cease use of and destroy Licensed Software, any copies thereof.”
• “It is understood and agreed that, notwithstanding any other provision of this Agreement, Licensee’s breach of this Agreement will cause Sun irreparable damage for which recovery of money damages would be inadequate…”

There is no reason for Sun to act in this anti-social way. There are free replacements for the previously encumbered binary blobs (thanks IcedTea!). So anybody that wants to can distribute a completely free version. And redistributing other peoples contributions under such draconian one-sided terms is totally unnecessary (and rude!). If someone wants to add the previously proprietary binary blobs one can do that under the terms of the OpenJDK Assembly Exception to the GPL that was specially drafted to allow this. So there is no need to not redistribute the free parts under the GPL as everybody else is doing.

Come on Sun, tear down this proprietary legal wall that divides users from the developers that contributed all this code to be free and open.

Since the various OpenJDK servers are seeing major ongoing outages the last couple of days it might be good to point out that at least the source code from the master jdk6 and jdk7 mercurial forests do have backups: http://classpath.wildebeest.org/hg/. These are read-only of course. You will also find backups of the various IcedTea mercurial repos there. The IcedTea builds can currently fail if you don’t have openjdk.tar.gz and/or hotspot.tar.gz around in some configurations. You can also get those from the IcedTea autobuilder/tester (and then use configure --with-openjdk-src-zip and/or --with-hotspot-src-zip). Besides creating binaries and test reports, the autobuilder also keeps copies of all the sources used to create the latest successful build run around: http://icedtea.classpath.org/builds/.

Wow, just wow! Look at all those new features in the just released GNU Debugger 7.0:

• Python scripting support
• Reverse debugging, Process record and replay
• Non-stop debugging
• Multi-architecture debugging
• Multi-inferior, multi-process debugging
• GDB now has an interface for JIT compilation
• Tracepoints may now be conditional
• Multi-byte and wide character set support
• New /r and /m modifiers for the “disassemble” command
• Automatic retrieval of shared library files from remote targets
• Inlined functions are now supported
• New remote protocal packets
• GDB is now able to read compressed debug sections
• Thread switching is now supported on Tru64
• Ada task switching is now supported
• New features in gdbserver, the GDB remote stub
• New command to stop execution when a system call is made

Of course no surprises if you have been following Nick Clifton’s GNU Toolchain Updates and some of these features (python scripting, better C++ support and disassemble /m are my personal favourites) have been backported to the Fedora GDB packages already. But it is really nice to see them all together now in a fresh GDB release. Congrats to the GDB hackers. This is really a must have update!

Josh announced SystemTap 1.0

Experimental support for unprivileged users, cross-compiling for foreign architectures, matching C++ class and namespace scopes, reduced runtime memory consumption, reduced dormant overhead in userspace markers, bug fixes, and more…

I am happy this is finally out the door. Designating something with 1.0 is always special. And it is always hard to decide when to cross that magical barrier. But after 0.9.9 it was hard to not just do it!

With the recent releases of Systemtap 0.9.9 and IcedTea7 1.11 some pieces of the systemtap whole system observability picture come together. On top of the “meta-probes” in hotspot (for garbage collection, monitors, object allocation, threads, etc.) we now also see the java language level tracing coming to life.

Always wondered what really was going on when your “simple” Hello world java program was running…

$ stap -e \
'probe hotspot.method_entry {log(thread_indent(1) . "=> " . class . "." . method . sig)}
probe hotspot.method_return {log(thread_indent(-1) . "<= " . class . "." . method . sig)}' \
-c 'java -XX:+ExtendedDTraceProbes Hello'
[...]
   0 java(26933):=> Hello.main([Ljava/lang/String;)V
   9 java(26933): => java/lang/ClassLoader.checkPackageAccess(Ljava/lang/Class;Ljava/security/ProtectionDomain;)V
  16 java(26933):  => java/lang/System.getSecurityManager()Ljava/lang/SecurityManager;
  23 java(26933):  <= java/lang/System.getSecurityManager()Ljava/lang/SecurityManager;
  30 java(26933):  => java/util/HashSet.add(Ljava/lang/Object;)Z
  37 java(26933):   => java/util/HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
  43 java(26933):    => java/lang/Object.hashCode()I
  50 java(26933):    <= java/lang/Object.hashCode()I
  56 java(26933):    => java/util/HashMap.hash(I)I
  63 java(26933):    <= java/util/HashMap.hash(I)I
  69 java(26933):    => java/util/HashMap.indexFor(II)I
  76 java(26933):    <= java/util/HashMap.indexFor(II)I
  83 java(26933):   <= java/util/HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
  89 java(26933):  <= java/util/HashSet.add(Ljava/lang/Object;)Z
  96 java(26933): <= java/lang/ClassLoader.checkPackageAccess(Ljava/lang/Class;Ljava/security/ProtectionDomain;)V
[...]
2611 java(26933):     => java/io/BufferedOutputStream.flush()V
2617 java(26933):      => java/io/BufferedOutputStream.flushBuffer()V
2624 java(26933):       <= java/io/BufferedOutputStream.flushBuffer()V
2630 java(26933):   <= java/io/BufferedOutputStream.flush()V
2636 java(26933):  <= java/io/PrintStream.newLine()V
2643 java(26933): <= java/io/PrintStream.println(Ljava/lang/String;)V
2649 java(26933):<= Hello.main([Ljava/lang/String;)V

The needed changes have also been added to IcedTea6 so expect them in a GNU/Linux distro near you soon.

Just did an aptitude dist-upgrade to get planet.classpath.org and the main IcedTea backup server from Debian Etch to Debian Lenny. Pretty smooth.

Needed to adjust some apache2 NameVirtualHost settings as outlined in the release notes. And sadly, had to pin mercurial to 0.9.5, since newer versions don’t work nicely with the forest extension and serving hg through http with hgwebdir. But that was it. Everything else went without any hiccups.

But if you use one of those servers and do notice something wrong, please let me know.

Another nice feature for Systemtap 0.9 was added by Josh Stone. Systemtap can collect data from any variable in scope at a probe point using the DWARF debug info. You can even dereference pointers, access struct members, array elements, etc. This is very powerful when collecting data during a trace and the systemtap runtime makes sure all data access is safe. But there were two issues making this less powerful than it could be.

First to the keep the tracing language simple systemtap only supports basic types (integers and strings), associative arrays or aggregates in stap scripts. This means that you could not easily pass program data around to an helper function to manipulate or format. Second sometimes programs “hide” the real type of a variable, or use a void * pointer that gets cast to the right type later on. You could work around this in the past by using embedded C and guru mode, but that wasn’t very nice, and made your script potentially unsafe.

So to make sure you can do this safely Josh added a @cast construct. This allows you to pass around a pointer to program data and interpret it as if it was any type described in the DWARF debuginfo for the program. All accesses are of course still checked for safety by the runtime.

A nice example of this feature in action is the following simple stap script to print the number of incoming connections for an executable by port number. We want to probe the kernel and get the inet_sock from the inet_csk_accept function when it returns successfully. Although this function handles inet sockets (it is part of inet_connection_sock.c), it passes around sock pointers. It can do this since an inet_sock struct starts with a sock pointer, later it will cast this to a full featured inet_sock pointer. So we do the same in our script:

global ports;

probe kernel.function("inet_csk_accept").return
{
  sock = $return
  if (sock != 0)
    {
      port = @cast(sock, "inet_sock")->num;
      ports[execname(), port]++;
    }
}

probe timer.s(30), end
{
  printf("Connections on ports: %s\n", ctime(gettimeofday_s()));
  foreach ([exec, port] in ports-)
    printf("%12s %4d: %4d\n", exec, port, ports[exec, port]);
  delete ports;
}

$ stap ports.stp
Connections on ports: Sat Feb 28 22:49:10 2009
       httpd   80:  172
       spamd  783:   30
        exim   25:   27
     portmap  111:   11
  imap-login  993:    8
      ypserv  818:    7
        sshd   22:    2

There are some more exciting network tracing examples in the Systemtap Examples collection.

We recently released Systemtap 0.9 and one of the nice new features included is the user space markers that Stan Cox has been working on. They were designed so that they should be compatible with dtrace static user space markers, so you can immediately take advantage of them if your program already has those included. It even comes with a little dtrace python script wrapper that automagically does the right thing during the build. A nice example of that is postgresql, which has a set of markers to observe transactions, database locks, etc. All you have to do is recompile postgresql with –enable-dtrace and tada, out roll systemtap enabled markers that you can use for getting some high level events from your database. Like for example (postgresql-transactions.stp) how many and how long transactions take:

$ stap -x `pgrep -n postgres` postgresql-transactions.stp
committed transactions:
transaction id: time
            34: 1593213ns
            36: 2817146ns
            37: 1463901ns
            38: 1427854ns

aborted transactions: 4

We are trying to get some of these static markers activated in packages compiled for Fedora as SystemtapStaticProbes F11 Feature, so you can use them out of the box.

But you can also add your own markers to existing code. Daniel Tralamazza has been experimenting with a small glibc patch to add markers around various pthread mutexes, for doing userland synchronization primitives analysis. Then you can easily get things like the top 10 most shared locks.

Sarah has been publishing sets of pictures from the Fosdem Free Java Meeting.

IcedTea 1.4 got released this week. And while it is full of new exciting stuff, you really should check out the XRender support by Clemens Eisserer. Especially if you often use java through remote X. It just flies! Trying it out is easy as soon as the new IcedTea hits a distro near you:

java -Dsun.java2d.xrender=True my.fancy.gui.HelloWorld

Also check out the JGears2 benchmark to compare your results.

So this speeds up the rendering pipeline to X enormously. Now the next step will be optimizing or rewriting the actual Render backend (pisces at this time) which seems to be the next bottleneck, at least for anti-aliased operations.

Clemens will give a talk about his work at Fosdem. Hope to see you all there.

Sometimes people ask me what Red Hat actually pays me for. Aren’t you working on something java related? Although my manager has been very generous and allows me to spend some (but not too much!) time on helping out the free java efforts, my main job is in the engineering tools group (gcc, gdb/Archer [formerly Frysk], binutils, elfutils, oprofile, etc.). Currently I am hacking on Systemtap which has been a lot of (low-level) fun.

LWN just published my article “A Systemtap update” which gives a high level overview of the project through some small examples (all work out of the box on Fedora 10 of course).  It is tucked away on the LWN kernel page, but I tried to show how Systemtap moved beyond the kernel and now provides complete system observability. At the end of the article I point out some of the other work that is being doing around debugging and tracing (elfutils dwarf framework, gcc vta branch, froggy/archer, the user-space breakpoint support layer) to show it is all connected to provide a better debugging and tracing environment for GNU/Linux.

Looking for a handy reference of all the talks in the Free Java developer room at Fosdem 2009 in Brussels, Saturday 7 and Sunday 8 February? Look no further! PDF Poster and ODG Poster.

Our libre-java meeting at Fosdem is approaching quickly. February 7 and 8 in Brussels, Belgium. If you want to give a talk, do a demo or have some general presentation/hack-session in our room, please act quickly so we can still schedule it (the deadline is end of this week!).

This is what will be at our disposal:

• the room “AW1.120″ with a capacity of 74 seats (in the building “AW”)
  • on Saturday 2009-02-07 from 12:00 to 18:00
  • on Sunday 2009-02-08 from 09:00 to 17:00
• a video projector with VGA cable
• Internet connectivity (wifi A and B only, no wired)

This is what we need from you:

• Talk proposals and abstracts for activities.
  Please submit these BEFORE Saturday January 10!
  Either by emailing the Fosdem meeting committee at:
  fosdem2009@developer.classpath.org
  Or adding your proposal to the wiki:
  http://wiki.debian.org/Java/DevJam/2009/Fosdem

Also please add you name to the wiki even if you don’t want to present something so we know roughly how many people to expect.

planet.classpath.org moved servers and if done correctly nobody will notice (except for the new server having a totally sweet favicon ). But if you do happen to notice anything odd with the planet after the move, then please do yell and scream.

Without much fanfare systemtap 0.8 was released a little while ago. There is one little tidbit in the release notes that does warrent some excitement though:

User space probing is supported at a prototype level, for kernels built with the utrace patches.

So what does that mean? Take for example the para-callgraph.stp script:

$ stap para-callgraph.stp 'process("/bin/ls").function("*")' -c /bin/ls
0 ls(12631):->main argc=0x1 argv=0x7fff1ec3b038
276 ls(12631): ->human_options spec=0x0 opts=0x61a28c block_size=0x61a290
365 ls(12631): <-human_options return=0x0
496 ls(12631): ->clone_quoting_options o=0x0
657 ls(12631):  ->xmemdup p=0x61a600 s=0x28
815 ls(12631):   ->xmalloc n=0x28
908 ls(12631):   <-xmalloc return=0x1efe540
950 ls(12631):  <-xmemdup return=0x1efe540
990 ls(12631): <-clone_quoting_options return=0x1efe540
1030 ls(12631): ->get_quoting_style o=0x1efe540
[...]
650290 ls(12631):  <-print_current_files
650330 ls(12631): <-print_dir
650456 ls(12631): ->free_pending_ent p=0x1f02d90
650539 ls(12631): <-free_pending_ent
650660 ls(12631): ->close_stdout
650821 ls(12631):  ->close_stream stream=0x376db6c780
650966 ls(12631):  <-close_stream return=0x0
651082 ls(12631):  ->close_stream stream=0x376db6c860
651164 ls(12631):  <-close_stream return=0x0
651205 ls(12631): <-close_stdout

That is timestamp, process name, tid, function entry and function exit with parameters and return values. Currently it relies on having the debuginfo files available, so make sure you install the coreutils-debuginfo package (if you want to trace /bin/ls and friends). Systemtap 0.8 should be in a distro near you soon. Fedora 10 already has it.

Another nice thing added in Fedora 10 is oprofile-jit, which enhances the system profiler with java support (for runtimes supporting jvmti/jvmpi, gcj native code was obviously already supported), just add -agentlib:jvmti_oprofile to your java invocation, and then opreport can give you stuff like:

samples  %        linenr info                 image name   app name    symbol name
136220   20.3345  (no location information)   21010.jo     java        Interpreter
15176     2.2654  indexSet.cpp:528            libjvm.so    libjvm.so   IndexSetIterator::advance_and_next()
12273     1.8321  (no location information)   21010.jo     java        int[] java.math.BigInteger.montReduce(int[], int[], int, int)
11129     1.6613  (no location information)   21010.jo     java        int java.text.CollationElementIterator.next()
9932      1.4826  (no location information)   21010.jo     java        java.lang.String com.sun.javatest.finder.JavaCommentStream.readComment()~1
9731      1.4526  (no location information)   21010.jo     java        java.nio.charset.CoderResult sun.nio.cs.UTF_8$Decoder.decodeArrayLoop(java.nio.ByteBuffer, java.nio.CharBuffer)
9239      1.3792  reg_split.cpp:409           libjvm.so    libjvm.so   PhaseChaitin::Split(unsigned int)
8617      1.2863  ifg.cpp:464                 libjvm.so    libjvm.so   PhaseChaitin::build_ifg_physical(ResourceArea*)

Note how you can see the percentages of time spend in the interpreter, compiled methods, hotspot (and if I would have it enabled, libc, kernel, etc). The above is clearly a somewhat short run (of the jtreg crypto tests) and you can see that most of the time is spend in the Interpreter because the methods haven’t been compiled yet.

Paul wrote about “Fedora 10 around the corner!” and said something really nice:

Last, but certainly not least, I want to thank you, the reader, if I haven’t already. You’re part of our community too, and without you we would be diminished. Free software isn’t just about bits and bytes, it’s about people, about doing something real, something tangible, something lasting for your fellow human beings. And with your help, the Fedora Project has been able to lead in free software innovation for over five years and ten releases now. Each and every one of you — pat yourself on the back for a job well done.

Thanks Paul, very well said.

This post by Kirill Grouchnikov from Pushing Pixels made me sad:

Trust is hard to build and easy to destroy

Are we really doing that badly? Ever since the full GPL release by Sun of the reference implementation of Java as OpenJDK and the positive wave that IcedTea brought to unite the existing libre java communities pushing Java into the core of the various GNU/Linux distros, I feel like the sky is the limit. We as a community now have the full freedom to cooperate with each other in whatever way we like. But for some the feeling is still not there. Are we afraid to really define “Java the next generation”? It seems the code and the spirit is there. But there is a leadership trust issue. How do we fix that?

There is always a lot going on in the low levels of the GNU Toolchain (gcc, binutils, gdb, etc) but since it is so low level and sometimes a bit specialised it is hard to keep up. Luckily Nick Clifton started a blog writing a monthly GNU Toolchain Update. This month saw lots of GCC updates (a new register allocator, a new set of loop transformation optimizations, a new picoChip port and more). Really recommended for those who want to keep informed about the latest GNU Toolchain improvements.