Observe, systemtap and oprofile updates

Posted on November 27, 2008, 18:21.

Without much fanfare systemtap 0.8 was released a little while ago. There is one little tidbit in the release notes that does warrent some excitement though:

User space probing is supported at a prototype level, for kernels built with the utrace patches.

So what does that mean? Take for example the para-callgraph.stp script:

$ stap para-callgraph.stp 'process("/bin/ls").function("*")' -c /bin/ls
0 ls(12631):->main argc=0x1 argv=0x7fff1ec3b038
276 ls(12631): ->human_options spec=0x0 opts=0x61a28c block_size=0x61a290
365 ls(12631): <-human_options return=0x0
496 ls(12631): ->clone_quoting_options o=0x0
657 ls(12631):  ->xmemdup p=0x61a600 s=0x28
815 ls(12631):   ->xmalloc n=0x28
908 ls(12631):   <-xmalloc return=0x1efe540
950 ls(12631):  <-xmemdup return=0x1efe540
990 ls(12631): <-clone_quoting_options return=0x1efe540
1030 ls(12631): ->get_quoting_style o=0x1efe540
[...]
650290 ls(12631):  <-print_current_files
650330 ls(12631): <-print_dir
650456 ls(12631): ->free_pending_ent p=0x1f02d90
650539 ls(12631): <-free_pending_ent
650660 ls(12631): ->close_stdout
650821 ls(12631):  ->close_stream stream=0x376db6c780
650966 ls(12631):  <-close_stream return=0x0
651082 ls(12631):  ->close_stream stream=0x376db6c860
651164 ls(12631):  <-close_stream return=0x0
651205 ls(12631): <-close_stdout

That is timestamp, process name, tid, function entry and function exit with parameters and return values. Currently it relies on having the debuginfo files available, so make sure you install the coreutils-debuginfo package (if you want to trace /bin/ls and friends). Systemtap 0.8 should be in a distro near you soon. Fedora 10 already has it.

Another nice thing added in Fedora 10 is oprofile-jit, which enhances the system profiler with java support (for runtimes supporting jvmti/jvmpi, gcj native code was obviously already supported), just add -agentlib:jvmti_oprofile to your java invocation, and then opreport can give you stuff like:

samples  %        linenr info                 image name   app name    symbol name
136220   20.3345  (no location information)   21010.jo     java        Interpreter
15176     2.2654  indexSet.cpp:528            libjvm.so    libjvm.so   IndexSetIterator::advance_and_next()
12273     1.8321  (no location information)   21010.jo     java        int[] java.math.BigInteger.montReduce(int[], int[], int, int)
11129     1.6613  (no location information)   21010.jo     java        int java.text.CollationElementIterator.next()
9932      1.4826  (no location information)   21010.jo     java        java.lang.String com.sun.javatest.finder.JavaCommentStream.readComment()~1
9731      1.4526  (no location information)   21010.jo     java        java.nio.charset.CoderResult sun.nio.cs.UTF_8$Decoder.decodeArrayLoop(java.nio.ByteBuffer, java.nio.CharBuffer)
9239      1.3792  reg_split.cpp:409           libjvm.so    libjvm.so   PhaseChaitin::Split(unsigned int)
8617      1.2863  ifg.cpp:464                 libjvm.so    libjvm.so   PhaseChaitin::build_ifg_physical(ResourceArea*)

Note how you can see the percentages of time spend in the interpreter, compiled methods, hotspot (and if I would have it enabled, libc, kernel, etc). The above is clearly a somewhat short run (of the jtreg crypto tests) and you can see that most of the time is spend in the Interpreter because the methods haven’t been compiled yet.

4 Comments

  1. dipplum says:
    July 31, 2009 at 08:12

    thanks for the user space example for systemtap! I’ve found everywhere for it.

  2. links for 2009-07-31 | 小李贼 says:
    July 31, 2009 at 16:36

    […] Mark J. Wielaard » Blog Archive » Observe, systemtap and oprofile updates (tags: systemtap) […]

  3. Albert Strasheim says:
    March 26, 2010 at 11:31

    Is there any way to convince SystemTap to trace into a shared library?

  4. Mark Wielaard says:
    April 3, 2010 at 15:11

    yes, tracing a shared library works as if tracing an executable, through process(“/path/to/library”).function/statement probes.

    So to find processes doing giant mallocs you could use something like: stap -e ‘probe process(“/lib/libc.so.6”).function(“malloc”) {if($bytes > 64 * 1024) log(execname() . ” malloc hog!”)}’

«
»