Observe, systemtap and oprofile updates
Without much fanfare systemtap 0.8 was released a little while ago. There is one little tidbit in the release notes that does warrent some excitement though:
User space probing is supported at a prototype level, for kernels built with the utrace patches.
So what does that mean? Take for example the para-callgraph.stp script:
$ stap para-callgraph.stp 'process("/bin/ls").function("*")' -c /bin/ls
0 ls(12631):->main argc=0x1 argv=0x7fff1ec3b038
276 ls(12631): ->human_options spec=0x0 opts=0x61a28c block_size=0x61a290
365 ls(12631): <-human_options return=0x0
496 ls(12631): ->clone_quoting_options o=0x0
657 ls(12631): ->xmemdup p=0x61a600 s=0x28
815 ls(12631): ->xmalloc n=0x28
908 ls(12631): <-xmalloc return=0x1efe540
950 ls(12631): <-xmemdup return=0x1efe540
990 ls(12631): <-clone_quoting_options return=0x1efe540
1030 ls(12631): ->get_quoting_style o=0x1efe540
[...]
650290 ls(12631): <-print_current_files
650330 ls(12631): <-print_dir
650456 ls(12631): ->free_pending_ent p=0x1f02d90
650539 ls(12631): <-free_pending_ent
650660 ls(12631): ->close_stdout
650821 ls(12631): ->close_stream stream=0x376db6c780
650966 ls(12631): <-close_stream return=0x0
651082 ls(12631): ->close_stream stream=0x376db6c860
651164 ls(12631): <-close_stream return=0x0
651205 ls(12631): <-close_stdout
That is timestamp, process name, tid, function entry and function exit with parameters and return values. Currently it relies on having the debuginfo files available, so make sure you install the coreutils-debuginfo package (if you want to trace /bin/ls and friends). Systemtap 0.8 should be in a distro near you soon. Fedora 10 already has it.
Another nice thing added in Fedora 10 is oprofile-jit, which enhances the system profiler with java support (for runtimes supporting jvmti/jvmpi, gcj native code was obviously already supported), just add -agentlib:jvmti_oprofile to your java invocation, and then opreport can give you stuff like:
samples % linenr info image name app name symbol name 136220 20.3345 (no location information) 21010.jo java Interpreter 15176 2.2654 indexSet.cpp:528 libjvm.so libjvm.so IndexSetIterator::advance_and_next() 12273 1.8321 (no location information) 21010.jo java int[] java.math.BigInteger.montReduce(int[], int[], int, int) 11129 1.6613 (no location information) 21010.jo java int java.text.CollationElementIterator.next() 9932 1.4826 (no location information) 21010.jo java java.lang.String com.sun.javatest.finder.JavaCommentStream.readComment()~1 9731 1.4526 (no location information) 21010.jo java java.nio.charset.CoderResult sun.nio.cs.UTF_8$Decoder.decodeArrayLoop(java.nio.ByteBuffer, java.nio.CharBuffer) 9239 1.3792 reg_split.cpp:409 libjvm.so libjvm.so PhaseChaitin::Split(unsigned int) 8617 1.2863 ifg.cpp:464 libjvm.so libjvm.so PhaseChaitin::build_ifg_physical(ResourceArea*)
Note how you can see the percentages of time spend in the interpreter, compiled methods, hotspot (and if I would have it enabled, libc, kernel, etc). The above is clearly a somewhat short run (of the jtreg crypto tests) and you can see that most of the time is spend in the Interpreter because the methods haven’t been compiled yet.
thanks for the user space example for systemtap! I’ve found everywhere for it.
[…] Mark J. Wielaard » Blog Archive » Observe, systemtap and oprofile updates (tags: systemtap) […]
Is there any way to convince SystemTap to trace into a shared library?
yes, tracing a shared library works as if tracing an executable, through process(“/path/to/library”).function/statement probes.
So to find processes doing giant mallocs you could use something like: stap -e ‘probe process(“/lib/libc.so.6”).function(“malloc”) {if($bytes > 64 * 1024) log(execname() . ” malloc hog!”)}’