Observe, systemtap and oprofile updates
Without much fanfare systemtap 0.8 was released a little while ago. There is one little tidbit in the release notes that does warrent some excitement though:
User space probing is supported at a prototype level, for kernels built with the utrace patches.
So what does that mean? Take for example the para-callgraph.stp script:
$ stap para-callgraph.stp 'process("/bin/ls").function("*")' -c /bin/ls 0 ls(12631):->main argc=0x1 argv=0x7fff1ec3b038 276 ls(12631): ->human_options spec=0x0 opts=0x61a28c block_size=0x61a290 365 ls(12631): <-human_options return=0x0 496 ls(12631): ->clone_quoting_options o=0x0 657 ls(12631): ->xmemdup p=0x61a600 s=0x28 815 ls(12631): ->xmalloc n=0x28 908 ls(12631): <-xmalloc return=0x1efe540 950 ls(12631): <-xmemdup return=0x1efe540 990 ls(12631): <-clone_quoting_options return=0x1efe540 1030 ls(12631): ->get_quoting_style o=0x1efe540 [...] 650290 ls(12631): <-print_current_files 650330 ls(12631): <-print_dir 650456 ls(12631): ->free_pending_ent p=0x1f02d90 650539 ls(12631): <-free_pending_ent 650660 ls(12631): ->close_stdout 650821 ls(12631): ->close_stream stream=0x376db6c780 650966 ls(12631): <-close_stream return=0x0 651082 ls(12631): ->close_stream stream=0x376db6c860 651164 ls(12631): <-close_stream return=0x0 651205 ls(12631): <-close_stdout
That is timestamp, process name, tid, function entry and function exit with parameters and return values. Currently it relies on having the debuginfo files available, so make sure you install the coreutils-debuginfo package (if you want to trace /bin/ls and friends). Systemtap 0.8 should be in a distro near you soon. Fedora 10 already has it.
Another nice thing added in Fedora 10 is oprofile-jit, which enhances the system profiler with java support (for runtimes supporting jvmti/jvmpi, gcj native code was obviously already supported), just add -agentlib:jvmti_oprofile
to your java invocation, and then opreport
can give you stuff like:
samples % linenr info image name app name symbol name 136220 20.3345 (no location information) 21010.jo java Interpreter 15176 2.2654 indexSet.cpp:528 libjvm.so libjvm.so IndexSetIterator::advance_and_next() 12273 1.8321 (no location information) 21010.jo java int[] java.math.BigInteger.montReduce(int[], int[], int, int) 11129 1.6613 (no location information) 21010.jo java int java.text.CollationElementIterator.next() 9932 1.4826 (no location information) 21010.jo java java.lang.String com.sun.javatest.finder.JavaCommentStream.readComment()~1 9731 1.4526 (no location information) 21010.jo java java.nio.charset.CoderResult sun.nio.cs.UTF_8$Decoder.decodeArrayLoop(java.nio.ByteBuffer, java.nio.CharBuffer) 9239 1.3792 reg_split.cpp:409 libjvm.so libjvm.so PhaseChaitin::Split(unsigned int) 8617 1.2863 ifg.cpp:464 libjvm.so libjvm.so PhaseChaitin::build_ifg_physical(ResourceArea*)
Note how you can see the percentages of time spend in the interpreter, compiled methods, hotspot (and if I would have it enabled, libc, kernel, etc). The above is clearly a somewhat short run (of the jtreg crypto tests) and you can see that most of the time is spend in the Interpreter because the methods haven’t been compiled yet.
thanks for the user space example for systemtap! I’ve found everywhere for it.
[…] Mark J. Wielaard » Blog Archive » Observe, systemtap and oprofile updates (tags: systemtap) […]
Is there any way to convince SystemTap to trace into a shared library?
yes, tracing a shared library works as if tracing an executable, through process(“/path/to/library”).function/statement probes.
So to find processes doing giant mallocs you could use something like: stap -e ‘probe process(“/lib/libc.so.6”).function(“malloc”) {if($bytes > 64 * 1024) log(execname() . ” malloc hog!”)}’