Mark J. Wielaard

Thomas Zander, who got a cool job at Trolltech, pointed me at Eskil’s blog who shows some cool integration between Qt, AWT and Swing. I quickly tested QT/Jambi (all released under the GPL!) against IcedTea and it works pretty nicely. There seems to be some JNI issue when running it against GCJ though. Needs some debugging. But Jambi can also be build against GCJ and QT/Jambi is already part of Mandrake (they even submitted a bug/patch for classpath jni.h, thanks!).

Ian Lance Taylor is working on a new ELF linker called gold which is designed for speed and will do incremental linking. He is writing notes on how linkers work (with specifics for ELF and i386) which are a must read for everybody interested in these kind of low level technicalities.

• Linkers part 1 – A Personal Introduction and A Technical Introduction.
• Linkers part 2 – Linker Technical Introduction, Basic Linker Data Types, Basic Linker Operations.
• Linkers part 3 – Address Spaces, Object File Formats.
• Linkers part 4 – Shared Libraries (Procedure Linkage Table – PLT and Global Offset Table – GOT).
• Linkers part 5 – Shared Libraries Redux, ELF Symbols.
• Linkers part 6 – Relocations, Position Dependent Shared Libraries.

There will be more notes in the future, but this should keep you occupied for a little while (should make a great weekend read) The writing does jump a little between subjects, just keep on reading and go back over all the notes again in the end. There are some assembler tricks in there so keep your x86 assembly manual handy. Have fun!

If you are running Fedora Rawhide (the Fedora development branch) you can now do:

yum install java-1.7.0-icedtea \
java-1.7.0-icedtea-devel \
java-1.7.0-icedtea-demo \
java-1.7.0-icedtea-plugin \
java-1.7.0-icedtea-src

Just in time for Fedora 8 test 2! This is an experimental package based on OpenJDK b18 and IcedTea 1.3 and includes the browser plugin!

Note: When using x86_64 the above will install both the x86_64 and i586 versions. You want to make sure that you run update-alternative --config java to select the x86_64 version explicitly or bad things will happen since most jni libraries only come in 64bit variants.

More info: http://fedoraproject.org/wiki/Features/IcedTea.

For Frysk I am working to add .debug_frame support to libunwind. Since I didn’t know that much about stack unwinding and since some of the details are hard to grasp without some historical context I wrote the following to summarize all the relevant documentation pointers that I could find on stack unwinding. Please do let me know if I got any details wrong (the following is mostly written with x86/x86_64 in mind since that is what I am currently interested in).

Unwinding the call stack used to be something only a debugger would do and relied on the executable having a frame pointer in a dedicated register that points to the bottom of the stack frame for the current function which also contained the return address. Having a frame pointer allows you to quickly walk the call stack and get all the addresses. If you can map those to the names of the relevant functions they are in you have a nice backtrace for the user.

If you want to get more of the state in each call frame then you could rely on each function having a prologue and epilogue that saved and restored the registers of the caller (some architectures like x86 even have special instructions to help push and pop the relevant registers on the stack on function entry and return). Given a calling convention for a particular architecture you could use these to reliably find the original registers on the stack, which in turn with some debug info would give you the values of variables and arguments of the functions on the call stack.

Unfortunately compilers got smart and optimized code might not keep a frame pointer (frees up one more register) and might reschedule the function prologue and epilogue instructions between the other instructions in the function. All making it pretty hard for an unwinder to reconstruct the previous call frames on the stack. In particular x86_64 does away with a standard frame pointer. You can still get some information back by conservatively approximating the instructions in the function and guessing at the actual way the various registers are stored but this becomes pretty messy pretty quickly.

To help debuggers still get all the information needed to unwind a stack and restore all needed registers the debugging information (DWARF) generated by compilers was extended to include Call Frame Information (CFI) that allows a debugger to reconstruct the calling pc and registers of a function (see the DWARF 3 spec – section 6.4). This information is stored in the .debug_frame section of an ELF file. It uses a simplified version of the DWARF instructions (not all operands are relevant for reconstructing the registers). This section is not guaranteed to be available, it is not necessarily loaded into memory and can even be split off into its own debug info file in some distributions.

At the same time different languages got constructs (exceptions, continuations, global gotos, asynchronous garbage collectors, etc) which required some sort of reliable unwinding (and in some cases rewinding) of the call stack. Since some optimizations and some newer architectures also did away with a standard frame pointer another way to reliably unwind the stack was needed. This became the exception handler framework (.eh_frame) which is based on the DWARF CFI work but which is slightly different. Unfortunately nobody seems to have documented the precise differences between the formats. So you will have to carefully read both the DWARF standard and the LSB core specification Exception Frames side-by-side.

Note that a debugger that wants to walk a stack and recover all registers might need more information than some of these language constructs, which might only need unwind information for specific call sites. Depending on optimizations, architecture and language compiled (and sometimes specific distribution default choices) no, full or partial exception handler unwind information and/or frame pointers are generated (see the GCC options -funwind-tables and -fasynchronous-unwind-tables).

Both the DWARF and the exception handler specs are architecture neutral. But since you do need to a mapping between the actual registers and the specs you also need to consult the relevant architecture abi that defines the actual mapping. Sometimes these architecture abi specs also define some DWARF/EH extensions. See for example the x86_64 abi spec (Section 3.6 and 3.7).

Note that in practice what gcc generates overrides any of the above specs, and if a discrepancy is found the spec usually gets updated. And that one should be careful about bugs in the old DWARF 2 spec and extension of DWARF specified by the LSB (which mostly augment DWARF 2 to be like DWARF 3, at least for the exception handler sections).

If an .eh_frame section is available in an ELF file it is guaranteed to be loaded into memory. But depending on architecture and language being compiled might not be available at all (and neither might the frame pointer or the .debug_frame section). This does also mean that unwind information might be stored differently for different components linked together into a program if they were compiled with different flags or have different source languages, making cross component/language unwinding an interesting exercise.

Once or twice a week my personal server had some weird trouble with httpd, spamassasin or named suddenly crashing. It seemed impossible to replicate but in recent weeks the problem became worse. First I thought it was some software problem. So I decided to upgrade to CentOS 5. But in the middle of the upgrade the installer suddenly crashed. Luckily the upgrade could be resumed after a reboot into the rescue CD, but it was a bit ugly. Even though the disk, CD and memory (hours of running MemTest86+) all looked fine I finally decided to replace the memory. And wow, the server is as stable as can be. Moral of the story, don’t blame your software (Free Software is uber-stable!) suspect hardware problems (and don’t think MemTest86+ can actually find all memory problems).

Now that my server was uber-stable again (and pretty modern because of the CentOS 5 upgrade) I played a bit with setting up a Mercurial and Trac mirror for Frysk. This was super easy with EPEL a Fedora spinoff to provide stable packages for the enterprise distros (RHEL, CentOS and Scientific Linux). The only thing missing was Tailor but that could be easily build from the Fedora source package. Trac is super nice, easy to configure and theme for small projects. It provides a wiki, roadmap, ticketing/bug system and source browser for various source code management systems (subversion, mercurial, git or bazaar). And you can disable anything you don’t need (for the Frysk mirror I am only using the source browser). I am definitely going to use Trac plus some distributed source management system for my next project.

Bruno Souza is the NetBeans Community Manager for Sun these days. And he had an interesting suggestion for a new license model for NetBeans 6.0: We are considering releasing a future early access version of NetBeans 6.0 under a dual licensing scheme of CDDL and GPL v2 with Classpath exception. Whoa!

From the NetBeans Why GPL v2 Frequently Asked Questions:

Why does Sun want to dual license NetBeans software under CDDL and GPLv2 with Classpath Exception?

• The GPL v2 license will provide an additional option to vendors that are unable to work with NetBeans software under the CDDL license.
• Adding GPLv2 as a license option will make NetBeans software even more Linux friendly.
• Adding GPLv2 with Classpath exception to NetBeans software will keep product portfolios and bundles consistent. Sun open sourced its JDK implementation under GPLv2 and the GlassFish project is dual-licensed under CDDL and GPLv2 with Classpath exception.

Good move Sun! I hope the rest of the NetBeans community agrees. Personally I would love to see this happening. With this there probably isn’t a corner of the Java ecosystem that isn’t covered by the GPL. Sun is radically changing the industry here. Allowing anybody to share, mix and replay anything java under the GPL is major. It will be facinating to see what will all come from this.

Reading Harald Welte‘s blog (who is sometimes painfully honest about the Neo1973 and OpenMoko developments) I got a happy surprise reading under the heading Progress with the new OpenMoko and FIC Mobility office:

Our internal software team finally has proper leadership and guidance from somebody who is both Taiwanese and has a thorough understanding of Free Software: jserv

Whoa! That is Jim Huang. Who is a Kaffe contributor and recently made the Cacao VM run on OpenMoko. Congrats Jim! And Congrats FIC! OpenMoko/Neo1973 will surely fly with him on board.

This is a public service announcement: DON’T FORGET TO RENEW YOUR DOMAINS IN TIME! Thanks to Brian for getting classpath.org back online.

The 2006 OLS paper Perfmon2: A flexible performance monitoring interface for Linux by Stéphane Eranian gives an overview of designing a generic interface for hardware monitoring for a diverse test of processors. Modern processors have all kinds of support to collect information about cpu cycles used, instruction pipelines, on-chip caches, etc.

Since the Performance Monitoring Unit (PMU) on different processor architectures are so different and support collecting very different sets of events designing a generic interface is pretty hard. But it looks like perfmon2 does present a nice, although somewhat complex, common interface to program the Performance Monitoring Configuration (PMC) registers and read the data collected in the Performance Monitoring Data (PMD) registers. And it provides a somewhat nicer way to program and collect data than having to go through the raw hardware. It provides things like generic 64-bit counters for events (and in gneeral makes sure that all data structures use fixed-size data types), even if the underlying hardware has smaller counters (doing emulation in software when the hardware counters overflow). And most importantly makes it available to user space in a secure manner, so one doesn’t need direct hardware access in privileged mode and prevents “data leaks” between untrusted processes.

The interface allows for both counting and profiling (sampling) events on a per-thread or per-cpu basis. But whole-process or whole-system profilling is left up to the user (through ptrace attaching a thread and tracking clone events, although exec events do automatically carry over monitoring contexts. Maybe a utrace based framework would make things simpler here, but currently it seems the perfmon2 and utrace patches don’t mix). The paper is somewhat vague on how and when one can use a mixed per-thread and per-cpu environment, which is somewhat unfortunate since it looks like if an admin is using per-cpu monitoring a self-monitoring process cannot simultaneously use the MPD counters. Self-monitoring is interesting since it means a dynamic runtime environment like hotspot that dynamically regenerates code can easily see “hot code paths”. One limitation seems to be that threads using this technique need to be tied to one processor since the monitoring context cannot migrate between CPUs.

The paper gives a good overview of the various techniques used to detect and access the various events supported by the CPU and expose the counters through new system calls, translation files in /sys/kernel from logical registers to actual register names and mapping in read only shared buffers between kernel and user space for self-monitoring threads. The event sets and multiplexing of events is interesting but very abstract. The paper doesn’t contain any code samples and one is assumed to know the kind of performance event counters modern CPUs support. Things become a little easier if one reads this paper while having access to a system with pfmon tool (and the perfmon2 kernel patch) installed or reading the pfmon manual to look at examples to make things a bit more concrete.

From Lillian on the OpenJDK distro-pkg-dev mailinglist about IcedTea 1.1

We are proud to announce the release of IcedTea 1.1. This release represents the stabilization of Crypto and SSL support.

The IcedTea project provides a harness to build the source code from OpenJDK (http://openjdk.java.net) using Free Software build tools and provides replacements libraries for the binary plugs with code from the GNU Classpath project. More information on IcedTea can be found here: http://icedtea.classpath.org

What’s new?

• GNU Crypto security providers have been merged from GNU Classpath.
• SSL support by means of the IcedTls security provider.
• DebugInfo is always generated.
• Initial graphics work: Color management and raster fixes for compatibility with LittleCMS, and work to satisfy libt2k requirements with Freetype.
• Uses OpenJDK build 15.
• Various bug fixes: http://icedtea.classpath.org/bugzilla/

The tarball and nosrc RPM can be downloaded here: http://icedtea.classpath.org/download/

The following people helped with this release:
Gary Benson, Thomas Fitzsimmons, Kyle Galloway, Andrew Haley, Francis Kung, Casey Marshall, Mark Wielaard and Lillian Angel.

We would also like to thank the bug reporters and testers!

Nice work!