• Linkers part 1 – A Personal Introduction and A Technical Introduction.
• Linkers part 2 – Linker Technical Introduction, Basic Linker Data Types, Basic Linker Operations.
• Linkers part 3 – Address Spaces, Object File Formats.
• Linkers part 4 – Shared Libraries (Procedure Linkage Table – PLT and Global Offset Table – GOT).
• Linkers part 5 – Shared Libraries Redux, ELF Symbols.
• Linkers part 6 – Relocations, Position Dependent Shared Libraries.

There will be more notes in the future, but this should keep you occupied for a little while (should make a great weekend read) The writing does jump a little between subjects, just keep on reading and go back over all the notes again in the end. There are some assembler tricks in there so keep your x86 assembly manual handy. Have fun!

Unwinding the call stack used to be something only a debugger would do and relied on the executable having a frame pointer in a dedicated register that points to the bottom of the stack frame for the current function which also contained the return address. Having a frame pointer allows you to quickly walk the call stack and get all the addresses. If you can map those to the names of the relevant functions they are in you have a nice backtrace for the user.

If you want to get more of the state in each call frame then you could rely on each function having a prologue and epilogue that saved and restored the registers of the caller (some architectures like x86 even have special instructions to help push and pop the relevant registers on the stack on function entry and return). Given a calling convention for a particular architecture you could use these to reliably find the original registers on the stack, which in turn with some debug info would give you the values of variables and arguments of the functions on the call stack.

Unfortunately compilers got smart and optimized code might not keep a frame pointer (frees up one more register) and might reschedule the function prologue and epilogue instructions between the other instructions in the function. All making it pretty hard for an unwinder to reconstruct the previous call frames on the stack. In particular x86_64 does away with a standard frame pointer. You can still get some information back by conservatively approximating the instructions in the function and guessing at the actual way the various registers are stored but this becomes pretty messy pretty quickly.

To help debuggers still get all the information needed to unwind a stack and restore all needed registers the debugging information (DWARF) generated by compilers was extended to include Call Frame Information (CFI) that allows a debugger to reconstruct the calling pc and registers of a function (see the DWARF 3 spec – section 6.4). This information is stored in the .debug_frame section of an ELF file. It uses a simplified version of the DWARF instructions (not all operands are relevant for reconstructing the registers). This section is not guaranteed to be available, it is not necessarily loaded into memory and can even be split off into its own debug info file in some distributions.

At the same time different languages got constructs (exceptions, continuations, global gotos, asynchronous garbage collectors, etc) which required some sort of reliable unwinding (and in some cases rewinding) of the call stack. Since some optimizations and some newer architectures also did away with a standard frame pointer another way to reliably unwind the stack was needed. This became the exception handler framework (.eh_frame) which is based on the DWARF CFI work but which is slightly different. Unfortunately nobody seems to have documented the precise differences between the formats. So you will have to carefully read both the DWARF standard and the LSB core specification Exception Frames side-by-side.

Note that a debugger that wants to walk a stack and recover all registers might need more information than some of these language constructs, which might only need unwind information for specific call sites. Depending on optimizations, architecture and language compiled (and sometimes specific distribution default choices) no, full or partial exception handler unwind information and/or frame pointers are generated (see the GCC options -funwind-tables and -fasynchronous-unwind-tables).

Both the DWARF and the exception handler specs are architecture neutral. But since you do need to a mapping between the actual registers and the specs you also need to consult the relevant architecture abi that defines the actual mapping. Sometimes these architecture abi specs also define some DWARF/EH extensions. See for example the x86_64 abi spec (Section 3.6 and 3.7).

Note that in practice what gcc generates overrides any of the above specs, and if a discrepancy is found the spec usually gets updated. And that one should be careful about bugs in the old DWARF 2 spec and extension of DWARF specified by the LSB (which mostly augment DWARF 2 to be like DWARF 3, at least for the exception handler sections).

If an .eh_frame section is available in an ELF file it is guaranteed to be loaded into memory. But depending on architecture and language being compiled might not be available at all (and neither might the frame pointer or the .debug_frame section). This does also mean that unwind information might be stored differently for different components linked together into a program if they were compiled with different flags or have different source languages, making cross component/language unwinding an interesting exercise.

Now that my server was uber-stable again (and pretty modern because of the CentOS 5 upgrade) I played a bit with setting up a Mercurial and Trac mirror for Frysk. This was super easy with EPEL a Fedora spinoff to provide stable packages for the enterprise distros (RHEL, CentOS and Scientific Linux). The only thing missing was Tailor but that could be easily build from the Fedora source package. Trac is super nice, easy to configure and theme for small projects. It provides a wiki, roadmap, ticketing/bug system and source browser for various source code management systems (subversion, mercurial, git or bazaar). And you can disable anything you don’t need (for the Frysk mirror I am only using the source browser). I am definitely going to use Trac plus some distributed source management system for my next project.

Since the Performance Monitoring Unit (PMU) on different processor architectures are so different and support collecting very different sets of events designing a generic interface is pretty hard. But it looks like perfmon2 does present a nice, although somewhat complex, common interface to program the Performance Monitoring Configuration (PMC) registers and read the data collected in the Performance Monitoring Data (PMD) registers. And it provides a somewhat nicer way to program and collect data than having to go through the raw hardware. It provides things like generic 64-bit counters for events (and in gneeral makes sure that all data structures use fixed-size data types), even if the underlying hardware has smaller counters (doing emulation in software when the hardware counters overflow). And most importantly makes it available to user space in a secure manner, so one doesn’t need direct hardware access in privileged mode and prevents “data leaks” between untrusted processes.

The interface allows for both counting and profiling (sampling) events on a per-thread or per-cpu basis. But whole-process or whole-system profilling is left up to the user (through ptrace attaching a thread and tracking clone events, although exec events do automatically carry over monitoring contexts. Maybe a utrace based framework would make things simpler here, but currently it seems the perfmon2 and utrace patches don’t mix). The paper is somewhat vague on how and when one can use a mixed per-thread and per-cpu environment, which is somewhat unfortunate since it looks like if an admin is using per-cpu monitoring a self-monitoring process cannot simultaneously use the MPD counters. Self-monitoring is interesting since it means a dynamic runtime environment like hotspot that dynamically regenerates code can easily see “hot code paths”. One limitation seems to be that threads using this technique need to be tied to one processor since the monitoring context cannot migrate between CPUs.

The paper gives a good overview of the various techniques used to detect and access the various events supported by the CPU and expose the counters through new system calls, translation files in /sys/kernel from logical registers to actual register names and mapping in read only shared buffers between kernel and user space for self-monitoring threads. The event sets and multiplexing of events is interesting but very abstract. The paper doesn’t contain any code samples and one is assumed to know the kind of performance event counters modern CPUs support. Things become a little easier if one reads this paper while having access to a system with pfmon tool (and the perfmon2 kernel patch) installed or reading the pfmon manual to look at examples to make things a bit more concrete.

Another nice paper about DWARF is Introduction to the DWARF Debugging Format, by Michael Eager

yumdownloader --enablerepo=core-source --source gcc

(don’t forget that you have to enable the correct source repo for the package, also don’t know whether that can be automated)
Now Seth Vidal wrote debuginfo-install to get the debug packages (and dependencies) for a package. Now if we could automate all these things in such a way that you automagically get all relevant debug-info and optionally the sources downloaded and installed from Frysk when inspecting a package that would be really cool.