dtrace for linux; Oracle does the right thing
At Fosdem we had a talk on dtrace for linux in the Debugging Tools devroom.
Not explicitly mentioned in that talk, but certainly the most exciting thing, is that Oracle is doing a proper linux kernel port:
commit e1744f50ee9bc1978d41db7cc93bcf30687853e6 Author: Tomas Jedlicka <tomas.jedlicka@oracle.com> Date: Tue Aug 1 09:15:44 2017 -0400 dtrace: Integrate DTrace Modules into kernel proper This changeset integrates DTrace module sources into the main kernel source tree under the GPLv2 license. Sources have been moved to appropriate locations in the kernel tree.
That is right, dtrace dropped the CDDL and switched to the GPL!
The user space code dtrace-utils and libdtrace-ctf (a combination of GPLv2 and UPL) can be found on the DTrace Project Source Control page. The NEWS file mentions the license switch (and that it is build upon elfutils, which I personally was pleased to find out).
The kernel sources (GPLv2+ for the core kernel and UPL for the uapi) are slightly harder to find because they are inside the uek kernel source tree, but following the above commit you can easily get at the whole linux kernel dtrace directory.
Update: There is now a dtrace-linux-kernel.git repository with all the dtrace commits rebased on top of recent upstream linux kernels.
The UPL is the Universal Permissive License, which according to the FSF is a lax, non-copyleft license that is compatible with the GNU GPL.
Thank you Oracle for making everyone’s life easier by waving your magic relicensing wand!
Now there is lots of hard work to do to actually properly integrate this. And I am sure there are a lot of technical hurdles when trying to get this upstreamed into the mainline kernel. But that is just hard work. Which we can now start collaborating on in earnest.
Like systemtap and the Dynamic Probes (dprobes) before it, dtrace is a whole system observability tool combining tracing, profiling and probing/debugging techniques. Something the upstream linux kernel hackers don’t always appreciate when presented as one large system. They prefer having separate small tweaks for tracing, profiling and probing which are mostly separate from each other. It took years for the various hooks, kprobes, uprobes, markers, etc. from systemtap (and other systems) to get upstream. But these days they are. And there is now even a byte code interpreter (eBPF) in the mainline kernel as originally envisioned by dprobes, which systemtap can now target through stapbpf. So with all those techniques now available in the linux kernel it will be exciting to see if dtrace for linux can unite them all.
All this really means is that the replacements developed by the Linux kernel community are now powerful enough for it to no longer constitute a competitive advantage to keep dtrace incompatibly licensed, isolating it in its own little Solaris ecosystem. So they are now trying to kill the community efforts in a desperate move to capture the large Linux (kernel) user base before it is too late.
The right thing to do would have been to do what they are doing now back when dtrace was still actually an improvement over the status quo.
If you will look closer you can find that perf still is waaay behind DTrace even in case of quite a simple tracing so the conclusion that Oracle is trying to kill anything is really off the target.
OL customers base is growing and without something like DTrace supporting Linux by Oracle is really way harder than the on Solaris.
perf has very limited functionalities and processing/correlating the tracing data sampled from multiple pints requires way bigger level of complexity using external scripting than it is possible to do using DTrace.
To the best of my knowledge, both DTrace and LinuxPerf to a large part come from efforts by the same engineer, Brendan Gregg (not to diminish less distinguished dozens of others who helped), as he worked with different teams across his career. And indeed, much of that started as he missed the lack of observability he was used to having in Solaris, when he was hired into a Linux-based ecosystem.
Of course not.
First, eBPF is entirely insecure, they even added non-associative arrays to their language, and the framework is rife for abuse. On a proper system you have to disable eBPF for security reasons.
Second, you cannot expect the large dtrace ecosystem in 3rdparty apps to be thrown away, just because linux devs thought they can do it better. They couldn’t. They can do less, slower and are terribly insecure. dtrace was years ahead, and there are still no proper stap probes anywhere. And I would refuse to do so on any public service.
dTrace for linux is the only proper way to continue, same as on darwin and bsd.
Nice argument Kevin, so even when Oracle does the Right Thing(TM) it’s of course just to be Evil?
How paranoid is it possible to be?