Software Freedom Conservancy 2022 Fundraiser

Please donate to the Software Freedom Conservancy this year. Software Freedom Conservancy has been growing and is able to take on the work it does thanks to the incredible support of individuals who care about an organization who stands up for the equitable, ethical and end user focused technologies.

They have been a great partner to Sourceware, helping with the GNU Toolchain Infrastructure, putting developers and community first.

New services for sourceware, SFC & FSF

The FSF was nice enough to host a video chat on Sourceware infrastructure – A presentation and community Q&A. Which was basically the BoF we had wanted to give about Sourceware Infrastructure at the Cauldron. Extended with some discussion on recent developments, Sourceware as Conservancy member project and collaboration with the FSF tech-team. It was less interactive than the in person BoF would have been, but there was some nice feedback afterwards.

Slides New services for sourceware projects. Markdown source plus presenter notes.

Valgrind 3.20.0

We are pleased to announce a new release of Valgrind, version 3.20.0, available from

This is mostly a bug fix release to make sure valgrind works well against the latest gcc, glibc and linux kernel, but also contains a lot of work to make valgrind work better on FreeBSD. Plus a couple of new features:

  • The option “–vgdb-stop-at=event1,event2,…” accepts the new value abexit. This indicates to invoke gdbserver when your program exits abnormally (i.e. with a non zero exit code).
  • Fix Rust v0 name demangling.
  • The Linux rseq syscall is now implemented as (silently) returning ENOSYS.
  • Add FreeBSD syscall wrappers for __specialfd and __realpathat.
  • Remove FreeBSD dependencies on COMPAT10, which fixes compatibility with HardenedBSD
  • The option –enable-debuginfod= [default: yes] has been added on Linux.
  • More DWARF5 support as generated by clang14.

The most interesting bug fixed (IMHO) is #458915 syscall sometimes returns its number instead of return code when vgdb is attached. It might not hit you, but if it did it was really confusing.

Sourceware Infrastructure / Conservancy / GNU Toolchain at Cauldron

Cauldron was fun, heard so many interesting talks, met so many fun people, had great conversations.

I also had a BoF about all the great infrastructure work we have been adding to Sourceware over that last year and had wanted to discuss how the different project hosted on Sourceware wanted to use it to improve their processes: New services for sourceware projects. The file also has the presenter notes and discussion items I had prepared.

But when I said that at the end I also would like to discuss the recent Sourceware as Conservancy member project proposal and that I had asked both Conservancy and FSF members to call in to help with that discussion there was what felt like a shouting match. It was the first time I felt an in-person event was worse than an email discussion. Hopefully people will calm down and restart this discussion on the sourceware overseers list.

Sourceware as Conservancy member project


Last month the Sourceware overseers started a discussion with the projects hosted on Sourceware and the Software Freedom Conservancy (SFC) to become a Conservancy member project (which means Conservancy would become the fiscal sponsor of Sourceware). After many positive responses the SFC’s Evaluations Committee has voted to accept Sourceware.

Read the rest of this entry »

Happy birthday, Valgrind!

It has been twenty years today since Valgrind 1.0 was released.

Make sure to read Nicholas Nethercote’s Twenty years of Valgrind to learn about the early days, Valgrind “skins”, the influence Valgrind had on raising the bar when it comes to correctness for C and C++ programs, and why a hacker on the Rust programming language still uses Valgrind.

Sourceware – GNU Toolchain Infrastructure roadmap

Making email/git based workflow more fun, secure and productive by automating contribution tracking and testing across different distros and architectures.

What is Sourceware?

Sourceware,, is community run infrastructure, mailinglists, git, bug trackers, wikis, etc. hosted in the Red Hat Open Source Community Infrastructure Community Cage together with servers from e.g. Ceph, CentOS, Fedora and Gnome.

Sourceware is mainly known for hosting the GNU Toolchain projects, like gcc at, glibc, binutils and gdb. But also hosts projects like annobin, bunsen, bzip2, cgen, cygwin at, debugedit, dwz, elfutils at, gccrs, gnu-abi, insight, kawa, libffi, libabigail, mauve, newlib, systemtap and valgrind at

A longer list of Sourceware projects, those without their own domain name, including several dormant projects, can be found here:

Most of these projects use a email/git based workflow using mailinglists for discussing patches in preference to web based “forges”.

Zero maintenance automation

Although email based git workflows are great for real patch discussions, they do not always make tracking the state of patches easy.

Just like our other services, such as bugzilla, mailinglists and git repos we like to provide zero maintenance infrastructure for tracking and automation of patches and testing.

So we are trying to consolidate around a shared buildbot for (test) automation and patchwork for tracking the state of contributions. By sharing experiences between the Sourceware projects and coordination and fully automating the infrastructure services.

A shared buildbot

We have a shared buildbot for Sourceware projects at This includes compute resources (buildbot-workers) for various architectures thanks to some generous sponsors. We have native/VM workers for x86_64, ppc64le, s390x, ppc64, i386, arm64 and armhf for debian, fedora and centos (although not all combinations yet) and x86_64 container builders for fedora, debian and opensuse.

There are currently 95 builders on 15 workers, doing ~300 builds a day (more on week days, less on weekends). There are a couple of full testsuite builders (for gcc and binutils-gdb), but most builders are “quick” CI builders, which will sent email whenever a regression is detected. It seems to catch and report a couple of issues a week across all projects.

Builder is its own project on Sourceware which comes with its own git repo, mailinglist and amazing community, that can help you integrate new builders, add workers, containers and get you access to systems to replicate any failures where the buildbot logs don’t give enough information.

And buildbot itself is automated, so whenever a change is made to add a new builder, or define a new container, the buildbot automatically reconfigures itself and the workers will start using the new container images starting with the next build.

The same mechanism can also be used to run tasks on specific commits or periodically. Tasks which are now often done by a cron job or git hook. For example to update documentation, websites, generate release tars or update bugzilla. The advantage over cron jobs is that it can be done more immediately and/or only when really needed based on specific commit files. The advantage over git hooks is that they run in the builder context, not in the context of the specific user that pushed a commit.

Picking your (CI) tests

Although the buildbot itself is zero maintenance, getting and acting on the results of course is not. We already divide the tests into quick CI tests and full test runs. And most tests upload all results to bunsendb. bunsen can help pick good CI tests by indicating which tests are flaky or compare results across different setups.

A prototype testsuite log comparison bunsenweb widget is running at

Lots of things will be coming here, including taking advantage of testrun cluster analysis that’s already being done, a per-testrun testcase search/browse engine, other search operators, testsuite summary (vs detail) grids, who knows, ideas welcome!

What about pre-commit checks?

The builder CI checks what has been committed on the main branch of the projects. This makes sure that what is checked out is in a good state and that any pushed regressions are found early and often.

There is also support for git user try branches. When a user pushes to their try branch the same builder CI checks are ran, so a project developer knows their proposed patch(es) won’t break the build or introduce regressions.

The binutils and gdb communities are currently trying this out. Once new builder resources from OSUOSL are installed we’ll roll this out to other Sourceware projects.

What about non-committers?

The above only helps developers that have commit access on sourceware, but not others who sent in patches. For that we have plus the CICD trybot that DJ wrote The glibc community is already using this. We would like to connect patchwork, buildbot and the trybot for other Sourceware projects

The current trybot doesn’t do authentication, this might not be OK for all builders. So we want to either require checking for known GPG keys on the patch emails or let a trusted developer set a flag in patchwork before the trybot triggers. Once we have public-inbox setup we could also use b4 for DKIM attestation for known/trusted hackers.

Some projects have already experimented with public-inbox. But we don’t have an instance running on sourceware itself yet. This would resolve complaints of not very usable mailman archives.

But I really like to have a webforge!

You are in luck. We already have a sourcehut mirror at This allows anybody to fork any sourceware project on sourcehut, prepare their patches and submit a merge request through email (without having to locally setup git send-email or smtp – the patch emails are generated server side).

Sourcehut is designed around email based workflows, fully Free Software, doesn’t use javascript and is much faster and resource constrained compared to (proprietary) alternatives.

The sourcehut mirror is currently read-only (but syncs automatically with any git updates made on sourceware). When sourcehut supports project groups (one of the beta goals) we will test a self-hosted instance to see whether this is a good way to attract more contributors without loosing the advantages of the email based workflow. The various components are very modular so we can only use those parts we need.

Valgrind 3.18.1

We are pleased to announce a new release of Valgrind, version 3.18.1, available from

3.18.1 fixes a number of bugs and adds support for glibc-2.34, and for new platforms x86/FreeBSD and amd64/FreeBSD.  Debuginfo reading is faster, and Rust demangling has been improved. For PPC64, ISA 3.1 support has been completed, and some newer ARM64 and S390 instructions are also supported.  See the release notes for details of changes.  Note, 3.18.0 had no formal release — it was pulled at the last minute due to a packaging problem.

Our thanks to all those who contribute to Valgrind’s development.  This
release represents a great deal of time, energy and effort on the part of many people.

Happy and productive debugging and profiling,

The Valgrind Developers

Valgrind Memcheck: Different ways to lose your memory

The Red Hat Developer blog posted an article I wrote. Valgrind Memcheck: Different ways to lose your memory.

It explains the different kinds of memory leaks (definitely, reachable, possibly and indirectly). Why which kinds are reported and/or marked as errors by default or not. How to generate suppressions. And how to run things in a way that is most helpful in a regression test suite.

It also includes some C code examples that will make you want to run screaming away to a more memory safe language. But till then, please make sure to run valgrind on your code.

FSF Associate Membership

To the FSF leadership,

Last year I wrote you the following:

My membership name is mark and I am FSF associate member #6. I am currently contributing $10 + $32, for a total of $42 dollars a month.

Besides membership fees of more than $500, I also donated $2000 last year to the FSF because I believe in the Free Software mission and want to support the GNU project.

But I am very disappointed in how the FSF has been handling the recent changes in leadership and the efforts of various GNU volunteers (which includes myself as a GNU maintainer since 2000) to be more welcoming and creating a more flexible governance structure for GNU. The FSF keeps ignoring our calls for a neutral discussion space. And I think it is unacceptable that people are afraid to publicly discuss these issues because they feel intimidated and fear to get personal threats or have to endure racist or sexist language. The FSF really has a responsibility to the GNU volunteers to be able to work and communicate with each other without having to feel harassed all the time

So I am reducing my membership fee to a minimum $10 a month and won’t be making any additional contributions till these issues have been resolved

Since then we setup the GNU Assembly without any resources from the FSF. And created the GNU Social Contract for GNU maintainers and developers to promote the GNU system and create a welcome environment for everybody to be able to create more user freedom. The FSF never officially helped or even replied to our requests to formulate an open and welcoming working relationship with us as GNU volunteers.

And now, instead of taking the opportunity to diversify the FSF leadership with a view towards the future and more inclusiveness, the FSF simply reinstated Richard Stallman as board member without any explanation how this addresses the original issues or the changes needed in the FSF.

This makes the FSF a negative force in promoting Free Software and it negatively impacts the GNU system, the GNU developers and users.

Hopefully the FSF will be able to find new leadership, so that it can again be a positive force for Free Software, support the GNU project as a welcoming community and promote user freedom. But for now I am canceling my FSF Associate Membership.


Mark Wielaard