Sourceware Infrastructure / Conservancy / GNU Toolchain at Cauldron

Cauldron was fun, heard so many interesting talks, met so many fun people, had great conversations.

I also had a BoF about all the great infrastructure work we have been adding to Sourceware over that last year and had wanted to discuss how the different project hosted on Sourceware wanted to use it to improve their processes: New services for sourceware projects. The file also has the presenter notes and discussion items I had prepared.

But when I said that at the end I also would like to discuss the recent Sourceware as Conservancy member project proposal and that I had asked both Conservancy and FSF members to call in to help with that discussion there was what felt like a shouting match. It was the first time I felt an in-person event was worse than an email discussion. Hopefully people will calm down and restart this discussion on the sourceware overseers list.

Sourceware as Conservancy member project


Last month the Sourceware overseers started a discussion with the projects hosted on Sourceware and the Software Freedom Conservancy (SFC) to become a Conservancy member project (which means Conservancy would become the fiscal sponsor of Sourceware). After many positive responses the SFC’s Evaluations Committee has voted to accept Sourceware.

Read the rest of this entry »

Happy birthday, Valgrind!

It has been twenty years today since Valgrind 1.0 was released.

Make sure to read Nicholas Nethercote’s Twenty years of Valgrind to learn about the early days, Valgrind “skins”, the influence Valgrind had on raising the bar when it comes to correctness for C and C++ programs, and why a hacker on the Rust programming language still uses Valgrind.

Sourceware – GNU Toolchain Infrastructure roadmap

Making email/git based workflow more fun, secure and productive by automating contribution tracking and testing across different distros and architectures.

What is Sourceware?

Sourceware,, is community run infrastructure, mailinglists, git, bug trackers, wikis, etc. hosted in the Red Hat Open Source Community Infrastructure Community Cage together with servers from e.g. Ceph, CentOS, Fedora and Gnome.

Sourceware is mainly known for hosting the GNU Toolchain projects, like gcc at, glibc, binutils and gdb. But also hosts projects like annobin, bunsen, bzip2, cgen, cygwin at, debugedit, dwz, elfutils at, gccrs, gnu-abi, insight, kawa, libffi, libabigail, mauve, newlib, systemtap and valgrind at

A longer list of Sourceware projects, those without their own domain name, including several dormant projects, can be found here:

Most of these projects use a email/git based workflow using mailinglists for discussing patches in preference to web based “forges”.

Zero maintenance automation

Although email based git workflows are great for real patch discussions, they do not always make tracking the state of patches easy.

Just like our other services, such as bugzilla, mailinglists and git repos we like to provide zero maintenance infrastructure for tracking and automation of patches and testing.

So we are trying to consolidate around a shared buildbot for (test) automation and patchwork for tracking the state of contributions. By sharing experiences between the Sourceware projects and coordination and fully automating the infrastructure services.

A shared buildbot

We have a shared buildbot for Sourceware projects at This includes compute resources (buildbot-workers) for various architectures thanks to some generous sponsors. We have native/VM workers for x86_64, ppc64le, s390x, ppc64, i386, arm64 and armhf for debian, fedora and centos (although not all combinations yet) and x86_64 container builders for fedora, debian and opensuse.

There are currently 95 builders on 15 workers, doing ~300 builds a day (more on week days, less on weekends). There are a couple of full testsuite builders (for gcc and binutils-gdb), but most builders are “quick” CI builders, which will sent email whenever a regression is detected. It seems to catch and report a couple of issues a week across all projects.

Builder is its own project on Sourceware which comes with its own git repo, mailinglist and amazing community, that can help you integrate new builders, add workers, containers and get you access to systems to replicate any failures where the buildbot logs don’t give enough information.

And buildbot itself is automated, so whenever a change is made to add a new builder, or define a new container, the buildbot automatically reconfigures itself and the workers will start using the new container images starting with the next build.

The same mechanism can also be used to run tasks on specific commits or periodically. Tasks which are now often done by a cron job or git hook. For example to update documentation, websites, generate release tars or update bugzilla. The advantage over cron jobs is that it can be done more immediately and/or only when really needed based on specific commit files. The advantage over git hooks is that they run in the builder context, not in the context of the specific user that pushed a commit.

Picking your (CI) tests

Although the buildbot itself is zero maintenance, getting and acting on the results of course is not. We already divide the tests into quick CI tests and full test runs. And most tests upload all results to bunsendb. bunsen can help pick good CI tests by indicating which tests are flaky or compare results across different setups.

A prototype testsuite log comparison bunsenweb widget is running at

Lots of things will be coming here, including taking advantage of testrun cluster analysis that’s already being done, a per-testrun testcase search/browse engine, other search operators, testsuite summary (vs detail) grids, who knows, ideas welcome!

What about pre-commit checks?

The builder CI checks what has been committed on the main branch of the projects. This makes sure that what is checked out is in a good state and that any pushed regressions are found early and often.

There is also support for git user try branches. When a user pushes to their try branch the same builder CI checks are ran, so a project developer knows their proposed patch(es) won’t break the build or introduce regressions.

The binutils and gdb communities are currently trying this out. Once new builder resources from OSUOSL are installed we’ll roll this out to other Sourceware projects.

What about non-committers?

The above only helps developers that have commit access on sourceware, but not others who sent in patches. For that we have plus the CICD trybot that DJ wrote The glibc community is already using this. We would like to connect patchwork, buildbot and the trybot for other Sourceware projects

The current trybot doesn’t do authentication, this might not be OK for all builders. So we want to either require checking for known GPG keys on the patch emails or let a trusted developer set a flag in patchwork before the trybot triggers. Once we have public-inbox setup we could also use b4 for DKIM attestation for known/trusted hackers.

Some projects have already experimented with public-inbox. But we don’t have an instance running on sourceware itself yet. This would resolve complaints of not very usable mailman archives.

But I really like to have a webforge!

You are in luck. We already have a sourcehut mirror at This allows anybody to fork any sourceware project on sourcehut, prepare their patches and submit a merge request through email (without having to locally setup git send-email or smtp – the patch emails are generated server side).

Sourcehut is designed around email based workflows, fully Free Software, doesn’t use javascript and is much faster and resource constrained compared to (proprietary) alternatives.

The sourcehut mirror is currently read-only (but syncs automatically with any git updates made on sourceware). When sourcehut supports project groups (one of the beta goals) we will test a self-hosted instance to see whether this is a good way to attract more contributors without loosing the advantages of the email based workflow. The various components are very modular so we can only use those parts we need.

Valgrind 3.18.1

We are pleased to announce a new release of Valgrind, version 3.18.1, available from

3.18.1 fixes a number of bugs and adds support for glibc-2.34, and for new platforms x86/FreeBSD and amd64/FreeBSD.  Debuginfo reading is faster, and Rust demangling has been improved. For PPC64, ISA 3.1 support has been completed, and some newer ARM64 and S390 instructions are also supported.  See the release notes for details of changes.  Note, 3.18.0 had no formal release — it was pulled at the last minute due to a packaging problem.

Our thanks to all those who contribute to Valgrind’s development.  This
release represents a great deal of time, energy and effort on the part of many people.

Happy and productive debugging and profiling,

The Valgrind Developers

Valgrind Memcheck: Different ways to lose your memory

The Red Hat Developer blog posted an article I wrote. Valgrind Memcheck: Different ways to lose your memory.

It explains the different kinds of memory leaks (definitely, reachable, possibly and indirectly). Why which kinds are reported and/or marked as errors by default or not. How to generate suppressions. And how to run things in a way that is most helpful in a regression test suite.

It also includes some C code examples that will make you want to run screaming away to a more memory safe language. But till then, please make sure to run valgrind on your code.

FSF Associate Membership

To the FSF leadership,

Last year I wrote you the following:

My membership name is mark and I am FSF associate member #6. I am currently contributing $10 + $32, for a total of $42 dollars a month.

Besides membership fees of more than $500, I also donated $2000 last year to the FSF because I believe in the Free Software mission and want to support the GNU project.

But I am very disappointed in how the FSF has been handling the recent changes in leadership and the efforts of various GNU volunteers (which includes myself as a GNU maintainer since 2000) to be more welcoming and creating a more flexible governance structure for GNU. The FSF keeps ignoring our calls for a neutral discussion space. And I think it is unacceptable that people are afraid to publicly discuss these issues because they feel intimidated and fear to get personal threats or have to endure racist or sexist language. The FSF really has a responsibility to the GNU volunteers to be able to work and communicate with each other without having to feel harassed all the time

So I am reducing my membership fee to a minimum $10 a month and won’t be making any additional contributions till these issues have been resolved

Since then we setup the GNU Assembly without any resources from the FSF. And created the GNU Social Contract for GNU maintainers and developers to promote the GNU system and create a welcome environment for everybody to be able to create more user freedom. The FSF never officially helped or even replied to our requests to formulate an open and welcoming working relationship with us as GNU volunteers.

And now, instead of taking the opportunity to diversify the FSF leadership with a view towards the future and more inclusiveness, the FSF simply reinstated Richard Stallman as board member without any explanation how this addresses the original issues or the changes needed in the FSF.

This makes the FSF a negative force in promoting Free Software and it negatively impacts the GNU system, the GNU developers and users.

Hopefully the FSF will be able to find new leadership, so that it can again be a positive force for Free Software, support the GNU project as a welcoming community and promote user freedom. But for now I am canceling my FSF Associate Membership.


Mark Wielaard

GNU Social Contract version 1.0

Andreas Enge announced the GNU Social Contract version 1.0:

Hello all,

just a public heads-up on progress on the GNU Social Contract. Following our initially announced timeline, we had put online the first draft at the end of January. The goal of the document is to formulate a common core set of values for the GNU Project, on which we can jointly build to form a stronger community. It is both an agreement among us, GNU contributors, and a pledge to the broader free software community. Additionally, we think it can be a first step towards formalising a transparent and collective governance of the GNU Project.

We received a number of questions and suggestions on the first draft of the document, witnesses to our collective approach to shaping a document that can help us go forward together. We discussed all the input with great care; it is documented, together with the adopted resolutions, at:

The result of all this is version 1.0 of the GNU Social Contract, see

We believe that the outcome is an even snappier document, which lays out our common foundations even more clearly, and thank everyone of you who contributed to improving it.

We have invited all GNU maintainers to send a message until February 24, the end of the endorsement period, to endorse this version 1.0 of the GNU Social Contract, or to declare they do not wish to adhere to it. The current status is maintained at:

Happy “I Love Free Software” day, and thank you for supporting GNU!


A mission statement and social contract for GNU

2019 was a difficult year for the Free Software Community with lots of questions about the future of GNU. It is hard to come up with good answers unless you know which shared principles you all value. After a very long discussion we finally have a first GNU Social Contract DRAFT and a new public wiki for GNU maintainers to share public discussion documents like this.

Update: Carlos wrote a nice introduction for the glibc community.

Proposals for the new GNU/FSF relationship


As volunteers for the GNU Project we are happy that the FSF provides GNU with services like fiscal sponsorship, technical infrastructure, promotion, copyright assignment, and volunteer management. And we note that the FSF is looking for feedback on this relationship going forward:

FSF and GNU:

To that end we have held discussions with other GNU maintainers, developers and other contributors, drafting a GNU mission statement and social contract, identifying stakeholders, delegation models and consensus based decision making. We would like to share some of the things we believe should happen to improve the shared understanding of the relationship for the future of the Free Software Foundation and the GNU Project.


We believe GNU leadership includes the GNU maintainers who should have this discussion together with the FSF. That way, the FSF can support the GNU Project as a whole.

More generally, we think it is time for the GNU Project to collectively define its governance structure, in a way that includes all stakeholders, and that the FSF should facilitate this process.

responsibility and delegation

We recognize that as the fiscal sponsor of the GNU project, the FSF is ultimately responsible for how GNU uses FSF-supported resources: the web site, mailing lists, earmarked financial accounts, sysadmin support, and so on. However, as the FSF is an organization that is more focused on advocacy than the day-to-day details of software production, we expect that the FSF delegates its responsibility for how GNU resources are allocated and used to the GNU project. The FSF should be supportive of requests from GNU maintainers for FSF-supported resources.

web site

The web site at is currently run by a team of volunteers, the GNU webmasters, according to the procedures described at

We would like to increase transparency on how the web site is run and changed, notably by using a publicly visible tool rather than the currently-used private ticket system.

We understand that traditionally the FSF has used the domain not just for the GNU project, but also to facilitate other programs of the FSF. We would like to see a better separation between pages maintained by GNU volunteers and FSF staff maintained pages for other FSF programs.

Specifically we like to give pages maintained by the FSF Licensing and Compliance Lab like the Free Software Definition, License List, Free System Distribution Guidelines and other FSF Compliance programs on which the larger Free Software community relies a special status or redirect them to the domain.

We would also like to discuss which (historical) FSF/GNU Philosophy pages could be better maintained by the FSF License Education program or the FSF Education and Outreach program.

domain names

The domain and its sub-domains are administrated by FSF employees. Upon request, they can delegate sub-domains, such as,,, etc.

We think the procedure for GNU maintainers to make use of sub-domains should be documented.

allocation of resources

The FSF provides hardware and sysadmin work time to support the GNU Project. We would like to have a clear procedure to request such support—e.g., a procedure by which developers of a GNU package could ask for a virtual machine (VM).

collecting donations on behalf of GNU

The FSF acts as fiscal sponsor of several GNU packages as part of its Working Together for Free Software Fund. We think this action is very welcome and should continue and we would appreciate a clear, documented and transparent procedure for GNU packages to join the Working Together for Free Software Fund.


We hope that the FSF, as the holder of the GNU trademark, will continue to use this trademark in a responsible manner in support of the GNU Project and GNU packages. We would like to see a public trademark policy guideline for GNU.


We think the FSF and the GNU Project should take advantage of recent changes in the FSF leadership to clarify their relation. Furthermore, we would like that relation to be as transparent as possible: procedures should be publicly documented, requests and replies should be logged and visible at least to members of the GNU Project, ownership of the various GNU resources should also be publicly documented.


Ludovic Courtès, Andy Wingo, Carlos O’Donell, Andreas Enge and Mark Wielaard