Sourceware – GNU Toolchain Infrastructure roadmap

Making email/git based workflow more fun, secure and productive by automating contribution tracking and testing across different distros and architectures.

What is Sourceware?

Sourceware,, is community run infrastructure, mailinglists, git, bug trackers, wikis, etc. hosted in the Red Hat Open Source Community Infrastructure Community Cage together with servers from e.g. Ceph, CentOS, Fedora and Gnome.

Sourceware is mainly known for hosting the GNU Toolchain projects, like gcc at, glibc, binutils and gdb. But also hosts projects like annobin, bunsen, bzip2, cgen, cygwin at, debugedit, dwz, elfutils at, gccrs, gnu-abi, insight, kawa, libffi, libabigail, mauve, newlib, systemtap and valgrind at

A longer list of Sourceware projects, those without their own domain name, including several dormant projects, can be found here:

Most of these projects use a email/git based workflow using mailinglists for discussing patches in preference to web based “forges”.

Zero maintenance automation

Although email based git workflows are great for real patch discussions, they do not always make tracking the state of patches easy.

Just like our other services, such as bugzilla, mailinglists and git repos we like to provide zero maintenance infrastructure for tracking and automation of patches and testing.

So we are trying to consolidate around a shared buildbot for (test) automation and patchwork for tracking the state of contributions. By sharing experiences between the Sourceware projects and coordination and fully automating the infrastructure services.

A shared buildbot

We have a shared buildbot for Sourceware projects at This includes compute resources (buildbot-workers) for various architectures thanks to some generous sponsors. We have native/VM workers for x86_64, ppc64le, s390x, ppc64, i386, arm64 and armhf for debian, fedora and centos (although not all combinations yet) and x86_64 container builders for fedora, debian and opensuse.

There are currently 95 builders on 15 workers, doing ~300 builds a day (more on week days, less on weekends). There are a couple of full testsuite builders (for gcc and binutils-gdb), but most builders are “quick” CI builders, which will sent email whenever a regression is detected. It seems to catch and report a couple of issues a week across all projects.

Builder is its own project on Sourceware which comes with its own git repo, mailinglist and amazing community, that can help you integrate new builders, add workers, containers and get you access to systems to replicate any failures where the buildbot logs don’t give enough information.

And buildbot itself is automated, so whenever a change is made to add a new builder, or define a new container, the buildbot automatically reconfigures itself and the workers will start using the new container images starting with the next build.

The same mechanism can also be used to run tasks on specific commits or periodically. Tasks which are now often done by a cron job or git hook. For example to update documentation, websites, generate release tars or update bugzilla. The advantage over cron jobs is that it can be done more immediately and/or only when really needed based on specific commit files. The advantage over git hooks is that they run in the builder context, not in the context of the specific user that pushed a commit.

Picking your (CI) tests

Although the buildbot itself is zero maintenance, getting and acting on the results of course is not. We already divide the tests into quick CI tests and full test runs. And most tests upload all results to bunsendb. bunsen can help pick good CI tests by indicating which tests are flaky or compare results across different setups.

A prototype testsuite log comparison bunsenweb widget is running at

Lots of things will be coming here, including taking advantage of testrun cluster analysis that’s already being done, a per-testrun testcase search/browse engine, other search operators, testsuite summary (vs detail) grids, who knows, ideas welcome!

What about pre-commit checks?

The builder CI checks what has been committed on the main branch of the projects. This makes sure that what is checked out is in a good state and that any pushed regressions are found early and often.

There is also support for git user try branches. When a user pushes to their try branch the same builder CI checks are ran, so a project developer knows their proposed patch(es) won’t break the build or introduce regressions.

The binutils and gdb communities are currently trying this out. Once new builder resources from OSUOSL are installed we’ll roll this out to other Sourceware projects.

What about non-committers?

The above only helps developers that have commit access on sourceware, but not others who sent in patches. For that we have plus the CICD trybot that DJ wrote The glibc community is already using this. We would like to connect patchwork, buildbot and the trybot for other Sourceware projects

The current trybot doesn’t do authentication, this might not be OK for all builders. So we want to either require checking for known GPG keys on the patch emails or let a trusted developer set a flag in patchwork before the trybot triggers. Once we have public-inbox setup we could also use b4 for DKIM attestation for known/trusted hackers.

Some projects have already experimented with public-inbox. But we don’t have an instance running on sourceware itself yet. This would resolve complaints of not very usable mailman archives.

But I really like to have a webforge!

You are in luck. We already have a sourcehut mirror at This allows anybody to fork any sourceware project on sourcehut, prepare their patches and submit a merge request through email (without having to locally setup git send-email or smtp – the patch emails are generated server side).

Sourcehut is designed around email based workflows, fully Free Software, doesn’t use javascript and is much faster and resource constrained compared to (proprietary) alternatives.

The sourcehut mirror is currently read-only (but syncs automatically with any git updates made on sourceware). When sourcehut supports project groups (one of the beta goals) we will test a self-hosted instance to see whether this is a good way to attract more contributors without loosing the advantages of the email based workflow. The various components are very modular so we can only use those parts we need.

Valgrind 3.18.1

We are pleased to announce a new release of Valgrind, version 3.18.1, available from

3.18.1 fixes a number of bugs and adds support for glibc-2.34, and for new platforms x86/FreeBSD and amd64/FreeBSD.  Debuginfo reading is faster, and Rust demangling has been improved. For PPC64, ISA 3.1 support has been completed, and some newer ARM64 and S390 instructions are also supported.  See the release notes for details of changes.  Note, 3.18.0 had no formal release — it was pulled at the last minute due to a packaging problem.

Our thanks to all those who contribute to Valgrind’s development.  This
release represents a great deal of time, energy and effort on the part of many people.

Happy and productive debugging and profiling,

The Valgrind Developers

Valgrind Memcheck: Different ways to lose your memory

The Red Hat Developer blog posted an article I wrote. Valgrind Memcheck: Different ways to lose your memory.

It explains the different kinds of memory leaks (definitely, reachable, possibly and indirectly). Why which kinds are reported and/or marked as errors by default or not. How to generate suppressions. And how to run things in a way that is most helpful in a regression test suite.

It also includes some C code examples that will make you want to run screaming away to a more memory safe language. But till then, please make sure to run valgrind on your code.

FSF Associate Membership

To the FSF leadership,

Last year I wrote you the following:

My membership name is mark and I am FSF associate member #6. I am currently contributing $10 + $32, for a total of $42 dollars a month.

Besides membership fees of more than $500, I also donated $2000 last year to the FSF because I believe in the Free Software mission and want to support the GNU project.

But I am very disappointed in how the FSF has been handling the recent changes in leadership and the efforts of various GNU volunteers (which includes myself as a GNU maintainer since 2000) to be more welcoming and creating a more flexible governance structure for GNU. The FSF keeps ignoring our calls for a neutral discussion space. And I think it is unacceptable that people are afraid to publicly discuss these issues because they feel intimidated and fear to get personal threats or have to endure racist or sexist language. The FSF really has a responsibility to the GNU volunteers to be able to work and communicate with each other without having to feel harassed all the time

So I am reducing my membership fee to a minimum $10 a month and won’t be making any additional contributions till these issues have been resolved

Since then we setup the GNU Assembly without any resources from the FSF. And created the GNU Social Contract for GNU maintainers and developers to promote the GNU system and create a welcome environment for everybody to be able to create more user freedom. The FSF never officially helped or even replied to our requests to formulate an open and welcoming working relationship with us as GNU volunteers.

And now, instead of taking the opportunity to diversify the FSF leadership with a view towards the future and more inclusiveness, the FSF simply reinstated Richard Stallman as board member without any explanation how this addresses the original issues or the changes needed in the FSF.

This makes the FSF a negative force in promoting Free Software and it negatively impacts the GNU system, the GNU developers and users.

Hopefully the FSF will be able to find new leadership, so that it can again be a positive force for Free Software, support the GNU project as a welcoming community and promote user freedom. But for now I am canceling my FSF Associate Membership.


Mark Wielaard

GNU Social Contract version 1.0

Andreas Enge announced the GNU Social Contract version 1.0:

Hello all,

just a public heads-up on progress on the GNU Social Contract. Following our initially announced timeline, we had put online the first draft at the end of January. The goal of the document is to formulate a common core set of values for the GNU Project, on which we can jointly build to form a stronger community. It is both an agreement among us, GNU contributors, and a pledge to the broader free software community. Additionally, we think it can be a first step towards formalising a transparent and collective governance of the GNU Project.

We received a number of questions and suggestions on the first draft of the document, witnesses to our collective approach to shaping a document that can help us go forward together. We discussed all the input with great care; it is documented, together with the adopted resolutions, at:

The result of all this is version 1.0 of the GNU Social Contract, see

We believe that the outcome is an even snappier document, which lays out our common foundations even more clearly, and thank everyone of you who contributed to improving it.

We have invited all GNU maintainers to send a message until February 24, the end of the endorsement period, to endorse this version 1.0 of the GNU Social Contract, or to declare they do not wish to adhere to it. The current status is maintained at:

Happy “I Love Free Software” day, and thank you for supporting GNU!


A mission statement and social contract for GNU

2019 was a difficult year for the Free Software Community with lots of questions about the future of GNU. It is hard to come up with good answers unless you know which shared principles you all value. After a very long discussion we finally have a first GNU Social Contract DRAFT and a new public wiki for GNU maintainers to share public discussion documents like this.

Update: Carlos wrote a nice introduction for the glibc community.

Proposals for the new GNU/FSF relationship


As volunteers for the GNU Project we are happy that the FSF provides GNU with services like fiscal sponsorship, technical infrastructure, promotion, copyright assignment, and volunteer management. And we note that the FSF is looking for feedback on this relationship going forward:

FSF and GNU:

To that end we have held discussions with other GNU maintainers, developers and other contributors, drafting a GNU mission statement and social contract, identifying stakeholders, delegation models and consensus based decision making. We would like to share some of the things we believe should happen to improve the shared understanding of the relationship for the future of the Free Software Foundation and the GNU Project.


We believe GNU leadership includes the GNU maintainers who should have this discussion together with the FSF. That way, the FSF can support the GNU Project as a whole.

More generally, we think it is time for the GNU Project to collectively define its governance structure, in a way that includes all stakeholders, and that the FSF should facilitate this process.

responsibility and delegation

We recognize that as the fiscal sponsor of the GNU project, the FSF is ultimately responsible for how GNU uses FSF-supported resources: the web site, mailing lists, earmarked financial accounts, sysadmin support, and so on. However, as the FSF is an organization that is more focused on advocacy than the day-to-day details of software production, we expect that the FSF delegates its responsibility for how GNU resources are allocated and used to the GNU project. The FSF should be supportive of requests from GNU maintainers for FSF-supported resources.

web site

The web site at is currently run by a team of volunteers, the GNU webmasters, according to the procedures described at

We would like to increase transparency on how the web site is run and changed, notably by using a publicly visible tool rather than the currently-used private ticket system.

We understand that traditionally the FSF has used the domain not just for the GNU project, but also to facilitate other programs of the FSF. We would like to see a better separation between pages maintained by GNU volunteers and FSF staff maintained pages for other FSF programs.

Specifically we like to give pages maintained by the FSF Licensing and Compliance Lab like the Free Software Definition, License List, Free System Distribution Guidelines and other FSF Compliance programs on which the larger Free Software community relies a special status or redirect them to the domain.

We would also like to discuss which (historical) FSF/GNU Philosophy pages could be better maintained by the FSF License Education program or the FSF Education and Outreach program.

domain names

The domain and its sub-domains are administrated by FSF employees. Upon request, they can delegate sub-domains, such as,,, etc.

We think the procedure for GNU maintainers to make use of sub-domains should be documented.

allocation of resources

The FSF provides hardware and sysadmin work time to support the GNU Project. We would like to have a clear procedure to request such support—e.g., a procedure by which developers of a GNU package could ask for a virtual machine (VM).

collecting donations on behalf of GNU

The FSF acts as fiscal sponsor of several GNU packages as part of its Working Together for Free Software Fund. We think this action is very welcome and should continue and we would appreciate a clear, documented and transparent procedure for GNU packages to join the Working Together for Free Software Fund.


We hope that the FSF, as the holder of the GNU trademark, will continue to use this trademark in a responsible manner in support of the GNU Project and GNU packages. We would like to see a public trademark policy guideline for GNU.


We think the FSF and the GNU Project should take advantage of recent changes in the FSF leadership to clarify their relation. Furthermore, we would like that relation to be as transparent as possible: procedures should be publicly documented, requests and replies should be logged and visible at least to members of the GNU Project, ownership of the various GNU resources should also be publicly documented.


Ludovic Courtès, Andy Wingo, Carlos O’Donell, Andreas Enge and Mark Wielaard

Software Freedom Conservancy Interview

As part of the Software Freedom Conservancy Donor Match they did a little interview with me. Please read it and get inspired to Donate or join the Conservancy as a Supporter.

Software Freedom Conservancy Donor Match

I decided to be part of the Software Freedom Conservancy Donor Match this year. Because I believe many more free software communities deserve to have a home for their project at the Conservancy. In their own words:

Software Freedom Conservancy is a not-for-profit organization that helps promote, improve, develop, and defend Free, Libre, and Open Source Software (FLOSS) projects. Conservancy provides a non-profit home and infrastructure for FLOSS projects. This allows FLOSS developers to focus on what they do best — writing and improving FLOSS for the general public — while Conservancy takes care of the projects’ needs that do not relate directly to software development and documentation.

Please support the Software Freedom Conservancy by donating so they will be able to provide a home to many more communities. A donation of 10 US dollars a month will make you an official sponsor. Donations will be matched and so count double. And new Supporters will even have their donations tripled!

Software Freedom Conservancy Member Projects

A public discussion about GNU

New GNU Governance

There is now a public discussion about GNU governance issues as described in this LWN article: Rethinking the governance of the GNU Project. We have had private discussion about GNU governance issues for the last couple of decades between GNU maintainers, but that never resulted in actual change. And recent events made things a bit more urgent. Since the Chief GNUisance is no longer the president of the FSF. The FSF is now asking for feedback on how their relationship with the GNU project should go forward with respect to fiscal sponsorship, technical infrastructure, promotion, copyright assignment, and volunteer management. So we need to answer a lot of questions.

Mentoring and apprenticeship

We started with a description of how various GNU projects handle mentoring and apprenticeship. Once a GNU maintainer is assigned as the FSF steward of a project/package there are lots of documents on coding standards and what it means for a project to be GNU and Free Software. But there is no core guideline and a GNU maintainer has almost complete freedom interpreting whether any guidelines are or aren’t applicable to their project. This results in GNU maintainers reinventing a lot of project maintenance, governance and delegation of tasks. It would be good to document the various (consensus based) development models that are the result.

GNU membership

The mentoring and apprenticeship discussion focused on the GNU maintainers as being the core of the GNU project. But as was pointed out there are also webmasters, translators, infrastructure maintainers (partially paid FSF staff and volunteers), education and conference organizers, etc. All these people are GNU stakeholders. And how we organize governance of the GNU project should also involve them. There are also already some committees to evaluate new GNU packages and give feedback on the GNU coding standards. But given these committees are advisory only and are sometimes ignored or overruled people have been demotivated to join them or don’t see them as legitimate. It isn’t clear who is actually a GNU member, or whether the FSF recognizes just the GNU maintainers or also other GNU volunteers as stakeholders.

FSF Philosophy or GNU Policy

Both the GNU membership and the new GNU governance discussions try to answer the question “What is GNU?“. The easy answer is “GNU is an operating system that is free software, put together by people working together for the freedom of all software users to control their computing“. That still leaves a lot to define. What is in an Operating System, who are these people that do all this work and how do we coordinate all that work?

But looking at it is much more complex than that. As you expect there is a people section and a software section. But then there is a lot of sections that blur the lines between the FSF and GNU. Most of that is simply historical. GNU used to be the only program the FSF ran. And some of these pages now have their own on The FSF now has a long list of programs besides GNU it runs. But things like the Free Software License List, Free Software Definition and Free System Distribution Guidelines are still maintained on It would be good to agree on who defines what.

And looking at the Philosophy of the GNU project page one could ask whether GNU is fundamentally about producing coherent, empowering free software systems, or whether it is fundamentally about developing and propagating an inspiring, liberatory philosophy? Or maybe it is both? And which Philosophy articles actually define Policy for the project and which are just personal opinions or preferences of the authors? How we are going to maintain these pages in the future (or maybe we are just going to mark them as historic?) depends on answers to these questions.


The FSF manages a lot of resources for the GNU project. It holds the trademark, it is entrusted with some of the copyrights, does fundraising and uses the money for technical infrastructure that GNU volunteers can use. Crucially it maintains the infrastructure for,,, and for GNU projects to publish their work and coordinate development. But this infrastructure doesn’t currently scale and several GNU projects have to maintain their own infrastructure. Some projects have their own (earmarked) funds through the FSF Working Together for Free Software program (or sometimes through other foundations like Software in the Public Interest). It would be nice if the FSF could provide a place to have a discussion about the use of FSF resources by all the GNU volunteers ( maybe) to help with these discussions and to make it more clear who can speak for GNU and which volunteers can use which mailinglists for what purposes.

GNU Social Contract

All the above discussions will be easier if we could agree on some guidelines that everybody would follow when acting on behalf of GNU. A mission statement about what it means to be GNU and what the values are that the GNU community respects when working together. Condensed to something that is easy to comprehend and follow by anybody who wishes to associate with GNU. Ludo posted a first (annotated) draft based on the idea of the Debian Social Contract. And after some discussion, Andreas posted a preliminary version of the GNU Social Contract based on four core principles:

  • The GNU Project respects users’ freedoms
  • The GNU Project provides a consistent system
  • The GNU Project collaborates with the broader free software community
  • The GNU Project welcomes contributions from all and everyone

If you are working on and/or participation in a GNU project we would love to hear your feedback on the proposed GNU Social Contract, the relation of the GNU project and the FSF, governance, membership and any of the other topics that we have been discussing. Together we can make sure that the GNU project will keep empowering all users to control their computing.