Mark J. Wielaard

Making email/git based workflow more fun, secure and productive by automating contribution tracking and testing across different distros and architectures.

What is Sourceware?

Sourceware, https://sourceware.org/, is community run infrastructure, mailinglists, git, bug trackers, wikis, etc. hosted in the Red Hat Open Source Community Infrastructure Community Cage together with servers from e.g. Ceph, CentOS, Fedora and Gnome.

Sourceware is mainly known for hosting the GNU Toolchain projects, like gcc at https://gcc.gnu.org/, glibc, binutils and gdb. But also hosts projects like annobin, bunsen, bzip2, cgen, cygwin at https://cygwin.org/, debugedit, dwz, elfutils at http://elfutils.org, gccrs, gnu-abi, insight, kawa, libffi, libabigail, mauve, newlib, systemtap and valgrind at https://valgrind.org/.

A longer list of Sourceware projects, those without their own domain name, including several dormant projects, can be found here: https://sourceware.org/mailman/listinfo.

Most of these projects use a email/git based workflow using mailinglists for discussing patches in preference to web based “forges”.

Zero maintenance automation

Although email based git workflows are great for real patch discussions, they do not always make tracking the state of patches easy.

Just like our other services, such as bugzilla, mailinglists and git repos we like to provide zero maintenance infrastructure for tracking and automation of patches and testing.

So we are trying to consolidate around a shared buildbot for (test) automation and patchwork for tracking the state of contributions. By sharing experiences between the Sourceware projects and coordination and fully automating the infrastructure services.

A shared buildbot

We have a shared buildbot for Sourceware projects at https://builder.sourceware.org/. This includes compute resources (buildbot-workers) for various architectures thanks to some generous sponsors. We have native/VM workers for x86_64, ppc64le, s390x, ppc64, i386, arm64 and armhf for debian, fedora and centos (although not all combinations yet) and x86_64 container builders for fedora, debian and opensuse.

There are currently 95 builders on 15 workers, doing ~300 builds a day (more on week days, less on weekends). There are a couple of full testsuite builders (for gcc and binutils-gdb), but most builders are “quick” CI builders, which will sent email whenever a regression is detected. It seems to catch and report a couple of issues a week across all projects.

Builder is its own project on Sourceware which comes with its own git repo, mailinglist and amazing community, that can help you integrate new builders, add workers, containers and get you access to systems to replicate any failures where the buildbot logs don’t give enough information.

And buildbot itself is automated, so whenever a change is made to add a new builder, or define a new container, the buildbot automatically reconfigures itself and the workers will start using the new container images starting with the next build.

The same mechanism can also be used to run tasks on specific commits or periodically. Tasks which are now often done by a cron job or git hook. For example to update documentation, websites, generate release tars or update bugzilla. The advantage over cron jobs is that it can be done more immediately and/or only when really needed based on specific commit files. The advantage over git hooks is that they run in the builder context, not in the context of the specific user that pushed a commit.

Picking your (CI) tests

Although the buildbot itself is zero maintenance, getting and acting on the results of course is not. We already divide the tests into quick CI tests and full test runs. And most tests upload all results to bunsendb. bunsen can help pick good CI tests by indicating which tests are flaky or compare results across different setups.

A prototype testsuite log comparison bunsenweb widget is running at https://builder.sourceware.org/testruns/

Lots of things will be coming here, including taking advantage of testrun cluster analysis that’s already being done, a per-testrun testcase search/browse engine, other search operators, testsuite summary (vs detail) grids, who knows, ideas welcome!

What about pre-commit checks?

The builder CI checks what has been committed on the main branch of the projects. This makes sure that what is checked out is in a good state and that any pushed regressions are found early and often.

There is also support for git user try branches. When a user pushes to their try branch the same builder CI checks are ran, so a project developer knows their proposed patch(es) won’t break the build or introduce regressions.

The binutils and gdb communities are currently trying this out. Once new builder resources from OSUOSL are installed we’ll roll this out to other Sourceware projects.

What about non-committers?

The above only helps developers that have commit access on sourceware, but not others who sent in patches. For that we have https://patchwork.sourceware.org/ plus the CICD trybot that DJ wrote https://sourceware.org/glibc/wiki/CICDDesign. The glibc community is already using this. We would like to connect patchwork, buildbot and the trybot for other Sourceware projects

The current trybot doesn’t do authentication, this might not be OK for all builders. So we want to either require checking for known GPG keys on the patch emails or let a trusted developer set a flag in patchwork before the trybot triggers. Once we have public-inbox setup we could also use b4 for DKIM attestation for known/trusted hackers.

Some projects have already experimented with public-inbox. But we don’t have an instance running on sourceware itself yet. This would resolve complaints of not very usable mailman archives.

But I really like to have a webforge!

You are in luck. We already have a sourcehut mirror at https://sr.ht/~sourceware/ This allows anybody to fork any sourceware project on sourcehut, prepare their patches and submit a merge request through email (without having to locally setup git send-email or smtp – the patch emails are generated server side).

Sourcehut is designed around email based workflows, fully Free Software, doesn’t use javascript and is much faster and resource constrained compared to (proprietary) alternatives.

The sourcehut mirror is currently read-only (but syncs automatically with any git updates made on sourceware). When sourcehut supports project groups (one of the beta goals) we will test a self-hosted instance to see whether this is a good way to attract more contributors without loosing the advantages of the email based workflow. The various sr.ht components are very modular so we can only use those parts we need.