New services for sourceware projects

Sourceware is one of the original free software project hosting services. It hosts dozens of projects, including some GNU projects.

It is community operated, (mostly) Red Hat subsidized infrastructure.

Presenter Notes


Historically (1998), (2000), now (since 2001).

Also known as,,,

Host to ~25 projects, annobin, binutils, builder, bunsen, bzip2, cgen, cygwin, debugedit, dwz, elfutils, insight, gcc, gccrs, gdb, glibc, gnu-gabi, kawa, libabigail, libffi, lvm2, newlib, poke, sid, systemtap, valgrind, ...

Plus another ~40 dormant or moved projects.

Worry-free, friendly, developer controlled, home for Free Software projects.

Presenter Notes


  • mailman based mailing lists
  • plus public-inbox (*)
  • git hosting
  • project websites, also git based
  • bug tracking using bugzilla
  • patch tracking via patchwork (*)
  • integration testing using buildbot (*)
  • bunsen test results database (*)
  • others, just ask!

Trying to provide zero maintenance infrastructure. Prefer packaged software for easy automated (security) updates. Otherwise virtual environments.

Various projects have admin groups to setup their own services.

(*) New or improved - this talk!

Presenter Notes

Goal and vision

Email is awesome, but how can we combine our discussion based patch reviews with patch tracking and test automation?

Provide infrastructure for tracking and automation of patches, testing and analyzing testresults.

And how do we keep improving and innovating as a community run free software infrastructure project?

Sourceware was started in 1998, what about the next 24 years?

Presenter Notes

An installation of Buildbot (python), along with a local community, mailinglist, git repo, container files, self-configuring.

Includes various compute resources:

  • virtual machines, bare metal, containers
  • distros (fedora, debian, centos, ubuntu, opensuse)
  • architectures (i386, x86_64, ppc64le, s390x, ppc64, armhf, arm64)
  • Thanks: Brno University, Marist University, Thomas Fitzsimmons, Mark Wielaard, Frank Eigler, IBM, The Works on Arm initiative and OSUOSL.

Presenter Notes

Three kinds of builders

  • Try builders - "pre-commit" - reports to author only. (binutils, elfutils, gdb and libabigail)
  • CI builders - "quick regression tests", reports to author and list on regression. (bzip2, dwz, debugedit, libabigail, elfutils, glibc, gcc, gccrs, valgrind, gdb, binutils)
  • Full builders - "all (slow) tests, not all commits, not all arches" (gcc x86_64/arm64/armhf, glibc x86_64/arm64, gdb-binutils x86_64, gccrust-bootstrap x86_64/arm64)

All results, reported or not, go into bunsen. 15.000+ builds a month.

Presenter Notes

Try Example

  • git checkout -b frob
  • hack, hack, hack... OK, looks good to submit
  • git commit -a -m "Awesome hack"
  • git push origin frob:users/mark/try-frob
  • ... wait for the emails to come in or watch buildbot logs ...
  • Send in patches and mention what the try bot reported

After your patches have been accepted you can delete the branch again:

  • git push origin :users/mark/try-frob

Currently enabled for binutils, elfutils, gdb and libabigail.

Presenter Notes

bunsen testrun storage & analysis

An installation of bunsen (git:// (python), under rapid development. irc: #bunsen on

Stores all test-related logs + metadata from each testrun into one commit of a ordinary dedicated git repo (git://

Small python analysis scripts parse new log files from git into an sqlite database. Analysis passes build on each other. Testruns are clustered by metadata similarities.

Small command line reporting tools browse/search git + sqlite data, show diffs/regressions. More coming.

Includes a simple web frontend (

Easy to run your own!;a=blob;f=README

Presenter Notes

bunsen web: list testruns

Presenter Notes

bunsen web: testrun metadata

Presenter Notes

bunsen web: compare two testruns

Presenter Notes

bunsen web: see details of one test case

Presenter Notes

An installation of patchwork (python/django).

git pw is awesome

patchwork plus CI/CD - Let's use those buildbot workers too

Presenter Notes

An installation of public-inbox (perl).

A better mail archive and so much more....

Allows people to "subscribe" to the list through atom, nntp, imap

Easy way to have mailinglist mirrors, including local (git like) mirror

Try out piem (public-inbox emacs mode) or b4 tools.

Presenter Notes

b4 example


    midmask =
    linkmask =
 $ b4 am
 Looking up
 Grabbing thread from
 Analyzing 1 messages in the thread
 Checking attestation on all messages, may take a moment...
   ✓ [PATCH] Use grep -E instead of egrep.
   ✓ Signed: DKIM/
Total patches: 1
 Base: applies clean to current tree
       git am ./20220907_mliska_use_grep_e_instead_of_egrep.mbx

Presenter Notes

b4 example (cont)

$ b4 ty --auto
Auto-thankanating commits in master
Found 2 of your commits since 1.week
Calculating patch hashes, may take a moment...
  Located: [PATCH] Use grep -E instead of egrep.
  Located: [PATCH] Fix executable stack warning from linker
Generating 2 thank-you letters
  Writing: ./mliska_suse_cz_patch_use_grep_e_instead_of_egrep_.thanks
  Writing: ./mliska_suse_cz_patch_fix_executable_stack_warning_from_linker.thanks
You can now run:
  git send-email ./*.thanks

Presenter Notes

Patch attestation


  • Awesome way to "close" the secure software supply chain
  • Security "theater" that will exclude people and reduce code reviews to "has the submitter jumped through code signing hoops"

Presenter Notes

Experiment: sourcehut

A more webby git workflow alternative

git send-email without the email

Presenter Notes

Sourceware as Conservancy member

First off: things are fine and stable.

The next 24 years of Sourceware?

Why? In case of future financial or organizational needs.

Reached out to Software Freedom Conservancy a few months ago. Wrote up an application a couple of months ago, reached out to "all" sourceware users. SFC offered project membership to Sourceware.

What is a fiscal sponsor?

Independent from any guest projects!

In particular the GNU toolchain projects have the FSF as fiscal sponsor, nothing changes about that.

Continue to have public discussions on overseers mailing list and public video chats with Conservancy.

Presenter Notes

The FSF tech-team

The various GNU projects hosted on sourceware also get support for the FSF tech-team hosting some websites, releases, etc.

They have a paid staff and can also provide backups, mirrors and have offered technical assistance to sourceware.

Run from which we are learning some tricks.

Presenter Notes

Future plans

bugzilla sourceware infrastructure

Offsite backup/system, BBB instance, Software Heritage, gitolite, cgit, signed release upload, dkim preserving lists...

Do this every 4 months?

Presenter Notes