Sourceware GNU Toolchain Infrastructure (*) and beyond
======================================================
Presenter Notes
---------------
The title is a bit of a lie, let's try again.
HTML created by python landslide.
Press 'h' for help.
For more background info see presentor notes. Press 'p'.
---
New services for sourceware projects
------------------------------------
sourceware.org is one of the original free software project hosting
services. It hosts dozens of projects, including some of the GNU
toolchain components.
It is community operated, Red Hat subsidized infrastructure.
Presenter Notes
---------------
This is about infrastructure that the GNU toolchain projects use on
Sourceware, together with many other projects. And it isn't directly
about the Infrastructure itself, but about how you want to use it and
maybe about which policies projects want to adopt that the
infrastructure makes possible (setting project policies isn't the job
of sourceware overseers, but we love to help making sure the
infrastructure is ready).
---
Goal and vision
---------------
Email is awesome, but how can we combine our discussion based patch
reviews with patch tracking and test automation?
Provide zero maintenance infrastructure for tracking and automation of
patches, testing and analyzing testresults.
And how do we keep improving and innovating as a community run free
software infrastructure project?
Sourceware was started in 1998, what about the next 24 years?
Presenter Notes
---------------
At Red Hat we are trying to push CI testing upstream. We have to push
fixes upstream if we find them in products. So try to test and find
regressions earlier. Younger generation has trouble contributing to
projects that primarily use email. For elfutils we got some new
contributors who struggle with providing patches. gccrs
frontend. project on github, but wants to integrate with gcc. There
are two main things that make them productive, automated testing and
easy patch tracking. Finally while setting up the new services this
year we started discussing a good structure for making sure Sourceware
keeps around for another 24 years. Best to have this discussion while
there are no known problems or issues.
---
Discussions
-----------
Discuss how to use the infrastructure, builder, try/ci/full buildbots,
bunsen testsuite analyzis, patchwork patch tracking, CICD
trybot/buildbot integration, handling patches/email with public-inbox
and the sourcehut mirror and other automation to make you more
productive and happy to contribute.
Discuss the future of the Sourceware infrastructure as a Software
Freedom Conservancy member project (having a fiscal sponsor).
Presenter notes
---------------
So we believe we are providing the infrastructure services for
projects to automate email based patch reviews with patch tracking,
but it is up to the projects to see how much they use the services and
whether they want to set policies on how to use them.
After discussion with the communty the Software Freedom Conservancy
voted to accepted us as member project so we should soon have a fiscal
sponsor.
---
builder.sourceware.org
----------------------
An installation of Buildbot (python), along with a local community,
mailinglist, git repo, container files, self-configuring.
Includes various compute resources:
- virtual machines, bare metal, containers
- distros (fedora, debian, centos, ubuntu, opensuse)
- architectures (i386, x86_64, ppc64le, s390x, ppc64, armhf, arm64)
- Thanks: Brno University, Marist University, Thomas Fitzsimmons, Mark
Wielaard, Frank Eigler, IBM, The Works on Arm initiative and OSUOSL.
- More please!
Presenter Notes
---------------
Show https://builder.sourceware.org/
---
Three kinds of builders
-----------------------
- Try builders - "pre-commit" - reports to author only.
(binutils, elfutils, gdb and libabigail)
- CI builders - "quick regression tests", reports to author and list
on regression.
(bzip2, dwz, debugedit, libabigail, elfutils, glibc, gcc, gccrs,
valgrind, gdb, binutils)
- Full builders - "all (slow) tests, not all commits, not all arches"
(gcc x86_64/arm64/armhf, glibc x86_64/arm64, gdb-binutils x86_64,
gccrust-bootstrap x86_64/arm64)
All results, reported or not, go into bunsen.
10.000+ builds a month.
Presenter Notes
---------------
Try builders normally run all tests as the CI builders. Full builders
are only needed for those projects that have heavy/slow tests.
---
Try Example
-----------
- git checkout -b frob
- hack, hack, hack... OK, looks good to submit
- git commit -a -m "Awesome hack"
- git push origin frob:users/mark/try-frob
- ... wait for the emails to come in or watch buildbot logs ...
- Send in patches and mention what the try bot reported
After your patches have been accepted you can delete the branch again:
- git push origin :users/mark/try-frob
Currently enabled for binutils, elfutils, gdb and libabigail.
Presenter Notes
---------------
My favorite kind of builder!
---
bunsen testrun storage & analysis
---------------------------------
An installation of bunsen (git://sourceware.org/git/bunsen.git)
(python), under rapid development. irc: #bunsen on irc.libera.chat.
Stores all test-related logs + metadata from each testrun into one commit
of a ordinary dedicated git repo (git://sourceware.org/git/bunsendb.git).
Small python analysis scripts parse new log files from git into an
sqlite database. Analysis passes build on each other. Testruns are
clustered by metadata similarities.
Small command line reporting tools browse/search git + sqlite data,
show diffs/regressions. More coming.
Includes a simple web frontend (https://builder.sourceware.org/testruns/).
Easy to run your own! https://sourceware.org/git/?p=bunsen.git;a=blob;f=README
Presenter Notes
---------------
Sourceware testrun logs includes numerous & huge dejagnu, automake,
autoconf log files, and more. Bunsen stores them in git.
Git compresses amazingly: 1TB of raw text from all 45.000 buildbot
runs fits into 10GB. Use your knowledge of ordinary git to replicate,
subset, garbage-collect, authenticate.
Let's flip through some screenshots of the web interface.
---
bunsen web: list testruns
-------------------------
Presenter Notes
---------------
Each testrun uses its git commit hash as identifier.
---
bunsen web: testrun metadata
----------------------------
Presenter Notes
---------------
Here we see the metadata for this particular testrun.
From the "filelist" and other tabs, one can explore what the system
has found out about this testrun: all the files, all the test cases.
The table's right columns let one navigate to related testruns that
share given metadata values, or else precede or follow it. Let's
compare this testrun to one that came from the previous elfutils
source version: "source.gitdescribe", "prev", "delta".
---
bunsen web: compare two testruns
------------------------------------
Presenter Notes
---------------
Here we see the differences in test results between the two builds.
Nothing too worrysome. Notice the autoconf values are also
differenced. We can click on each log file to instantly see the exact
line context of each test case.
The regression mode limits differences to only pass->fail type
transitions, omitting skipped results and other such less serious
results.
---
bunsen web: see details of one test case
----------------------------------------
Presenter Notes
---------------
Here we see the specific logfile region that accounts for the skipped
run-large-elf-file.sh.trs case.
There is much more to see, clicking around. Please excuse the plain
looks, it's old school HTML with simple self-contained server tech.
---
patchwork.sourceware.org
------------------------
An installation of patchwork (python/django).
git pw is awesome
patchwork plus CI/CD - Let's use those buildbot workers too
Presenter Notes
---------------
I use it as a personal TODO for elfutils, not really a team effort.
gdb used it but abandoned
glibc does seem to use it as a group with weekly patch reviews
some gcc hackers use it?
Why does patchwork work for some, but not others?
---
inbox.sourceware.org
--------------------
An installation of public-inbox (perl).
A better mail archive and so much more....
Allows people to "subscribe" to the list through atom, nntp, imap
Easy way to have mailinglist mirrors, including local (git like) mirror
Try out piem (public-inbox emacs mode) or b4 tools.
Presenter Notes
---------------
All based on Message-IDs.
Make email lists like git.
Not a push, but a pull resource.
---
b4 example
----------
.git/config
[b4]
midmask = https://inbox.sourceware.org/dwz/%s
linkmask = https://inbox.sourceware.org.org/dwz/%s
$ b4 am a2bf576d-3598-385d-2139-cae0d4f11074@suse.cz
Looking up https://inbox.sourceware.org/dwz/a2bf576d-3598-385d-2139-cae0d4f11074%40suse.cz
Grabbing thread from inbox.sourceware.org/dwz/a2bf576d-3598-385d-2139-cae0d4f11074%40suse.cz/t.mbox.gz
Analyzing 1 messages in the thread
Checking attestation on all messages, may take a moment...
---
✓ [PATCH] Use grep -E instead of egrep.
---
✓ Signed: DKIM/suse.cz
---
Total patches: 1
---
Link: https://inbox.sourceware.org.org/dwz/a2bf576d-3598-385d-2139-cae0d4f11074@suse.cz
Base: applies clean to current tree
git am ./20220907_mliska_use_grep_e_instead_of_egrep.mbx
---
b4 example (cont)
-----------------
$ b4 ty --auto
Auto-thankanating commits in master
Found 2 of your commits since 1.week
Calculating patch hashes, may take a moment...
Located: [PATCH] Use grep -E instead of egrep.
Located: [PATCH] Fix executable stack warning from linker
---
Generating 2 thank-you letters
Writing: ./mliska_suse_cz_patch_use_grep_e_instead_of_egrep_.thanks
Writing: ./mliska_suse_cz_patch_fix_executable_stack_warning_from_linker.thanks
---
You can now run:
git send-email ./*.thanks
---
Patch attestation
-----------------
Either:
- Awesome way to "close" the secure software supply chain
- Security "theater" that will exclude people and reduce code
reviews to "has the submitter jumped through code signing hoops"
Presenter Notes
---------------
Example showed DKIM verification, but this can also be done for gpg
signed email messages or special patchatt "signed" patch emails.
---
Experiment: sourcehut
---------------------
https://sr.ht/~sourceware/
A more webby git workflow alternative
git send-email without the email
Presenter Notes
---------------
Not many people seem to use it, it might not be known well enough.
Was tried to submit gccrs patches, but it wasn't a success?
If it doesn't work out we can still keep the git mirrors.
---
Sourceware as Conservancy member
--------------------------------
First off: things are fine and stable.
The next 24 years of Sourceware?
Why? In case of future financial or organizational needs.
Reached out to Software Freedom Conservancy a few months ago. Wrote up
an application last month, reached out to "all" sourceware users. SFC
offered project membership to Sourceware last week.
What is a fiscal sponsor?
Does anything change? Hardly.
Independent from any guest projects!
In particular the GNU toolchain projects have the FSF as fiscal
sponsor, nothing changes about that.
Continue to have public discussions on overseers mailing list and
public video chats with Conservancy.
Presenter notes
---------------
There currently isn't really a need, so now is a good time to set
something up. This is just about sourceware providing free software
project hosting for the next few decades. Projects still decide how or
if they want to use the services. We don't want to decide any policy
for any guest project. Talked to FSF to make sure this isn't in
conflict with the FSF providing fiscal sponsorship to the GNU
Toolchain projects or a way to undermine GNU projects leadership. We
believe this is a non-event. But we want to be as transparent as
possible. If this isn't a non-event we are doing it wrong. We want to
know if any of this is controversial. All discussions are open and
public on the overseers mailinglist. Video chats with Conservancy have
been open to all on BBB.