status update and project summary

The long story short

  • Linus recently mentioned regzbot positively again

  • regression tracking once again helped to get a last-minute fix into 5.17 that otherwise would have missed it

  • regzbot now understands a command to put some slow-moving entries on "back burner" to get them out of the spotlight

  • my texts on regressions and regzbot for the Linux kernel's documentation are on track to get merged into Linux 5.18

  • I gave a talk that mentioned regzbot and my regression tracking work and will soon give another that will focus around this topic

  • I recently reworked some code in preparation for better integration of external bug trackers like bugzilla, but I'm not finished yet

  • sponsorship for this project is ending, but the project, regzbot, and Thorstens regression tracking work will live on – but how exactly is still unclear

  • I reached all the major goals I wanted to realize with the funding; but as always there is a lot more that could be done

The details

Working well

Regzbot is still far from perfect and could really need a few more features. Nevertheless I'm quite happy with it, as it does the basic job just as it's supposed to. :-D

Seems the Linux kernel's lead developer, Linus Torvalds, regularly looks into the reports I compile using regzbot, as can be seen from messages like this and this; to quote from the latter::

So if you are a subsystem maintainer, and you have one of those regressions on your list, please go make them a priority. And if you don't know what I'm talking about, please do look up the reports by regzbot and Guenter Roeck. I added links below to make it really easy.

For context: Guenter Roeck does build tests and keeps track of any regressions he find himself for now.

Regzbot also helped to yet again spot a regression fix that would have missed the Linux 5.17 if I hadn't intervened. It's a small fix and kinda unimportant when compared to the last minute fix in 5.16 enabled by regzbot, but nevertheless good to see.

There are a lot of other situations where regzbot and my regression tracking work I do with regzbot helped to get regressions fixed that otherwise likely would have been forgotten – or wouldn't have been noticed by any developer at all, which happens quite often forreports submitted via Hence a quick reminder: most of the time it's a bad idea to report issues there, as explained in "reporting issues".

It also seems at least in part due to my regression tracking work one subsystem got better at fixing regressions in a timely manner, as it's expected for regressions in the Linux kernel.

Documentation on regressions for users and developers

In my last status report I mentioned a text I was working on for the Linux kernel's documentation covering regressions and usage of regzbot. I had to do quite a lot of changes and split the text into two documents (one for users, one for kernel developers) to get it approved; but that is normal for the kernel and really improved the text a lot!. In the end the texts were accepted by the docs maintainer a while ago and now on track to get merged for the kernel release (Linux 5.18) 1, 2 – that's normal, too, as big changes like this only get merged in the beginning on the development phase for a new version. The only thing that can go wrong now: Linus objects when the docs maintainers ask him to pull the changes, but that rarely happens – and Linus was CCed on all reversions of the patches, so he likely has seen them already and no objections, as he likely would have spoken up otherwise. But such a silence is normal in situations like this, as he has lots on his plate.

Back burner

Regressions normally should be fixed quite quickly, but sometimes there are good reason why a slower approach needs to be taken. That's why it's now possible to mark them as "on back-burner" in regzbot, which make regzbot show such reports in a separate section. That reduces the noise and ensures regressions that need to be dealt with urgently get more of the spotlight. This was something I had on my mind already, as can be seen in my last status report, where I mentioned it as "something I sooner or later will work on"; when Linus said something like that would be nice to have I sad down and implemented it.

Talks about regressions and regzbot

I also gave a talk about reporting Linux kernel bugs on the Chemnitzer Linux-Tage 2022, a German based conference dedicated to Free/Libre & Open-Source Software that was hold virtually this year mid-March. That talk had a section on regressions and briefly mentioned my work on regression tracking and regzbot.

Looks like it'll also give a talk about regression tracking which will mention a regzbot a bit more on this year's Kernel Recipes (the program was not published yet, but I submitted a talk about this and the speaker page already lists me).

Polishing and new features

The regzbot code once again got some polishing in the past few weeks to shake out a few bugs. I also reworked a few things in preparation for supporting bug trackers like properly, as some people report problems there, despite the kernels docs telling everyone that it's a bad idea in most cases. Implementing that takes more time than expected, as doing regression tracking consumes quite a bit of my time already. Another reason: I chose to go a more complicated route to realize this feature, as that will allow a few other things as well that would be nice to have in regzbot. Not sure yet when I’ll find time to continue working on this, I had to take a break from it as other things became more important (in fact I likely shouldn't have started on this at all for now, but that is water under the bridge now).

project summary

I started working on regzbot in April 2021 (see 1, 2) with funding provided by NGI pointer. From the start off it was clear that the funding is limited to one year without a chance for renewal. It's thus running out now and the project thus kinda comes to an end. So time for a summary.

Did I achieve everything I wanted to achieve? Not everything, but definitely all the important things. Mainly: write regzbot, get it running (it does that since October) and use it to track regressions to get it battle tested in the field. The basic idea worked as expected. The amount of things I didn't foresee and required course adjustments was in fact smaller than expected.

A lot of kernel developers seem to like the general concept of regzbot, OTOH I get a feeling that some don't like it when their regressions are tracked, but that's kinda.

But in the end what matters are two things: Linus seems kinda happy that someone is tracking regressions again; the tracking enabled by regzbot definitely helped to get regressions fixed that otherwise would have fallen through the cracks or would only be fixed much slower.

Getting the docs ready took a bit longer than expected. I guess they'll help to get more developers to use regzbot directly, but that remains to be seen. I had also hoped to write an article about regzbot and regression tracking in general (okay, maybe two articles) for, but didn't find time for this.

And as always: over the course of the last few months a few "this would be nice to have in regzbot" things evolved. I mentioned a few of them in my last status report. One of them: making regzbot more useful for subsystem maintainers. But these things have to wait for now.

the end of the project?

The obvious question is: does the project end now that funding is running out? No, it definitely won't at the end of the month, but how the future exactly looks like remains to be seen.

I'm currently trying to find a way to continue doing what I did in the past year: work on regzbot and as the Linux kernel's regression tracker. Remains to be seen if that works out. I also hope that I can extend my work slightly into encouraging users to help with testing and reporting new kernel versions, for example by making testing and reporting easier; I think that's needed because it's a related area where a lot of things are way harder than they have to. The "reporting issues" text I contributed to the kernel in 2020 was one of the many things I already did to improve things, but that way only meant as a start and there are only so many hours in a day.

But whatever happens: I still want to finish a few things (like the article) wrt to regzbot and regression tracking, so I plan to continue working on it for a little while in the next few weeks. But obviously as everyone else I need an income to make a living and get something on the table. So at some point I'll be forced to shift my focus to other things, if I don't find anyone to back my efforts. We'll see. But whatever happens: I plan to continue doing regression tracking for the Linux kernel in the foreseeable future, even if I need to take a job in a unrelated area; but in that case I'll likely won't find the same amount of time for it, which likely will have an impact on the quality of my efforts. And you know how it is with plans: sometimes life gets in the way, so I can't promise how long I'll be able to work as the Linux kernel's regression tracker in my spare time.

But that's just hypothetical for now, I hope I really find some way to make a living with the work I've been doing last year.