Thorsten Leemhuis

Regression tracking: state of the union early 2024

Tue, 09 Jan 2024 11:00:00 +0100

[Posted the text on the regressions list as well: https://lore.kernel.org/all/7613e402-894a-4d38-8cef-7263630c1c57@leemhuis.info/; if you want to comment on it, please reply there.] The long story short I'm not really happy with my performance wrt to my regression tracking efforts during the last year. To counter that, I've already shifted my focus somewhat ~in October. With the new year I will shift it some more. Top-priority will be "make regzbot more useful for kernel subsystem maintainers" from now on.

Why your Linux kernel bug report might be fruitless

Mon, 05 Jun 2023 00:00:00 +0000

[last updated: 2024-01-06] Prevalent reasons why Linux developers do not fix reported issues or ignore bug reports completely These are the most frequent reasons why Linux kernel developers ignore bug reports or only react scantly to them: Your kernel was apparently built by a Linux distributor or hardware vendor. Your kernel was apparently built from modified Linux sources. Your kernel apparently loaded add-on drivers. You submitted the report to the wrong place (bugzilla.

about page rewritten

Sat, 31 Dec 2022 11:54:17 +0100

Just to increase visibility: I've rewritten the 'about' page, which now contains much more details about these efforts -- especially for anyone that is new to them. Enjoy! Thorsten

status update and project summary

Wed, 16 Mar 2022 10:04:17 +0100

The long story short Linus recently mentioned regzbot positively again regression tracking once again helped to get a last-minute fix into 5.17 that otherwise would have missed it regzbot now understands a command to put some slow-moving entries on "back burner" to get them out of the spotlight my texts on regressions and regzbot for the Linux kernel's documentation are on track to get merged into Linux 5.18 I gave a talk that mentioned regzbot and my regression tracking work and will soon give another that will focus around this topic

status update: polishing, problems, further work

Wed, 12 Jan 2022 10:04:17 +0100

A (late) "Happy new year". Time for a status update, which is overdue anyway. The long story short regzbot in general is working as intended and regularly sending weekly reports now the code got a lot of polishing, but more is needed (as always) four additional features would make regzbot a lot more useful, but they have to wait a bit, as development slows down for a while sadly, fewer subsystems and developer than hoped for use regzbot directly (at least for now); I also have to re-educate more developers than expected about placing "Link:" tags, despite them being mandated for years now

status update: field-testing, webview, getting started, and occasional reports

Wed, 27 Oct 2021 07:49:50 +0200

The long story short publishing the code and field-testing regzbot as expected found bugs and deficiencies that got fixed the webview was improved and is online now there is now a "getting started" document for users and Linux kernel developers the third iteration of a patch to improve something in the Linux kernel's documentation relevant to the bot was accepted for 5.16 regzbot is now able to sent occasional (say weekly) reports next week it will start to get more into the spot-light, which hopefully makes some users and developers actually start to use it directly The details Field-testing In late August I finally published the regzbot code I had been working on for months.

Regzbot code published and running now to see how it works

Mon, 30 Aug 2021 09:37:55 +0200

Tick tick, is this thing still on? Sorry for being quiet many weeks, I was busy writing regzbot, the "Linux kernel regression tracking bot". And you know how it is: after "ohh, this is going quicker than expected" this thing called reality catches up. Yeah, that things that presents lots of obstacles and tells you to throw away and reimplement some parts to make things really fly. And yes, there were other distractions as well – like Twitter and this "to earn enough money, I occasionally still need to write an article or two" thing.

regzbot: 'the design' aka 'how is this thing supposed to work'

Mon, 19 Apr 2021 09:03:52 +0200

The bot this project will build got a name: "regzbot". That makes it a good time to explain how it will look like and how it's supposed to establish proper regression tracking in the Linux kernel development process. So here is a simple example that is more concrete than the vague introduction post. Below text was also posted to Linus, Greg and a few mailing list as RFC. The ideal case is simple.

First steps: Linux now has a mailing list for regression reports

Thu, 08 Apr 2021 15:03:52 +0200

I finally started to work on this process a few days ago. There is one highlight I can show to the world already: I requested the creation of a mailing list for regression reports and it was granted, so now regressions@lists.linux.dev exists. I also submitted a patchset to get it mentioned in the documentation and the document about "reporting issues" to the Linux kernel developers; the latter will tell everyone that they should CC that list or forward their reports there if it happened in one of the rare subsystems where reports need to go to some bug tracker.

Hello world from the project to properly establish regression tracking in the Linux kernel development process

Wed, 04 Nov 2020 00:00:00 +0000

Welcome to the website about a project that wants to properly establish regression tracking in the Linux kernel development process. It's a project by Thorsten Leemhuis, who will start to work on it in April 2021. This site is mainly a placeholder until then, but for the impatient, here are a few glimpses of what this will be about:

Goal

The project aims to establish a solution to ensure all regressions reported to the Linux kernel developers get acted upon and do not fall through the cracks. Regressions in the scope of the Linux kernel means: something broke or works seriously worse after switching to a newer Linux kernel version that got built from a similar configuration.

Avoiding such problems is pretty important to Linus Torvalds and the world, as it's a cornerstone to make future Linux versions work as good as today's. The project wants to improve this with one goal: make it less risky and thus more attractive for vendors to ship newer and more secure kernel versions regularly as normal update to their Linux-based operating systems.

Approach & Motivation

To ensure all regressions reported by CI systems or humans get resolved, the effort will mainly work on two things:

Build a bot that monitors different locations for regressions reports, as they are reported in various places; if it find them, assign some unique ID.
Make the bot monitor the changes merged into the mainline Linux kernel to notice if one of makes a particular regression ID as resolved by that change.
Make the bot generate a few static webpages that shows the various regressions being tracked and their current status.
Create and establish procedures and documentation how to use the bot and its website in the Linux kernel development process.

The tracking bot is not existing yet and will likely be called ~~"pursuebot"~~ "regzbot". It will be tailored to suit exactly into the Linux kernel development process, which is quite different from most other Open Source projects. Special care will be taken to ensure the bot does not create significant overhead for the developers and the development process; that crucial for the success, as the bot likely won't be adapted by the developers otherwise. They are unlikely to be forced, as using tool like the bot is nearly always optional in Linux kernel development (strictly speaking not even git is required to contribute!).

Due to the quite unusual development approach the Linux kernel has most (all?) existing solutions are unsuitable to be used here; it's also why pursuebot won't be yet another bug-tracker (which the world does not need anyway!). It instead will just track regressions reported, which get reported like any other Linux kernel issue to various different places; often that will mean some mailing list, rarely one of a few different regular bug trackers. Due to this and the heavily mail-based the Linux kernel development process the bot will be fed and controlled by mail only.

The bot will provide a web-view into the data, to give release manager insights into open regressions and their current status.

To increase chances developers will use the bot the project will work the design out in close relationship with the maintainers. The same holds true for the fourth area of this effort, which will be worked on in parallel and likely the biggest part of this project: make the kernel developers use the bot. That among others means procedures and documentation will need to be written how to actually use the bot in Linux kernel development.

Together with the existing “no regression” rule enforced in Linux kernel development all this will help to make sure all valid regressions report get handled in the future and never forgotten (which sadly happens these days :-/ ). This will hopefully help to ensure new releases with their improved security techniques work as good as their predecessors. That's important, as the kernel is it the very heart of many devices (Routers, Smartphones, IoT, …) that drive the internet or connected to it. But many of those devices use outdated Linux kernel versions, as vendors and admins fear switching to a later version might break something. In the end these kernels thus have known vulnerabilities and lack state-of the art security techniques, just because people fear regressions. This projects wants to reduce that fear and thus motivate people to switch to newer release. In the end it will thus indirectly help to make the internet more secure.

Additional information

The goal is to establish the solution so well that it can work without constant hand holding by an individual acting as 'regression tracker'. This will increase chances the Linux kernel developers continue to use the solution in case this project ends due to lack of funding. Until now, only one year of funding secure. It's provided by the NGI pointer project, which won't extend the grant or provide an additional, so other funding providers need to be found after that time.

The project does not want to build a new bot from the ground up; instead it is hoped it can adjust an existing solution to the specific needs for the mail based kernel development process. One possible solution that's high on the short-list: Syzbot, as it's controlled via email already and used by many Linux kernel developers already. Sadly it's written in Go, which is a problem, as python is the language preferred by the kernel.org admins.

The idea for a mail-centered, low-overhead tracking bot is roughly what a lot of kernel developers wished for in 2017. Back then Thorsten Leemhuis worked as human Linux kernel regression tracker for a while and discussed the topic in two sessions with core Linux kernel developers. For details see these LWN.net reports from the Kernel summit 2017 and the maintainers summit 2017: Kernel regression tracking, part 1 and Kernel regression tracking, part 2.

More details to follow here after the project started.

This website is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871528.