Welcome to the website about a project that wants to properly establish regression tracking in the Linux kernel development process. It's a project by Thorsten Leemhuis, who will start to work on it in April 2021. This site is mainly a placeholder until then, but for the impatient, here are a few glimpses of what this will be about:

Goal

The project aims to establish a solution to ensure all regressions reported to the Linux kernel developers get acted upon and do not fall through the cracks. Regressions in the scope of the Linux kernel means: something broke or works seriously worse after switching to a newer Linux kernel version that got built from a similar configuration.

Avoiding such problems is pretty important to Linus Torvalds and the world, as it's a cornerstone to make future Linux versions work as good as today's. The project wants to improve this with one goal: make it less risky and thus more attractive for vendors to ship newer and more secure kernel versions regularly as normal update to their Linux-based operating systems.

Approach & Motivation

To ensure all regressions reported by CI systems or humans get resolved, the effort will mainly work on two things:

1. Build a bot that monitors different locations for regressions reports, as they are reported in various places; if it find them, assign some unique ID.
2. Make the bot monitor the changes merged into the mainline Linux kernel to notice if one of makes a particular regression ID as resolved by that change.
3. Make the bot generate a few static webpages that shows the various regressions being tracked and their current status.
4. Create and establish procedures and documentation how to use the bot and its website in the Linux kernel development process.

The tracking bot is not existing yet and will likely be called "pursuebot" "regzbot". It will be tailored to suit exactly into the Linux kernel development process, which is quite different from most other Open Source projects. Special care will be taken to ensure the bot does not create significant overhead for the developers and the development process; that crucial for the success, as the bot likely won't be adapted by the developers otherwise. They are unlikely to be forced, as using tool like the bot is nearly always optional in Linux kernel development (strictly speaking not even git is required to contribute!).

Due to the quite unusual development approach the Linux kernel has most (all?) existing solutions are unsuitable to be used here; it's also why pursuebot won't be yet another bug-tracker (which the world does not need anyway!). It instead will just track regressions reported, which get reported like any other Linux kernel issue to various different places; often that will mean some mailing list, rarely one of a few different regular bug trackers. Due to this and the heavily mail-based the Linux kernel development process the bot will be fed and controlled by mail only.

The bot will provide a web-view into the data, to give release manager insights into open regressions and their current status.

To increase chances developers will use the bot the project will work the design out in close relationship with the maintainers. The same holds true for the fourth area of this effort, which will be worked on in parallel and likely the biggest part of this project: make the kernel developers use the bot. That among others means procedures and documentation will need to be written how to actually use the bot in Linux kernel development.

Together with the existing “no regression” rule enforced in Linux kernel development all this will help to make sure all valid regressions report get handled in the future and never forgotten (which sadly happens these days :-/ ). This will hopefully help to ensure new releases with their improved security techniques work as good as their predecessors. That's important, as the kernel is it the very heart of many devices (Routers, Smartphones, IoT, …) that drive the internet or connected to it. But many of those devices use outdated Linux kernel versions, as vendors and admins fear switching to a later version might break something. In the end these kernels thus have known vulnerabilities and lack state-of the art security techniques, just because people fear regressions. This projects wants to reduce that fear and thus motivate people to switch to newer release. In the end it will thus indirectly help to make the internet more secure.

Additional information

The goal is to establish the solution so well that it can work without constant hand holding by an individual acting as 'regression tracker'. This will increase chances the Linux kernel developers continue to use the solution in case this project ends due to lack of funding. Until now, only one year of funding secure. It's provided by the NGI pointer project, which won't extend the grant or provide an additional, so other funding providers need to be found after that time.

The project does not want to build a new bot from the ground up; instead it is hoped it can adjust an existing solution to the specific needs for the mail based kernel development process. One possible solution that's high on the short-list: Syzbot, as it's controlled via email already and used by many Linux kernel developers already. Sadly it's written in Go, which is a problem, as python is the language preferred by the kernel.org admins.

The idea for a mail-centered, low-overhead tracking bot is roughly what a lot of kernel developers wished for in 2017. Back then Thorsten Leemhuis worked as human Linux kernel regression tracker for a while and discussed the topic in two sessions with core Linux kernel developers. For details see these LWN.net reports from the Kernel summit 2017 and the maintainers summit 2017: Kernel regression tracking, part 1 and Kernel regression tracking, part 2.

More details to follow here after the project started.

This website is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871528.