[MAINTAINERS SUMMIT] [0/4] Common scenario for four proposals regarding regressions

Thorsten Leemhuis
2024-06-13T08:22:34Z

Lo! I prepared four proposals for the maintainers summit regarding
regressions I'll send in reply to this mail. They are somewhat related
and address different aspects of one scenario I see frequently in
different variations; so instead of repeating that scenario in slightly
modified form in each of the proposals, I'm putting it out here once:

---

A change makes it into mainline (case "a") right before a release (say
of 6.7) or (case "b") shortly after (e.g. during a merge window of 6.8).
That change then within a few days might even be backported to the
latest stable releases (case "c") deemed for end users (say 6.6.13 or
[in case "b"] to 6.7.3). Only then it becomes known that the change
causes a regression (e.g. in both mainline and the stable trees).

Once reported, the problem then is quickly debugged and within two or
three days a tested fix ready for review emerges[1]. But then that fix
only makes it to a new mainline -rc after more than two, three, or many
more weeks due to one or a combination of the following factors:

* Review takes a long time, as nobody feels urged to take a closer look
soon[2].

* Maintainers take quite some time to commit the fix to their subsystem
tree[2].

* Maintainers take quite some time to submit another PR to Linus or only
send the fix shortly before[2, 3] or after[4] the next proper mainline
release (e.g. 6.8).

A few days later the stable team then backports the fix (for case "a"
and "c" -- or "b", if the fix was only merged in the following merge
window) and after a stable-rc phase fixes the problem in their trees as
well[5] -- which takes at least three days, usually close to about one
week, and if the timing is bad easily 10 days or longer.

Despite an available fix, users of mainline then in the end were exposed
to the regression for many weeks[6] -- often more than 1 month, but I've
seen 2+ months quite a few times, too (e.g. when the culprit was merged
shortly before the 6.7 release and the fix only in the merge window for
6.9).

For users of stable trees it is often about as long or a little bit
longer depending on how well the mainline merge of the fix aligns with
the release of the next stable-rc[7]; that's because the stable team is
not allowed[8] or usually won't[9] do anything to resolve regressions
that also happen in mainline before a fix is mainlined.

[1] That's obviously not always the case, but surprisingly often, which
is great; thx for that!
[2] Because they are simply not aware that the patch fixes a regression
that bothers users or due to stances like "the next mainline release
is still weeks away".
[3] For example due to stances like "because I did not want to send
Linus a PR with just one fix"; I recently even had a case of "the
-next rules forbids to commit new changes during the merge window"
(which is not true when it comes to fixes) that delayed things.
[4] Disclaimer: for fixes that bear big risks or fixes for regressions
introduced more than a year ago waiting can be the right thing.
[5] Assuming the stable team notices that it's fixing a regression in
their trees.
[6] Which can discourage or hinder testers and CIs from testing and mean
that they miss other bugs affecting the same platform. IOW: even in
a simplified case of this scenario where the fix was not backported
to stable trees this would be a problem for some folks that could
easily be avoided by merging the fix faster.
[7] Until the issue is fixed, users thus have four options: (1) live
with the regression, which might be impossible if it breaks
something crucial like booting; (2) switch back to the
second-to-last stable series, which often will be EOL and thus prone
to vulnerabilities; (3) switch to a older longterm kernel series,
which might be impossible due to missing drivers or because the
culprit was backported there, too; (4) switch to a unstable kernel
(e.g. mainline) once the issue is fixed there, which they might be
afraid to do or if unlucky contains another regression that causes
trouble.
[8] Our rules forbid the stable maintainer to revert a mainline commit
or accept a fix for it before an equivalent revert or fix is
mainlined.
[9] The stable team could temporarily revert a backport of a mainline
commit that is causing a regression in both mainline and stable
which they later could reapply once a fix for it was mainlined --
but it almost never does that. Which I can understand, as that would
complicate things and might be unwise, as the commit might fix some
security issue; this approach also might be the right strategy in
general to ensure mainline is fixed quickly as well[6].

---

Ciao, Thorsten