[MAINTAINERS SUMMIT] [1/4] Create written down guidelines for handling regressions

Thorsten Leemhuis
2024-06-13T08:26:41Z

Different assumptions about the appropriate handling of regressions
frequently lead to friction and time consuming discussions during my
regression tracking and prodding work. That is frustrating, demotivating
and exhausting for everyone involved and even brought us to situations
like "then I'm stepping down as maintainer". To avoid things like this,
I propose we try to pin down guidelines together and ideally make Linus
bless them.

The "Expectations and best practices for fixing regressions" in
Documentation/process/handling-regressions.rst (
https://docs.kernel.org/process/handling-regressions.html#expectations-and-best-practices-for-fixing-regressions
) could be a start for such guidelines -- but I'm obviously biased here,
as I wrote that text, so feel free to propose something new.

That text is based on generalized interpretations of statements and
actions from Linus while keeping practical application and our workflows
in mind -- including the maintenance of stable trees. I have no idea if
I went too far somewhere: the submission of that text was addressed to
Linus, but he did not react; otoh he merged it later after Greg ACKed it
and it came to his doorstep through the docs tree.

But in the end it seems most people do not know about this text or do
not take it for real. That's why I'd like to make this more official or
create something new that is blessed and widely accepted, despite all
the downsides that "writing things down" sometimes has. That means that
the text likely should be moved somewhere else closer to
Documentation/process/submitting-patches.rst and/or to
Documentation/process/6.Followthrough.rst maybe.

That "Expectations and best practices for fixing regressions" has a
short section that basically says everything crucial in generic way already:

> As a Linux kernel developer, you are expected to give your best to prevent
> situations where a regression caused by a recent change of yours leaves users
> only these options:
>
> * Run a kernel with a regression that impacts usage.
>
> * Switch to an older or newer kernel series.
>
> * Continue running an outdated and thus potentially insecure kernel for more
> than three weeks after the regression's culprit was identified. Ideally it
> should be less than two. And it ought to be just a few days, if the issue is
> severe or affects many users -- either in general or in prevalent
> environments.

This thus should already prevent all variations of the example scenario
the mail at the start of this thread covered. So maybe this is enough
already.

Fun fact: in an earlier version of that text (which was in mainline for
about a year and also ACKed by Greg) it was more like "within two weeks,
ideally one"; but that afaics turned out to be too demanding, especially
for subsystem maintainers, as then they would definitely have to send
PRs more often to reach that target. This and other practical aspects
are also the reason why the text continues with more detailed instructions:

> How to realize that in practice depends on various factors. Use the following
> rules of thumb as a guide.

Let me comment on some of the other points in case we want to use them
as a base for guidelines. Again, many of the underlying problems that
lead to the following points can be seen in the scenario at the start of
the thread.

> In general:
>
> * Prioritize work on regressions over all other Linux kernel work, unless the
> latter concerns a severe issue (e.g. acute security vulnerability, data loss,
> bricked hardware, ...).
>
> * Expedite fixing mainline regressions that recently made it into a proper
> mainline, stable, or longterm release (either directly or via backport).

Note: The "stable, or longterm release (either directly or via
backport)" in this point is just from my interpretation, not sure what
Linus thinks about it. Proposal 3/4 will focus on that, so maybe ignore
this part here.

Another note: That "recently" in the second point becomes more concrete
in later points (see quotes below; yes, this is not really ideal and
maybe should be fixed) and is based on
https://lore.kernel.org/all/CAHk-=wis_qQy4oDNynNKi5b7Qhosmxtoj1jxo5wmB6SRUwQUBQ@mail.gmail.com/

> * Do not consider regressions from the current cycle as something that can wait
> till the end of the cycle, as the issue might discourage or prevent users and
> CI systems from testing mainline now or generally.

Not sure if this and the two points before it are really what Linus
wants, but from his actions it seemed to me it's something like that.

> * Work with the required care to avoid additional or bigger damage, even if
> resolving an issue then might take longer than outlined below.

FWIW, this obviously provides a loophole that could be used in any
situation -- but at the same time I think it's wise to have it here for
cases where reverts are not an option and a proper fix takes time to get
right.

> On timing once the culprit of a regression is known:
>
> * Aim to mainline a fix within two or three days, if the issue is severe or
> bothering many users -- either in general or in prevalent conditions like a
> particular hardware environment, distribution, or stable/longterm series.

Note, ignore...

> * Aim to mainline a fix by Sunday after the next, if the culprit made it
> into a recent mainline, stable, or longterm release (either directly or via
> backport); if the culprit became known early during a week and is simple to
> resolve, try to mainline the fix within the same week.

...this point here, as proposal 3/4 will cover that in more detail.

> * For other regressions, aim to mainline fixes before the hindmost Sunday
> within the next three weeks. One or two Sundays later are acceptable, if the
> regression is something people can live with easily for a while -- like a
> mild performance regression.

FWIW, I chose this "Sunday after the next", "hindmost Sunday within the
next three weeks", … approach so that subsystem maintainers normally
have no extra work if they flush their fixes at the end of the week,
which quite a few do. And when it comes to regression that imho is a
wise approach, as that ensures the fix makes it into the next -rc. It
often also aligns nicely with stable trees, as that way the fix one or
two days later might already be in the next stable-rc, as they are often
released on Mondays or Tuesdays (at least they usually did before Greg
put even more load on his already overburdened shoulders with the CVE
stuff a few months ago...).

> * It's strongly discouraged to delay mainlining regression fixes till the next
> merge window, except when the fix is extraordinarily risky or when the
> culprit was mainlined more than a year ago.

This used to be a big problem and already got somewhat better, but there
is still quite a bit of room for improvement from what I see.

> On procedure:
>
> * Always consider reverting the culprit, as it's often the quickest and least
> dangerous way to fix a regression. Don't worry about mainlining a fixed
> variant later: that should be straight-forward, as most of the code went
> through review once already.

This was meant to encourage reverts, as some people see them as
something bad -- when in reality it in kernel context Linus afaics wants
them to be seen as "this was not ready, no big deal, just revert and
reapply in a few weeks together with a fix". Wondering how if we should
do something to get that better across.

> * Try to resolve any regressions introduced in mainline during the past
> twelve months before the current development cycle ends: Linus wants such
> regressions to be handled like those from the current cycle, unless fixing
> bears unusual risks.
>
> * Consider CCing Linus on discussions or patch review, if a regression seems
> tangly. Do the same in precarious or urgent cases -- especially if the
> subsystem maintainer might be unavailable. Also CC the stable team, when you
> know such a regression made it into a mainline, stable, or longterm release.
>
> * For urgent regressions, consider asking Linus to pick up the fix straight
> from the mailing list: he is totally fine with that for uncontroversial
> fixes. Ideally though such requests should happen in accordance with the
> subsystem maintainers or come directly from them.
>
> * In case you are unsure if a fix is worth the risk applying just days before
> a new mainline release, send Linus a mail with the usual lists and people in
> CC; in it, summarize the situation while asking him to consider picking up
> the fix straight from the list. He then himself can make the call and when
> needed even postpone the release. Such requests again should ideally happen
> in accordance with the subsystem maintainers or come directly from them.
>
> Regarding stable and longterm kernels:
>
> * You are free to leave regressions to the stable team, if they at no point in
> time occurred with mainline or were fixed there already.

Ignore...

> * If a regression made it into a proper mainline release during the past
> twelve months, ensure to tag the fix with "Cc: stable@vger.kernel.org", as a
> "Fixes:" tag alone does not guarantee a backport. Please add the same tag,
> in case you know the culprit was backported to stable or longterm kernels.

...this point here, as proposal 2/4 will cover it.

> * When receiving reports about regressions in recent stable or longterm kernel
> series, please evaluate at least briefly if the issue might happen in current
> mainline as well -- and if that seems likely, take hold of the report. If in
> doubt, ask the reporter to check mainline.
>
> * Whenever you want to swiftly resolve a regression that recently also made it
> into a proper mainline, stable, or longterm release, fix it quickly in
> mainline; when appropriate thus involve Linus to fast-track the fix (see
> above). That's because the stable team normally does neither revert nor fix
> any changes that cause the same problems in mainline.
>
> * In case of urgent regression fixes you might want to ensure prompt
> backporting by dropping the stable team a note once the fix was mainlined;
> this is especially advisable during merge windows and shortly thereafter, as
> the fix otherwise might land at the end of a huge patch queue.
>
> On patch flow:
>
> * Developers, when trying to reach the time periods mentioned above, remember
> to account for the time it takes to get fixes tested, reviewed, and merged by
> Linus, ideally with them being in linux-next at least briefly. Hence, if a
> fix is urgent, make it obvious to ensure others handle it appropriately.
>
> * Reviewers, you are kindly asked to assist developers in reaching the time
> periods mentioned above by reviewing regression fixes in a timely manner.
>
> * Subsystem maintainers,

FWIW, there is one problem related to this and the previous point that I
haven't written a proposal for, but maybe should have: reviewers and
subsystem maintainers have no dead simple and reliable way to detect
"ohh, this is a regression fix I maybe should prioritize".

Some agreed on tag in the subject could help. [REGFIX]? [URGENT]?
[FASTTRACK]? Hmmm, do not really like any of them, except maybe the
last... :-/

A `Cc: regressions@lists.linux.dev` as well, but would be yet another
tag and harder to spot. :-/ Yes, I mentioned the latter idea two years
ago already without success. But some people started doing it since
then, which is nice, as it helps me keep an eye on things or to become
aware of regressions; it would also allow me to easily spot regression
fixes that are queued for the next cycle that instead might better be
merged in the current cycle.

> you likewise are encouraged to expedite the handling
> of regression fixes. Thus evaluate if skipping linux-next is an option for
> the particular fix. Also consider sending git pull requests more often than
> usual when needed. And try to avoid holding onto regression fixes over
> weekends -- especially when the fix is marked for backporting.

If all subsystems would usually abide by this point, many regressions
would be fixed quite a bit faster from what I see, as "fixes are sitting
in subsystem trees for one or more weeks and are sometimes flushed
shortly after a new rc" is the aspect that from my point of view
currently is the one that is causing most delays (note: for a few
subsystems that is not a problem at all, they are good at this!).