Re: [MAINTAINERS SUMMIT] [4/4] Discuss how to better prevent backports of commits that turn out to cause regressions

James Bottomley
2024-06-13T13:57:02Z

On Thu, 2024-06-13 at 09:06 -0400, Sasha Levin wrote:
> On Thu, Jun 13, 2024 at 07:58:58AM -0400, James Bottomley wrote:
> > On Thu, 2024-06-13 at 10:42 +0200, Thorsten Leemhuis wrote:
> > > The scenario shown at the start of the thread illustrates a
> > > problem I see frequently: commits with a Fixes: tag end up in new
> > > to stable series releases just days after being mainlined and
> > > cause regressions -- just like they do in mainline, which just
> > > was not known yet at the time of backporting. This happens
> > > extremely often right after merge windows when huge piles of
> > > changes are backported to the stable trees each cycle shortly
> > > after -rc1 is out (which even some kernel developers apparently
> > > are somewhat afraid to test from what I've
> > > seen).
> >
> > I haven't really observed this for curated fixes.  For most
> > subsystems, patches with Fixes tags that are cc'd to stable tend to
> > go steadily outside the merge window.  Obviously a few arrive
> > within it, but usually at roughly the rate they arrive outside it.
> >
> > What I observe in the merge window is huge piles of patches go into
> > stable *without* a cc:stable tag from the autosel machinery (and
> > quite a few even without fixes: tags).
>
> Could you provide a concrete example? This shouldn't happen.

This one has no fixes or cc stable:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37f1663c91934f664fb850306708094a324c227c

Yet here it is backported:

Message-id: 20240603121056.1837607-1-sashal@kernel.org

(I can't give a lore reference because conveniently it was a personal
cc with no tracked mailing list).

I picked that one because we discovered a bug with the strlcpy to
strscpy conversions in SCSI which it looks like this backport has.

> > So this does beg a couple of questions:
> >
> > Since you have the figures, what's the proportion of regressions
> > caused by backports to stable without cc:stable tags?
>
> This question came up two years ago and we had statistics around
> this. Autosel patches didn't cause more (actually, it was *less*)
> regressions than stable tagged ones.

OK, so Thorsten's stats should bear this out then ...

> > Could we fix a lot of this by delaying autosel?  It tends to ramp
> > up in the merge window when everyone is concentrating on other
> > things, so any regressions it causes naturally get ignored for a
> > couple of weeks.
>
> autosel is currently delayed about 3-4 weeks, sometimes more.

That's fairly recent then. When I look at 6.8 autosel began its flood
pretty much the moment the first SCSI pull request went in to the merge
window. Checking 6.9 it seems to be about 10 days after ... has that
made a difference, or is it too early to tell?

James