public inbox for ksummit@lists.linux.dev
 help / color / mirror / Atom feed
From: James Bottomley <James.Bottomley@HansenPartnership.com>
To: Thorsten Leemhuis <linux@leemhuis.info>,
	"ksummit@lists.linux.dev" <ksummit@lists.linux.dev>
Subject: Re: [MAINTAINERS SUMMIT] [4/4] Discuss how to better prevent backports of commits that turn out to cause regressions
Date: Thu, 13 Jun 2024 07:58:58 -0400	[thread overview]
Message-ID: <7794a2b09ae4fa73ac35fdaec4858145a665efea.camel@HansenPartnership.com> (raw)
In-Reply-To: <e7f9ae0f-7635-4bf7-827b-bad2d58bf228@leemhuis.info>

On Thu, 2024-06-13 at 10:42 +0200, Thorsten Leemhuis wrote:
> The scenario shown at the start of the thread illustrates a problem I
> see frequently: commits with a Fixes: tag end up in new to stable
> series releases just days after being mainlined and cause regressions
> -- just like they do in mainline, which just was not known yet at the
> time of backporting. This happens extremely often right after merge
> windows when huge piles of changes are backported to the stable trees
> each cycle shortly after -rc1 is out (which even some kernel
> developers apparently are somewhat afraid to test from what I've
> seen).

I haven't really observed this for curated fixes.  For most subsystems,
patches with Fixes tags that are cc'd to stable tend to go steadily
outside the merge window.  Obviously a few arrive within it, but
usually at roughly the rate they arrive outside it.

What I observe in the merge window is huge piles of patches go into
stable *without* a cc:stable tag from the autosel machinery (and quite
a few even without fixes: tags).

So this does beg a couple of questions:

Since you have the figures, what's the proportion of regressions caused
by backports to stable without cc:stable tags?

Could we fix a lot of this by delaying autosel?  It tends to ramp up in
the merge window when everyone is concentrating on other things, so any
regressions it causes naturally get ignored for a couple of weeks.

James


  parent reply	other threads:[~2024-06-13 11:59 UTC|newest]

Thread overview: 107+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-06-13  8:22 [MAINTAINERS SUMMIT] [0/4] Common scenario for four proposals regarding regressions Thorsten Leemhuis
2024-06-13  8:26 ` [MAINTAINERS SUMMIT] [1/4] Create written down guidelines for handling regressions Thorsten Leemhuis
2024-09-12 13:33   ` Thorsten Leemhuis
2024-06-13  8:32 ` [MAINTAINERS SUMMIT] [2/4] Ensure recent mainline regression are fixed in latest stable series Thorsten Leemhuis
2024-06-13 11:02   ` Johannes Berg
2024-06-13 11:21     ` Greg KH
2024-06-13 13:18       ` Sasha Levin
2024-06-13 11:17   ` Jiri Kosina
2024-06-13 11:28   ` Laurent Pinchart
2024-06-14  0:50     ` Steven Rostedt
2024-06-14 14:01   ` Mark Brown
2024-06-14 14:32     ` Rafael J. Wysocki
2024-06-13  8:34 ` [MAINTAINERS SUMMIT] [3/4] Elevate handling of regressions that made it to releases deemed for end users Thorsten Leemhuis
2024-06-13 11:34   ` Laurent Pinchart
2024-06-13 11:39     ` Jiri Kosina
2024-06-14 14:10       ` Mark Brown
2024-06-18 12:58         ` Thorsten Leemhuis
2024-06-19 20:25           ` Laurent Pinchart
2024-06-20 10:47             ` Thorsten Leemhuis
2024-06-13 15:56     ` Liam R. Howlett
2024-06-18 12:24     ` Thorsten Leemhuis
2024-06-20 13:20       ` Jani Nikula
2024-06-20 13:35         ` Thorsten Leemhuis
2024-06-20 14:16           ` Mark Brown
2024-06-21  6:47           ` Jiri Kosina
2024-06-21 10:19             ` Thorsten Leemhuis
2024-06-13  8:42 ` [MAINTAINERS SUMMIT] [4/4] Discuss how to better prevent backports of commits that turn out to cause regressions Thorsten Leemhuis
2024-06-13  9:59   ` Jan Kara
2024-06-13 10:18     ` Thorsten Leemhuis
2024-06-13 14:08     ` Konstantin Ryabitsev
2024-06-14  9:19       ` Lee Jones
2024-06-14  9:24         ` Lee Jones
2024-06-14 12:27         ` Konstantin Ryabitsev
2024-06-14 14:26           ` Konstantin Ryabitsev
2024-06-14 14:36             ` Lee Jones
2024-06-14 14:29       ` Michael Ellerman
2024-06-14 14:38         ` Konstantin Ryabitsev
2024-06-14 14:44           ` Rafael J. Wysocki
2024-06-14 15:08           ` Geert Uytterhoeven
2024-06-15 11:29             ` Michael Ellerman
2024-06-17 10:15             ` Jani Nikula
2024-06-17 12:42               ` Geert Uytterhoeven
2024-06-14 15:45           ` Mark Brown
2024-06-14 14:43         ` Mark Brown
2024-06-14 14:51           ` Konstantin Ryabitsev
2024-06-14 15:42             ` Mark Brown
2024-06-14 14:43         ` Steven Rostedt
2024-06-14 14:57           ` Laurent Pinchart
2024-06-16  1:13         ` Linus Torvalds
2024-06-16  3:28           ` Steven Rostedt
2024-06-16  4:59             ` Linus Torvalds
2024-06-16  8:22               ` Paolo Bonzini
2024-06-16  9:05               ` Geert Uytterhoeven
2024-06-16 15:07               ` Steven Rostedt
2024-06-17 13:48               ` Dan Carpenter
2024-06-17 15:23                 ` Dan Carpenter
2024-06-17 14:39               ` Konstantin Ryabitsev
2024-06-17 16:04                 ` Paul E. McKenney
2024-06-17 16:06                   ` Konstantin Ryabitsev
2024-06-17 16:14                     ` Paolo Bonzini
2024-06-17 16:18                       ` Konstantin Ryabitsev
2024-06-17 17:11                         ` Geert Uytterhoeven
2024-06-18 12:05                 ` Michael Ellerman
2024-06-16  7:26           ` Takashi Iwai
2024-06-16  8:10           ` Paolo Bonzini
2024-06-16 11:31             ` Laurent Pinchart
2024-06-16 11:39             ` Takashi Iwai
2024-06-16 16:40             ` Linus Torvalds
2024-06-16  8:31           ` Jiri Kosina
2024-06-16  8:54             ` Geert Uytterhoeven
2024-06-13 19:39     ` Dan Carpenter
2024-06-14  1:00       ` Steven Rostedt
2024-06-13 11:58   ` James Bottomley [this message]
2024-06-13 13:06     ` Sasha Levin
2024-06-13 13:56       ` James Bottomley
2024-06-13 14:02         ` Greg KH
2024-06-13 15:11           ` James Bottomley
2024-06-13 16:27             ` Greg KH
2024-06-14 18:47             ` Sasha Levin
2024-06-17 10:59               ` Vlastimil Babka
2024-06-13 18:08         ` Sasha Levin
2024-06-13 13:45     ` Greg KH
2024-06-13 13:40   ` Sasha Levin
2024-06-18 13:12     ` Thorsten Leemhuis
2024-06-13 14:28   ` Andrew Lunn
2024-06-13 18:14     ` Sasha Levin
2024-06-14 14:41       ` Jan Kara
2024-06-14 15:03         ` Rafael J. Wysocki
2024-06-14 17:46           ` Sasha Levin
2024-06-18 14:43 ` [MAINTAINERS SUMMIT] [0/4] Common scenario for four proposals regarding regressions James Bottomley
2024-06-18 15:50   ` Mark Brown
2024-06-20 10:32   ` Thorsten Leemhuis
2024-06-20 12:57     ` James Bottomley
2024-06-20 13:55       ` Mark Brown
2024-06-20 14:01         ` James Bottomley
2024-06-20 14:42           ` Mark Brown
2024-06-20 16:02             ` James Bottomley
2024-06-20 17:15               ` Mark Brown
2024-06-20 23:25               ` Sasha Levin
2024-06-21  6:33                 ` Thorsten Leemhuis
     [not found]               ` <20240625175131.672d14a4@rorschach.local.home>
2024-06-26  7:36                 ` Greg KH
2024-06-26 18:32                   ` Steven Rostedt
2024-06-26 19:05                     ` James Bottomley
2024-07-25 10:14                   ` Thorsten Leemhuis
2024-07-25 13:14                     ` Greg KH
2024-06-20 16:59       ` Thorsten Leemhuis
2024-06-20 23:18         ` Sasha Levin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=7794a2b09ae4fa73ac35fdaec4858145a665efea.camel@HansenPartnership.com \
    --to=james.bottomley@hansenpartnership.com \
    --cc=ksummit@lists.linux.dev \
    --cc=linux@leemhuis.info \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox