All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: James Bottomley <James.Bottomley@hansenpartnership.com>
Cc: Thorsten Leemhuis <linux@leemhuis.info>,
	"ksummit@lists.linux.dev" <ksummit@lists.linux.dev>
Subject: Re: [MAINTAINERS SUMMIT] [4/4] Discuss how to better prevent backports of commits that turn out to cause regressions
Date: Thu, 13 Jun 2024 09:06:54 -0400	[thread overview]
Message-ID: <ZmruqWDTG2PK-rbu@sashalap> (raw)
In-Reply-To: <7794a2b09ae4fa73ac35fdaec4858145a665efea.camel@HansenPartnership.com>

On Thu, Jun 13, 2024 at 07:58:58AM -0400, James Bottomley wrote:
>On Thu, 2024-06-13 at 10:42 +0200, Thorsten Leemhuis wrote:
>> The scenario shown at the start of the thread illustrates a problem I
>> see frequently: commits with a Fixes: tag end up in new to stable
>> series releases just days after being mainlined and cause regressions
>> -- just like they do in mainline, which just was not known yet at the
>> time of backporting. This happens extremely often right after merge
>> windows when huge piles of changes are backported to the stable trees
>> each cycle shortly after -rc1 is out (which even some kernel
>> developers apparently are somewhat afraid to test from what I've
>> seen).
>
>I haven't really observed this for curated fixes.  For most subsystems,
>patches with Fixes tags that are cc'd to stable tend to go steadily
>outside the merge window.  Obviously a few arrive within it, but
>usually at roughly the rate they arrive outside it.
>
>What I observe in the merge window is huge piles of patches go into
>stable *without* a cc:stable tag from the autosel machinery (and quite
>a few even without fixes: tags).

Could you provide a concrete example? This shouldn't happen.

>So this does beg a couple of questions:
>
>Since you have the figures, what's the proportion of regressions caused
>by backports to stable without cc:stable tags?

This question came up two years ago and we had statistics around this.
Autosel patches didn't cause more (actually, it was *less*) regressions
than stable tagged ones.

>Could we fix a lot of this by delaying autosel?  It tends to ramp up in
>the merge window when everyone is concentrating on other things, so any
>regressions it causes naturally get ignored for a couple of weeks.

autosel is currently delayed about 3-4 weeks, sometimes more.

-- 
Thanks,
Sasha

  reply	other threads:[~2024-06-13 13:27 UTC|newest]

Thread overview: 107+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-06-13  8:22 [MAINTAINERS SUMMIT] [0/4] Common scenario for four proposals regarding regressions Thorsten Leemhuis
2024-06-13  8:26 ` [MAINTAINERS SUMMIT] [1/4] Create written down guidelines for handling regressions Thorsten Leemhuis
2024-09-12 13:33   ` Thorsten Leemhuis
2024-06-13  8:32 ` [MAINTAINERS SUMMIT] [2/4] Ensure recent mainline regression are fixed in latest stable series Thorsten Leemhuis
2024-06-13 11:02   ` Johannes Berg
2024-06-13 11:21     ` Greg KH
2024-06-13 13:18       ` Sasha Levin
2024-06-13 11:17   ` Jiri Kosina
2024-06-13 11:28   ` Laurent Pinchart
2024-06-14  0:50     ` Steven Rostedt
2024-06-14 14:01   ` Mark Brown
2024-06-14 14:32     ` Rafael J. Wysocki
2024-06-13  8:34 ` [MAINTAINERS SUMMIT] [3/4] Elevate handling of regressions that made it to releases deemed for end users Thorsten Leemhuis
2024-06-13 11:34   ` Laurent Pinchart
2024-06-13 11:39     ` Jiri Kosina
2024-06-14 14:10       ` Mark Brown
2024-06-18 12:58         ` Thorsten Leemhuis
2024-06-19 20:25           ` Laurent Pinchart
2024-06-20 10:47             ` Thorsten Leemhuis
2024-06-13 15:56     ` Liam R. Howlett
2024-06-18 12:24     ` Thorsten Leemhuis
2024-06-20 13:20       ` Jani Nikula
2024-06-20 13:35         ` Thorsten Leemhuis
2024-06-20 14:16           ` Mark Brown
2024-06-21  6:47           ` Jiri Kosina
2024-06-21 10:19             ` Thorsten Leemhuis
2024-06-13  8:42 ` [MAINTAINERS SUMMIT] [4/4] Discuss how to better prevent backports of commits that turn out to cause regressions Thorsten Leemhuis
2024-06-13  9:59   ` Jan Kara
2024-06-13 10:18     ` Thorsten Leemhuis
2024-06-13 14:08     ` Konstantin Ryabitsev
2024-06-14  9:19       ` Lee Jones
2024-06-14  9:24         ` Lee Jones
2024-06-14 12:27         ` Konstantin Ryabitsev
2024-06-14 14:26           ` Konstantin Ryabitsev
2024-06-14 14:36             ` Lee Jones
2024-06-14 14:29       ` Michael Ellerman
2024-06-14 14:38         ` Konstantin Ryabitsev
2024-06-14 14:44           ` Rafael J. Wysocki
2024-06-14 15:08           ` Geert Uytterhoeven
2024-06-15 11:29             ` Michael Ellerman
2024-06-17 10:15             ` Jani Nikula
2024-06-17 12:42               ` Geert Uytterhoeven
2024-06-14 15:45           ` Mark Brown
2024-06-14 14:43         ` Mark Brown
2024-06-14 14:51           ` Konstantin Ryabitsev
2024-06-14 15:42             ` Mark Brown
2024-06-14 14:43         ` Steven Rostedt
2024-06-14 14:57           ` Laurent Pinchart
2024-06-16  1:13         ` Linus Torvalds
2024-06-16  3:28           ` Steven Rostedt
2024-06-16  4:59             ` Linus Torvalds
2024-06-16  8:22               ` Paolo Bonzini
2024-06-16  9:05               ` Geert Uytterhoeven
2024-06-16 15:07               ` Steven Rostedt
2024-06-17 13:48               ` Dan Carpenter
2024-06-17 15:23                 ` Dan Carpenter
2024-06-17 14:39               ` Konstantin Ryabitsev
2024-06-17 16:04                 ` Paul E. McKenney
2024-06-17 16:06                   ` Konstantin Ryabitsev
2024-06-17 16:14                     ` Paolo Bonzini
2024-06-17 16:18                       ` Konstantin Ryabitsev
2024-06-17 17:11                         ` Geert Uytterhoeven
2024-06-18 12:05                 ` Michael Ellerman
2024-06-16  7:26           ` Takashi Iwai
2024-06-16  8:10           ` Paolo Bonzini
2024-06-16 11:31             ` Laurent Pinchart
2024-06-16 11:39             ` Takashi Iwai
2024-06-16 16:40             ` Linus Torvalds
2024-06-16  8:31           ` Jiri Kosina
2024-06-16  8:54             ` Geert Uytterhoeven
2024-06-13 19:39     ` Dan Carpenter
2024-06-14  1:00       ` Steven Rostedt
2024-06-13 11:58   ` James Bottomley
2024-06-13 13:06     ` Sasha Levin [this message]
2024-06-13 13:56       ` James Bottomley
2024-06-13 14:02         ` Greg KH
2024-06-13 15:11           ` James Bottomley
2024-06-13 16:27             ` Greg KH
2024-06-14 18:47             ` Sasha Levin
2024-06-17 10:59               ` Vlastimil Babka
2024-06-13 18:08         ` Sasha Levin
2024-06-13 13:45     ` Greg KH
2024-06-13 13:40   ` Sasha Levin
2024-06-18 13:12     ` Thorsten Leemhuis
2024-06-13 14:28   ` Andrew Lunn
2024-06-13 18:14     ` Sasha Levin
2024-06-14 14:41       ` Jan Kara
2024-06-14 15:03         ` Rafael J. Wysocki
2024-06-14 17:46           ` Sasha Levin
2024-06-18 14:43 ` [MAINTAINERS SUMMIT] [0/4] Common scenario for four proposals regarding regressions James Bottomley
2024-06-18 15:50   ` Mark Brown
2024-06-20 10:32   ` Thorsten Leemhuis
2024-06-20 12:57     ` James Bottomley
2024-06-20 13:55       ` Mark Brown
2024-06-20 14:01         ` James Bottomley
2024-06-20 14:42           ` Mark Brown
2024-06-20 16:02             ` James Bottomley
2024-06-20 17:15               ` Mark Brown
2024-06-20 23:25               ` Sasha Levin
2024-06-21  6:33                 ` Thorsten Leemhuis
     [not found]               ` <20240625175131.672d14a4@rorschach.local.home>
2024-06-26  7:36                 ` Greg KH
2024-06-26 18:32                   ` Steven Rostedt
2024-06-26 19:05                     ` James Bottomley
2024-07-25 10:14                   ` Thorsten Leemhuis
2024-07-25 13:14                     ` Greg KH
2024-06-20 16:59       ` Thorsten Leemhuis
2024-06-20 23:18         ` Sasha Levin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ZmruqWDTG2PK-rbu@sashalap \
    --to=sashal@kernel.org \
    --cc=James.Bottomley@hansenpartnership.com \
    --cc=ksummit@lists.linux.dev \
    --cc=linux@leemhuis.info \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.