[LSF/MM/BPF TOPIC] Filesystem backporting to stable

[LSF/MM/BPF TOPIC] Filesystem backporting to stable

[Leah Rumancik]

The current standard for backporting to stable is rather ad hoc. Not
everyone will cc stable on bug fixes, and even if they do, if the
patch does not apply cleanly to the stable branches, it often gets
dropped if no one notices or cares enough to do the backporting.

Historically, the XFS community has avoided backports via cc’ing
stable and via AUTOSEL due to concerns of accidentally introducing new
bugs to the stable branches. However, over the last year, XFS has
developed a new strategy for managing stable backports. A “stable
maintainer” role has been created for each stable branch to identify,
apply, and test potential backports before they are sent to the stable
mailing list. This provides better monitoring of the stable branches
which reduces the risk of introducing new bugs to stable and lessens
the possibility of bug fixes getting dropped on their way to stable.
XFS has benefited from this new backporting procedure and perhaps
other filesystems would as well.

- Leah

Re: [LSF/MM/BPF TOPIC] Filesystem backporting to stable

[Darrick J. Wong]

On Tue, Feb 28, 2023 at 03:46:11PM -0800, Leah Rumancik wrote:
> The current standard for backporting to stable is rather ad hoc. Not
> everyone will cc stable on bug fixes, and even if they do, if the
> patch does not apply cleanly to the stable branches, it often gets
> dropped if no one notices or cares enough to do the backporting.

I'm very glad to see you stepping forward!

> Historically, the XFS community has avoided backports via cc’ing
> stable and via AUTOSEL due to concerns of accidentally introducing new
> bugs to the stable branches. However, over the last year, XFS has
> developed a new strategy for managing stable backports. A “stable
> maintainer” role has been created for each stable branch to identify,
> apply, and test potential backports before they are sent to the stable
> mailing list. This provides better monitoring of the stable branches
> which reduces the risk of introducing new bugs to stable and lessens
> the possibility of bug fixes getting dropped on their way to stable.
> XFS has benefited from this new backporting procedure and perhaps
> other filesystems would as well.

Given the recent roaring[1] between Sasha and Eric Biggers, I think this
is a very apropos topic.  It's probably all right for robots to pick
over driver patches and fling them into the LTS kernels, but filesystems
are so complex that they really need experienced people to make
judgement calls.

Another knot that I think we need to slice through is the question of
what to do when support for LTS kernels becomes sparse.  Case in point:

Oracle produces a kernel based on 5.4 and 5.15.  We're not going to
introduce one based on 5.10.  If Amir stops supporting 5.10, what do we
do about 5.4?  If a bugfix appears that needs to be applied to 5.4-6.1,
Greg will not let Chandan backport it to 5.4 until Chandan either takes
responsibility for 5.10, or tricks someone else into do it.  That's not
fair to him, and Oracle isn't going to start supporting 5.10.  Preening
online fsck parts 1 and 2 now consumes so much time that I don't have
the bandwidth to embark on any of the iomap cleanups that are slowly
getting more urgent.

[I am very very grateful that you all came along!]

So what do we do?  Do I beg the three of you to try to do 5.10 in your
spare time, which isn't really fair to you or Amir.

Proposal:

How about we EOL the XFS code in the releases that nobody wants so that
patches can keep flowing to the ones that are still wanted?

(Hypothetically speaking, since this isn't a problem /today/.)

<rant>
Really this is a resource allocation problem stemming from the ability
of the LTS leaders to saddle the rest of the fsdevel community with
responsibilities that we weren't asked about nor agreed to.

--D

[1] https://lore.kernel.org/linux-fsdevel/Y%2F4z2NyGgwG4zvYq@sashalap/T/#maa4db2ea8187e5e047a59bbb6411d916173367b9

> - Leah

Re: [LSF/MM/BPF TOPIC] Filesystem backporting to stable

[Dave Chinner]

On Tue, Feb 28, 2023 at 05:48:23PM -0800, Darrick J. Wong wrote:
> Another knot that I think we need to slice through is the question of
> what to do when support for LTS kernels becomes sparse.  Case in point:
> 
> Oracle produces a kernel based on 5.4 and 5.15.  We're not going to
> introduce one based on 5.10.  If Amir stops supporting 5.10, what do we
> do about 5.4?  If a bugfix appears that needs to be applied to 5.4-6.1,
> Greg will not let Chandan backport it to 5.4 until Chandan either takes
> responsibility for 5.10, or tricks someone else into do it.

And this is soon going to become a much bigger problem with 6.1-lts
in the near future will start gating fixes flowing back to 5.15,
5.10 and 5.4....

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/MM/BPF TOPIC] Filesystem backporting to stable

[Sasha Levin]

On Tue, Feb 28, 2023 at 05:48:23PM -0800, Darrick J. Wong wrote:
>On Tue, Feb 28, 2023 at 03:46:11PM -0800, Leah Rumancik wrote:
>> The current standard for backporting to stable is rather ad hoc. Not
>> everyone will cc stable on bug fixes, and even if they do, if the
>> patch does not apply cleanly to the stable branches, it often gets
>> dropped if no one notices or cares enough to do the backporting.
>
>I'm very glad to see you stepping forward!
>
>> Historically, the XFS community has avoided backports via cc’ing
>> stable and via AUTOSEL due to concerns of accidentally introducing new
>> bugs to the stable branches. However, over the last year, XFS has
>> developed a new strategy for managing stable backports. A “stable
>> maintainer” role has been created for each stable branch to identify,
>> apply, and test potential backports before they are sent to the stable
>> mailing list. This provides better monitoring of the stable branches
>> which reduces the risk of introducing new bugs to stable and lessens
>> the possibility of bug fixes getting dropped on their way to stable.
>> XFS has benefited from this new backporting procedure and perhaps
>> other filesystems would as well.
>
>Given the recent roaring[1] between Sasha and Eric Biggers, I think this
>is a very apropos topic.  It's probably all right for robots to pick
>over driver patches and fling them into the LTS kernels, but filesystems
>are so complex that they really need experienced people to make
>judgement calls.

Which is fine in today's world, no? If a certain subsystem wants to do
it's own thing, it just makes our lives that much easier, and on that
note: thanks Leah & Amir!

>Another knot that I think we need to slice through is the question of
>what to do when support for LTS kernels becomes sparse.  Case in point:
>
>Oracle produces a kernel based on 5.4 and 5.15.  We're not going to
>introduce one based on 5.10.  If Amir stops supporting 5.10, what do we
>do about 5.4?  If a bugfix appears that needs to be applied to 5.4-6.1,
>Greg will not let Chandan backport it to 5.4 until Chandan either takes
>responsibility for 5.10, or tricks someone else into do it.  That's not
>fair to him, and Oracle isn't going to start supporting 5.10.  Preening
>online fsck parts 1 and 2 now consumes so much time that I don't have
>the bandwidth to embark on any of the iomap cleanups that are slowly
>getting more urgent.
>
>[I am very very grateful that you all came along!]
>
>So what do we do?  Do I beg the three of you to try to do 5.10 in your
>spare time, which isn't really fair to you or Amir.
>
>Proposal:
>
>How about we EOL the XFS code in the releases that nobody wants so that
>patches can keep flowing to the ones that are still wanted?

We're in the process of going away from 6 year kernels which will in
turn lessen the number of parallel trees we support.

OTOH, this raises the question of what Oracle plans to do after 5.4 goes
EOL?

-- 
Thanks,
Sasha

Re: [LSF/MM/BPF TOPIC] Filesystem backporting to stable

[Jan Kara]

On Sun 12-03-23 22:34:00, Sasha Levin wrote:
> > How about we EOL the XFS code in the releases that nobody wants so that
> > patches can keep flowing to the ones that are still wanted?
> 
> We're in the process of going away from 6 year kernels which will in
> turn lessen the number of parallel trees we support.
> 
> OTOH, this raises the question of what Oracle plans to do after 5.4 goes
> EOL?

Well, I guess what every other distro does with old kernels. Fix whatever
customers report + a bunch of CVE fixes. We still support 2.6.16 so 5.4 is
"piece'a cake" ;)

								Honza

-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR