Improving Linux Commit Backporting

Improving Linux Commit Backporting

[Manthey, Norbert]

Dear Linux Stable Maintainers,

while maintaining downstream Linux releases, we noticed that we have to
backport some patches manually, because they are not picked up by your
automated backporting. Some of these backports can be done with
improved cherry-pick tooling. We have implemented a script/tool "git-
fuzzy-pick" which we would like to share. Besides picking more commits,
the tool also supports executing a validation script right after
picking, e.g. compiling the modified source file. Picking stats and
details are presented below.

We would like to discuss whether you can integrate this improved tool
into into your daily workflows. We already found the stable-tools
repository [1] with some scripts that help automate backporting. To
contribute git-fuzzy-pick there, we would need you to declare a license
for the current state of this repository.

To test backporting performance, we tried to backport stable-candidate
patches from 6.12 to 6.1. Specifically, on tag 6.1.125 we executed the
command stable show-missing-stable v6.12.12..v6.12.17 to collect
patches considered for backporting. This results in 431 backport
candidates. When using git-fuzzy-pick, we can pick 9 patches more than
with default cherry-picking. All modifications have been validated by
attempting to build the
object files of the modified C source files with make using the kernels
“allyesconfig” configuration.

196 Cherry-picked with --strategy=recursive --Xpatience -x
  1 Applied with patch -p1 ... --fuzz=1
  8 Applied with patch -p1 ... --fuzz=2

Please let us know how to best share the tool with you! Long term, we
would like to integrate it into your backporting workflow, so that more
kernel commits can be applied automatically.

Best,
Norbert


[1]
https://kernel.googlesource.com/pub/scm/linux/kernel/git/sashal/stable-tools/

Re: Improving Linux Commit Backporting

[gregkh]

On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
> Dear Linux Stable Maintainers,
> 
> while maintaining downstream Linux releases, we noticed that we have to
> backport some patches manually, because they are not picked up by your
> automated backporting. Some of these backports can be done with
> improved cherry-pick tooling. We have implemented a script/tool "git-
> fuzzy-pick" which we would like to share. Besides picking more commits,
> the tool also supports executing a validation script right after
> picking, e.g. compiling the modified source file. Picking stats and
> details are presented below.
> 
> We would like to discuss whether you can integrate this improved tool
> into into your daily workflows. We already found the stable-tools
> repository [1] with some scripts that help automate backporting. To
> contribute git-fuzzy-pick there, we would need you to declare a license
> for the current state of this repository.

There's no need for us to declare the license for the whole repo, you
just need to pick a license for your script to be under.  Anything
that's under a valid open source license is fine with me.

That being said, I did just go and add SPDX license lines to all of the
scripts that I wrote, or that was already defined in the comments of the
files, to make it more obvious what they are under.

> To test backporting performance, we tried to backport stable-candidate
> patches from 6.12 to 6.1. Specifically, on tag 6.1.125 we executed the
> command stable show-missing-stable v6.12.12..v6.12.17 to collect
> patches considered for backporting. This results in 431 backport
> candidates. When using git-fuzzy-pick, we can pick 9 patches more than
> with default cherry-picking. All modifications have been validated by
> attempting to build the
> object files of the modified C source files with make using the kernels
> “allyesconfig” configuration.
> 
> 196 Cherry-picked with --strategy=recursive --Xpatience -x
>   1 Applied with patch -p1 ... --fuzz=1
>   8 Applied with patch -p1 ... --fuzz=2
> 
> Please let us know how to best share the tool with you! Long term, we
> would like to integrate it into your backporting workflow, so that more
> kernel commits can be applied automatically.

A long long time ago I used to apply patches with a whole lot of fuzz in
a semi-automated way much like this, but things slipped through all the
time.  So now, if we have a failure, I throw it back to the
developers/maintainers involved and let them and/or the community deal
with it if they want to have the commit backported.  That way they can
test and manually verify that the backport is correct.

So no, I don't recommend auto-fuzz-forcing any commits WITHOUT manually
looking and verifying and checking and taking responsibility for the
backport being correct.  Right now we are full of work just keeping up
with 40 commits a day in the stable trees.  If others wish to help out,
we are more than willing to take the backports that have been submitted
to us, and we now can handle them directly from the mailing lists even
easier than before, with Sasha's bot reporting any potential problems,
and my local scripts that properly apply them to the specific queues in
a "very few" key-stroke command from my email client.

So in summary, I'm more than willing to take a patch that adds your tool
to our repo, for others to use, but I don't want to be the one
responsible for running it.  I want others to take that responsibility
if they care about applying those patches to older kernels properly, as
it's the companies that care about those kernels that should be the ones
helping us out the most, instead of asking others to do this work for
them, don't you think?  :)

thanks,

greg k-h

Re: Improving Linux Commit Backporting

[gregkh]

On Thu, Apr 03, 2025 at 02:45:35PM +0100, gregkh@linuxfoundation.org wrote:
> On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
> > Dear Linux Stable Maintainers,
> > 
> > while maintaining downstream Linux releases, we noticed that we have to
> > backport some patches manually, because they are not picked up by your
> > automated backporting. Some of these backports can be done with
> > improved cherry-pick tooling. We have implemented a script/tool "git-
> > fuzzy-pick" which we would like to share. Besides picking more commits,
> > the tool also supports executing a validation script right after
> > picking, e.g. compiling the modified source file. Picking stats and
> > details are presented below.
> > 
> > We would like to discuss whether you can integrate this improved tool
> > into into your daily workflows. We already found the stable-tools
> > repository [1] with some scripts that help automate backporting. To
> > contribute git-fuzzy-pick there, we would need you to declare a license
> > for the current state of this repository.
> 
> There's no need for us to declare the license for the whole repo, you
> just need to pick a license for your script to be under.  Anything
> that's under a valid open source license is fine with me.
> 
> That being said, I did just go and add SPDX license lines to all of the
> scripts that I wrote, or that was already defined in the comments of the
> files, to make it more obvious what they are under.

Wait, you should be looking at the scripts in the stable-queue.git tree
in the scripts/ directory.  You pointed at a private repo of some things
that Sasha uses for his work, which is specific to his workflow.

thanks,

greg k-h

Re: Improving Linux Commit Backporting

[gregkh]

On Thu, Apr 03, 2025 at 02:57:34PM +0100, gregkh@linuxfoundation.org wrote:
> On Thu, Apr 03, 2025 at 02:45:35PM +0100, gregkh@linuxfoundation.org wrote:
> > On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
> > > Dear Linux Stable Maintainers,
> > > 
> > > while maintaining downstream Linux releases, we noticed that we have to
> > > backport some patches manually, because they are not picked up by your
> > > automated backporting. Some of these backports can be done with
> > > improved cherry-pick tooling. We have implemented a script/tool "git-
> > > fuzzy-pick" which we would like to share. Besides picking more commits,
> > > the tool also supports executing a validation script right after
> > > picking, e.g. compiling the modified source file. Picking stats and
> > > details are presented below.
> > > 
> > > We would like to discuss whether you can integrate this improved tool
> > > into into your daily workflows. We already found the stable-tools
> > > repository [1] with some scripts that help automate backporting. To
> > > contribute git-fuzzy-pick there, we would need you to declare a license
> > > for the current state of this repository.
> > 
> > There's no need for us to declare the license for the whole repo, you
> > just need to pick a license for your script to be under.  Anything
> > that's under a valid open source license is fine with me.
> > 
> > That being said, I did just go and add SPDX license lines to all of the
> > scripts that I wrote, or that was already defined in the comments of the
> > files, to make it more obvious what they are under.
> 
> Wait, you should be looking at the scripts in the stable-queue.git tree
> in the scripts/ directory.  You pointed at a private repo of some things
> that Sasha uses for his work, which is specific to his workflow.

Also, one final things.  Doing backports to older kernels is a harder
task than doing it for newer kernels.  This means you need to do more
work, and have a more experienced developer do that work, as the nuances
are tricky and slight and they must understand the code base really
well.

Attempting to automate this, and make it a "junior developer" task
assignment is ripe for errors and problems and tears (on my side and
yours.)  We have loads of examples of this in the past, please don't
duplicate the errors of others and think that "somehow, this time it
will be different!", but rather "learn from our past mistakes and only
make new ones."

Good luck with backporting, as I know just how hard of a task this
really is.  And obviously, you are learning that too :)

greg k-h

Re: Improving Linux Commit Backporting

[gregkh]

On Fri, Apr 04, 2025 at 12:28:06PM +0000, Manthey, Norbert wrote:

<snip>

You sent this in html format, which is rejected by the kernel community
mailing lists.  Please fix you email client if you wish to work with the
kernel community properly.

thanks,

greg k-h

Re: Improving Linux Commit Backporting

[Manthey, Norbert]

On Thu, 2025-04-03 at 15:51 +0100, gregkh@linuxfoundation.org wrote:
...snip ...
> On Thu, Apr 03, 2025 at 02:57:34PM +0100, gregkh@linuxfoundation.org wrote:
> > On Thu, Apr 03, 2025 at 02:45:35PM +0100, gregkh@linuxfoundation.org wrote:
> > > On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
...snip...
> > > > We would like to discuss whether you can integrate this improved tool
> > > > into into your daily workflows. We already found the stable-tools
> > > > repository [1] with some scripts that help automate backporting. To
> > > > contribute git-fuzzy-pick there, we would need you to declare a license
> > > > for the current state of this repository.
> > > 
> > > There's no need for us to declare the license for the whole repo, you
> > > just need to pick a license for your script to be under.  Anything
> > > that's under a valid open source license is fine with me.
> > > 
> > > That being said, I did just go and add SPDX license lines to all of the
> > > scripts that I wrote, or that was already defined in the comments of the
> > > files, to make it more obvious what they are under.
Thanks!
> > 
> > Wait, you should be looking at the scripts in the stable-queue.git tree
> > in the scripts/ directory.  You pointed at a private repo of some things
> > that Sasha uses for his work, which is specific to his workflow.
I had a look at those scripts too. Looks like you use git am, and abort in case this operation
fails.
> 
> Also, one final things.  Doing backports to older kernels is a harder
> task than doing it for newer kernels.  This means you need to do more
> work, and have a more experienced developer do that work, as the nuances
> are tricky and slight and they must understand the code base really
> well.
> 
> Attempting to automate this, and make it a "junior developer" task
> assignment is ripe for errors and problems and tears (on my side and
> yours.)  We have loads of examples of this in the past, please don't
> duplicate the errors of others and think that "somehow, this time it
> will be different!", but rather "learn from our past mistakes and only
> make new ones."
We understand. We might make the tool available to help simplify the human effort of backporting. To
make this more successful, is there a way to identify the errors and learnings you mention from the
past? Avoiding them automatically early in the process helps keeping the errors away.
> 
> Good luck with backporting, as I know just how hard of a task this
> really is.  And obviously, you are learning that too :)
Thanks.

Norbert

Re: Improving Linux Commit Backporting

[gregkh]

On Mon, Apr 07, 2025 at 07:26:59AM +0000, Manthey, Norbert wrote:
> On Thu, 2025-04-03 at 15:51 +0100, gregkh@linuxfoundation.org wrote:
> ...snip ...
> > On Thu, Apr 03, 2025 at 02:57:34PM +0100, gregkh@linuxfoundation.org wrote:
> > > On Thu, Apr 03, 2025 at 02:45:35PM +0100, gregkh@linuxfoundation.org wrote:
> > > > On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
> ...snip...
> > > > > We would like to discuss whether you can integrate this improved tool
> > > > > into into your daily workflows. We already found the stable-tools
> > > > > repository [1] with some scripts that help automate backporting. To
> > > > > contribute git-fuzzy-pick there, we would need you to declare a license
> > > > > for the current state of this repository.
> > > > 
> > > > There's no need for us to declare the license for the whole repo, you
> > > > just need to pick a license for your script to be under.  Anything
> > > > that's under a valid open source license is fine with me.
> > > > 
> > > > That being said, I did just go and add SPDX license lines to all of the
> > > > scripts that I wrote, or that was already defined in the comments of the
> > > > files, to make it more obvious what they are under.
> Thanks!
> > > 
> > > Wait, you should be looking at the scripts in the stable-queue.git tree
> > > in the scripts/ directory.  You pointed at a private repo of some things
> > > that Sasha uses for his work, which is specific to his workflow.
> I had a look at those scripts too. Looks like you use git am, and abort in case this operation
> fails.

We only use 'git am' to apply the quilt series to the tree when doing
a release (the final one or the -rc ones).  We use quilt to manage
everything in the stable-queue.git tree as that provides us with the
needed flexibility.

> > Also, one final things.  Doing backports to older kernels is a harder
> > task than doing it for newer kernels.  This means you need to do more
> > work, and have a more experienced developer do that work, as the nuances
> > are tricky and slight and they must understand the code base really
> > well.
> > 
> > Attempting to automate this, and make it a "junior developer" task
> > assignment is ripe for errors and problems and tears (on my side and
> > yours.)  We have loads of examples of this in the past, please don't
> > duplicate the errors of others and think that "somehow, this time it
> > will be different!", but rather "learn from our past mistakes and only
> > make new ones."
> We understand. We might make the tool available to help simplify the human effort of backporting. To
> make this more successful, is there a way to identify the errors and learnings you mention from the
> past? Avoiding them automatically early in the process helps keeping the errors away.

Don't ignore fuzz, manually check, and verify, everything.

good luck!

greg k-h

Re: Improving Linux Commit Backporting

[Willy Tarreau]

On Mon, Apr 07, 2025 at 09:31:08AM +0200, gregkh@linuxfoundation.org wrote:
> > We understand. We might make the tool available to help simplify the human effort of backporting. To
> > make this more successful, is there a way to identify the errors and learnings you mention from the
> > past? Avoiding them automatically early in the process helps keeping the errors away.
> 
> Don't ignore fuzz, manually check, and verify, everything.

Also, diffing the file between the latest kernel and the one you're
backporting to helps discover changed assumptions. For example, a
group of functions might be called with similar assumptions in the
old kernel, with some tests replicated everywhere (say pointer foo
must not be NULL). In newer kernels this test is moved up in the
caller chain and is no longer performed in the lower functions. When
you want to backport a fix from this kernel to the old one, you may
need to reimplement yourself the nullity check that the old kernel
requires. And that's valid for locking and many other things in
general. There's no way to automatically discover these ones, aside
comparing the older and newer states of the file to see how it
evolved over time, and developing your own habits of remembering
that certain areas are different in your kernel, by doing lots of
backports there, as well as following LKML to try to spot some
changes that may affect your areas of interest. After a few years
on old kernel you can start to develop some reflexes but that's not
rocket science.

> good luck!

Seconded!

> greg k-h

Willy

Re: Improving Linux Commit Backporting

[Sasha Levin]

On Thu, Apr 03, 2025 at 03:51:25PM +0100, gregkh@linuxfoundation.org wrote:
>On Thu, Apr 03, 2025 at 02:57:34PM +0100, gregkh@linuxfoundation.org wrote:
>> On Thu, Apr 03, 2025 at 02:45:35PM +0100, gregkh@linuxfoundation.org wrote:
>> > On Thu, Apr 03, 2025 at 01:15:28PM +0000, Manthey, Norbert wrote:
>> > > Dear Linux Stable Maintainers,
>> > >
>> > > while maintaining downstream Linux releases, we noticed that we have to
>> > > backport some patches manually, because they are not picked up by your
>> > > automated backporting. Some of these backports can be done with
>> > > improved cherry-pick tooling. We have implemented a script/tool "git-
>> > > fuzzy-pick" which we would like to share. Besides picking more commits,
>> > > the tool also supports executing a validation script right after
>> > > picking, e.g. compiling the modified source file. Picking stats and
>> > > details are presented below.
>> > >
>> > > We would like to discuss whether you can integrate this improved tool
>> > > into into your daily workflows. We already found the stable-tools
>> > > repository [1] with some scripts that help automate backporting. To
>> > > contribute git-fuzzy-pick there, we would need you to declare a license
>> > > for the current state of this repository.
>> >
>> > There's no need for us to declare the license for the whole repo, you
>> > just need to pick a license for your script to be under.  Anything
>> > that's under a valid open source license is fine with me.
>> >
>> > That being said, I did just go and add SPDX license lines to all of the
>> > scripts that I wrote, or that was already defined in the comments of the
>> > files, to make it more obvious what they are under.
>>
>> Wait, you should be looking at the scripts in the stable-queue.git tree
>> in the scripts/ directory.  You pointed at a private repo of some things
>> that Sasha uses for his work, which is specific to his workflow.
>
>Also, one final things.  Doing backports to older kernels is a harder
>task than doing it for newer kernels.  This means you need to do more
>work, and have a more experienced developer do that work, as the nuances
>are tricky and slight and they must understand the code base really
>well.
>
>Attempting to automate this, and make it a "junior developer" task
>assignment is ripe for errors and problems and tears (on my side and
>yours.)  We have loads of examples of this in the past, please don't
>duplicate the errors of others and think that "somehow, this time it
>will be different!", but rather "learn from our past mistakes and only
>make new ones."
>
>Good luck with backporting, as I know just how hard of a task this
>really is.  And obviously, you are learning that too :)

I've played with wiggle[1] in the past, which does all what you've
described and more.

It introduced too many issues, where it wasn't worth doing it. I really
think that the better solution here is to figure out dependencies and
bring them in.

I'd advise against using such tool at the scale of -stable :)

[1] https://github.com/neilbrown/wiggle

-- 
Thanks,
Sasha