Backport dependencies helper

Backport dependencies helper

[Sasha Levin]

Hi all,

I wanted to share a resource I've been using to help me with doing
backports to various stable kernels.

In the Stable Kernel world, when we need to backport a patch, we'd
rather take any relevant dependencies to make the patch work cleanly on
an older kernel, rather than modifying the patch and diverging from
upstream.

This raises an interesting problem: how do we figure out which other
patches might be "interesting" to look at? git-blame is a great tool,
but it takes a while to go through the history of a patch, and given the
volume of patches we need to look at, it just isn't enough.

So here's a tool in the form of a git repo that can help point out these
interesting patches:

        https://git.kernel.org/pub/scm/linux/kernel/git/sashal/deps.git/

How does it work, you might ask? It's actually quite simple: Each
directory represents a kernel version which we'll call K, and each file
inside that directory is named after an upstream commit we'll call C,
and it's content are the list of commits one would need to apply on top
of kernel K to "reach" commit C.

For example, let's say we want to apply:

        f8788d86ab28 ("Linux 5.6-rc3")

On top of the v5.5 kernel tree. All we need to do is:

        $ cat v5.5/f8788d86ab28f61f7b46eb6be375f8a726783636
        f8788d86ab28 ("Linux 5.6-rc3")
        11a48a5a18c6 ("Linux 5.6-rc2")
        bb6d3fb354c5 ("Linux 5.6-rc1")

If you don't feel like cloning the repo (which contains quite a few
files), you can also use kernel.org's web interface in a script that
might look something like this:

        #!/bin/bash
        curl https://git.kernel.org/pub/scm/linux/kernel/git/sashal/deps.git/plain/$1/$2

And then simply:

        $ ./deps.sh v5.5 f8788d86ab28f61f7b46eb6be375f8a726783636
        f8788d86ab28 ("Linux 5.6-rc3")
        11a48a5a18c6 ("Linux 5.6-rc2")
        bb6d3fb354c5 ("Linux 5.6-rc1")

Caveats:

 - Each file is limited to 50 entries. I feel that at that point it
   stops being useful.
 - Each file stops if a merge commit is hit.
 - I might have bugs in my scripts and some entries are broken, please
   report those if you see them.

-- 
Thanks,
Sasha

Re: Backport dependencies helper

[Willy Tarreau]

Hi Sasha,

On Tue, Mar 31, 2020 at 08:32:17AM -0400, Sasha Levin wrote:
> Each
> directory represents a kernel version which we'll call K, and each file
> inside that directory is named after an upstream commit we'll call C,
> and it's content are the list of commits one would need to apply on top
> of kernel K to "reach" commit C.

That's very interesting! I still have nightmare-like memories or
complete week-ends spent trying to address this using heuristics
when I was maintaining 2.6.32 and 3.10. However how do you produce
these ? Is this related to the stable-deps utility in your stable-tools
repository ?

Thanks,
Willy

Re: Backport dependencies helper

[Sasha Levin]

On Tue, Mar 31, 2020 at 03:44:00PM +0200, Willy Tarreau wrote:
>Hi Sasha,
>
>On Tue, Mar 31, 2020 at 08:32:17AM -0400, Sasha Levin wrote:
>> Each
>> directory represents a kernel version which we'll call K, and each file
>> inside that directory is named after an upstream commit we'll call C,
>> and it's content are the list of commits one would need to apply on top
>> of kernel K to "reach" commit C.
>
>That's very interesting! I still have nightmare-like memories or
>complete week-ends spent trying to address this using heuristics
>when I was maintaining 2.6.32 and 3.10. However how do you produce
>these ? Is this related to the stable-deps utility in your stable-tools
>repository ?

No, those tools try to do the same thing, but work differently.
stable-deps attempts to look at context lines surrounding the patch
itself to guess which other patches might be interesting.

While here, I use git-bisect to create a list of commits required to be
applied before any given commit.

-- 
Thanks,
Sasha

Re: Backport dependencies helper

[Willy Tarreau]

On Tue, Mar 31, 2020 at 10:08:30AM -0400, Sasha Levin wrote:
> No, those tools try to do the same thing, but work differently.
> stable-deps attempts to look at context lines surrounding the patch
> itself to guess which other patches might be interesting.

OK!

> While here, I use git-bisect to create a list of commits required to be
> applied before any given commit.

I see, this sounds like a great idea!

Thanks,
Willy

Re: Backport dependencies helper

[Greg KH]

On Tue, Mar 31, 2020 at 08:32:17AM -0400, Sasha Levin wrote:
> Hi all,
> 
> I wanted to share a resource I've been using to help me with doing
> backports to various stable kernels.
> 
> In the Stable Kernel world, when we need to backport a patch, we'd
> rather take any relevant dependencies to make the patch work cleanly on
> an older kernel, rather than modifying the patch and diverging from
> upstream.
> 
> This raises an interesting problem: how do we figure out which other
> patches might be "interesting" to look at? git-blame is a great tool,
> but it takes a while to go through the history of a patch, and given the
> volume of patches we need to look at, it just isn't enough.
> 
> So here's a tool in the form of a git repo that can help point out these
> interesting patches:
> 
>        https://git.kernel.org/pub/scm/linux/kernel/git/sashal/deps.git/
> 
> How does it work, you might ask? It's actually quite simple: Each
> directory represents a kernel version which we'll call K, and each file
> inside that directory is named after an upstream commit we'll call C,
> and it's content are the list of commits one would need to apply on top
> of kernel K to "reach" commit C.
> 
> For example, let's say we want to apply:
> 
>        f8788d86ab28 ("Linux 5.6-rc3")
> 
> On top of the v5.5 kernel tree. All we need to do is:
> 
>        $ cat v5.5/f8788d86ab28f61f7b46eb6be375f8a726783636
>        f8788d86ab28 ("Linux 5.6-rc3")
>        11a48a5a18c6 ("Linux 5.6-rc2")
>        bb6d3fb354c5 ("Linux 5.6-rc1")
> 
> If you don't feel like cloning the repo (which contains quite a few
> files), you can also use kernel.org's web interface in a script that
> might look something like this:
> 
>        #!/bin/bash
>        curl https://git.kernel.org/pub/scm/linux/kernel/git/sashal/deps.git/plain/$1/$2
> 
> And then simply:
> 
>        $ ./deps.sh v5.5 f8788d86ab28f61f7b46eb6be375f8a726783636
>        f8788d86ab28 ("Linux 5.6-rc3")
>        11a48a5a18c6 ("Linux 5.6-rc2")
>        bb6d3fb354c5 ("Linux 5.6-rc1")
> 
> Caveats:
> 
> - Each file is limited to 50 entries. I feel that at that point it
>   stops being useful.
> - Each file stops if a merge commit is hit.
> - I might have bugs in my scripts and some entries are broken, please
>   report those if you see them.

This is really cool, thanks for posting this, and for doing this work.

greg k-h