Re: CI for LTS kernels

Re: CI for LTS kernels

[Ihor Solodrai]

On 3/24/25 8:49 PM, Shung-Hsi Yu wrote:
> Hi Ihor,
>
> On Mon, Mar 24, 2025 at 10:16:38PM +0000, Ihor Solodrai wrote:
>> On Monday, March 24th, 2025 at 1:55 PM, Eduard Zingerman <eddyz87@gmail.com> wrote:
>>> Hi All,
>>>
>>> The question of testing LTS kernel on BPF CI was
>>> raised by Shung-Hsi on the LSFMM today.
>>> I think, Ihor in CC can guide through the process
>>> of adding such support to the CI if decision would
>>> be made to commit to this.
>
> Thank you Eduard for start the thread.
>
> Attaching the link to the slides[1] for reference.
>
>>> Eduard.
>>
>> Hi Eduard, thanks for pinging me.
>>
>> I actually thought about implementing LTS kernel testing for libbpf,
>> but so far it was not a priority.
>
> I'm not too familiar with BPF CI, but I assuming this meant having
> libbpf's GitHub action test stable/linux.git as well, along side the
> current bpf/bpf-next.git target?
>
> It if it's that then it would be great. Exactly what I'm looking for.

Hi Shung-Hsi.

In short, yes: it's possible to set up Github Actions to run BPF
selftests on LTS kernels using current BPF CI code.

I'm afraid we don't have enough bandwidth on BPF side to maintain LTS
kernel testing within our CI infrastructure. But I will share with you
some pointers in case you're willing to take a stab at it.

All the CI code is public and you should be able to figure out how to
modify it for your needs.

https://github.com/libbpf/ci has a collection of callable Github
Actions that can be used as building blocks for kernel testing.
For example, you can use `get-linux-source` action to download any
source revision:

      - uses: libbpf/ci/get-linux-source@v3
        with:
          repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
          rev: master

The actual BPF CI workflows implementation is located at
https://github.com/kernel-patches/vmtest repo.  In particular, see
`.github/workflows` directory.

https://github.com/libbpf/libbpf contains a simpler version of a very
similar workflow, although it has additional code to test libbpf
builds on various distros.

>
>> ... One thing we'd need to figure out
>> is a way of determining which subset of the selftests is supposed to
>> work on a given revision.
>
> I think I might be able to help.
>
> Some question though, is the plan to run bpf-next BPF selftests on LTS
> kernels? (hence both bpf/bpf-next.git and stable/linux.git will both
> need to cloned, separately)

Well, that's a design decision for the LTS testers. I don't think it
makes sense to run all bpf-next tests on older kernels, as usually
tests are added with new features, which would be absent. But it seems
like a good idea to backport and run tests (if there are any) for
backported fixes. It feels like a lot of work though.

>
> Suppose we have some per test case annotation of the kernel release that
> its depending feature is introduced, would that work? (we might have to
> start with an even coarser grain and ignore annotating feature
> introduced long time ago)
>
> Thinking out loud here. Starting with 6.12 is likely the easiest, for
> that I'd just need annotations like like SINCE("6.14"), SINCE("6.13"),
> SINCE(BPF_LTS_TEST_BASE) where BPF_LTS_TEST_BASE is "6.12". And have
> most test group/cases annotated with SINCE(BPF_LTS_TEST_BASE). Anything
> not annotated would be considered to be bpf-next-only.

I'd be reluctant to introduce annotations directly in test
definitions. I think a better approach is to maintain allow/denylists
that control what tests to run on what revision. BPF selftests runners
already support allow/denylisting. The hard part is to actually
produce and maintain those lists.

>
> OTOH we could just use the LTS BPF selftests found in the same code base
> as the LTS kernel themselves. That seems to be eaiser as a POC.

Yes, that's a reasonable approach in the beginning.

>
>> I definitely could help setting this up, if there is a need.
>
> Definitely. If you could also give me some pointers on where to start (I
> guess I'd need to clone libbpf repo in GitHub and hack on the .github
> files), and what to watch out for that would be deeply appreciated.

One thing that you might want to explore is setting up a Kernel Patches
Daemon (KPD) instance. It can be  integrated with patchwork and github to
automatically collect submitted patches and open PRs to a github
repository for CI testing.

Here is a public repository:
https://github.com/facebookincubator/kernel-patches-daemon

I don't know the details about how to set it up, but I know a couple
of subsystems besides BPF are already using it.

>
> Thanks,
> Shung-Hsi
>
> 1: https://speakerdeck.com/shunghsiyu/bpf-in-stable-kernels