BPF CI for Stable

BPF CI for Stable

[Shung-Hsi Yu]

Hi,

Following up on the LSF/MM/BPF 2026 session "BPF in Stable Kernels, an
Update"[1]. Since the event, 6.1 coverage has also been added[2] (thanks
to Paul). With a collaborator on my side, it seem like a good time to
ask:

  Is it possible to get BPF CI for stable in its own repository under a
  GitHub organization (e.g. libbpf/stable-bpf-ci)?

Currently it lives as a personal repo that is fork of libbpf/libbpf[3].
As someone (I think it was Andrii) pointed out during the session, the
vast files currently in my fork are not needed at all. The GitHub Action
workflow only needs the .github/, plus the supplement files under ci/.

Technically BPF CI for stable can also simply be merged back to
libbpf/libbpf, but that felt less ideal because:
- kbuilder-debian container image currently doesn't work when testing
  stable kernels (I haven't look at reason of failure, might not be hard
  to fix)
- actions that tests stable kernel will be mixed with actions targeting
  libbpf itself 
- BPF CI for stable will have to contain version-specific tweaks (e.g.
  [4])

None of the above are fatal flaws that prevent libbf/libbpf to host
files that run BPF selftests on stable kernel, so I'd say having a
standalone repository is more of a personal preference.

OTOH we can also try kernel-patches/bpf-like approach and have a
kernel-patches/stable, providing proper, fully-fledged BPF CI for stable
that tests incoming patchset to stable (not sure if Daniel was asking
about this during the session). But I find this to be much more work.
The current approach of applying queued patches from stable-queue[5],
while comes with quite some delay between the moment patch is proposed
to stable mailing vs the moment said patch gets tested, seem like a
workable compromise.


Thanks,
Shung-Hsi

1: https://speakerdeck.com/shunghsiyu/bpf-in-stable-kernels-an-update
2: https://github.com/shunghsiyu/libbpf/pull/5
3: https://github.com/shunghsiyu/libbpf/
4: https://github.com/shunghsiyu/libbpf/tree/master/ci/diffs/6.1
5: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/

Re: BPF CI for Stable

[Ihor Solodrai]

On 6/5/26 12:45 AM, Shung-Hsi Yu wrote:
> Hi,
> 
> Following up on the LSF/MM/BPF 2026 session "BPF in Stable Kernels, an
> Update"[1]. Since the event, 6.1 coverage has also been added[2] (thanks
> to Paul). With a collaborator on my side, it seem like a good time to
> ask:
> 
>   Is it possible to get BPF CI for stable in its own repository under a
>   GitHub organization (e.g. libbpf/stable-bpf-ci)?

Hi Shung-Hsi,

I think a better home for the BPF CI targeting stable kernels is
kernel-patches org [1] for a couple of reasons. First, repos in the
kernel-patches org have access to hardware - generic runners provided
by Meta and s390x runners provided by IBM. Second reason is that this
org hosts source of truth repos for most of BPF CI code, even though a
big chunk of it lives in libbpf/ci.

Let me know if this makes sense, and I'll create a new repository and
grant you write permissions there.

> 
> Currently it lives as a personal repo that is fork of libbpf/libbpf[3].
> As someone (I think it was Andrii) pointed out during the session, the
> vast files currently in my fork are not needed at all. The GitHub Action
> workflow only needs the .github/, plus the supplement files under ci/.
> 
> Technically BPF CI for stable can also simply be merged back to
> libbpf/libbpf

No, that's a bad idea.

>, but that felt less ideal because:
> - kbuilder-debian container image currently doesn't work when testing
>   stable kernels (I haven't look at reason of failure, might not be hard
>   to fix)

Technically, it's not a hard requirement to use the container.
It makes the build jobs a little faster and a little more reliable, but
it was crafted for bpf-next testing, which may not be ideal for stable.
If necessary you could develop a separate "kbuilder-stable" container, but
it's up to you to decide whether it's needed.

> - actions that tests stable kernel will be mixed with actions targeting
>   libbpf itself 
> - BPF CI for stable will have to contain version-specific tweaks (e.g.
>   [4])

This is fine and expected. It is of course great to be able to reuse code,
but practically speaking, I think in this case it may be more work to
keep the common parts compatible with both stable and bpf-next workflows,
than maintaining two diverged CI trees.

The rule of thumb should be this: if you can use a libbpf/ci action, do
it. If there is something very specific to the workflow, keep it in place.
Don't be shy copy-pasting yaml fragments that work for you, it's fine.

> 
> None of the above are fatal flaws that prevent libbf/libbpf to host
> files that run BPF selftests on stable kernel, so I'd say having a
> standalone repository is more of a personal preference.

I agree that a standalone repository is the way to go.

> 
> OTOH we can also try kernel-patches/bpf-like approach and have a
> kernel-patches/stable, providing proper, fully-fledged BPF CI for stable
> that tests incoming patchset to stable (not sure if Daniel was asking
> about this during the session). But I find this to be much more work.

We don't have to do this work right away (or ever).

First let's set up the kernel-patches/stable (or whatever the name) with
the implementation you already have.

We can enhance and improve it when there are cycles and willingness to do
so in the future, 

> The current approach of applying queued patches from stable-queue[5],
> while comes with quite some delay between the moment patch is proposed
> to stable mailing vs the moment said patch gets tested, seem like a
> workable compromise.

This can be automated by the way.
Check out linux-next sync on current BPF CI, it's straightforward [2].
There is a job periodically syncing a branch, and the CI pipeline is
triggered on push.

[1] https://github.com/kernel-patches/
[2] https://github.com/kernel-patches/vmtest/blob/master/.github/workflows/linux-next-sync.yml


> 
> 
> Thanks,
> Shung-Hsi
> 
> 1: https://speakerdeck.com/shunghsiyu/bpf-in-stable-kernels-an-update
> 2: https://github.com/shunghsiyu/libbpf/pull/5
> 3: https://github.com/shunghsiyu/libbpf/
> 4: https://github.com/shunghsiyu/libbpf/tree/master/ci/diffs/6.1
> 5: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/

Re: BPF CI for Stable

[Shung-Hsi Yu]

On Fri, Jun 05, 2026 at 10:12:14AM -0700, Ihor Solodrai wrote:
> On 6/5/26 12:45 AM, Shung-Hsi Yu wrote:
[...]
> >   Is it possible to get BPF CI for stable in its own repository under a
> >   GitHub organization (e.g. libbpf/stable-bpf-ci)?
> 
> Hi Shung-Hsi,
> 
> I think a better home for the BPF CI targeting stable kernels is
> kernel-patches org [1] for a couple of reasons. First, repos in the
> kernel-patches org have access to hardware - generic runners provided
> by Meta and s390x runners provided by IBM. Second reason is that this
> org hosts source of truth repos for most of BPF CI code, even though a
> big chunk of it lives in libbpf/ci.
> 
> Let me know if this makes sense, and I'll create a new repository and
> grant you write permissions there.

Make sense, let's have it in kernel-patches org.

[...]
> > Technically BPF CI for stable can also simply be merged back to
> > libbpf/libbpf
> 
> No, that's a bad idea.
> 
> >, but that felt less ideal because:
> > - kbuilder-debian container image currently doesn't work when testing
> >   stable kernels (I haven't look at reason of failure, might not be hard
> >   to fix)
> 
> Technically, it's not a hard requirement to use the container.
> It makes the build jobs a little faster and a little more reliable, but
> it was crafted for bpf-next testing, which may not be ideal for stable.
> If necessary you could develop a separate "kbuilder-stable" container, but
> it's up to you to decide whether it's needed.

Understood, I'll stick with Ubuntu 24.04.

> > - actions that tests stable kernel will be mixed with actions targeting
> >   libbpf itself 
> > - BPF CI for stable will have to contain version-specific tweaks (e.g.
> >   [4])
> 
> This is fine and expected. It is of course great to be able to reuse code,
> but practically speaking, I think in this case it may be more work to
> keep the common parts compatible with both stable and bpf-next workflows,
> than maintaining two diverged CI trees.
> 
> The rule of thumb should be this: if you can use a libbpf/ci action, do
> it. If there is something very specific to the workflow, keep it in place.
> Don't be shy copy-pasting yaml fragments that work for you, it's fine.
[...]
> > OTOH we can also try kernel-patches/bpf-like approach and have a
> > kernel-patches/stable, providing proper, fully-fledged BPF CI for stable
> > that tests incoming patchset to stable (not sure if Daniel was asking
> > about this during the session). But I find this to be much more work.
> 
> We don't have to do this work right away (or ever).
> 
> First let's set up the kernel-patches/stable (or whatever the name) with
> the implementation you already have.
> 
> We can enhance and improve it when there are cycles and willingness to do
> so in the future, 

Sounds good to me, let's go with kernel-patches/stable. Thanks!

> > The current approach of applying queued patches from stable-queue[5],
> > while comes with quite some delay between the moment patch is proposed
> > to stable mailing vs the moment said patch gets tested, seem like a
> > workable compromise.
> 
> This can be automated by the way.
> Check out linux-next sync on current BPF CI, it's straightforward [2].
> There is a job periodically syncing a branch, and the CI pipeline is
> triggered on push.

I'll looking this, thanks for the pointer.

Shung-Hsi

> [1] https://github.com/kernel-patches/
> [2] https://github.com/kernel-patches/vmtest/blob/master/.github/workflows/linux-next-sync.yml
[...]

Re: BPF CI for Stable

[Ihor Solodrai]

On 6/10/26 1:18 AM, Shung-Hsi Yu wrote:
> On Fri, Jun 05, 2026 at 10:12:14AM -0700, Ihor Solodrai wrote:
>> On 6/5/26 12:45 AM, Shung-Hsi Yu wrote:
> [...]
>>>   Is it possible to get BPF CI for stable in its own repository under a
>>>   GitHub organization (e.g. libbpf/stable-bpf-ci)?
>>
>> Hi Shung-Hsi,
>>
>> I think a better home for the BPF CI targeting stable kernels is
>> kernel-patches org [1] for a couple of reasons. First, repos in the
>> kernel-patches org have access to hardware - generic runners provided
>> by Meta and s390x runners provided by IBM. Second reason is that this
>> org hosts source of truth repos for most of BPF CI code, even though a
>> big chunk of it lives in libbpf/ci.
>>
>> Let me know if this makes sense, and I'll create a new repository and
>> grant you write permissions there.
> 
> Make sense, let's have it in kernel-patches org.

Here you go: https://github.com/kernel-patches/linux-stable

I've sent you an invite.

I vibe-code basic sync with upstream. README describes how it works, but
I'll reiterate here for visibility on the list.

The repository has a main branch with only the CI code, test branches
for each tracked stable version, and Linux mainline mirror.

There is a sync job that runs daily. It:
  * syncs torvalds/master
  * then for each listed stable version
    * pulls in the tree
    * applies the patches from the queue repo
    * overlays .github/ and ci/ from the main branch, adding a commit
      * this is similar to what KPD does for bpf-next, but without patchwork and KPD
    * pushes the result to linux-X.Y.y branch

The testing workflow (test.yml), which currently doesn't do anything
interesting, is triggered on push to linux-* branches.

The scripts are straightforward, though a little verbose as is typical
for vibe code. Feel free to edit it as you see fit, or even scrap it.
I wanted to have something working before handing over the repo to you.

This way we have an automated sync + trigger running on schedule right
out of the box. With that, you can add code to test.yml to actually 
build and run tests.

I think you can copy most of the stuff from your libbpf fork, but it'll
have to be tested and debugged of course.

And the best part is you can make use of the hardware available to the
kernel-patches org: beefy bare-metal machines for testing, s390x runners,
codebuild. Oh, and AI reviews of course, if you want to experiment with that.

My understanding is you'll be maintaining the stable part of the CI, but
feel free to reach out if you need help.

Have fun!

> 
> [...]
>>> Technically BPF CI for stable can also simply be merged back to
>>> libbpf/libbpf
>>
>> No, that's a bad idea.
>>
>>> , but that felt less ideal because:
>>> - kbuilder-debian container image currently doesn't work when testing
>>>   stable kernels (I haven't look at reason of failure, might not be hard
>>>   to fix)
>>
>> Technically, it's not a hard requirement to use the container.
>> It makes the build jobs a little faster and a little more reliable, but
>> it was crafted for bpf-next testing, which may not be ideal for stable.
>> If necessary you could develop a separate "kbuilder-stable" container, but
>> it's up to you to decide whether it's needed.
> 
> Understood, I'll stick with Ubuntu 24.04.
> 
>>> - actions that tests stable kernel will be mixed with actions targeting
>>>   libbpf itself 
>>> - BPF CI for stable will have to contain version-specific tweaks (e.g.
>>>   [4])
>>
>> This is fine and expected. It is of course great to be able to reuse code,
>> but practically speaking, I think in this case it may be more work to
>> keep the common parts compatible with both stable and bpf-next workflows,
>> than maintaining two diverged CI trees.
>>
>> The rule of thumb should be this: if you can use a libbpf/ci action, do
>> it. If there is something very specific to the workflow, keep it in place.
>> Don't be shy copy-pasting yaml fragments that work for you, it's fine.
> [...]
>>> OTOH we can also try kernel-patches/bpf-like approach and have a
>>> kernel-patches/stable, providing proper, fully-fledged BPF CI for stable
>>> that tests incoming patchset to stable (not sure if Daniel was asking
>>> about this during the session). But I find this to be much more work.
>>
>> We don't have to do this work right away (or ever).
>>
>> First let's set up the kernel-patches/stable (or whatever the name) with
>> the implementation you already have.
>>
>> We can enhance and improve it when there are cycles and willingness to do
>> so in the future, 
> 
> Sounds good to me, let's go with kernel-patches/stable. Thanks!
> 
>>> The current approach of applying queued patches from stable-queue[5],
>>> while comes with quite some delay between the moment patch is proposed
>>> to stable mailing vs the moment said patch gets tested, seem like a
>>> workable compromise.
>>
>> This can be automated by the way.
>> Check out linux-next sync on current BPF CI, it's straightforward [2].
>> There is a job periodically syncing a branch, and the CI pipeline is
>> triggered on push.
> 
> I'll looking this, thanks for the pointer.
> 
> Shung-Hsi
> 
>> [1] https://github.com/kernel-patches/
>> [2] https://github.com/kernel-patches/vmtest/blob/master/.github/workflows/linux-next-sync.yml
> [...]