bpf-next experiment

bpf-next experiment

[Alexei Starovoitov]

Hi All,

Couple years ago folks suggested that bpf-next should be
a separate pull request to increase subsystem visibility.
Back then we rejected the idea since many networking related
changes required bpf core changes. Things are different now.
bpf kfuncs can be added independently by various subsystems,
verifier additions are mainly driven by sched-ext,
so it's time to give it a shot. It's an experiment.
If things don't work out as expected we will go back to
the old model of feeding bpf trees through net/net-next trees.

So here is the plan:

1. bpf fixes go directly to Linus (skipping net tree) and
net/bpf trees are fast forwarded afterwards as usual.

2. Non-networking bpf commits land in bpf-next/master branch.
It will form bpf-next PR during the merge window.

3. Networking related commits (like XDP) land in bpf-next/net branch.
They will be PR-ed to net-next and ffwded from net-next
as we do today. All these patches will get to mainline
via net-next PR.

4. bpf-next/master and bpf-next/net branches are manually
merged into bpf-next/for-next branch.
This step achieves two objectives:
- bpf maintainers watch for conflicts between /master and /net
- Stephen Rothwell continues taking /for-next branch into linux-next
as usual

bpf CI will run tests against 4 trees (instead of 2):
bpf, bpf-next/master, bpf-next/net, bpf-next/for-next.
This is wip. Watch for more "Checks" in patchwork.

By the merge window in September we will reassess
the situation and if it's still worth doing we will
proceed with PR formed from bpf-next/master.
If not, we will PR bpf-next/master into net-next and
call it a failed experiment.

We feel that there are more positives to this process
than headaches, so fingers crossed.

Re: bpf-next experiment

[Jakub Kicinski]

On Wed, 14 Aug 2024 12:32:00 -0700 Alexei Starovoitov wrote:
> Couple years ago folks suggested that bpf-next should be
> a separate pull request to increase subsystem visibility.
> Back then we rejected the idea since many networking related
> changes required bpf core changes. Things are different now.
> bpf kfuncs can be added independently by various subsystems,
> verifier additions are mainly driven by sched-ext,
> so it's time to give it a shot. It's an experiment.
> If things don't work out as expected we will go back to
> the old model of feeding bpf trees through net/net-next trees.

Excellent, fingers crossed :)

Re: bpf-next experiment

[Toke Høiland-Jørgensen]

Alexei Starovoitov <alexei.starovoitov@gmail.com> writes:

> 2. Non-networking bpf commits land in bpf-next/master branch.
> It will form bpf-next PR during the merge window.
>
> 3. Networking related commits (like XDP) land in bpf-next/net branch.
> They will be PR-ed to net-next and ffwded from net-next
> as we do today. All these patches will get to mainline
> via net-next PR.

So from a submitter PoV, someone submitting an XDP-related patch (say),
should base this off of bpf-next/net, and tag it as bpf-next in the
subject? Or should it also be tagged as bpf-next/net?

-Toke

Re: bpf-next experiment

[Alexei Starovoitov]

On Thu, Aug 15, 2024 at 12:15 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Alexei Starovoitov <alexei.starovoitov@gmail.com> writes:
>
> > 2. Non-networking bpf commits land in bpf-next/master branch.
> > It will form bpf-next PR during the merge window.
> >
> > 3. Networking related commits (like XDP) land in bpf-next/net branch.
> > They will be PR-ed to net-next and ffwded from net-next
> > as we do today. All these patches will get to mainline
> > via net-next PR.
>
> So from a submitter PoV, someone submitting an XDP-related patch (say),
> should base this off of bpf-next/net, and tag it as bpf-next in the
> subject? Or should it also be tagged as bpf-next/net?

This part we're still figuring out.
There are few considerations...
it's certainly easier for bpf CI when the patch set
is tagged with [PATCH bpf-next/net] then CI won't try
to find the branch,
but it will take a long time to teach all contributors
to tag things differently,
so CI would need to get smart anyway and would need
to apply to /master, run tests, apply to /net, run tests too.
Currently when there is no tag CI attempts to apply to bpf.git,
if it fails, it tries to apply to bpf-next/master and only
then reports back "merge conflict".
It will do this for bpf, bpf-next/master, bpf-next/net now.

Sometimes devs think that the patch is a fix, so they
tag it with [PATCH bpf], but it might not be,
and after review we apply it to bpf-next instead.

So tree/branch to base patches off and tag don't
matter that much.
So I hope, in practice, we won't need to teach all
developers about new tag and about new branch.
We certainly won't be asking to resubmit if patches
are not tagged one way or the other,
but if you want to help CI and tell maintainers
your preferences then certainly start using
[PATCH bpf-next] and [PATCH bpf-next/net] when necessary.
Or don't :) and instead help us make CI smarter :)

Re: bpf-next experiment

[Simon Horman]

On Wed, Aug 14, 2024 at 12:32:00PM -0700, Alexei Starovoitov wrote:
> Hi All,
> 
> Couple years ago folks suggested that bpf-next should be
> a separate pull request to increase subsystem visibility.
> Back then we rejected the idea since many networking related
> changes required bpf core changes. Things are different now.
> bpf kfuncs can be added independently by various subsystems,
> verifier additions are mainly driven by sched-ext,
> so it's time to give it a shot. It's an experiment.
> If things don't work out as expected we will go back to
> the old model of feeding bpf trees through net/net-next trees.
> 
> So here is the plan:
> 
> 1. bpf fixes go directly to Linus (skipping net tree) and
> net/bpf trees are fast forwarded afterwards as usual.
> 
> 2. Non-networking bpf commits land in bpf-next/master branch.
> It will form bpf-next PR during the merge window.
> 
> 3. Networking related commits (like XDP) land in bpf-next/net branch.
> They will be PR-ed to net-next and ffwded from net-next
> as we do today. All these patches will get to mainline
> via net-next PR.

Hi Alexei,

Nice plan :)

I wonder if, bpf-next/net-next might be a more intuitive name, as the
proposed branch is closely related to net-next.

OTOH, mabey one '-next', as per your proposal, is enough :)

> 
> 4. bpf-next/master and bpf-next/net branches are manually
> merged into bpf-next/for-next branch.
> This step achieves two objectives:
> - bpf maintainers watch for conflicts between /master and /net
> - Stephen Rothwell continues taking /for-next branch into linux-next
> as usual
> 
> bpf CI will run tests against 4 trees (instead of 2):
> bpf, bpf-next/master, bpf-next/net, bpf-next/for-next.
> This is wip. Watch for more "Checks" in patchwork.
> 
> By the merge window in September we will reassess
> the situation and if it's still worth doing we will
> proceed with PR formed from bpf-next/master.
> If not, we will PR bpf-next/master into net-next and
> call it a failed experiment.
> 
> We feel that there are more positives to this process
> than headaches, so fingers crossed.
> 

Re: bpf-next experiment

[Andrii Nakryiko]

On Thu, Aug 15, 2024 at 8:54 AM Simon Horman <horms@kernel.org> wrote:
>
> On Wed, Aug 14, 2024 at 12:32:00PM -0700, Alexei Starovoitov wrote:
> > Hi All,
> >
> > Couple years ago folks suggested that bpf-next should be
> > a separate pull request to increase subsystem visibility.
> > Back then we rejected the idea since many networking related
> > changes required bpf core changes. Things are different now.
> > bpf kfuncs can be added independently by various subsystems,
> > verifier additions are mainly driven by sched-ext,
> > so it's time to give it a shot. It's an experiment.
> > If things don't work out as expected we will go back to
> > the old model of feeding bpf trees through net/net-next trees.
> >
> > So here is the plan:
> >
> > 1. bpf fixes go directly to Linus (skipping net tree) and
> > net/bpf trees are fast forwarded afterwards as usual.
> >
> > 2. Non-networking bpf commits land in bpf-next/master branch.
> > It will form bpf-next PR during the merge window.
> >
> > 3. Networking related commits (like XDP) land in bpf-next/net branch.
> > They will be PR-ed to net-next and ffwded from net-next
> > as we do today. All these patches will get to mainline
> > via net-next PR.
>
> Hi Alexei,
>
> Nice plan :)
>
> I wonder if, bpf-next/net-next might be a more intuitive name, as the
> proposed branch is closely related to net-next.
>
> OTOH, mabey one '-next', as per your proposal, is enough :)
>
> >
> > 4. bpf-next/master and bpf-next/net branches are manually
> > merged into bpf-next/for-next branch.
> > This step achieves two objectives:
> > - bpf maintainers watch for conflicts between /master and /net
> > - Stephen Rothwell continues taking /for-next branch into linux-next
> > as usual
> >
> > bpf CI will run tests against 4 trees (instead of 2):
> > bpf, bpf-next/master, bpf-next/net, bpf-next/for-next.

BPF CI has been set up to recognize "bpf-net" as a marker for
bpf-next/net branch. We can update that, of course, but just FYI what
is currently recognized.

> > This is wip. Watch for more "Checks" in patchwork.
> >
> > By the merge window in September we will reassess
> > the situation and if it's still worth doing we will
> > proceed with PR formed from bpf-next/master.
> > If not, we will PR bpf-next/master into net-next and
> > call it a failed experiment.
> >
> > We feel that there are more positives to this process
> > than headaches, so fingers crossed.
> >

Re: bpf-next experiment

[Toke Høiland-Jørgensen]

Alexei Starovoitov <alexei.starovoitov@gmail.com> writes:

> On Thu, Aug 15, 2024 at 12:15 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>>
>> Alexei Starovoitov <alexei.starovoitov@gmail.com> writes:
>>
>> > 2. Non-networking bpf commits land in bpf-next/master branch.
>> > It will form bpf-next PR during the merge window.
>> >
>> > 3. Networking related commits (like XDP) land in bpf-next/net branch.
>> > They will be PR-ed to net-next and ffwded from net-next
>> > as we do today. All these patches will get to mainline
>> > via net-next PR.
>>
>> So from a submitter PoV, someone submitting an XDP-related patch (say),
>> should base this off of bpf-next/net, and tag it as bpf-next in the
>> subject? Or should it also be tagged as bpf-next/net?
>
> This part we're still figuring out.
> There are few considerations...
> it's certainly easier for bpf CI when the patch set
> is tagged with [PATCH bpf-next/net] then CI won't try
> to find the branch,
> but it will take a long time to teach all contributors
> to tag things differently,
> so CI would need to get smart anyway and would need
> to apply to /master, run tests, apply to /net, run tests too.
> Currently when there is no tag CI attempts to apply to bpf.git,
> if it fails, it tries to apply to bpf-next/master and only
> then reports back "merge conflict".
> It will do this for bpf, bpf-next/master, bpf-next/net now.
>
> Sometimes devs think that the patch is a fix, so they
> tag it with [PATCH bpf], but it might not be,
> and after review we apply it to bpf-next instead.
>
> So tree/branch to base patches off and tag don't
> matter that much.
> So I hope, in practice, we won't need to teach all
> developers about new tag and about new branch.
> We certainly won't be asking to resubmit if patches
> are not tagged one way or the other,
> but if you want to help CI and tell maintainers
> your preferences then certainly start using
> [PATCH bpf-next] and [PATCH bpf-next/net] when necessary.
> Or don't :) and instead help us make CI smarter :)

Alright, sounds good, thanks for clarifying! And exciting change in
general :)

-Toke