XDP packet queueing and scheduling capabilities

XDP packet queueing and scheduling capabilities

[team lnx]

Hello everyone,

https://lwn.net/Articles/901046/
would this be resumed any time soon this year ?

Regards,
teamlnx

Re: XDP packet queueing and scheduling capabilities

[Toke Høiland-Jørgensen]

team lnx <teamlnxi8@gmail.com> writes:

> Hello everyone,
>
> https://lwn.net/Articles/901046/
> would this be resumed any time soon this year ?

The latest version of those patches are here:
https://git.kernel.org/pub/scm/linux/kernel/git/toke/linux.git/log/?h=xdp-queueing-08

They are a bit rough around the edges, and there are some missing pieces
still; I'm working on getting a userspace/BPF implementation to the
point where those can be filled in. After that, and some thorough
testing, it should be possible to post another revision.

What's your interest in this? I.e., what do you want to use them for?
Would be useful to know so I can make sure this is covered :)

-Toke

Re: XDP packet queueing and scheduling capabilities

[Alexandre Cassen]

Hi Toke,

here is a target with lot of interest for it : www.gtp-guard.org

When dealing with mobile network data-plane, at some point you have 
ordering issues and shaping needs, so queuing is truly needed.
Alternatively ones can  implement PIFO or others built on AF_XDP but if 
dedicated bpf map covers the use-case, would be nice.

Watching at your LPC 2022 presentation, at the end, discussions where 
made around using existing Qdisc kernel framework and find
a way to share the path between XDP and netstack. Is it a target for 
adding PIFO, or more generally getting queuing support for XDP ?

regs,
Alexandre


On 13/02/2024 14:03, Toke Høiland-Jørgensen wrote:
> team lnx <teamlnxi8@gmail.com> writes:
> 
>> Hello everyone,
>>
>> https://lwn.net/Articles/901046/
>> would this be resumed any time soon this year ?
> 
> The latest version of those patches are here:
> https://git.kernel.org/pub/scm/linux/kernel/git/toke/linux.git/log/?h=xdp-queueing-08
> 
> They are a bit rough around the edges, and there are some missing pieces
> still; I'm working on getting a userspace/BPF implementation to the
> point where those can be filled in. After that, and some thorough
> testing, it should be possible to post another revision.
> 
> What's your interest in this? I.e., what do you want to use them for?
> Would be useful to know so I can make sure this is covered :)
> 
> -Toke
> 
> 

Re: XDP packet queueing and scheduling capabilities

[Dave Taht]

On Tue, Feb 13, 2024 at 9:33 AM Alexandre Cassen <acassen@gmail.com> wrote:
>
> Hi Toke,
>
> here is a target with lot of interest for it : www.gtp-guard.org

Occasionally I point at the complementary use case that is libreqos. I
would love to make a dent in the 5G world.

> When dealing with mobile network data-plane, at some point you have
> ordering issues and shaping needs, so queuing is truly needed.

yep. But see libreqos.

> Alternatively ones can  implement PIFO or others built on AF_XDP but if
> dedicated bpf map covers the use-case, would be nice.
>
> Watching at your LPC 2022 presentation, at the end, discussions where
> made around using existing Qdisc kernel framework and find
> a way to share the path between XDP and netstack. Is it a target for
> adding PIFO, or more generally getting queuing support for XDP ?
>
> regs,
> Alexandre
>
>
> On 13/02/2024 14:03, Toke Høiland-Jørgensen wrote:
> > team lnx <teamlnxi8@gmail.com> writes:
> >
> >> Hello everyone,
> >>
> >> https://lwn.net/Articles/901046/
> >> would this be resumed any time soon this year ?
> >
> > The latest version of those patches are here:
> > https://git.kernel.org/pub/scm/linux/kernel/git/toke/linux.git/log/?h=xdp-queueing-08
> >
> > They are a bit rough around the edges, and there are some missing pieces
> > still; I'm working on getting a userspace/BPF implementation to the
> > point where those can be filled in. After that, and some thorough
> > testing, it should be possible to post another revision.
> >
> > What's your interest in this? I.e., what do you want to use them for?
> > Would be useful to know so I can make sure this is covered :)
> >
> > -Toke
> >
> >
>


-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: XDP packet queueing and scheduling capabilities

[Toke Høiland-Jørgensen]

Alexandre Cassen <acassen@gmail.com> writes:

> Hi Toke,
>
> here is a target with lot of interest for it : www.gtp-guard.org

Ah, seems like a cool project; thanks for the pointer!

> When dealing with mobile network data-plane, at some point you have
> ordering issues and shaping needs, so queuing is truly needed.
> Alternatively ones can implement PIFO or others built on AF_XDP but if
> dedicated bpf map covers the use-case, would be nice.

Right, I'm kinda thinking about the map type that is part of the XDP
queueing series as a general-purpose packet buffer that will enable all
kinds of features, not just queueing for forwarding. Whether it'll end
up being the PIFO map type, or a simpler one, I'm less certain about.
The PIFO abstraction may end up being too special-purpose. Opinions
welcome!

> Watching at your LPC 2022 presentation, at the end, discussions where
> made around using existing Qdisc kernel framework and find a way to
> share the path between XDP and netstack. Is it a target for adding
> PIFO, or more generally getting queuing support for XDP ?

I don't personally consider it feasible to have forwarded XDP frames
share the qdisc path. The presence of an sk_buff is too simply too
fundamentally baked into the qdisc layer. I'm hoping that the addition
of an eBPF-based qdisc will instead make it feasible to share queueing
algorithm code between the two layers (and even build forwarding paths
that can handle both by having the different BPF implementations
cooperate). And of course co-existence between XDP and stack forwarding
is important to avoid starvation, but that is already an issue for XDP
forwarding today.

-Toke

Re: XDP packet queueing and scheduling capabilities

[Alexandre Cassen]

On 13/02/2024 17:07, Toke Høiland-Jørgensen wrote:
> Alexandre Cassen <acassen@gmail.com> writes:
> 
>> Hi Toke,
>>
>> here is a target with lot of interest for it : www.gtp-guard.org
> 
> Ah, seems like a cool project; thanks for the pointer!

Well, right now focus is much more on code rather than documentation :D

but doc will happen at one time. In a hurry right now adding support to 
XDP routing data-path that will deal with GTP-U decap on one side and 
PPPoE encap the other side, and reciprocally. This is one use-case to 
make mobile access network converge to existing isp access network 
infrastructure (not L2TP since it just create scalling issues for bunch 
of customers). another one will be SRv6 later on... Anyway, having fun 
hacking around on it :)


>> When dealing with mobile network data-plane, at some point you have
>> ordering issues and shaping needs, so queuing is truly needed.
>> Alternatively ones can implement PIFO or others built on AF_XDP but if
>> dedicated bpf map covers the use-case, would be nice.
> 
> Right, I'm kinda thinking about the map type that is part of the XDP
> queueing series as a general-purpose packet buffer that will enable all
> kinds of features, not just queueing for forwarding. Whether it'll end
> up being the PIFO map type, or a simpler one, I'm less certain about.
> The PIFO abstraction may end up being too special-purpose. Opinions
> welcome!


Read the code last night, that is exactly what is required here, 
bpf_timer trick is fun.

(IP fragmentation in gtp-guard is using bpf_timer for ephemeral id 
tracking handling, this is just here re-ordering pop-up... on 
constructor vendors equipements where normal path is using diffrent 
processing-unit than frag handling this happen !)

Agreed on your point, having a general purpose map type enable 
inheritage option for specific feature instead of multiplying meaningful 
map type for each feature/use-case.


>> Watching at your LPC 2022 presentation, at the end, discussions where
>> made around using existing Qdisc kernel framework and find a way to
>> share the path between XDP and netstack. Is it a target for adding
>> PIFO, or more generally getting queuing support for XDP ?
> 
> I don't personally consider it feasible to have forwarded XDP frames
> share the qdisc path. The presence of an sk_buff is too simply too
> fundamentally baked into the qdisc layer. I'm hoping that the addition
> of an eBPF-based qdisc will instead make it feasible to share queueing
> algorithm code between the two layers (and even build forwarding paths
> that can handle both by having the different BPF implementations
> cooperate). And of course co-existence between XDP and stack forwarding
> is important to avoid starvation, but that is already an issue for XDP
> forwarding today.

Agreed too, eBPF backed Qdisc 'proxy' sounds great idea. latency 
forecast impact ?


- Alexandre

Re: XDP packet queueing and scheduling capabilities

[Toke Høiland-Jørgensen]

Alexandre Cassen <acassen@gmail.com> writes:

>>> Watching at your LPC 2022 presentation, at the end, discussions where
>>> made around using existing Qdisc kernel framework and find a way to
>>> share the path between XDP and netstack. Is it a target for adding
>>> PIFO, or more generally getting queuing support for XDP ?
>> 
>> I don't personally consider it feasible to have forwarded XDP frames
>> share the qdisc path. The presence of an sk_buff is too simply too
>> fundamentally baked into the qdisc layer. I'm hoping that the addition
>> of an eBPF-based qdisc will instead make it feasible to share queueing
>> algorithm code between the two layers (and even build forwarding paths
>> that can handle both by having the different BPF implementations
>> cooperate). And of course co-existence between XDP and stack forwarding
>> is important to avoid starvation, but that is already an issue for XDP
>> forwarding today.
>
> Agreed too, eBPF backed Qdisc 'proxy' sounds great idea. latency 
> forecast impact ?

Of writing the qdisc in eBPF instead of as a regular kernel module?
Negligible; the overhead shown in the last posting of those patches[0]
is not nil, but it seems there's a path to getting rid of it (teaching
BPF how to put skbs directly into list/rbtree data structures instead of
having to allocate a container for it).

The latency impact of mixing XDP and qdisc traffic? Dunno, depends on
the traffic and the algorithms managing it. I don't think there's
anything inherent in the BPF side of things that would impact latency
(it's all just code in the end), as long as we make sure that the APIs
and primitives can express all the things we need to effectively
implement good algorithms. Which is why I'm asking for examples of use
cases :)

-Toke

[0] https://lore.kernel.org/r/cover.1705432850.git.amery.hung@bytedance.com

Re: XDP packet queueing and scheduling capabilities

[Marcus Wichelmann]

Am 13.02.24 um 14:03 schrieb Toke Høiland-Jørgensen:
> What's your interest in this? I.e., what do you want to use them for?
> Would be useful to know so I can make sure this is covered :)

Hi, just wanting to let you know that we're also still very interested 
in this for our cloud-provider use case.

We use XDP to route packets to and from VMs and want to also apply 
firewall rules on the way. For this, we need to buffer/delay reordered 
IP fragments that arrived before the first fragment until we have seen 
the required L4 header.

Having a packet queue for this, like in your patch, that can also be 
dequeued again, would definitively be helpful for that.

We'd also like to implement shaping in the future, so this is also very 
interesting for us.

-- 
Best regards,
Marcus Wichelmann
SDN

______________________________

Hetzner Cloud GmbH
Feringastraße 12A
85774 Unterföhring
Germany

Phone: +49 89 381690 150
E-Mail: marcus.wichelmann@hetzner-cloud.de

Handelsregister München HRB 226782
Geschäftsführer: Sebastian Färber, Markus Kalmuk

------------------
For information on the processing of your personal
data in the context of this contact, please see
https://hetzner-cloud.de/datenschutz
------------------

Re: XDP packet queueing and scheduling capabilities

[Toke Høiland-Jørgensen]

Marcus Wichelmann <marcus.wichelmann@hetzner-cloud.de> writes:

> Am 13.02.24 um 14:03 schrieb Toke Høiland-Jørgensen:
>> What's your interest in this? I.e., what do you want to use them for?
>> Would be useful to know so I can make sure this is covered :)
>
> Hi, just wanting to let you know that we're also still very interested 
> in this for our cloud-provider use case.
>
> We use XDP to route packets to and from VMs and want to also apply 
> firewall rules on the way. For this, we need to buffer/delay reordered 
> IP fragments that arrived before the first fragment until we have seen 
> the required L4 header.
>
> Having a packet queue for this, like in your patch, that can also be 
> dequeued again, would definitively be helpful for that.
>
> We'd also like to implement shaping in the future, so this is also very 
> interesting for us.

Great, thanks for the feedback! Hadn't thought about frag reassembly,
but that should definitely be possible as well; will keep that in mind! :)

-Toke

Re: XDP packet queueing and scheduling capabilities

[team lnx]

I see a lot of discussion happening already, one use case I am looking
at is shaping of traffic which is discussed already also to control
traffic
flow (to avoid packet drops) if an Rx interface slows down on Tx path.


On Tue, Feb 13, 2024 at 5:03 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> team lnx <teamlnxi8@gmail.com> writes:
>
> > Hello everyone,
> >
> > https://lwn.net/Articles/901046/
> > would this be resumed any time soon this year ?
>
> The latest version of those patches are here:
> https://git.kernel.org/pub/scm/linux/kernel/git/toke/linux.git/log/?h=xdp-queueing-08
>
> They are a bit rough around the edges, and there are some missing pieces
> still; I'm working on getting a userspace/BPF implementation to the
> point where those can be filled in. After that, and some thorough
> testing, it should be possible to post another revision.
>
> What's your interest in this? I.e., what do you want to use them for?
> Would be useful to know so I can make sure this is covered :)
>
> -Toke
>