XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

[Sophia Yoo]

Hello,

I am relatively new to BPF/XDP, and I’m currently trying to use a TC
egress program to redirect a packet back to ingress, where I have an
XDP program attached. When the XDP program is attached in generic mode
(XDP_SKB_MODE), the redirect occurs properly and the packet is seen on
the ingress of the interface, but when the program is attached in
native mode (XDP_DRV_MODE), the packet never redirects and is just
seen exiting the interface, even though the return code of the
redirect function is “success”.

I've read previous threads that seemed related to this issue, but
could not find the answer to my question. I’m trying to understand the
difference in behavior between generic versus driver mode, and how to
properly redirect the packet in driver mode (if this is possible). I
would really appreciate any insight. Here are more details on my
setup. When my TC program sees an outgoing packet, it should grab it,
make a clone, and redirect the clone to the ingress direction XDP
program using bpf_clone_redirect(skb, IFINDEX, BPF_F_INGRESS); When
the XDP program is attached in generic mode the packets get redirected
successfully and can be seen on the XDP ingress interface (and on
tcpdump etc.). However, when the XDP program is attached in xdp native
mode (with the NIC provided driver), the packet does not get
redirected and simply exits (in this case since I am using the
bpf_clone_redirect() helper, two packets are seen exiting). I
understand that XDP native mode occurs sooner in the packet processing
pipeline than XDP generic mode, and that in native mode there is no
skb, but I’m having some trouble understanding exactly what is
happening under the hood and how to “fix” it, if possible. My NIC
driver is Intel xgbe. I am using bcc to attach and load my xdp and tc
programs.

I read in this thread
https://marc.info/?l=xdp-newbies&m=162344478620095&w=2 that Netronome
NICs do not support XDP_REDIRECT, but it seems from this Feb’21 thread
https://marc.info/?l=xdp-newbies&m=161362777730255&w=2 that Intel xgbe
drivers should support it, although there is additional discussion
about potential buggy behavior in a May’21 thread
https://marc.info/?l=xdp-newbies&m=162024487722215&w=2.

These threads regarding XDP_REDIRECT support in relation to xdp frags
also seemed relevant, but since I am new to this area I had some
trouble understanding the significance for solving the issue I’m
working on. https://marc.info/?l=linux-netdev&m=164275984217304&w=2

https://lore.kernel.org/bpf/YgeUFb4LIP7VfeL9@lore-desk/T/

I would appreciate any insight or advice on understanding

the difference between what is happening in xdp generic versus xdp
driver modes during a packet redirect from TC egress to XDP ingress

whether it is possible to perform a redirect in xdp driver mode with
an Intel xgbe driver

if yes to 2, how to successfully redirect a packet from TC egress
program to XDP ingress program attached in xdp driver mode

Thank you very much for your time.

Best,

Sophia Yoo

Re: XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

[Toke Høiland-Jørgensen]

Sophia Yoo <sy6@princeton.edu> writes:

> Hello,
>
> I am relatively new to BPF/XDP, and I’m currently trying to use a TC
> egress program to redirect a packet back to ingress, where I have an
> XDP program attached. When the XDP program is attached in generic mode
> (XDP_SKB_MODE), the redirect occurs properly and the packet is seen on
> the ingress of the interface, but when the program is attached in
> native mode (XDP_DRV_MODE), the packet never redirects and is just
> seen exiting the interface, even though the return code of the
> redirect function is “success”.

The difference between XDP generic and driver mode is that the generic
mode is hooked into the core networking stack whereas driver mode runs
in the driver as the very first thing after packets are physically
received from the network.

This also explains why what you're trying to do doesn't work: the TC
hook runs in the core networking stack, and when it does a redirect, the
packet does not actually pass through the network driver, it appears
further up in the stack, where only the generic XDP hook will see it.

So this has nothing to do with the support in the driver; XDP is simply
not suitable for what you're trying to do. Why are you trying to do this
in the first place? I.e., what's the higher-level use case here?

-Toke

Re: XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

[Sophia Yoo]

On Mon, Feb 14, 2022 at 1:28 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Sophia Yoo <sy6@princeton.edu> writes:
>
> > Hello,
> >
> > I am relatively new to BPF/XDP, and I’m currently trying to use a TC
> > egress program to redirect a packet back to ingress, where I have an
> > XDP program attached. When the XDP program is attached in generic mode
> > (XDP_SKB_MODE), the redirect occurs properly and the packet is seen on
> > the ingress of the interface, but when the program is attached in
> > native mode (XDP_DRV_MODE), the packet never redirects and is just
> > seen exiting the interface, even though the return code of the
> > redirect function is “success”.
>
> The difference between XDP generic and driver mode is that the generic
> mode is hooked into the core networking stack whereas driver mode runs
> in the driver as the very first thing after packets are physically
> received from the network.
>
> This also explains why what you're trying to do doesn't work: the TC
> hook runs in the core networking stack, and when it does a redirect, the
> packet does not actually pass through the network driver, it appears
> further up in the stack, where only the generic XDP hook will see it.
>
Thanks for your quick response! I see, I didn't realize that the
packet was being redirected before it passed through the network
driver. However, in this scenario, shouldn't the packets "disappear"
after being redirected from tc egress to xdp ingress? In other words,
even if the packet doesn't actually show up on the xdp ingress hook in
driver mode, it should at least not be seen exiting on the egress path
(e.g., on tcpdump)?

> So this has nothing to do with the support in the driver; XDP is simply
> not suitable for what you're trying to do. Why are you trying to do this
> in the first place? I.e., what's the higher-level use case here?

The higher-level design goal (simplified to what's relevant) is that I
am trying to perform a TCP 3WHS between the kernel network stack and
my xdp ingress/tc egress hooks, where some trigger on xdp ingress
sends a SYN packet to the network stack, and when the network stack
returns a SYN-ACK packet the egress program redirects the packet to
the ingress interface, where the XDP program converts the packet to an
ACK packet, then completing the handshake with the network stack.

Initially, I had wanted to use an XDP program instead of a TC program
on the egress path to do the redirect to ingress (which I believe
would bypass the current issue if both ingress and egress xdp programs
were attached in driver mode), but as of now it seems that redirect
from egress to ingress simply isn't supported in XDP, which is why I'm
using a TC program.

Thanks for the help!
Sophia

>
> -Toke
>

Re: XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

[Toke Høiland-Jørgensen]

Sophia Yoo <sy6@princeton.edu> writes:

> On Mon, Feb 14, 2022 at 1:28 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>>
>> Sophia Yoo <sy6@princeton.edu> writes:
>>
>> > Hello,
>> >
>> > I am relatively new to BPF/XDP, and I’m currently trying to use a TC
>> > egress program to redirect a packet back to ingress, where I have an
>> > XDP program attached. When the XDP program is attached in generic mode
>> > (XDP_SKB_MODE), the redirect occurs properly and the packet is seen on
>> > the ingress of the interface, but when the program is attached in
>> > native mode (XDP_DRV_MODE), the packet never redirects and is just
>> > seen exiting the interface, even though the return code of the
>> > redirect function is “success”.
>>
>> The difference between XDP generic and driver mode is that the generic
>> mode is hooked into the core networking stack whereas driver mode runs
>> in the driver as the very first thing after packets are physically
>> received from the network.
>>
>> This also explains why what you're trying to do doesn't work: the TC
>> hook runs in the core networking stack, and when it does a redirect, the
>> packet does not actually pass through the network driver, it appears
>> further up in the stack, where only the generic XDP hook will see it.
>>
> Thanks for your quick response! I see, I didn't realize that the
> packet was being redirected before it passed through the network
> driver. However, in this scenario, shouldn't the packets "disappear"
> after being redirected from tc egress to xdp ingress? In other words,
> even if the packet doesn't actually show up on the xdp ingress hook in
> driver mode, it should at least not be seen exiting on the egress path
> (e.g., on tcpdump)?

Is it possible the packet is looping? I.e., it goes out eth0, you
redirect it to egress on eth1, it goes all the way up the stack and the
kernel ends up forwarding it back out eth0?

>> So this has nothing to do with the support in the driver; XDP is simply
>> not suitable for what you're trying to do. Why are you trying to do this
>> in the first place? I.e., what's the higher-level use case here?
>
> The higher-level design goal (simplified to what's relevant) is that I
> am trying to perform a TCP 3WHS between the kernel network stack and
> my xdp ingress/tc egress hooks, where some trigger on xdp ingress
> sends a SYN packet to the network stack, and when the network stack
> returns a SYN-ACK packet the egress program redirects the packet to
> the ingress interface, where the XDP program converts the packet to an
> ACK packet, then completing the handshake with the network stack.

It's trying to handshake with itself (on the same box)? Why? Is this a
real use case, or a test for something else? If the latter, have you
considered using a veth pair in going to a different namespace (that
gets the "direction" right)?

> Initially, I had wanted to use an XDP program instead of a TC program
> on the egress path to do the redirect to ingress (which I believe
> would bypass the current issue if both ingress and egress xdp programs
> were attached in driver mode), but as of now it seems that redirect
> from egress to ingress simply isn't supported in XDP, which is why I'm
> using a TC program.

Well, there's also an ingress hook for TC you could look at (or just
stay with generic XDP).

-Toke

Re: XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

[Sophia Yoo]

> Is it possible the packet is looping? I.e., it goes out eth0, you
> redirect it to egress on eth1, it goes all the way up the stack and the
> kernel ends up forwarding it back out eth0?

I don't believe so, as I am redirecting from the egress on eth0 to the
ingress on the same eth0 interface.

> >> So this has nothing to do with the support in the driver; XDP is simply
> >> not suitable for what you're trying to do. Why are you trying to do this
> >> in the first place? I.e., what's the higher-level use case here?

> > The higher-level design goal (simplified to what's relevant) is that I
> > am trying to perform a TCP 3WHS between the kernel network stack and
> > my xdp ingress/tc egress hooks, where some trigger on xdp ingress
> > sends a SYN packet to the network stack, and when the network stack
> > returns a SYN-ACK packet the egress program redirects the packet to
> > the ingress interface, where the XDP program converts the packet to an
> > ACK packet, then completing the handshake with the network stack.

> It's trying to handshake with itself (on the same box)? Why? Is this a
> real use case, or a test for something else? If the latter, have you
> considered using a veth pair in going to a different namespace (that
> gets the "direction" right)?

This is a real use case as part of a larger design I am working on.
However, given that redirect from TC egress to XDP_DRV_MODE ingress
fundamentally doesn't work, I am rethinking other options to the
design. Alternatively, it seems that support for XDP_REDIRECT from
egress to ingress is currently in the works, although I don't believe
the patch has yet been applied to the mainstream kernel. So I will
also try building the kernel myself with the patch and hopefully get
the egress to ingress redirect working with 2 XDP programs attached in
driver mode. Thank you very much for all your help!


On Mon, Feb 14, 2022 at 5:28 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Sophia Yoo <sy6@princeton.edu> writes:
>
> > On Mon, Feb 14, 2022 at 1:28 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> >>
> >> Sophia Yoo <sy6@princeton.edu> writes:
> >>
> >> > Hello,
> >> >
> >> > I am relatively new to BPF/XDP, and I’m currently trying to use a TC
> >> > egress program to redirect a packet back to ingress, where I have an
> >> > XDP program attached. When the XDP program is attached in generic mode
> >> > (XDP_SKB_MODE), the redirect occurs properly and the packet is seen on
> >> > the ingress of the interface, but when the program is attached in
> >> > native mode (XDP_DRV_MODE), the packet never redirects and is just
> >> > seen exiting the interface, even though the return code of the
> >> > redirect function is “success”.
> >>
> >> The difference between XDP generic and driver mode is that the generic
> >> mode is hooked into the core networking stack whereas driver mode runs
> >> in the driver as the very first thing after packets are physically
> >> received from the network.
> >>
> >> This also explains why what you're trying to do doesn't work: the TC
> >> hook runs in the core networking stack, and when it does a redirect, the
> >> packet does not actually pass through the network driver, it appears
> >> further up in the stack, where only the generic XDP hook will see it.
> >>
> > Thanks for your quick response! I see, I didn't realize that the
> > packet was being redirected before it passed through the network
> > driver. However, in this scenario, shouldn't the packets "disappear"
> > after being redirected from tc egress to xdp ingress? In other words,
> > even if the packet doesn't actually show up on the xdp ingress hook in
> > driver mode, it should at least not be seen exiting on the egress path
> > (e.g., on tcpdump)?
>
> Is it possible the packet is looping? I.e., it goes out eth0, you
> redirect it to egress on eth1, it goes all the way up the stack and the
> kernel ends up forwarding it back out eth0?
>
> >> So this has nothing to do with the support in the driver; XDP is simply
> >> not suitable for what you're trying to do. Why are you trying to do this
> >> in the first place? I.e., what's the higher-level use case here?
> >
> > The higher-level design goal (simplified to what's relevant) is that I
> > am trying to perform a TCP 3WHS between the kernel network stack and
> > my xdp ingress/tc egress hooks, where some trigger on xdp ingress
> > sends a SYN packet to the network stack, and when the network stack
> > returns a SYN-ACK packet the egress program redirects the packet to
> > the ingress interface, where the XDP program converts the packet to an
> > ACK packet, then completing the handshake with the network stack.
>
> It's trying to handshake with itself (on the same box)? Why? Is this a
> real use case, or a test for something else? If the latter, have you
> considered using a veth pair in going to a different namespace (that
> gets the "direction" right)?
>
> > Initially, I had wanted to use an XDP program instead of a TC program
> > on the egress path to do the redirect to ingress (which I believe
> > would bypass the current issue if both ingress and egress xdp programs
> > were attached in driver mode), but as of now it seems that redirect
> > from egress to ingress simply isn't supported in XDP, which is why I'm
> > using a TC program.
>
> Well, there's also an ingress hook for TC you could look at (or just
> stay with generic XDP).
>
> -Toke
>

Re: XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

[Toke Høiland-Jørgensen]

Sophia Yoo <sy6@princeton.edu> writes:

>> >> So this has nothing to do with the support in the driver; XDP is simply
>> >> not suitable for what you're trying to do. Why are you trying to do this
>> >> in the first place? I.e., what's the higher-level use case here?
>
>> > The higher-level design goal (simplified to what's relevant) is that I
>> > am trying to perform a TCP 3WHS between the kernel network stack and
>> > my xdp ingress/tc egress hooks, where some trigger on xdp ingress
>> > sends a SYN packet to the network stack, and when the network stack
>> > returns a SYN-ACK packet the egress program redirects the packet to
>> > the ingress interface, where the XDP program converts the packet to an
>> > ACK packet, then completing the handshake with the network stack.
>
>> It's trying to handshake with itself (on the same box)? Why? Is this a
>> real use case, or a test for something else? If the latter, have you
>> considered using a veth pair in going to a different namespace (that
>> gets the "direction" right)?
>
> This is a real use case as part of a larger design I am working on.
> However, given that redirect from TC egress to XDP_DRV_MODE ingress
> fundamentally doesn't work, I am rethinking other options to the
> design. Alternatively, it seems that support for XDP_REDIRECT from
> egress to ingress is currently in the works, although I don't believe
> the patch has yet been applied to the mainstream kernel.

Huh, really? Haven't seen that, got a link?

> So I will also try building the kernel myself with the patch and
> hopefully get the egress to ingress redirect working with 2 XDP
> programs attached in driver mode. Thank you very much for all your
> help!

You're welcome :)

-Toke

Re: XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

[Sophia Yoo]

> > This is a real use case as part of a larger design I am working on.
> > However, given that redirect from TC egress to XDP_DRV_MODE ingress
> > fundamentally doesn't work, I am rethinking other options to the
> > design. Alternatively, it seems that support for XDP_REDIRECT from
> > egress to ingress is currently in the works, although I don't believe
> > the patch has yet been applied to the mainstream kernel.
>
> Huh, really? Haven't seen that, got a link?

This is where I saw the discussion for adding XDP support in the
egress path https://lwn.net/Articles/818591/, which if I am
understanding correctly is also proposing to add the XDP_REDIRECT
support on egress. (Also, for completeness sake, this is the man page
for bpf_redirect() that says currently XDP only supports redirection
from ingress to egress, and does not have egress to ingress
redirection. https://man7.org/linux/man-pages/man7/bpf-helpers.7.html)
Do you have any insight into whether/when this XDP_REDIRECT
functionality on egress will be added to the kernel?

>
> > So I will also try building the kernel myself with the patch and
> > hopefully get the egress to ingress redirect working with 2 XDP
> > programs attached in driver mode. Thank you very much for all your
> > help!
>
> You're welcome :)
>
> -Toke
>

Re: XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

[Toke Høiland-Jørgensen]

Sophia Yoo <sy6@princeton.edu> writes:

>> > This is a real use case as part of a larger design I am working on.
>> > However, given that redirect from TC egress to XDP_DRV_MODE ingress
>> > fundamentally doesn't work, I am rethinking other options to the
>> > design. Alternatively, it seems that support for XDP_REDIRECT from
>> > egress to ingress is currently in the works, although I don't believe
>> > the patch has yet been applied to the mainstream kernel.
>>
>> Huh, really? Haven't seen that, got a link?
>
> This is where I saw the discussion for adding XDP support in the
> egress path https://lwn.net/Articles/818591/, which if I am
> understanding correctly is also proposing to add the XDP_REDIRECT
> support on egress.

Ah, right, that series. There was extensive discussion on a later
version:

https://lore.kernel.org/r/20200513014607.40418-1-dsahern@kernel.org

we ended up dropping it in favour of having programs in devmap entries.
So that effort is abandoned, and I don't think you'll get very far
experimenting with that series.

> (Also, for completeness sake, this is the man page
> for bpf_redirect() that says currently XDP only supports redirection
> from ingress to egress, and does not have egress to ingress
> redirection. https://man7.org/linux/man-pages/man7/bpf-helpers.7.html)
> Do you have any insight into whether/when this XDP_REDIRECT
> functionality on egress will be added to the kernel?

It may happen eventually (there are use cases for some kind of egress
hook), but whether it will end up covering data coming from the host (as
opposed to XDP-forwarded traffic) is less certain, cf the discussion I
linked above... As for a timeline, just as soon as someone sits down and
adds it :)

-Toke

Re: XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

[Jesper Dangaard Brouer]

On 14/02/2022 19.10, Sophia Yoo wrote:

> I am relatively new to BPF/XDP [...]

I see you come for academia, so I will recommend reading our XDP paper 
as a start:
  https://github.com/xdp-project/xdp-paper

Title: “The eXpress Data Path: Fast Programmable Packet Processing in 
the Operating System Kernel” which was presented at ACM CoNEXT 2018 on 
December 5th, 2018.


[...]
> My NIC driver is Intel xgbe. I am using bcc to attach and load my xdp
> and tc programs.

This "xgbe" driver doesn't exist in the kernel tree under Intel.
Are you sure this is the correct driver?

--Jesper

Re: XDP_REDIRECT not working in XDP_DRV_MODE with Intel xgbe driver

[Sophia Yoo]

On Tue, Feb 15, 2022 at 5:04 AM Jesper Dangaard Brouer
<jbrouer@redhat.com> wrote:

> On 14/02/2022 19.10, Sophia Yoo wrote:

> > I am relatively new to BPF/XDP [...]

> I see you come for academia, so I will recommend reading our XDP paper
> as a start:
>   https://github.com/xdp-project/xdp-paper

> Title: “The eXpress Data Path: Fast Programmable Packet Processing in
> the Operating System Kernel” which was presented at ACM CoNEXT 2018 on
> December 5th, 2018.

Cool! I had briefly read this paper before but didn't realize you and
Toke were authors of this work. Great to chat with you. I'll also take
a closer look at this paper again, it looks like a great resource.

> [...]
> > My NIC driver is Intel xgbe. I am using bcc to attach and load my xdp
> > and tc programs.

> This "xgbe" driver doesn't exist in the kernel tree under Intel.
> Are you sure this is the correct driver?

Whoops, I misspelled the driver name, the driver is actually "ixgbe"
(confirmed with the output of "lspci -v").

03:00.0 Ethernet controller: Intel Corporation Ethernet Connection
X552 10 GbE SFP+
Subsystem: Intel Corporation Ethernet Connection X552 10 GbE SFP+
Physical Slot: 0-2
Flags: bus master, fast devsel, latency 0, IRQ 35, NUMA node 0
Memory at 383fffc00000 (64-bit, prefetchable) [size=2M]
Memory at 383fffe04000 (64-bit, prefetchable) [size=16K]
Expansion ROM at fb980000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1b0] Access Control Services
Capabilities: [1c0] Latency Tolerance Reporting
Kernel driver in use: ixgbe
Kernel modules: ixgbe


--Sophia