bidirectional: => AF_XDP , <= XDP_REDIRECT

bidirectional: => AF_XDP , <= XDP_REDIRECT

[Ilya Goslhtein]

Hello,

I am trying to use a network interface to
(1) send packets via AF_XDP
(2) receive packets and redirect to another interface via xdp_redirect_map

Basically, I am making xdpbridge https://github.com/ilejn/xdpbridge.
  bidirectional. My goal is to process packets in one direction in 
kernelspace (xdp_redirect_map) and to pass packets in opposite direction 
via userspace (recieve via AF_XDP then send via AF_XDP, rings are not 
shared, so I perform memcpy).

For test purposes I ran two applications: my xdpbridge and 
xdp_redirect_map bpf sample.

It does not work. Only few packets are received, then the thing stops. 
Redirecting works until xdpbridge does 'bind' against the interface.

 From XDP point of view we have:
application one: bind with PF_XDP
application two: XDP program

What is the nature of the problem? Is it fundamental, 
environment/hardware specific, or I am simply doing something wrong?

ubuntu 18.04, 5.0.0-27-generic,
Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ 
Network Connection (rev 01)

Thanks.

-- 
Best regards,
Ilya Golshtein

Re: bidirectional: => AF_XDP , <= XDP_REDIRECT

[Toke Høiland-Jørgensen]

Ilya Goslhtein <ilejn@yandex.ru> writes:

> Hello,
>
> I am trying to use a network interface to
> (1) send packets via AF_XDP
> (2) receive packets and redirect to another interface via xdp_redirect_map
>
> Basically, I am making xdpbridge https://github.com/ilejn/xdpbridge.
>   bidirectional. My goal is to process packets in one direction in 
> kernelspace (xdp_redirect_map) and to pass packets in opposite direction 
> via userspace (recieve via AF_XDP then send via AF_XDP, rings are not 
> shared, so I perform memcpy).
>
> For test purposes I ran two applications: my xdpbridge and 
> xdp_redirect_map bpf sample.
>
> It does not work. Only few packets are received, then the thing stops. 
> Redirecting works until xdpbridge does 'bind' against the interface.

This is because the AF_XDP application loads another XDP program on the
interface that redirects the traffic into the socket. It is possible to
write an XDP program that can do both (i.e., for each packet, decide
whether to redirect it into the AF_XDP socket, or to another interface),
but obviously none of the example programs know how to do this, so they
end up stepping on each other's toes...

-Toke

Re: bidirectional: => AF_XDP , <= XDP_REDIRECT

[Ilya Goslhtein]

Hello Toke,

thanks for the response.

I do not think that it is the case.

The interface I am trying to share is outgoing for xdpbridge and 
incoming for xdp_bridge_map.

xdpbridge does not load xdp program for outgoing interface (while loads 
for incoming).

xdp_bridge_map loads dummy XDP program for outgoing redirect, while it 
is Ok if it is already exists.


It seems that if I use different queues for incoming and outgoing 
packets, everything is Ok, while I am not 100% sure yet. Does it look 
realistic?



On 10.09.2019 17:21, Toke Høiland-Jørgensen wrote:
> Ilya Goslhtein <ilejn@yandex.ru> writes:
>
>> Hello,
>>
>> I am trying to use a network interface to
>> (1) send packets via AF_XDP
>> (2) receive packets and redirect to another interface via xdp_redirect_map
>>
>> Basically, I am making xdpbridge https://github.com/ilejn/xdpbridge.
>>    bidirectional. My goal is to process packets in one direction in
>> kernelspace (xdp_redirect_map) and to pass packets in opposite direction
>> via userspace (recieve via AF_XDP then send via AF_XDP, rings are not
>> shared, so I perform memcpy).
>>
>> For test purposes I ran two applications: my xdpbridge and
>> xdp_redirect_map bpf sample.
>>
>> It does not work. Only few packets are received, then the thing stops.
>> Redirecting works until xdpbridge does 'bind' against the interface.
> This is because the AF_XDP application loads another XDP program on the
> interface that redirects the traffic into the socket. It is possible to
> write an XDP program that can do both (i.e., for each packet, decide
> whether to redirect it into the AF_XDP socket, or to another interface),
> but obviously none of the example programs know how to do this, so they
> end up stepping on each other's toes...
>
> -Toke

-- 
Best regards,
Ilya Golshtein

Re: bidirectional: => AF_XDP , <= XDP_REDIRECT

[Toke Høiland-Jørgensen]

Ilya Goslhtein <ilejn@yandex.ru> writes:

> Hello Toke,
>
> thanks for the response.
>
> I do not think that it is the case.
>
> The interface I am trying to share is outgoing for xdpbridge and 
> incoming for xdp_bridge_map.
>
> xdpbridge does not load xdp program for outgoing interface (while loads 
> for incoming).
>
> xdp_bridge_map loads dummy XDP program for outgoing redirect, while it 
> is Ok if it is already exists.
>
>
> It seems that if I use different queues for incoming and outgoing 
> packets, everything is Ok, while I am not 100% sure yet. Does it look 
> realistic?

Oh, right, yeah, the AF_XDP socket will need to configure a hardware
queue to use; depending on your hardware, that could be incompatible
with running a regular XDP program on the same hardware queue.

Incidently, we are working on a way to make this work better; talk
starts in five minutes at LPC:
https://linuxplumbersconf.org/event/4/contributions/462/

-Toke

Re: bidirectional: => AF_XDP , <= XDP_REDIRECT

[Ilya Goslhtein]

Hello,

playing with xdpbridge https://github.com/ilejn/xdpbridge/ I've observed 
a limitation which is serious for me.

xdpbridge is basically a combination of xdp_redirect_map sample 
(world=>client path) and slightly more advanced l2fwd mode of xdpsock 
(client=>world path),
where AF_XDP sockets are bound to two different interfaces (vanilla 
l2fwd mode uses same interface as ingress and egress) and where 
multithreading is supported.

xdpbridge sets options for client and egress interfaces independently.

So, setting zerocopy bind flag for world interface (-z command line 
parameter) prevents this interface from working as ingress for 
world=>client path, although different queues are used. No errors, just 
no data transfer.

It would be nice to understand if this issue fundamental or specific for 
the kernel (5.0.0-31-generic #33~18.04.1-Ubuntu) or card (Intel 
Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)) or 
driver.
Or, may be, it is just my mistake.

I do appreciate any comments or suggestions.

On 10.09.2019 17:38, Toke Høiland-Jørgensen wrote:
> Ilya Goslhtein <ilejn@yandex.ru> writes:
>
>> Hello Toke,
>>
>> thanks for the response.
>>
>> I do not think that it is the case.
>>
>> The interface I am trying to share is outgoing for xdpbridge and
>> incoming for xdp_bridge_map.
>>
>> xdpbridge does not load xdp program for outgoing interface (while loads
>> for incoming).
>>
>> xdp_bridge_map loads dummy XDP program for outgoing redirect, while it
>> is Ok if it is already exists.
>>
>>
>> It seems that if I use different queues for incoming and outgoing
>> packets, everything is Ok, while I am not 100% sure yet. Does it look
>> realistic?
> Oh, right, yeah, the AF_XDP socket will need to configure a hardware
> queue to use; depending on your hardware, that could be incompatible
> with running a regular XDP program on the same hardware queue.
>
> Incidently, we are working on a way to make this work better; talk
> starts in five minutes at LPC:
> https://linuxplumbersconf.org/event/4/contributions/462/
>
> -Toke

-- 
Best regards,
Ilya Golshtein

Re: bidirectional: => AF_XDP , <= XDP_REDIRECT

[Jesper Dangaard Brouer]

On Wed, 30 Oct 2019 18:44:27 +0300 Ilya Goslhtein <ilejn@yandex.ru> wrote:

> Hello,
> 
> playing with xdpbridge https://github.com/ilejn/xdpbridge/ I've observed 
> a limitation which is serious for me.

Hmm... I took a short look at your code.

Sorry for this candid/blunt evaluation: I think you are doing it wrong.
Both on how you use other FOSS code and how you use BPF/XDP.

First of all you are using bpf_load.c[1], which everybody should stop
using (sorry, I know I sort of started that trend).  Instead everybody
should use libbpf[2].  Second you have copied over bpf_load.c and
re-indented the entire file, which is bad, because it makes it very
hard to track the difference between the original FOSS project and your
project.

You *also* use libbpf, but directly from the kernel tree.  We don't
recommend doing it this way.  Facebook engineers are maintaining a
libbpf mirror on github[2], which can be used as a git-submodule.
Before all distros start shipping libbpf, the easiest way to get
started is to use libbpf as a git-submodule.

[1] https://github.com/ilejn/xdpbridge/blob/master/bpf_load.c
[2] https://github.com/libbpf/libbpf


> xdpbridge is basically a combination of xdp_redirect_map sample 
> (world=>client path) and slightly more advanced l2fwd mode of xdpsock 
> (client=>world path),  
> where AF_XDP sockets are bound to two different interfaces (vanilla 
> l2fwd mode uses same interface as ingress and egress) and where 
> multithreading is supported.

IMHO choosing AF_XDP for a bridge implementation is the wrong approach.
I would implement this in the XDP BPF-program, and use fallback to the
normal Linux bridging code (for broadcasting, ARP etc).


> xdpbridge sets options for client and egress interfaces independently.
> 
> So, setting zerocopy bind flag for world interface (-z command line 
> parameter) prevents this interface from working as ingress for 
> world=>client path, although different queues are used. No errors, just   
> no data transfer.

Sounds like you want part of the traffic to reach the normal kernel
networking stack.  In that case, the XDP program should sort that out
(and call XDP_PASS) before doing the XDP_REDIRECT step into AF_XDP.

 
> It would be nice to understand if this issue fundamental or specific
> for the kernel (5.0.0-31-generic #33~18.04.1-Ubuntu) or card (Intel 
> Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01))
> or driver.
> Or, may be, it is just my mistake.
> 
> I do appreciate any comments or suggestions.

I'm glad that you are open to feedback.

I recommend you look at our XDP-tutorial[3], to see an example of how
Toke and I recommend structuring a project that want to use XDP/BPF.


[3] https://github.com/xdp-project/xdp-tutorial
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer


 
> On 10.09.2019 17:38, Toke Høiland-Jørgensen wrote:
> > Ilya Goslhtein <ilejn@yandex.ru> writes:
> >  
> >> Hello Toke,
> >>
> >> thanks for the response.
> >>
> >> I do not think that it is the case.
> >>
> >> The interface I am trying to share is outgoing for xdpbridge and
> >> incoming for xdp_bridge_map.
> >>
> >> xdpbridge does not load xdp program for outgoing interface (while loads
> >> for incoming).
> >>
> >> xdp_bridge_map loads dummy XDP program for outgoing redirect, while it
> >> is Ok if it is already exists.
> >>
> >>
> >> It seems that if I use different queues for incoming and outgoing
> >> packets, everything is Ok, while I am not 100% sure yet. Does it look
> >> realistic?  
> > Oh, right, yeah, the AF_XDP socket will need to configure a hardware
> > queue to use; depending on your hardware, that could be incompatible
> > with running a regular XDP program on the same hardware queue.
> >
> > Incidently, we are working on a way to make this work better; talk
> > starts in five minutes at LPC:
> > https://linuxplumbersconf.org/event/4/contributions/462/
> >
> > -Toke  
> 

Re: bidirectional: => AF_XDP , <= XDP_REDIRECT

[Ilya Goslhtein]

Hello Jesper,

thank you for your comments.

I agree that it is strange to use AF_XDP as a bridge, while it might be 
not so strange if you consider some userspace application logic.

xdpbridge is basically a sample. With the help of this sample I can ask 
questions about one or another XDP specific issue, e.g. to clarify if 
zerocopy is not compatible with XDP_REDIRECT.

Initially I avoided using libbpf but currently I use it anyway so I 
agree that using it properly is a plus.


On 30.10.2019 19:39, Jesper Dangaard Brouer wrote:
> On Wed, 30 Oct 2019 18:44:27 +0300 Ilya Goslhtein <ilejn@yandex.ru> wrote:
>
>> Hello,
>>
>> playing with xdpbridge https://github.com/ilejn/xdpbridge/ I've observed
>> a limitation which is serious for me.
> Hmm... I took a short look at your code.
>
> Sorry for this candid/blunt evaluation: I think you are doing it wrong.
> Both on how you use other FOSS code and how you use BPF/XDP.
>
> First of all you are using bpf_load.c[1], which everybody should stop
> using (sorry, I know I sort of started that trend).  Instead everybody
> should use libbpf[2].  Second you have copied over bpf_load.c and
> re-indented the entire file, which is bad, because it makes it very
> hard to track the difference between the original FOSS project and your
> project.
>
> You *also* use libbpf, but directly from the kernel tree.  We don't
> recommend doing it this way.  Facebook engineers are maintaining a
> libbpf mirror on github[2], which can be used as a git-submodule.
> Before all distros start shipping libbpf, the easiest way to get
> started is to use libbpf as a git-submodule.
>
> [1] https://github.com/ilejn/xdpbridge/blob/master/bpf_load.c
> [2] https://github.com/libbpf/libbpf
>
>
>> xdpbridge is basically a combination of xdp_redirect_map sample
>> (world=>client path) and slightly more advanced l2fwd mode of xdpsock
>> (client=>world path),
>> where AF_XDP sockets are bound to two different interfaces (vanilla
>> l2fwd mode uses same interface as ingress and egress) and where
>> multithreading is supported.
> IMHO choosing AF_XDP for a bridge implementation is the wrong approach.
> I would implement this in the XDP BPF-program, and use fallback to the
> normal Linux bridging code (for broadcasting, ARP etc).
>
>
>> xdpbridge sets options for client and egress interfaces independently.
>>
>> So, setting zerocopy bind flag for world interface (-z command line
>> parameter) prevents this interface from working as ingress for
>> world=>client path, although different queues are used. No errors, just
>> no data transfer.
> Sounds like you want part of the traffic to reach the normal kernel
> networking stack.  In that case, the XDP program should sort that out
> (and call XDP_PASS) before doing the XDP_REDIRECT step into AF_XDP.
>
>   
>> It would be nice to understand if this issue fundamental or specific
>> for the kernel (5.0.0-31-generic #33~18.04.1-Ubuntu) or card (Intel
>> Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01))
>> or driver.
>> Or, may be, it is just my mistake.
>>
>> I do appreciate any comments or suggestions.
> I'm glad that you are open to feedback.
>
> I recommend you look at our XDP-tutorial[3], to see an example of how
> Toke and I recommend structuring a project that want to use XDP/BPF.
>
>
> [3] https://github.com/xdp-project/xdp-tutorial

-- 
Best regards,
Ilya Golshtein