Re: bidirectional: => AF_XDP , <= XDP_REDIRECT

[Jesper Dangaard Brouer <brouer@redhat.com>]

On Wed, 30 Oct 2019 18:44:27 +0300 Ilya Goslhtein <ilejn@yandex.ru> wrote:

> Hello,
> 
> playing with xdpbridge https://github.com/ilejn/xdpbridge/ I've observed 
> a limitation which is serious for me.

Hmm... I took a short look at your code.

Sorry for this candid/blunt evaluation: I think you are doing it wrong.
Both on how you use other FOSS code and how you use BPF/XDP.

First of all you are using bpf_load.c[1], which everybody should stop
using (sorry, I know I sort of started that trend).  Instead everybody
should use libbpf[2].  Second you have copied over bpf_load.c and
re-indented the entire file, which is bad, because it makes it very
hard to track the difference between the original FOSS project and your
project.

You *also* use libbpf, but directly from the kernel tree.  We don't
recommend doing it this way.  Facebook engineers are maintaining a
libbpf mirror on github[2], which can be used as a git-submodule.
Before all distros start shipping libbpf, the easiest way to get
started is to use libbpf as a git-submodule.

[1] https://github.com/ilejn/xdpbridge/blob/master/bpf_load.c
[2] https://github.com/libbpf/libbpf


> xdpbridge is basically a combination of xdp_redirect_map sample 
> (world=>client path) and slightly more advanced l2fwd mode of xdpsock 
> (client=>world path),  
> where AF_XDP sockets are bound to two different interfaces (vanilla 
> l2fwd mode uses same interface as ingress and egress) and where 
> multithreading is supported.

IMHO choosing AF_XDP for a bridge implementation is the wrong approach.
I would implement this in the XDP BPF-program, and use fallback to the
normal Linux bridging code (for broadcasting, ARP etc).


> xdpbridge sets options for client and egress interfaces independently.
> 
> So, setting zerocopy bind flag for world interface (-z command line 
> parameter) prevents this interface from working as ingress for 
> world=>client path, although different queues are used. No errors, just   
> no data transfer.

Sounds like you want part of the traffic to reach the normal kernel
networking stack.  In that case, the XDP program should sort that out
(and call XDP_PASS) before doing the XDP_REDIRECT step into AF_XDP.

 
> It would be nice to understand if this issue fundamental or specific
> for the kernel (5.0.0-31-generic #33~18.04.1-Ubuntu) or card (Intel 
> Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01))
> or driver.
> Or, may be, it is just my mistake.
> 
> I do appreciate any comments or suggestions.

I'm glad that you are open to feedback.

I recommend you look at our XDP-tutorial[3], to see an example of how
Toke and I recommend structuring a project that want to use XDP/BPF.


[3] https://github.com/xdp-project/xdp-tutorial
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer


 
> On 10.09.2019 17:38, Toke Høiland-Jørgensen wrote:
> > Ilya Goslhtein <ilejn@yandex.ru> writes:
> >  
> >> Hello Toke,
> >>
> >> thanks for the response.
> >>
> >> I do not think that it is the case.
> >>
> >> The interface I am trying to share is outgoing for xdpbridge and
> >> incoming for xdp_bridge_map.
> >>
> >> xdpbridge does not load xdp program for outgoing interface (while loads
> >> for incoming).
> >>
> >> xdp_bridge_map loads dummy XDP program for outgoing redirect, while it
> >> is Ok if it is already exists.
> >>
> >>
> >> It seems that if I use different queues for incoming and outgoing
> >> packets, everything is Ok, while I am not 100% sure yet. Does it look
> >> realistic?  
> > Oh, right, yeah, the AF_XDP socket will need to configure a hardware
> > queue to use; depending on your hardware, that could be incompatible
> > with running a regular XDP program on the same hardware queue.
> >
> > Incidently, we are working on a way to make this work better; talk
> > starts in five minutes at LPC:
> > https://linuxplumbersconf.org/event/4/contributions/462/
> >
> > -Toke  
>