AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Jesper Dangaard Brouer <brouer@redhat.com>]

Bringing these questions to the xdp-newbies list, where they belong.
Answers inlined below.

On Tue, 20 Aug 2019 21:17:57 +0200 Július Milan <Julius.Milan@pantheon.tech>
> 
> I am writing AF_XDP driver for FDio VPP. I have 2 questions.
>

That sounds excellent.  I was hoping someone would do this for FDio VPP.
Do notice that DPDK now also got AF_XDP support.  IHMO it makes a lot
of sense to implement AF_XDP for FDio, and avoid the DPDK dependency.
(AFAIK FDio already got other back-ends than DPDK).
 

> 1 - I created a simple driver according to sample in kernel. I load my XDP
> program and pin the maps.
> 
>   Then in user application I create a socket, mmap the memory and
> push it to xskmap in program. All fine yet.
> 
>   Then I start another instance of user application and do the
> same, create socket, mmap the memory and trying to
> 
>   push it somewhere else into the map. But I got  errno: 16
> "Device or resource busy" when trying to bind.
> 
>   I guess the memory can’t be mmaped 2 times, but should be
> shared, is that correct?

I'm cc'ing the AF_XDP experts, as I'm not sure myself.  I mostly deal
with the in-kernel XDP path.  (AF_XDP is essentially kernel bypass :-O)

 
>   If so, I am wondering how to solve this nicely.
> 
>   Can I store the value of first socket (that created the mmaped
> memory) in some special map in my XDP program to avoid complicated
> inter-process communication?
> 
>   And what happens if this first socket is closed while any other
> sockets are still alive (using its shared mmaped memory)?
> 
>   What would you recommend? Maybe you have some sample.

We just added a sample (by Eelco Cc'ed) into XDP-tutorial:
 https://github.com/xdp-project/xdp-tutorial/tree/master/advanced03-AF_XDP  

At-least read the README.org file... to get over the common gotchas.

AFAIK the sample doesn't cover your use-case.  I guess, we/someone
should extend the sample, to illustrate how how multiple interfaces can
share the same UMEM.

The official documentation is:
 https://www.kernel.org/doc/html/latest/networking/af_xdp.html


>   Can I do also atomic operations? (I want it just for such rare
> cases as initialization of next socket, to check if there already is
> one, that mmaped the memory)
> 
> 
> 
> 2 – We want to do also some decap/encap on XDP layer, before
> redirecting it to the socket.
> 

Decap on XDP layer is an excellent use-case, that demonstrate
cooperation between XDP and AF_XDP kernel-bypass facility.


>   On RX way it is easy, I do what I want and redirect it to the
> socket, but can I achieve the same also on TX?
> 

(Yes, RX case is easy)

We don't have an XDP TX hook yet... but so many people have requested
this, that we should add this.

>   Can I catch the packet while TX in XDP and do something with it
> (encapsulate it) before sending it out?

Usually, we recommend people use the TC egress BPF hook to do the encap
in TX.  For the AF_XDP use-case, the TC hook isn't there... so that is
not an option.  Again an argument for an XDP-TX hook.  You, could
of-cause add the encap header in your AF_XDP userspace program, but I
do understand it would make architectural sense that in-kernel XDP
would act as a decap/encap layer.

 
>   If so what about performance?
> 

For AF_XDP RX-side is really really fast, even in copy-mode.

For AF_XDP TX-side in copy-mode, it is rather slow, as it allocates
SKBs etc.  We could optimize this further but we have not.  When
enabling AF_XDP zero-copy mode, the TX-side is also super fast.

Another hint, for AF_XDP TX-side, remember to "produce" several packets
before doing the sendmsg system call.  Thus, effectively doing bulking
on the TX-ring.


>
> By the way, great job with XDP ;)

Thanks! 

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer