Re: [PATCH bpf-next] xsk: support AF_PACKET

[Jesper Dangaard Brouer <brouer@redhat.com>]

On Fri, 28 May 2021 12:22:40 +0200
Daniel Borkmann <daniel@iogearbox.net> wrote:

> On 5/28/21 12:00 PM, Magnus Karlsson wrote:
> > On Fri, May 28, 2021 at 11:52 AM Jesper Dangaard Brouer
> > <brouer@redhat.com> wrote:  
> >> On Fri, 28 May 2021 17:02:01 +0800
> >> Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:  
> >>> On Fri, 28 May 2021 10:55:58 +0200, Toke Høiland-Jørgensen <toke@redhat.com> wrote:  
> >>>> Xuan Zhuo <xuanzhuo@linux.alibaba.com> writes:
> >>>>  
> >>>>> In xsk mode, users cannot use AF_PACKET(tcpdump) to observe the current
> >>>>> rx/tx data packets. This feature is very important in many cases. So
> >>>>> this patch allows AF_PACKET to obtain xsk packages.  
> >>>>
> >>>> You can use xdpdump to dump the packets from the XDP program before it
> >>>> gets redirected into the XSK:
> >>>> https://github.com/xdp-project/xdp-tools/tree/master/xdp-dump  
> >>>
> >>> Wow, this is a good idea.  
> >>
> >> Yes, it is rather cool (credit to Eelco).  Notice the extra info you
> >> can capture from 'exit', like XDP return codes, if_index, rx_queue.
> >>
> >> The tool uses the perf ring-buffer to send/copy data to userspace.
> >> This is actually surprisingly fast, but I still think AF_XDP will be
> >> faster (but it usually 'steals' the packet).
> >>
> >> Another (crazy?) idea is to extend this (and xdpdump), is to leverage
> >> Hangbin's recent XDP_REDIRECT extension e624d4ed4aa8 ("xdp: Extend
> >> xdp_redirect_map with broadcast support").  We now have a
> >> xdp_redirect_map flag BPF_F_BROADCAST, what if we create a
> >> BPF_F_CLONE_PASS flag?
> >>
> >> The semantic meaning of BPF_F_CLONE_PASS flag is to copy/clone the
> >> packet for the specified map target index (e.g AF_XDP map), but
> >> afterwards it does like veth/cpumap and creates an SKB from the
> >> xdp_frame (see __xdp_build_skb_from_frame()) and send to netstack.
> >> (Feel free to kick me if this doesn't make any sense)  
> > 
> > This would be a smooth way to implement clone support for AF_XDP. If
> > we had this and someone added AF_XDP support to libpcap, we could both
> > capture AF_XDP traffic with tcpdump (using this clone functionality in
> > the XDP program) and speed up tcpdump for dumping traffic destined for
> > regular sockets. Would that solve your use case Xuan? Note that I have
> > not looked into the BPF_F_CLONE_PASS code, so do not know at this
> > point what it would take to support this for XSKMAPs.  

There is no spoon... the BPF_F_CLONE_PASS code is an idea.

> 
> Recently also ended up with something similar for our XDP LB to record pcaps [0] ;)
> My question is.. tcpdump doesn't really care where the packet data comes from,
> so why not extending libpcap's Linux-related internals to either capture from
> perf RB or BPF ringbuf 

Just want to first mention, that I do like adding a perf ring-buffer
(BPF ringbuf) interface to AF_PACKET.  But this is basically what
xdpdump already does.  The cool thing is that it is super flexible for
adding extra info like xdpdump does with XDP-return codes.


> rather than AF_PACKET sockets? Cloning is slow, and if
> you need to end up creating an skb which is then cloned once again inside AF_PACKET
> it's even worse. Just relying and reading out, say, perf RB you don't need any
> clones at all.

Well, this is exactly what we avoid with my idea of BPF_F_CLONE_PASS
when combined with AF_XDP. 

I should explain this idea better.  The trick is that AF_XDP have
preallocated all the packets it will every use (at setup time).   Thus,
the AF_XDP copy-mode does no allocations, which is why it is fast
(of-cause ZC mode is faster, but copy-mode AF_XDP is also VERY fast!).

(Details and step with AF_XDP code notes:)
When the xdp_do_redirect happens with ri->flags BPF_F_CLONE_PASS, then
the map specific enqueue (e.g. __xsk_map_redirect), will do a copy of
the xdp_buff (AF_XDP calls xsk_copy_xdp()) and for AF_XDP we don't need
to do a (real) allocation.  Instead of freeing the xdp_buff in xsk_rcv()
(see call to xdp_return_buff()) then we do the xdp_frame to SKB work.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer