Re: AF_XDP not transmitting frames immediately

[Maciej Fijalkowski <maciej.fijalkowski@intel.com>]

On Tue, Dec 14, 2021 at 10:40:05AM +0000, Karlsson, Magnus wrote:
> Adding Ederson and Maciej.
> 
> > On 14/12/2021 09.07, Karlsson, Magnus wrote:
> > >
> > >
> > >> -----Original Message----- From: Jesper Dangaard Brouer
> > >> <jbrouer@redhat.com> Sent: Monday, December 13, 2021 10:04 PM To:
> > >> Karlsson, Magnus <magnus.karlsson@intel.com>; Björn Töpel
> > >> <bjorn@kernel.org> Cc: Brouer, Jesper <brouer@redhat.com>; Xdp
> > >> <xdp- newbies@vger.kernel.org>; Ong, Boon Leong
> > >> <boon.leong.ong@intel.com>; Joao Pedro Barros Silva
> > >> <jopbs@vestas.com>; Diogo Alexandre Da Silva Lima <dioli@vestas.com>
> > >> Subject: AF_XDP not transmitting frames immediately
> > >>
> > >> Hi Magnus and Bjørn,
> > >>
> > >> I'm coding on an AF_XDP program[1] that need to send (a bulk of
> > >> packets) in a short time-window (related to Time-Triggered Ethernet).
> > >>
> > >> My observations are that AF_XDP doesn't send the frames immediately.
> > >> And yes, I do call sendto() to trigger a TX kick. In zero-copy mode
> > >> this is particular bad.  My program want to send 4 packets in a
> > >> burst, but I'm observing 8 packets grouped together on the receiving
> > >> host.
> > >>
> > >> Is the a known property of AF_XDP?
> > >
> > > Nope! It is supposed to be able to send one packet at a time, though I
> > > have several times seen bugs in the drivers where the batching
> > > behavior shines through like this, and once a bug in the core code.
> > > There is even a test these days for just sending a single packet,
> >
> > Where is that test in the kernel tree?
> 
> In tools/testing/selftests/bpf/xdpxceiver.c. It is the RUN_TO_COMPLETION_SINGLE_PKT test. But the framework only operates on veth currently.

I'd say it's driver's fault. Magnus fixed something similar for i40e:
https://lore.kernel.org/netdev/20210401172107.1191618-3-anthony.l.nguyen@intel.com/

We don't have currently igc HW on our side to dig this :<

> 
> > > since we have had issues with this in the past. That test does pass in
> > > bpf-next, but it is only run with the veth driver that does not
> > > support zero-copy so could still be an issue. What driver are you
> > > using in zero-copy mode and what kernel version are you on?
> >
> > Driver: igc with Intel chip i225
> 
> Have never tried this one personally. Do not know if I have one in the lab but let me check.
> 
> Ederson, do you have any experience with this card and if so, have you seen something similar?
> 
> > Kernel version: 5.15.0-net-next+ #618 SMP PREEMPT
> >   - Devel branch at commit 6d3b1b069946 (v5.15-12802-g6d3b1b069946)
> >
> > >> How can I get AF_XDP to "flush" TX packets when calling sendto()?
> > >> Should we add another flag than the current MSG_DONTWAIT?
> > >
> > > In zero-copy mode with softirq driver processing (not busy poll), a
> > > sendto will just trigger the xsk_wakeup ndo that schedules napi unless
> > > it is already executing. It is up to the driver to then get packets
> > > from the Tx ring and put them on the HW and make sure they are sent.
> > > Barring any HW quirks, sending one packets should be perfectly fine.
> >
> > I will investigate driver level issues.
> >
> > I have other (100G) NICs in my testlab, but I'm using these 1G NICs because
> > they support hardware timestamping, which allows me to investigate these
> > timing issues.
> > I'll find a way to see of other drivers behave differently.
> 
> Would be great if you could check if the problem also exists on e.g. ice.
> 
> > >> Hint, I'm using tcpdump hardware timestamping on receiving hist via
> > >> cmdline:
> > >>
> > >> tcpdump -vv -s0 -ni eth1 -j adapter_unsynced
> > >> --time-stamp-precision=nano -w af_xdp_tx_cyclic.dump42
> > >>
> > >> Notice[1] on specific branch: [1]
> > >> https://github.com/xdp-project/bpf-
> > examples/tree/vestas03_AF_XDP_exam
> > >> ple/AF_XDP-interaction
> > >
> >
> > Thanks for your feedback,
> > --Jesper
>