From: Jesper Dangaard Brouer <brouer@redhat.com>
To: "Björn Töpel" <bjorn.topel@intel.com>
Cc: "Björn Töpel" <bjorn.topel@gmail.com>,
ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org,
bpf@vger.kernel.org, magnus.karlsson@intel.com,
davem@davemloft.net, kuba@kernel.org, hawk@kernel.org,
john.fastabend@gmail.com, intel-wired-lan@lists.osuosl.org,
brouer@redhat.com, "Toke Høiland-Jørgensen" <toke@redhat.com>
Subject: Re: [PATCH bpf-next 3/6] xsk: introduce xsk_do_redirect_rx_full() helper
Date: Mon, 7 Sep 2020 14:45:08 +0200 [thread overview]
Message-ID: <20200907144508.3ddda938@carbon> (raw)
In-Reply-To: <dfa75afc-ceb7-76ce-6ba3-3b89c53f92f3@intel.com>
On Fri, 4 Sep 2020 17:39:17 +0200
Björn Töpel <bjorn.topel@intel.com> wrote:
> On 2020-09-04 17:11, Jesper Dangaard Brouer wrote:
> > On Fri, 4 Sep 2020 15:53:28 +0200 Björn Töpel
> > <bjorn.topel@gmail.com> wrote:
> >
> >> From: Björn Töpel <bjorn.topel@intel.com>
> >>
> >> The xsk_do_redirect_rx_full() helper can be used to check if a
> >> failure of xdp_do_redirect() was due to the AF_XDP socket had a
> >> full Rx ring.
> >
> > This is very AF_XDP specific. I think that the cpumap could likely
> > benefit from similar approach? e.g. if the cpumap kthread is
> > scheduled on the same CPU.
> >
>
> At least I thought this was *very* AF_XDP specific, since the kernel is
> dependent of that userland runs. Allocation (source) and Rx ring (sink).
> Maybe I was wrong! :-)
>
> The thing with AF_XDP zero-copy, is that we sort of assume that if a
> user enabled that most packets will have XDP_REDIRECT to an AF_XDP socket.
>
>
> > But for cpumap we only want this behavior if sched on the same CPU
> > as RX-NAPI. This could be "seen" by the cpumap code itself in the
> > case bq_flush_to_queue() drops packets, check if rcpu->cpu equal
> > smp_processor_id(). Maybe I'm taking this too far?
> >
>
> Interesting. So, if you're running on the same core, and redirect fail
> for CPUMAP, you'd like to yield the NAPI loop? Is that really OK from a
> fairness perspective? I mean, with AF_XDP zero-copy we pretty much know
> that all actions will be redirect to socket. For CPUMAP type of
> applications, can that assumption be made?
Yes, you are right. The RX NAPI loop could be doing something else,
and yielding the NAPI loop due to detecting same-CPU is stalling on
cpumap delivery might not be correct action.
I just tested the same-CPU processing case for cpumap (result below
signature), and it doesn't exhibit the bad 'dropping-off-edge'
performance slowdown. The cpumap code also already tries to mitigate
this, by calling wake_up_process() for every 8 packets (CPU_MAP_BULK_SIZE).
I find your patchset very interesting, as I believe we do need some
kind of general push-back "flow-control" mechanism for XDP. Maybe I
should solve this differently in our XDP-TX-QoS pipe dream ;-)
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
Quick benchmark of cpumap.
Same CPU RX and cpumap processing:
----------------------------------
(Doing XDP_DROP on CPU)
Running XDP/eBPF prog_name:xdp_cpu_map0
XDP-cpumap CPU:to pps drop-pps extra-info
XDP-RX 4 9,189,700 0 0
XDP-RX total 9,189,700 0
cpumap-enqueue 4:4 9,189,696 0 8.00 bulk-average
cpumap-enqueue sum:4 9,189,696 0 8.00 bulk-average
cpumap_kthread 4 9,189,702 0 143,582 sched
cpumap_kthread total 9,189,702 0 143,582 sched-sum
redirect_err total 0 0
xdp_exception total 0 0
2nd remote XDP/eBPF prog_name: xdp1
XDP-cpumap CPU:to xdp-pass xdp-drop xdp-redir
xdp-in-kthread 4 0 9,189,702 0
xdp-in-kthread total 0 9,189,702 0
%CPU
51,8 ksoftirqd/4
48,2 cpumap/4/map:17
(Doing XDP_PASS on CPU)
Running XDP/eBPF prog_name:xdp_cpu_map0
XDP-cpumap CPU:to pps drop-pps extra-info
XDP-RX 4 8,593,822 0 0
XDP-RX total 8,593,822 0
cpumap-enqueue 4:4 8,593,888 7,714,949 8.00 bulk-average
cpumap-enqueue sum:4 8,593,888 7,714,949 8.00 bulk-average
cpumap_kthread 4 878,930 0 13,732 sched
cpumap_kthread total 878,930 0 13,732 sched-sum
redirect_err total 0 0
xdp_exception total 0 0
2nd remote XDP/eBPF prog_name: xdp_redirect_dummy
XDP-cpumap CPU:to xdp-pass xdp-drop xdp-redir
xdp-in-kthread 4 878,931 0 0
xdp-in-kthread total 878,931 0 0
Another CPU getting cpumap redirected packets:
----------------------------------------------
(Doing XDP_DROP on CPU)
Running XDP/eBPF prog_name:xdp_cpu_map0
XDP-cpumap CPU:to pps drop-pps extra-info
XDP-RX 4 17,526,797 0 0
XDP-RX total 17,526,797 0
cpumap-enqueue 4:0 17,526,796 245,811 8.00 bulk-average
cpumap-enqueue sum:0 17,526,796 245,811 8.00 bulk-average
cpumap_kthread 0 17,281,001 0 16,351 sched
cpumap_kthread total 17,281,001 0 16,351 sched-sum
redirect_err total 0 0
xdp_exception total 0 0
2nd remote XDP/eBPF prog_name: xdp1
XDP-cpumap CPU:to xdp-pass xdp-drop xdp-redir
xdp-in-kthread 0 0 17,281,001 0
xdp-in-kthread total 0 17,281,001 0
(Doing XDP_PASS on CPU)
Running XDP/eBPF prog_name:xdp_cpu_map0
XDP-cpumap CPU:to pps drop-pps extra-info
XDP-RX 4 14,603,587 0 0
XDP-RX total 14,603,587 0
cpumap-enqueue 4:0 14,603,582 12,999,248 8.00 bulk-average
cpumap-enqueue sum:0 14,603,582 12,999,248 8.00 bulk-average
cpumap_kthread 0 1,604,338 0 0
cpumap_kthread total 1,604,338 0 0
redirect_err total 0 0
xdp_exception total 0 0
2nd remote XDP/eBPF prog_name: xdp_redirect_dummy
XDP-cpumap CPU:to xdp-pass xdp-drop xdp-redir
xdp-in-kthread 0 1,604,338 0 0
xdp-in-kthread total 1,604,338 0 0
next prev parent reply other threads:[~2020-09-07 17:49 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-04 13:53 [PATCH bpf-next 0/6] xsk: exit NAPI loop when AF_XDP Rx ring is full Björn Töpel
2020-09-04 13:53 ` [PATCH bpf-next 1/6] xsk: improve xdp_do_redirect() error codes Björn Töpel
2020-09-04 13:53 ` [PATCH bpf-next 2/6] xdp: introduce xdp_do_redirect_ext() function Björn Töpel
2020-09-04 13:53 ` [PATCH bpf-next 3/6] xsk: introduce xsk_do_redirect_rx_full() helper Björn Töpel
2020-09-04 15:11 ` Jesper Dangaard Brouer
2020-09-04 15:39 ` Björn Töpel
2020-09-07 12:45 ` Jesper Dangaard Brouer [this message]
2020-09-04 13:53 ` [PATCH bpf-next 4/6] i40e, xsk: finish napi loop if AF_XDP Rx queue is full Björn Töpel
2020-09-04 13:53 ` [PATCH bpf-next 5/6] ice, " Björn Töpel
2020-09-04 13:53 ` [PATCH bpf-next 6/6] ixgbe, " Björn Töpel
2020-09-04 15:35 ` Jesper Dangaard Brouer
2020-09-04 15:54 ` Björn Töpel
2020-09-04 13:59 ` [PATCH bpf-next 0/6] xsk: exit NAPI loop when AF_XDP Rx ring " Björn Töpel
2020-09-08 10:32 ` Maxim Mikityanskiy
2020-09-08 11:37 ` Magnus Karlsson
2020-09-08 12:21 ` Björn Töpel
2020-09-09 15:37 ` Jesper Dangaard Brouer
2020-09-04 14:27 ` Jesper Dangaard Brouer
2020-09-04 14:32 ` Björn Töpel
2020-09-04 23:58 ` Jakub Kicinski
2020-09-07 13:37 ` Björn Töpel
2020-09-07 18:40 ` Jakub Kicinski
2020-09-08 6:58 ` Björn Töpel
2020-09-08 17:24 ` Jakub Kicinski
2020-09-08 18:28 ` Björn Töpel
2020-09-08 18:34 ` Jakub Kicinski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200907144508.3ddda938@carbon \
--to=brouer@redhat.com \
--cc=ast@kernel.org \
--cc=bjorn.topel@gmail.com \
--cc=bjorn.topel@intel.com \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=hawk@kernel.org \
--cc=intel-wired-lan@lists.osuosl.org \
--cc=john.fastabend@gmail.com \
--cc=kuba@kernel.org \
--cc=magnus.karlsson@intel.com \
--cc=netdev@vger.kernel.org \
--cc=toke@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).