From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Federico Parola <fede.parola@hotmail.it>
Cc: brouer@redhat.com, xdp-newbies@vger.kernel.org
Subject: Re: Multi-core scalability problems
Date: Tue, 13 Oct 2020 18:41:00 +0200 [thread overview]
Message-ID: <20201013184100.0704963d@carbon> (raw)
In-Reply-To: <VI1PR04MB3104C89EF8DCB98F5330F36C9E040@VI1PR04MB3104.eurprd04.prod.outlook.com>
On Tue, 13 Oct 2020 15:49:03 +0200
Federico Parola <fede.parola@hotmail.it> wrote:
> Hello,
> I'm testing the performance of XDP when dropping packets using multiple
> cores and I'm getting unexpected results.
> My machine is equipped with a dual port Intel XL710 40 GbE and an Intel
> Xeon Gold 5120 CPU @ 2.20GHz with 14 cores (HyperThreading disabled),
> running Ubuntu server 18.04 with kernel 5.8.12.
> I'm using the xdp_rxq_info program from the kernel tree samples to drop
> packets.
> I generate 64 bytes UDP packets with MoonGen for a total of 42 Mpps.
> Packets are uniformly distributed in different flows (different src
> port) and I use flow direction rules on the rx NIC to send these flows
> to different queues/cores.
> Here are my results:
>
> 1 FLOW:
> Running XDP on dev:enp101s0f0 (ifindex:3) action:XDP_DROP options:no_touch
> XDP stats CPU pps issue-pps
> XDP-RX CPU 0 17784270 0
> XDP-RX CPU total 17784270
>
> RXQ stats RXQ:CPU pps issue-pps
> rx_queue_index 0:0 17784270 0
> rx_queue_index 0:sum 17784270
> ---
>
> 2 FLOWS:
> Running XDP on dev:enp101s0f0 (ifindex:3) action:XDP_DROP options:no_touch
> XDP stats CPU pps issue-pps
> XDP-RX CPU 0 7016363 0
> XDP-RX CPU 1 7017291 0
> XDP-RX CPU total 14033655
>
> RXQ stats RXQ:CPU pps issue-pps
> rx_queue_index 0:0 7016366 0
> rx_queue_index 0:sum 7016366
> rx_queue_index 1:1 7017294 0
> rx_queue_index 1:sum 7017294
> ---
>
> 4 FLOWS:
> Running XDP on dev:enp101s0f0 (ifindex:3) action:XDP_DROP options:no_touch
> XDP stats CPU pps issue-pps
> XDP-RX CPU 0 2359478 0
> XDP-RX CPU 1 2358508 0
> XDP-RX CPU 2 2357042 0
> XDP-RX CPU 3 2355396 0
> XDP-RX CPU total 9430425
>
> RXQ stats RXQ:CPU pps issue-pps
> rx_queue_index 0:0 2359474 0
> rx_queue_index 0:sum 2359474
> rx_queue_index 1:1 2358504 0
> rx_queue_index 1:sum 2358504
> rx_queue_index 2:2 2357040 0
> rx_queue_index 2:sum 2357040
> rx_queue_index 3:3 2355392 0
> rx_queue_index 3:sum 2355392
>
This is what I see with i40e:
unning XDP on dev:i40e2 (ifindex:6) action:XDP_DROP options:no_touch
XDP stats CPU pps issue-pps
XDP-RX CPU 1 8,411,547 0
XDP-RX CPU 2 2,804,016 0
XDP-RX CPU 3 2,803,600 0
XDP-RX CPU 4 5,608,380 0
XDP-RX CPU 5 13,999,125 0
XDP-RX CPU total 33,626,671
RXQ stats RXQ:CPU pps issue-pps
rx_queue_index 0:3 2,803,600 0
rx_queue_index 0:sum 2,803,600
rx_queue_index 1:1 8,411,540 0
rx_queue_index 1:sum 8,411,540
rx_queue_index 2:2 2,804,015 0
rx_queue_index 2:sum 2,804,015
rx_queue_index 3:5 8,399,326 0
rx_queue_index 3:sum 8,399,326
rx_queue_index 4:4 5,608,372 0
rx_queue_index 4:sum 5,608,372
rx_queue_index 5:5 5,599,809 0
rx_queue_index 5:sum 5,599,809
> I don't understand why overall performance is reducing with the number
> of cores, according to [1] I would expect it to increase until reaching
> a maximum value. Is there any parameter I should tune to overcome the
> problem?
That is strange, as my results above show that it does scale on my
testlab on same NIC i40e (Intel Corporation Ethernet Controller XL710
for 40GbE QSFP+ (rev 02)).
Can you try to use this[2] tool:
ethtool_stats.pl --dev enp101s0f0
And notice if there are any strange counters.
[2] https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
> [1]
> https://github.com/tohojo/xdp-paper/blob/master/benchmarks/bench02_xdp_drop.org
My best guess is that you have Ethernet flow-control enabled.
Some ethtool counter might show if that is the case.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
next prev parent reply other threads:[~2020-10-13 16:41 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-13 13:49 Multi-core scalability problems Federico Parola
2020-10-13 16:41 ` Jesper Dangaard Brouer [this message]
2020-10-13 16:44 ` Toke Høiland-Jørgensen
2020-10-14 6:56 ` Federico Parola
2020-10-14 9:15 ` Jesper Dangaard Brouer
2020-10-14 12:17 ` Federico Parola
2020-10-14 14:26 ` Jesper Dangaard Brouer
2020-10-15 12:04 ` Federico Parola
2020-10-15 13:22 ` Jesper Dangaard Brouer
2020-10-19 15:23 ` Federico Parola
2020-10-19 18:26 ` Jesper Dangaard Brouer
2020-10-24 13:57 ` Federico Parola
2020-10-26 8:14 ` Jesper Dangaard Brouer
[not found] <VI1PR04MB3104C1D86BDC113F4AC0CF4A9E050@VI1PR04MB3104.eurprd04.prod.outlook.com>
2020-10-14 8:35 ` Federico Parola
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201013184100.0704963d@carbon \
--to=brouer@redhat.com \
--cc=fede.parola@hotmail.it \
--cc=xdp-newbies@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.