From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeff Kirsher Subject: Re: [Intel-wired-lan] [PATCH bpf-next 00/11] AF_XDP: introducing zero-copy support Date: Mon, 04 Jun 2018 13:29:01 -0700 Message-ID: References: <20180604120601.18123-1-bjorn.topel@gmail.com> <20180604163838.5pzojvzrxd2cusny@ast-mbp> Reply-To: jeffrey.t.kirsher@intel.com Mime-Version: 1.0 Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-+Y2VCv5msGypA0c+WjaL" Cc: mykyta.iziumtsev@linaro.org, mst@redhat.com, brian.brooks@linaro.org, magnus.karlsson@gmail.com, andy@greyhouse.net, francois.ozog@linaro.org, willemdebruijn.kernel@gmail.com, daniel@iogearbox.net, ast@fb.com, intel-wired-lan@lists.osuosl.org, brouer@redhat.com, =?ISO-8859-1?Q?Bj=F6rn_T=F6pel?= , michael.lundkvist@ericsson.com, qi.z.zhang@intel.com, michael.chan@broadcom.com, magnus.karlsson@intel.com, netdev@vger.kernel.org, ilias.apalodimas@linaro.org To: Alexei Starovoitov , =?ISO-8859-1?Q?Bj=F6rn_T=F6pel?= Return-path: Received: from mga02.intel.com ([134.134.136.20]:63115 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750950AbeFDU1c (ORCPT ); Mon, 4 Jun 2018 16:27:32 -0400 In-Reply-To: <20180604163838.5pzojvzrxd2cusny@ast-mbp> Sender: netdev-owner@vger.kernel.org List-ID: --=-+Y2VCv5msGypA0c+WjaL Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2018-06-04 at 09:38 -0700, Alexei Starovoitov wrote: > On Mon, Jun 04, 2018 at 02:05:50PM +0200, Bj=C3=B6rn T=C3=B6pel wrote: > > From: Bj=C3=B6rn T=C3=B6pel > >=20 > > This patch serie introduces zerocopy (ZC) support for > > AF_XDP. Programs using AF_XDP sockets will now receive RX packets > > without any copies and can also transmit packets without incurring > > any > > copies. No modifications to the application are needed, but the NIC > > driver needs to be modified to support ZC. If ZC is not supported > > by > > the driver, the modes introduced in the AF_XDP patch will be > > used. Using ZC in our micro benchmarks results in significantly > > improved performance as can be seen in the performance section > > later > > in this cover letter. > >=20 > > Note that for an untrusted application, HW packet steering to a > > specific queue pair (the one associated with the application) is a > > requirement when using ZC, as the application would otherwise be > > able > > to see other user space processes' packets. If the HW cannot > > support > > the required packet steering you need to use the XDP_SKB mode or > > the > > XDP_DRV mode without ZC turned on. The XSKMAP introduced in the > > AF_XDP > > patch set can be used to do load balancing in that case. > >=20 > > For benchmarking, you can use the xdpsock application from the > > AF_XDP > > patch set without any modifications. Say that you would like your > > UDP > > traffic from port 4242 to end up in queue 16, that we will enable > > AF_XDP on. Here, we use ethtool for this: > >=20 > > ethtool -N p3p2 rx-flow-hash udp4 fn > > ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \ > > action 16 > >=20 > > Running the rxdrop benchmark in XDP_DRV mode with zerocopy can then > > be > > done using: > >=20 > > samples/bpf/xdpsock -i p3p2 -q 16 -r -N > >=20 > > We have run some benchmarks on a dual socket system with two > > Broadwell > > E5 2660 @ 2.0 GHz with hyperthreading turned off. Each socket has > > 14 > > cores which gives a total of 28, but only two cores are used in > > these > > experiments. One for TR/RX and one for the user space application. > > The > > memory is DDR4 @ 2133 MT/s (1067 MHz) and the size of each DIMM is > > 8192MB and with 8 of those DIMMs in the system we have 64 GB of > > total > > memory. The compiler used is gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0. > > The > > NIC is Intel I40E 40Gbit/s using the i40e driver. > >=20 > > Below are the results in Mpps of the I40E NIC benchmark runs for 64 > > and 1500 byte packets, generated by a commercial packet generator > > HW > > outputing packets at full 40 Gbit/s line rate. The results are > > without > > retpoline so that we can compare against previous numbers.=20 > >=20 > > AF_XDP performance 64 byte packets. Results from the AF_XDP V3 > > patch > > set are also reported for ease of reference. The numbers within > > parantheses are from the RFC V1 ZC patch set. > > Benchmark XDP_SKB XDP_DRV XDP_DRV with zerocopy > > rxdrop 2.9* 9.6* 21.1(21.5) > > txpush 2.6* - 22.0(21.6) > > l2fwd 1.9* 2.5* 15.3(15.0) > >=20 > > AF_XDP performance 1500 byte packets: > > Benchmark XDP_SKB XDP_DRV XDP_DRV with zerocopy > > rxdrop 2.1* 3.3* 3.3(3.3) > > l2fwd 1.4* 1.8* 3.1(3.1) > >=20 > > * From AF_XDP V3 patch set and cover letter. > >=20 > > So why do we not get higher values for RX similar to the 34 Mpps we > > had in AF_PACKET V4? We made an experiment running the rxdrop > > benchmark without using the xdp_do_redirect/flush infrastructure > > nor > > using an XDP program (all traffic on a queue goes to one > > socket). Instead the driver acts directly on the AF_XDP socket. > > With > > this we got 36.9 Mpps, a significant improvement without any change > > to > > the uapi. So not forcing users to have an XDP program if they do > > not > > need it, might be a good idea. This measurement is actually higher > > than what we got with AF_PACKET V4. > >=20 > > XDP performance on our system as a base line: > >=20 > > 64 byte packets: > > XDP stats CPU pps issue-pps > > XDP-RX CPU 16 32.3M 0 > >=20 > > 1500 byte packets: > > XDP stats CPU pps issue-pps > > XDP-RX CPU 16 3.3M 0 > >=20 > > The structure of the patch set is as follows: > >=20 > > Patches 1-3: Plumbing for AF_XDP ZC support > > Patches 4-5: AF_XDP ZC for RX > > Patches 6-7: AF_XDP ZC for TX >=20 > Acked-by: Alexei Starovoitov > for above patches >=20 > > Patch 8-10: ZC support for i40e. >=20 > these also look good to me. > would be great if i40e experts take a look at them asap. >=20 > If there are no major objections we'd like to merge all of it > for this merge window. We would like a bit more time to review and test the changes, I understand your eagerness for wanting this to get into 4.18 but this change is large enough that a 24-48 hour review time is not prudent, IMHO. Alex also has requested for more time so that he can review the changes as well. I will go ahead and put the entire series in my tree so that our validation team can start to "kick the tires". --=-+Y2VCv5msGypA0c+WjaL Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEiTyZWz+nnTrOJ1LZ5W/vlVpL7c4FAlsVoQ0ACgkQ5W/vlVpL 7c42QRAAmf/JXboTh/ReHnjD874IvJM/9enpr+lFJIDlHr3qRzXF6hyLxkPQPQHZ zf/qRT37nJt7Yvvc0jvrNN1OBBW+zJWMGzH3iK7WKM4BCGl8wR+7DySwDkA+EWnN xS+60szhSgJ5pOi12g8zoygHFpC6Zl9P4l8sLqSnpOc3GAumt67bqGrk9uS1fQYS topJKreex9/z9ZmLRDEmGHzOcI82w9kaVK2Uxx3TpXln/Q2rkLlE5DwEJlfI+EFd 3u1H85To31hPYl+B8DPBw70dsVOxvpco6kMvjopPxB3+C6q9r8DHHFhopeSiYZ8b FzKQ3nqrq6kLbjzf68Dhwx1MHwNJlKCTi6esi4IIeIVL8eHgTUiGhly3IY09hWuJ sYZyx1QjCLwnRwTKY6Irj/CeXEZX9+6eIvudD4/NLyh5UMnOg8sfA0+GMsKnL8vT AwEiy9Qa1jbPwe/VeqiTsbtYP6BrzVUAn7kAiAsS5EjbJv/8aMxbj7DCoVve0q1O 5Hv06gM36T4UEPnmQiXY0iZc1ZFqesM40uXXORxiuXZiDv3TBaXzVHYnLaA5mhm+ jYW7B9A5u7dOzU4DOmjDVw7rh0MTYqXA0cIS/mR9Kz5xcY5F2A00hIm6LR/u2ReD Xdzv5QMOV9r6NTog884luy5yQo1kBhAgLZNoU7MG0daf5BvHRVw= =l1pg -----END PGP SIGNATURE----- --=-+Y2VCv5msGypA0c+WjaL--