From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 569923E0C58 for ; Fri, 8 May 2026 12:43:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.49 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778244229; cv=none; b=KcFX8uGyiMx2sCojuLSnweIWGOHZOsllSUOVa6dHmegKgT8sLS/EYrOdD/VK+Y1FXBnl2pGlPVzPWrb4HMWZTH+xjdrCuL7yO1YjGTvYY3eWNP7Sm81/GTqHHXPEyTCYDYjZu6iG9Yt+3dyBPUwRx4msnnBjySwLErf0xWr26A0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778244229; c=relaxed/simple; bh=DLlT5gJ88mWG0cJW0N7+9cR72wBKSizYhJaXd2JhuOk=; h=Date:From:To:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ru8tEW4AMbz6oXu3omyPehJfe5DxgfhHwSv0dFCLR/1yvY66cj4yPGwlPLdatNoL4qjaJxCrrKpM1/g3WrhLkHf6YnXGX2tE0eCfusniVj0Q3nUnCTyYUWEYY4lucaAbHj+QyV2kZW+egmeef0qb5flpQ2gP68vOm0CTMgz3Z1E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AXT1anL4; arc=none smtp.client-ip=209.85.221.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AXT1anL4" Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-445795cf6f1so1269125f8f.1 for ; Fri, 08 May 2026 05:43:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1778244227; x=1778849027; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=hHaDVqGuSgw1dOeDgbNYynn5tMPklJkBwGnBT4ff2H8=; b=AXT1anL4AKNnlwziF3Q7sYLFiXkBWXLAHCCX9N0TdwRf9v3cfPnfs2FNAN57YIMrQ7 nSlf1ngjvt6pD5UprJQtl6paw6Mw0b8O5/TTzZsb1KE6omRqGSI6+jf+9Mih4Nycjcjw IA79tlt195S0Xb6FhAcX5qltOndyWTlmFhB6Pv7lDz6SfsWbJk5lyAD4RE+N8s55wTFA R286tN2KHbgnDPClJgup51n0iczaLRhcrlD8SWTpwYzVee42/kXaLGnuUAB/dqJRuiuH gw4GV7Zkfg806tEpvLsslf000ce/SjScTOzhpv7aeEfkqcnFVMd1J5L8ec3yd9QPaM+K bSmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778244227; x=1778849027; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=hHaDVqGuSgw1dOeDgbNYynn5tMPklJkBwGnBT4ff2H8=; b=IFyYQjSrLdOIXZx++fP8D/141aOdQBrjZQUhR3vOIJnRRDkQVFl9wvJFQhLK4+e0iN SlFBS2kQxJ6HMZl5Zu13I3XcVcPhaNyXW6hBadZP/8YrY1eqdRZBOihOYnWaH2z0oFoR mEjNYZOY21m78/Ad91cl+HUWtwU8Xmx00lIK0KduYfsANz18yurn1Q2LeudUzyFWb8Wb aGVwthSu08YdYH4yNsO6YxYA/+owpmMSDkpaCNaJAw91QWPg3f3WOhP6rdpRhWOAyH/E fRXeOJiF96jcYEqCwslNBt23GY4Lm4maQ5opTd+H3hxBqPEcbz55w3Cx77q87Gip0aWv GRew== X-Forwarded-Encrypted: i=1; AFNElJ+UrVJVjuHurXuVzp6OtK1pGmRHevvO0KcaVHzqU4lCjdafeNynTvE44Bi3aAbtTHwWuRp1jvZiG9/xLeE=@vger.kernel.org X-Gm-Message-State: AOJu0YxWurugzDQF0pKpXRiofflKWw0i85X8lNlNQ3pqt1VSCXrkM88M ey58kSKFsLykUpWTnFH+WU/Jrjem6BNPqiiZIPtPNr1VGPSzrJ0espFr X-Gm-Gg: Acq92OHVACoVtnHH/ot8Kb/WgjfHFW9oArHT63BJ67AY2HYaXaLnJryDcdgdZxeaUJv az2YC4EV/PIhCB7vo5MInG8W0jCetyFr6neQA9Ouq6YjgkdUmn39GJ+tPv7qmjjP/oj69W1gqY6 mlWGnwFD4yFlk5Aont95xgezcoNrDALOPgc2K6w/SRa6Y8D4Zna35X/5Y6EU6O9So/8MZLLAWD1 k43WP221UObvPQe+V1y2jnJoQ9Q0sIEKsLNBcXEGf4ndBO6skh7toRrNe8UckqDerp1JYgwu9ou 87Yt16fIoYlksxTWlMZPQnWwGT9inclUsM/2x7gOMb8ZfVKrt1Hg4ZiQRvgCMHUBR0NPSc74JSt LQ9uid1mjxziPDE0VcRfdfvpRB3is/ZR607txGQ7Ta/dEbYt8EpDWBxkgjixeqQR5ZzLor8Ex4b RdvmjzK05v3S21QsUxfo6q1dGo6NElHPoSQuXc7cXO+0yANBHAFZxMWfWU8uw/B2Vf X-Received: by 2002:a05:6000:4010:b0:44b:aca1:f480 with SMTP id ffacd0b85a97d-4515a6c3361mr19344538f8f.8.1778244226283; Fri, 08 May 2026 05:43:46 -0700 (PDT) Received: from pumpkin (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4548eb75c29sm4514358f8f.9.2026.05.08.05.43.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 May 2026 05:43:45 -0700 (PDT) Date: Fri, 8 May 2026 13:43:43 +0100 From: David Laight To: Tariq Toukan Cc: Christoph Paasch , Eric Dumazet , Jakub Kicinski , Paolo Abeni , "Andrew Lunn" , "David S. Miller" , Saeed Mahameed , "Mark Bloch" , Leon Romanovsky , , , , Gal Pressman , Dragos Tatulea , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , Stanislav Fomichev , Amery Hung , Alexei Starovoitov Subject: Re: [PATCH net-next V6 2/3] net/mlx5e: Avoid copying payload to the skb's linear part Message-ID: <20260508134343.6651d7c6@pumpkin> In-Reply-To: <20260507095330.318892-3-tariqt@nvidia.com> References: <20260507095330.318892-1-tariqt@nvidia.com> <20260507095330.318892-3-tariqt@nvidia.com> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.38; arm-unknown-linux-gnueabihf) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Thu, 7 May 2026 12:53:29 +0300 Tariq Toukan wrote: > From: Christoph Paasch > > mlx5e_skb_from_cqe_mpwrq_nonlinear() copies MLX5E_RX_MAX_HEAD (256) > bytes from the page-pool to the skb's linear part. Those 256 bytes > include part of the payload. > > When attempting to do GRO in skb_gro_receive, if headlen > data_offset > (and skb->head_frag is not set), we end up aggregating packets in the > frag_list. > > This is of course not good when we are CPU-limited. Also causes a worse > skb->len/truesize ratio,... > > So, let's avoid copying parts of the payload to the linear part. We use > eth_get_headlen() to parse the headers and compute the length of the > protocol headers, which will be used to copy the relevant bits of the > skb's linear part. > > We still allocate MLX5E_RX_MAX_HEAD for the skb so that if the networking > stack needs to call pskb_may_pull() later on, we don't need to reallocate > memory. > > This gives a nice throughput increase (ARM Neoverse-V2 with CX-7 NIC and > LRO enabled): > > BEFORE: > ======= > (netserver pinned to core receiving interrupts) > $ netperf -H 10.221.81.118 -T 80,9 -P 0 -l 60 -- -m 256K -M 256K > 87380 16384 262144 60.01 32547.82 > > (netserver pinned to adjacent core receiving interrupts) > $ netperf -H 10.221.81.118 -T 80,10 -P 0 -l 60 -- -m 256K -M 256K > 87380 16384 262144 60.00 52531.67 > > AFTER: > ====== > (netserver pinned to core receiving interrupts) > $ netperf -H 10.221.81.118 -T 80,9 -P 0 -l 60 -- -m 256K -M 256K > 87380 16384 262144 60.00 52896.06 > > (netserver pinned to adjacent core receiving interrupts) > $ netperf -H 10.221.81.118 -T 80,10 -P 0 -l 60 -- -m 256K -M 256K > 87380 16384 262144 60.00 85094.90 > > Additional tests across a larger range of parameters w/ and w/o LRO, w/ > and w/o IPv6-encapsulation, different MTUs (1500, 4096, 9000), different > TCP read/write-sizes as well as UDP benchmarks, all have shown equal or > better performance with this patch. > > Reviewed-by: Eric Dumazet > Reviewed-by: Saeed Mahameed > Signed-off-by: Christoph Paasch > Signed-off-by: Dragos Tatulea > Signed-off-by: Tariq Toukan > --- > drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 9 +++++++-- > 1 file changed, 7 insertions(+), 2 deletions(-) > > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c > index 75ccf40a7f17..301b33419207 100644 > --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c > +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c ... > @@ -2060,8 +2066,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w > pagep->frags++; > while (++pagep < frag_page); > > - headlen = min_t(u16, MLX5E_RX_MAX_HEAD - len, > - skb->data_len); > + headlen = min_t(u16, headlen - len, skb->data_len); That looks entirely broken. skb->data_len can be larger than 65535 so (u16)skb->data_len can discard significant bits. I can't quite see why the subtract can't overflow either. It is entirely non-obvious. There seem to be far too many u16 local variables in this code. Typically they just make the code larger because they require the compiler mask arithmetic results to 16bits all the time. (Only x86 and m68k have instructions for 8 and 16bit arithmetic.) The same is true for function parameters and results. I think all the min_t() in this file can easily be changed to min(). -- David > __pskb_pull_tail(skb, headlen); > } > } else {