From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E86A0277CAF for ; Thu, 23 Oct 2025 21:00:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761253233; cv=none; b=apl7KcD+aPGJdEfYVZwPnHolhQLrvlPM76SInGYflGoM5MWAuSBYKPpKZ+p6bSaNn6EEUyy0s9aR+sKX5cJfelMmEFj+4wABJJ4LLQJhBgCZBmtjjSrkU97bWBShgzDisyGLJjd/+vKmQcNVFhDB+baY/jJluqcC3zHPWhZneIY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761253233; c=relaxed/simple; bh=mAxysNeIrHgaruiWFWQZqnXGyaXHdWk1zacyuJSvdq4=; h=From:Subject:Date:Message-Id:MIME-Version:Content-Type:To:Cc; b=Ou/yQDB/6mmqk1CUVklfYyh9uIaWR9F/9G4ZDoGcoR85Xm8nXWN/boj4Pss6UBPGajcYHeubbouPClAqIBZpOHYfUKro6nt7iw05wyWI5BFK03TzUTfuf0aqw3p5KqqWg3JTNjy4F8C0YcFg6DrFcTwBOL9LsYnhiA/ucscSOGw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PDDAjSts; arc=none smtp.client-ip=209.85.128.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PDDAjSts" Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-784826b775aso15295837b3.2 for ; Thu, 23 Oct 2025 14:00:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1761253231; x=1761858031; darn=vger.kernel.org; h=cc:to:content-transfer-encoding:mime-version:message-id:date :subject:from:from:to:cc:subject:date:message-id:reply-to; bh=KUE2fgwmLwSez/53F3+hW+9Ssseh5miklnRUVCVob1w=; b=PDDAjStsBRozLIz2miDBzBJwtpxxEL1Py0de7ebAWKaE15UIp1PfqfaSRgDiIRyfIg ji+PSNG8kaKDvzw0sVCOyhjo7cbzQ9VUwIiJ3Lmps+9vkPHnVbu01kgZSOk/O/3dCfHU VIW5egR4ZitjyXe90lSBfVSSFLtMedkiVTsN80Oum1pByYptOOhMpWRK+BohpoRS1SlQ zLDMQMkePNGE2pJHz/pUAOB3UcjvseHw//foy7r40kHSZXFGC7prQ6GK1YMdYX2oradr i+vxJL4WXu5s0xie/XqvDGh3brHp/WUDBiAxxGKb09hUKjwspTcEsRN4RJG9tDOVzHcW eN8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761253231; x=1761858031; h=cc:to:content-transfer-encoding:mime-version:message-id:date :subject:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=KUE2fgwmLwSez/53F3+hW+9Ssseh5miklnRUVCVob1w=; b=fc0ZCv9Z5byXFnfp1ABzfCNznIIHBaCgw/AzxRhVQdxz3/nFfG36U8CQ574RDD+Rdg oWHwO78fzD+NO6Jrmy3R/TcHwtO/mLOZm8kuXwb9Lt8dZkheIQGTnNOjPd737kytdVCh VtoHsm5DgEmjkSKol8LU54TXiWOwLj456KMag96y0cCw9VnLgGWjx6zisegyHZmRhyFX e980A4BKw1nbalKKAmmF8PjdYPTjuN3puJe0YXIk211K0WmL82Nd0oLZZHC590U1zA8m MhGKJSEfHHCt6NAdERMfLdxsjvUrLcgNA23R9MQ3OBCiCeJ2Cw4THbUNdqoTDuLhJDu9 v7dw== X-Forwarded-Encrypted: i=1; AJvYcCWwAK2rCr3r33P7IFWy/+ivi2UpJ3PAUytrzJKWpNluwpGQYYr9RTTZCuYDgU1HjVHd+dkLW8sqnAwQNuM=@vger.kernel.org X-Gm-Message-State: AOJu0YzoRm661tCwlNFgrqbPIqB04AXaQ4rJd6xHUgVdPy4hnZRkiHzG RwfkXIuNLNqqyHnOznQn8XiUffs4DLeCT4KNc+1jMODnP8T9XrEDV4/OYTbMBAXc X-Gm-Gg: ASbGncuZR09Eq+vXo16Y/uNrri9dwxrabHpHgacpFrrbGg3EvoNHwfYT8eJu64ASB+w mYNPvhG0+x0Ep+HymyYvvZ260TMnLxjjdOTIGRLP+Rz6HMx5R8WM0efvXGEJKMNeLNjPeJdRnPO 5Gtsv1z00zlpSLkS0a2YTHLFXymIPj2Ry59g0TpE2xmHa/9TMlUcz9Qu3BSJqIfU70d2b90N0FC 025xyd4shvUMUvdFP9jBs1UtbVt9Y8cRT6SXoOy77HIHR4xs4njOhM1AK26wlnQW/CaEMPccmdl kKO9F8lMbWOl9NkItPoWv51bqSm9T1s0xNtU0kG6l12uuKMmAOUS3YhaB57Vz6Hr0WGI2W934hL 9cnltlUQMCy3CpXYrVTxieQn315Sbhr0XtgchsCO0DfpIqZ12dif5Pw+Ehe3ltlWuZHR3jg452U 6MgGwpd9VCX3I= X-Google-Smtp-Source: AGHT+IE60+L8TG1H5hAGOW+FcGHHNvbExdUzbyadSWQCoD6D3xa2yre0fB7tmeRoBJZcTO57oPVImw== X-Received: by 2002:a05:690c:600d:b0:782:f343:62af with SMTP id 00721157ae682-7836d374562mr228723197b3.61.1761253230843; Thu, 23 Oct 2025 14:00:30 -0700 (PDT) Received: from localhost ([2a03:2880:25ff:4e::]) by smtp.gmail.com with ESMTPSA id 00721157ae682-785cd6ec6besm8578607b3.50.2025.10.23.14.00.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Oct 2025 14:00:30 -0700 (PDT) From: Bobby Eshleman Subject: [PATCH net-next v5 0/4] net: devmem: improve cpu cost of RX token management Date: Thu, 23 Oct 2025 13:58:19 -0700 Message-Id: <20251023-scratch-bobbyeshleman-devmem-tcp-token-upstream-v5-0-47cb85f5259e@meta.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-B4-Tracking: v=1; b=H4sIAOuW+mgC/5XOTW7DIBCG4atErDsVYHCgq96j6oKfSY1awAJqJ Yp89yJWVndefhrpeedJKpaAlbxdnqTgFmrIqQ/5ciFuMekLIfi+CadcUsU1VFdMcwvYbO0D6/K D0STwuEWM0NwKLX9jgt+1toImAtfcIrsKL+lEuroWvIX7KH6QhA0S3hv57Jcl1JbLY7yysXEfV U356erGgILXYmaztlJK+h6xmVeX40ht/MAzdp7nnXeK+uskrRdy+sdPB57P5/mp81QJK2brjVK 3A7/v+x8pILDmtQEAAA== To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Kuniyuki Iwashima , Willem de Bruijn , Neal Cardwell , David Ahern , Mina Almasry Cc: Stanislav Fomichev , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Bobby Eshleman X-Mailer: b4 0.13.0 This series improves the CPU cost of RX token management by replacing the xarray allocator with an niov array and a uref field in niov. Improvement is ~13% cpu util per RX user thread. Using kperf, the following results were observed: Before: Average RX worker idle %: 13.13, flows 4, test runs 11 After: Average RX worker idle %: 26.32, flows 4, test runs 11 Two other approaches were tested, but with no improvement. Namely, 1) using a hashmap for tokens and 2) keeping an xarray of atomic counters but using RCU so that the hotpath could be mostly lockless. Neither of these approaches proved better than the simple array in terms of CPU. The sysfs /proc/sys/net/core/devmem_autorelease is added to opt-out of the optimization, but give users the performance gain by default. Note that prior revs reported only a 5% gain. This lower gain was measured with cpu frequency boosting (unknowingly) disabled. A consistent ~13% is measured for both kperf and nccl workloads with cpu frequency boosting on. Signed-off-by: Bobby Eshleman --- Changes in v5: - add sysctl to opt-out of performance benefit, back to old token release - Link to v4: https://lore.kernel.org/all/20250926-scratch-bobbyeshleman-devmem-tcp-token-upstream-v4-0-39156563c3ea@meta.com Changes in v4: - rebase to net-next - Link to v3: https://lore.kernel.org/r/20250926-scratch-bobbyeshleman-devmem-tcp-token-upstream-v3-0-084b46bda88f@meta.com Changes in v3: - make urefs per-binding instead of per-socket, reducing memory footprint - fallback to cleaning up references in dmabuf unbind if socket leaked tokens - drop ethtool patch - Link to v2: https://lore.kernel.org/r/20250911-scratch-bobbyeshleman-devmem-tcp-token-upstream-v2-0-c80d735bd453@meta.com Changes in v2: - net: ethtool: prevent user from breaking devmem single-binding rule (Mina) - pre-assign niovs in binding->vec for RX case (Mina) - remove WARNs on invalid user input (Mina) - remove extraneous binding ref get (Mina) - remove WARN for changed binding (Mina) - always use GFP_ZERO for binding->vec (Mina) - fix length of alloc for urefs - use atomic_set(, 0) to initialize sk_user_frags.urefs - Link to v1: https://lore.kernel.org/r/20250902-scratch-bobbyeshleman-devmem-tcp-token-upstream-v1-0-d946169b5550@meta.com --- Bobby Eshleman (4): net: devmem: rename tx_vec to vec in dmabuf binding net: devmem: refactor sock_devmem_dontneed for autorelease split net: devmem: use niov array for token management net: add per-netns sysctl for devmem autorelease include/net/netmem.h | 1 + include/net/netns/core.h | 1 + include/net/sock.h | 8 +++- net/core/devmem.c | 57 +++++++++++++++------- net/core/devmem.h | 13 ++++- net/core/net_namespace.c | 1 + net/core/sock.c | 115 +++++++++++++++++++++++++++++++++++++-------- net/core/sysctl_net_core.c | 9 ++++ net/ipv4/tcp.c | 69 ++++++++++++++++++++------- net/ipv4/tcp_ipv4.c | 12 +++-- net/ipv4/tcp_minisocks.c | 3 +- 11 files changed, 229 insertions(+), 60 deletions(-) --- base-commit: 61b7ade9ba8c3b16867e25411b5f7cf1abe35879 change-id: 20250829-scratch-bobbyeshleman-devmem-tcp-token-upstream-292be174d503 Best regards, -- Bobby Eshleman