From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl1-f41.google.com (mail-dl1-f41.google.com [74.125.82.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 453A93DF01C for ; Fri, 24 Apr 2026 16:24:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.41 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777047864; cv=none; b=lV6yla5+f5NyMpuoYBO0oxA9DKnwQFRW+Q9dRqD5amWSWlqVX6t2ZzQ6ZH/6A+P16etPaLJNwiPubBjJ9CuiesSDbaLn34ZgQpSf5h8qmiYGwUGEGS9UxF/GG07hX/o1WG1xeLqeNQ1xIqSUBOGIK1iedabcmeGeuUkDPL0dNJ0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777047864; c=relaxed/simple; bh=rKnGZyadXRPn7+yUgtk6UZv9kH+5IP0bOXa8EOkSS0U=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=I+IrSDwKx1CMdFR2jB+CcDg8/VCHNxHJ6i5v+rvXITyD0N8U4OmOJszbacfU3XsHfeDHiyFk6RVE9Mz5ALa58URblsmVyBnHazMz5unCXqWdiFd8T8w2EnTDQdWlxvGE6D6tmJIuYu1T4UX1inUb/QOYwVf/GzRkeG9GbPJdsys= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20251104.gappssmtp.com header.i=@davidwei-uk.20251104.gappssmtp.com header.b=K+S9Wv+L; arc=none smtp.client-ip=74.125.82.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20251104.gappssmtp.com header.i=@davidwei-uk.20251104.gappssmtp.com header.b="K+S9Wv+L" Received: by mail-dl1-f41.google.com with SMTP id a92af1059eb24-12dca45ca21so468289c88.1 for ; Fri, 24 Apr 2026 09:24:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20251104.gappssmtp.com; s=20251104; t=1777047862; x=1777652662; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=+6Od9QgYAcJsV8ihusavCJrmScuqDKBApgvlcBNX0xU=; b=K+S9Wv+Lx4FumHi/iwwLlqeJdpw1H/kIa67Ay1IH9c7VEfD9bxGb5GuxgZFG3t7Id0 evelc8FXc/RrUozpk/zeh/xh/Iyai7lcZFpXCDee9MiXz5FsVAuKVBgx19sXIAnW4Ppu 7xJvG9VedojkNJv9FIerT5jXcPSaIGrNqHEQZBCdUjqJ9iwv5391yJRpj6UAaQnYw9JZ nakVc6pI+56xRCFObv/yLpGfDu1KwP/hJroI1F7A+zg+m5Van15pmgTQb9m9JccOgaFz E2cCUiFEb0Cna/QKzCEqkJXaYf3a16j0VKx36CAC2AWPQ8iSCrHg37alSbRwgHy4fEfw 101A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777047862; x=1777652662; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+6Od9QgYAcJsV8ihusavCJrmScuqDKBApgvlcBNX0xU=; b=iVCsU74rrJNjsFu8UzCDsG8R24weY2LLPvdS4GbmxnBpPABEI1rfvvyfKyzuzjuDp7 ZJTWt0zcPC+uYQ2E5VY++UzKLuNnHDF73EDX7KJntfPm6RKNXmNYKybKRaDdrHmfadZH 8NVwd3mRBLTHiHT51dUF611of/jOf5ar4v7OS0z6/fixOWQH04C+1uSQcp9zGKdjGu9w shfg3CVcLr7B1qnfDhI5ZMB9+kgf0knxME3I4VBxrDMt0FQkkeBG0atVZRXTR4jI6Sti H3n9KL1iL3zuVqntalJx7WkbDGhKnOnYA93ULbBPLRoMmlpYUum1844olpYO74kDXGb8 mnvQ== X-Forwarded-Encrypted: i=1; AFNElJ+xSSa1g+bBlzXdz3OadrovZvKmpaaR1KVf5jKqpDjJ3g818NLk73vCo4cSw/xRJFF5g4/qH4U=@vger.kernel.org X-Gm-Message-State: AOJu0YyxGsu5flCU/8skr5FQpCE5C99S2R6BTceXSKDnLN2c87+RTPZH WYMNJYTn+vmT8KtQ/QZQYJsbzWrwWDktb9hLaGGpIhMBdmZQXk5MqNzzC5h9RmVUBqs= X-Gm-Gg: AeBDiesq1WG4D47BXqJdTTSoLj+cBBomndIxxWx/KKh31Bzo5TyckZw8ikfDwhmM2rR N7/JyvAo0vkzLd+mySN29spIi5QknDQwcyVrYOqHttIVCcTr5ZJJD7dCMOoUk70aQn+kGbxGN73 65bkUAUZoJIZWz0b2PcJ4dZoTxCzLtZOTSzGhVGNP2MtmXhNT5ee0BBUTs1CksqORhZBZ+1adMK OLuzA/fb49jr0lNDToow6tx87RwdqPl9C2j0N68GEBIYkbhQuH9ni+hCGrBl3FVLT2llFkVyKea nE0mifzzt8UKpRTIl1J91/AbPyXdOmHKKVXdO4M0byFEPqjgH42wN+yrfboAVEoafmq61H7y8Ng h7eB3ejJZk1TBQVkvB16b4jFHADCceG72xhBEF1SWdTJ4aUZurAXHWv7ofy7mnKhicSLfGd8gu0 8d9vELIRibdxBDNYtrPtq8KvBS+FqRMMU0g6yKxeRFa2VTzZfInU+fchXVoglr6+yuLcfAVUEgK 4GcQuX9yr4Oi9DnxaJTIn5666jrx5Bykbg= X-Received: by 2002:a05:7300:8607:b0:2d4:94cc:eebb with SMTP id 5a478bee46e88-2e477c9bc50mr17941859eec.13.1777047862086; Fri, 24 Apr 2026 09:24:22 -0700 (PDT) Received: from ?IPV6:2a03:83e0:1156:1:c8f:b917:4342:fa09? ([2620:10d:c090:500::1:eae7]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-2e53d8aed43sm32348927eec.26.2026.04.24.09.24.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 24 Apr 2026 09:24:21 -0700 (PDT) Message-ID: Date: Fri, 24 Apr 2026 09:24:18 -0700 Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH net-next v6 0/2] net: mana: add ethtool private flag for full-page RX buffers To: Dipayaan Roy , Jakub Kicinski Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, pabeni@redhat.com, leon@kernel.org, longli@microsoft.com, kotaranov@microsoft.com, horms@kernel.org, shradhagupta@linux.microsoft.com, ssengar@linux.microsoft.com, ernis@linux.microsoft.com, shirazsaleem@microsoft.com, linux-hyperv@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, stephen@networkplumber.org, jacob.e.keller@intel.com, leitao@debian.org, kees@kernel.org, john.fastabend@gmail.com, hawk@kernel.org, bpf@vger.kernel.org, daniel@iogearbox.net, ast@kernel.org, sdf@fomichev.me, dipayanroy@microsoft.com References: <20260407200216.272659-1-dipayanroy@linux.microsoft.com> <20260409183509.0b24dea6@kernel.org> <20260412125917.4fa8fc8d@kernel.org> <20260416083146.0bb94d2b@kernel.org> Content-Language: en-US From: David Wei In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2026-04-23 05:48, Dipayaan Roy wrote: > On Thu, Apr 16, 2026 at 08:31:46AM -0700, Jakub Kicinski wrote: >> On Tue, 14 Apr 2026 09:00:56 -0700 Dipayaan Roy wrote: >>> I still see roughly a 5% overhead from the atomic refcount operation >>> itself, but on that platform there is no throughput drop when using >>> page fragments versus full-page mode. >> >> That seems to contradict your claim that it's a problem with a specific >> platform.. Since we're in the merge window I asked David Wei to try to >> experiment with disabling page fragmentation on the ARM64 platforms we >> have at Meta. If it repros we should use the generic rx-buf-len >> ringparam because more NICs may want to implement this strategy. > > Hi Jakub, > > Thanks. I think I was not precise enough in my previous reply. > > What I meant is that the atomic refcount cost itself does not appear to > be unique to the affected platform. I see a similar ~5% overhead on > another ARM64 platformi (different vendor) as well. However, on that platform > there is no throughput delta between fragment mode and full-page mode; both reach > line rate. > > On the affected platform, fragment mode shows an additional ~15% > throughput drop versus full-page mode. So the current data suggests that > the atomic overhead is common, but the throughput regression is not > explained by that overhead alone and likely depends on an additional > platform-specific factor. > > Separately, the hardware team collected PCIe traces on the affected > platform and reported stalls in the fragment-mode case that are not seen > in full-page mode. They are still investigating the root cause, but > their current hypothesis is that this is related to that platform’s > PCIe/root-port microarchitecture rather than to page_pool refcounting > alone. > > That said, I agree the right direction depends on whether this > reproduces on other ARM64 platforms. If David is able to reproduce the > same behavior, then using the generic rx-buf-len ringparam sounds like > the better direction. > > Please let me know what David finds, and I can rework the patch > accordingly. Hi Dipayaan. Can you please share more details on your testing setup? * What are you using as the test client/server? iperf3 or something else? * What do you mean specifically by "5% overhead from the atomic refcount operation"? Some specific function? * What are you using to measure? perf? * How many queues, what is the napi softirq affinity? * How many NUMA nodes? Does the problem only appear when crossing? Thanks, David > > > Regards > Dipayaan Roy