From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 50E53C2BA18 for ; Fri, 21 Jun 2024 02:26:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Subject:Cc:To: From:Date:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=3EOjQvHBNQ4HSObNyapAYwFrypjXAGViXwaERjXA6IA=; b=k/KcM00MrjH/6zzOiD/kixnAx0 m1eklJHNpH6bXtPLroatVicEJCqYz+9Zmtc5Hhggg7WVL/KE2/S5nSF/mXE67SPWk8vxXpKNjhDXb 53tXcmCvHY6PwHGOZuUOXrMey2mr6UAiBXrc3Y2BSPSYAmnUSbKnY5iISpFgWyHzffpcM7lO3Qd4F LCVv18zNq9RCy/sPlhtN9uvir0WVhDJ4rqgpsgeAt+sGTm9W7OJih9OBAJMrdDGF6LT/Gy8Id4dfG U9vQOV99Pa0ytFn2tamjlAiIe1lD1SCb2SM5rIHcWCxVkUhFR8Bv39/ooYWBcPWnJ+KUqMkQa3xLS 0eg/E3xA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sKTzA-00000007Rgz-2O5X; Fri, 21 Jun 2024 02:26:44 +0000 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sKTz7-00000007RgT-0kPy for linux-arm-kernel@lists.infradead.org; Fri, 21 Jun 2024 02:26:42 +0000 Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-7046211e455so1099140b3a.3 for ; Thu, 20 Jun 2024 19:26:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718936800; x=1719541600; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=3EOjQvHBNQ4HSObNyapAYwFrypjXAGViXwaERjXA6IA=; b=l5QWvm/0JbxPLB84nNWZSnUKKbGW+y9sjQOpH5F7YNiHaYmCZuHtxeT7Fie2GVnfXZ p3QlcBcPtu842tyE1n0RZbOFOHtkNkZyNhkxyYJJupRYYdRBU9s+c/+DXn7I1lZiVpXQ 8r1CPxKl0tpvOTCQl484pfjRJPiLw/PdIK+Sy0QyM7cII4Zi/+kcS59uNUKPHS6fJqU9 qqsdRjkEnfMQBN7Y0j7eYX2m7C3FJG5iSIHaKe0jc5mFN27wJade7CbBg/wlP2YLC2/u +I1pkS00MTTUUHefqtPxJqcmP/1uWmxwToHfo4nNw6d/ZqHaU8LESkMiJOFr0b2n3Ag7 vsCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718936800; x=1719541600; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3EOjQvHBNQ4HSObNyapAYwFrypjXAGViXwaERjXA6IA=; b=SO9iuPoP1uKCErccywDuZkZUue2L4c+UjYrnah+Ai0v3KTJ0Lh1wk2HYeARlBbtr+B WeGmSdAd1rvoUjKctzdjeFtzPQ0bno7ujxDsWY/CFrevPhXReOtLy0/vgbpMAOSyD02W vQcHt9ThfceKMSwuO6cL1VV49Oe4yCWdlAYo/QP7eE5kDF9tUtsJjncb4wj2VwZRZYRi XcEAIA1S5CZi91BdqSnMIRU0YXMcv930QhCUtk1eCI5MDNwm711kKDp6+brwXMyYK0s8 N8t65Tbzdolv4irXUobQ1ubX5hBVZcPCcdZEFbwTZ5wuiM4+8KFh5fhOk8DAFRiz5l8s lcWw== X-Forwarded-Encrypted: i=1; AJvYcCXSlhhGfU2Y9vNS7BRoZ9GM0wnVQ9RMIIOrHCfOrZNFz6+uFuGpb9yPlus4Nsxhkae+KpPA+fFktutqj3i/5XC/6E3tVX8SXXCORAU26wWQ81uJCwg= X-Gm-Message-State: AOJu0Yx5sJ9BHspptTFce/j3c9x05FhdEC8WDBDlWAt/yEq+PBmWLEaV IkOOPbhnv6WCgyssRLgGrB/tyd4PDZRl0Fv9pOeWOoWs4x0dsYQv X-Google-Smtp-Source: AGHT+IEaTUQSQHEf8JO1QEEzvL12qgq6Mth9Alo5Y74gEn+JuuGA4loLO41+inOPhXbfBBplIF26gQ== X-Received: by 2002:aa7:9d0b:0:b0:704:3251:1b67 with SMTP id d2e1a72fcca58-70629c1d46amr7588666b3a.2.1718936799570; Thu, 20 Jun 2024 19:26:39 -0700 (PDT) Received: from localhost ([129.146.253.192]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7065130c204sm321525b3a.211.2024.06.20.19.26.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Jun 2024 19:26:39 -0700 (PDT) Date: Fri, 21 Jun 2024 10:26:27 +0800 From: Furong Xu <0x1207@gmail.com> To: Serge Semin Cc: "David S. Miller" , Alexandre Torgue , Jose Abreu , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Maxime Coquelin , Joao Pinto , Simon Horman , netdev@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, xfr@outlook.com, rock.xu@nio.com Subject: Re: [PATCH net-next v1] net: stmmac: xgmac: increase length limit of descriptor ring Message-ID: <20240621102627.000060d6@gmail.com> In-Reply-To: References: <20240620085200.583709-1-0x1207@gmail.com> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.34; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240620_192641_432401_8A9D31D2 X-CRM114-Status: GOOD ( 34.01 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, 20 Jun 2024 15:06:57 +0300 Serge Semin wrote: > Hi Furong > > On Thu, Jun 20, 2024 at 04:52:00PM +0800, Furong Xu wrote: > > DWXGMAC CORE supports a ring length of 65536 descriptors, bump max length > > from 1024 to 65536 > > What XGMAC IP-core version are you talking about? The DW XGMAC > IP-core databooks I have define the upper limit much lesser than that. > Hi Serge Thanks for this information. I double checked 3.10a and 3.20a, 3.10a do have a limit to 16K, and 3.20a bump the limit to 64K. So we need to lower the limit to fit all XGMAC versions. And what about your advice to set this limit? > Do you understand that specifying 65K descriptors will cause a huge > amount of memory consumed, right? Each descriptor is equipped with at > least 1-page buffer. If QoS/XGMAC SPH is enabled then each descriptor > is equipped with a second buffer. So 65K-descriptor will cause > allocation of at least 65536 * (4 * 4) bytes + 65536 * PAGE_SIZE > bytes. So it's ~256MB for the smallest possible 4K-pages. Not to > mention that there can be more than one queue, two buffers assigned to > each descriptor and more than a single page allocated for each buffer > in case of jumbos. All of that will multiply the basic ~256MB memory > consumption. > Fully agree with you. This patch is trying to make it possible for ethtool to set a longer descriptor length against XGMAC. All MAC cores still have 512 descriptors allocated by default for both TX and RX, which is defined by DMA_DEFAULT_TX_SIZE and DMA_DEFAULT_RX_SIZE in drivers/net/ethernet/stmicro/stmmac/common.h This patch does not change the default descriptor length for XGMAC core, but give ethtool a chance to set a bigger value than DMA_MAX_TX_SIZE and DMA_MAX_RX_SIZE defined in drivers/net/ethernet/stmicro/stmmac/common.h > Taking all of the above into account, what is the practical reason of > having so many descriptors allocated? Are you afraid your CPU won't > keep up with some heavy incoming traffic? > Heavy incoming traffic on some heavy load system, the max 1024 limit defined by DMA_MAX_RX_SIZE in drivers/net/ethernet/stmicro/stmmac/common.h is too few to achieve high throughput for XGMAC. With this patch, ethtool can set a new length than 1024 > Just a note about GMACs. The only GMAC having the ring-length limited > is DW QoS Eth (v4.x/v5.x). It may have up to 1K descriptors in the > ring. DW GMAC v3.73a doesn't have the descriptors array length constraint. > The last descriptor is marked by a special flag TDESC0.21 and > RDESC1.15, after meeting which the DMA-engine gets back to the first > descriptor in the ring. > > -Serge(y) > > > > > Signed-off-by: Furong Xu <0x1207@gmail.com> > > --- > > .../net/ethernet/stmicro/stmmac/dwxgmac2.h | 2 ++ > > .../ethernet/stmicro/stmmac/stmmac_ethtool.c | 24 +++++++++++++++---- > > 2 files changed, 22 insertions(+), 4 deletions(-) > > > > diff --git a/drivers/net/ethernet/stmicro/stmmac/dwxgmac2.h b/drivers/net/ethernet/stmicro/stmmac/dwxgmac2.h > > index 6a2c7d22df1e..264f4f876c74 100644 > > --- a/drivers/net/ethernet/stmicro/stmmac/dwxgmac2.h > > +++ b/drivers/net/ethernet/stmicro/stmmac/dwxgmac2.h > > @@ -11,6 +11,8 @@ > > > > /* Misc */ > > #define XGMAC_JUMBO_LEN 16368 > > +#define XGMAC_DMA_MAX_TX_SIZE 65536 > > +#define XGMAC_DMA_MAX_RX_SIZE 65536 > > > > /* MAC Registers */ > > #define XGMAC_TX_CONFIG 0x00000000 > > diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c > > index 18468c0228f0..3ae465c5a712 100644 > > --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c > > +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c > > @@ -491,9 +491,16 @@ static void stmmac_get_ringparam(struct net_device *netdev, > > struct netlink_ext_ack *extack) > > { > > struct stmmac_priv *priv = netdev_priv(netdev); > > > + u32 dma_max_rx_size = DMA_MAX_RX_SIZE; > > + u32 dma_max_tx_size = DMA_MAX_TX_SIZE; > > > > - ring->rx_max_pending = DMA_MAX_RX_SIZE; > > - ring->tx_max_pending = DMA_MAX_TX_SIZE; > > + if (priv->plat->has_xgmac) { > > + dma_max_rx_size = XGMAC_DMA_MAX_RX_SIZE; > > + dma_max_tx_size = XGMAC_DMA_MAX_TX_SIZE; > > + } > > + > > + ring->rx_max_pending = dma_max_rx_size; > > + ring->tx_max_pending = dma_max_tx_size; > > Do you understand the consequence of this change, right? > De > > > ring->rx_pending = priv->dma_conf.dma_rx_size; > > ring->tx_pending = priv->dma_conf.dma_tx_size; > > } > > @@ -503,12 +510,21 @@ static int stmmac_set_ringparam(struct net_device *netdev, > > struct kernel_ethtool_ringparam *kernel_ring, > > struct netlink_ext_ack *extack) > > { > > + struct stmmac_priv *priv = netdev_priv(netdev); > > + u32 dma_max_rx_size = DMA_MAX_RX_SIZE; > > + u32 dma_max_tx_size = DMA_MAX_TX_SIZE; > > + > > + if (priv->plat->has_xgmac) { > > + dma_max_rx_size = XGMAC_DMA_MAX_RX_SIZE; > > + dma_max_tx_size = XGMAC_DMA_MAX_TX_SIZE; > > + } > > + > > if (ring->rx_mini_pending || ring->rx_jumbo_pending || > > ring->rx_pending < DMA_MIN_RX_SIZE || > > - ring->rx_pending > DMA_MAX_RX_SIZE || > > + ring->rx_pending > dma_max_rx_size || > > !is_power_of_2(ring->rx_pending) || > > ring->tx_pending < DMA_MIN_TX_SIZE || > > - ring->tx_pending > DMA_MAX_TX_SIZE || > > + ring->tx_pending > dma_max_tx_size || > > !is_power_of_2(ring->tx_pending)) > > return -EINVAL; > > > > -- > > 2.34.1 > > > >