From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4FB5BE7718B for ; Thu, 2 Jan 2025 16:39:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id EAC756077C; Thu, 2 Jan 2025 16:39:20 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id KpXrtWlTsgjo; Thu, 2 Jan 2025 16:39:20 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.166.142; helo=lists1.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 4BBAF60758 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1735835960; bh=9g8PyF99klNIKBD2d1euNvk1LC5LNQ1Lnn6m+dnR3R4=; h=Date:From:To:Cc:In-Reply-To:References:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From; b=ipLEwu4gppHAxSX6orkzsli7NjdXeixaK/S3tD4T+RpAuJqUl3MUXNrDv2mO/cZ+x I1jSXn5tNBz3xClgZDH5fxzMM2sH2wVG3/862izQSyn2ncgHjJWoa4HGrkl4YIkkT9 Ronr07Ulg9EggqT6GPhdXtDPtQAhRviphnJg+AxWz7QOkRf24r9IqEjuMCOWqDJZHo aQB3vdZ+A7Gjl3eQh3z3UoTDz4/6YuJwo9sj5EiipoEXGa57IV7XuzqF7nExTnqfpI syVbGaK8H1BhaIgXaCumoqZaQ+jybYu8XreJsjaqOu2I8EmD8GCAVrhS6MxRIeFskf uLd/tI0GbkgEw== Received: from lists1.osuosl.org (lists1.osuosl.org [140.211.166.142]) by smtp3.osuosl.org (Postfix) with ESMTP id 4BBAF60758; Thu, 2 Jan 2025 16:39:20 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists1.osuosl.org (Postfix) with ESMTP id 4F69DE0F for ; Thu, 2 Jan 2025 16:39:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 41F7B6074F for ; Thu, 2 Jan 2025 16:39:18 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id NyOF-d95y2J6 for ; Thu, 2 Jan 2025 16:39:17 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=139.178.84.217; helo=dfw.source.kernel.org; envelope-from=kuba@kernel.org; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org 8EF9660758 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 8EF9660758 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by smtp3.osuosl.org (Postfix) with ESMTPS id 8EF9660758 for ; Thu, 2 Jan 2025 16:39:17 +0000 (UTC) Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1C60C5C6105; Thu, 2 Jan 2025 16:38:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB695C4CED0; Thu, 2 Jan 2025 16:39:15 +0000 (UTC) Date: Thu, 2 Jan 2025 08:39:15 -0800 From: Jakub Kicinski To: Edward Cree Cc: Haifeng Xu , Eric Dumazet , Tony Nguyen , Przemek Kitszel , "David S. Miller" , Paolo Abeni , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org Message-ID: <20250102083915.6e5375a1@kernel.org> In-Reply-To: <87e945f6-2811-0ddb-1666-06accd126efb@gmail.com> References: <87e945f6-2811-0ddb-1666-06accd126efb@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735835956; bh=jPFAeCo60VSV9qQBwJyjLnkLChMzPpcMojnaVDgF5bo=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=mPgqiowk1OW4WsKuS6FSbfUca+/aMwcVgL4mGSx4J4diTEKRVM/na8PIzOCZARtpa EwrXOHSvDO0HHO/cWJkUv8XyL44Mu+FE4TmI0CJ8XjMSJJRtz3b50B/rwFSwC+xRnq 3CQxyBcqgied41PFEWIZhHOlXpV553LDdsYMxWt2PhsqzERXpSQAx50jj7Dn/vs3rW wXivo60R0RqELBvIXsIfMvXYkk17jlpTpBKtvY4/PP3tdDvlKLJ2vcRL8gBEGD1KJc mhMtbOyg2m489tyNSpF7cjIXFWjsqicSmc4R9se6JApJRxDDlnIeDmlODxy4VQduZH Nz7Pf5vQF3eOw== X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dmarc=pass (p=quarantine dis=none) header.from=kernel.org X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=mPgqiowk Subject: Re: [Intel-wired-lan] =?utf-8?b?W1F1ZXN0aW9uXSBpeGdiZe+8mk1lY2hhbmlz?= =?utf-8?q?m_of_RSS?= X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" On Thu, 2 Jan 2025 16:01:18 +0000 Edward Cree wrote: > On 02/01/2025 11:23, Haifeng Xu wrote: > > We want to make full use of cpu resources to receive packets. So > > we enable 63 rx queues. But we found the rate of interrupt growth > > on cpu 0~15 is faster than other cpus(almost twice). > ... > > I am confused that why ixgbe NIC can dispatch the packets > > to the rx queues that not specified in RSS configuration. > > Hypothesis: it isn't doing so, RX is only happening on cpus (and > queues) 0-15, but the other CPUs are still sending traffic and > thus getting TX completion interrupts from their TX queues. > `ethtool -S` output has per-queue traffic stats which should > confirm this. > > (But Eric is right that if you _want_ RX to use every CPU you > should just change the indirection table.) IIRC Niantic had 4 bit entries in the RSS table or some such. It wasn't possible to RSS across more than 16 queues at a time. It's a great NIC but a bit dated at this point. From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B410714BFA2; Thu, 2 Jan 2025 16:39:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735835956; cv=none; b=LvGU7q8SLM1Ce/Da7A+EdW0BXJg6sqNtb5DJpY/Mt4zceCImfJVPogMruWeZDWElcBGj8fRZuLzwIoDxbV5ES0hrGJBHhX4zbZED4NmfZZmb9RZLrUOKPmfcsz7p1BWIZxgLGZLCYu/j6CFxL9kcuMQs8CkD4lWvV533xferulM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735835956; c=relaxed/simple; bh=jPFAeCo60VSV9qQBwJyjLnkLChMzPpcMojnaVDgF5bo=; h=Date:From:To:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=AIOPUqPH8+5Kt34RM6keQ90Y+llVZeNfYG+g1QWOl6mc0RQksioTrIjKyaYQ7ApUHAF8Riw3P9MZwdU1OKVpHS5aXnSUA3g92xkQ90h8yoVZL5DxbFQ4TpfIOGhQej4sD2Dsj41pRHcJFWMDCi1czZWwcb3zsViIKVnVUvnDxzw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mPgqiowk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mPgqiowk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB695C4CED0; Thu, 2 Jan 2025 16:39:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735835956; bh=jPFAeCo60VSV9qQBwJyjLnkLChMzPpcMojnaVDgF5bo=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=mPgqiowk1OW4WsKuS6FSbfUca+/aMwcVgL4mGSx4J4diTEKRVM/na8PIzOCZARtpa EwrXOHSvDO0HHO/cWJkUv8XyL44Mu+FE4TmI0CJ8XjMSJJRtz3b50B/rwFSwC+xRnq 3CQxyBcqgied41PFEWIZhHOlXpV553LDdsYMxWt2PhsqzERXpSQAx50jj7Dn/vs3rW wXivo60R0RqELBvIXsIfMvXYkk17jlpTpBKtvY4/PP3tdDvlKLJ2vcRL8gBEGD1KJc mhMtbOyg2m489tyNSpF7cjIXFWjsqicSmc4R9se6JApJRxDDlnIeDmlODxy4VQduZH Nz7Pf5vQF3eOw== Date: Thu, 2 Jan 2025 08:39:15 -0800 From: Jakub Kicinski To: Edward Cree Cc: Haifeng Xu , Eric Dumazet , Tony Nguyen , Przemek Kitszel , "David S. Miller" , Paolo Abeni , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org Subject: Re: [Question] =?UTF-8?B?aXhnYmXvvJpNZWNoYW5pc20=?= of RSS Message-ID: <20250102083915.6e5375a1@kernel.org> In-Reply-To: <87e945f6-2811-0ddb-1666-06accd126efb@gmail.com> References: <87e945f6-2811-0ddb-1666-06accd126efb@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Thu, 2 Jan 2025 16:01:18 +0000 Edward Cree wrote: > On 02/01/2025 11:23, Haifeng Xu wrote: > > We want to make full use of cpu resources to receive packets. So > > we enable 63 rx queues. But we found the rate of interrupt growth > > on cpu 0~15 is faster than other cpus(almost twice). > ... > > I am confused that why ixgbe NIC can dispatch the packets > > to the rx queues that not specified in RSS configuration. > > Hypothesis: it isn't doing so, RX is only happening on cpus (and > queues) 0-15, but the other CPUs are still sending traffic and > thus getting TX completion interrupts from their TX queues. > `ethtool -S` output has per-queue traffic stats which should > confirm this. > > (But Eric is right that if you _want_ RX to use every CPU you > should just change the indirection table.) IIRC Niantic had 4 bit entries in the RSS table or some such. It wasn't possible to RSS across more than 16 queues at a time. It's a great NIC but a bit dated at this point.