From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B66FD1A38D0; Thu, 14 Mar 2024 18:27:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710440856; cv=none; b=uEMakciQGoqU96NluZsVAosGoACRx+mI9vMfM7NJeFG+TojDvQYNZhkcWnj03SwKyA0OBW9Ipa+tdsCVqxTrs/acGoVl1s+uDvKIhxLhbKEM3tyxHOafQAqi/WwVl8wpn+NLJfT7UuadOVUOuBZOO76vKf18KoxwXlUrR1DytLc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710440856; c=relaxed/simple; bh=wvZQ6EkK3Zr78gZKY6mbHSxOYjMQ/6C07pm5tHfcalc=; h=Date:From:To:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=PPG6gHrMruWzQSGGUi2CRrahFsAQjZChMQUDwgvtCpIRtmSZ6DfWFhzJ4BUViennWyiRskm9HBiAwVsCwCJfkGax/xMHEyJM309smc7JfhqW2K//wFBo+02pMZTFneYCgg2ujxD4KmYJxSNWte8thVXU3pv/Xa/xbIL4HsY6J/Q= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pcgIDNo6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pcgIDNo6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5FD3CC433C7; Thu, 14 Mar 2024 18:27:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1710440856; bh=wvZQ6EkK3Zr78gZKY6mbHSxOYjMQ/6C07pm5tHfcalc=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=pcgIDNo6+Jx9g1FVdyq37yqerQyrpShtUOjQ9n27jtT7YMvFQeCuV1qZCgb3XcWQK Q/v3tSCHmPIW56Zim33p+Env+4YdDQ5ztgPOHSZblgNt+Rv3GNKuzlRGtbcyC6jgj4 /af/RGyDWDWZirAiycP/RBEe58UHQvEe45XUragt5UhSnnTk3kM0COQ25ickIbMGkP EGQ5Mq8rpuCWU09X2cKeOeJiuTCSnT6SdndVccRrpIm9eGM15t5TpZmzKZ8ANms1Om 3nW5JwaAM1NxFDPdn05JGGb0VsDv2KFc3PU+Z+qi0kPracqutTqUlTyvdwEtCHres0 BA13Ud52d9jNA== Date: Thu, 14 Mar 2024 11:27:34 -0700 From: Jakub Kicinski To: Shradha Gupta Cc: Haiyang Zhang , Shradha Gupta , "linux-kernel@vger.kernel.org" , "linux-hyperv@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "netdev@vger.kernel.org" , Eric Dumazet , Paolo Abeni , Ajay Sharma , Leon Romanovsky , Thomas Gleixner , Sebastian Andrzej Siewior , KY Srinivasan , Wei Liu , Dexuan Cui , Long Li , Michael Kelley Subject: Re: [PATCH] net :mana : Add per-cpu stats for MANA device Message-ID: <20240314112734.5f1c9f7e@kernel.org> In-Reply-To: <20240314025720.GA13853@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net> References: <1709823132-22411-1-git-send-email-shradhagupta@linux.microsoft.com> <20240307072923.6cc8a2ba@kernel.org> <20240307090145.2fc7aa2e@kernel.org> <20240308112244.391b3779@kernel.org> <20240311041950.GA19647@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net> <20240314025720.GA13853@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Wed, 13 Mar 2024 19:57:20 -0700 Shradha Gupta wrote: > Default interrupts affinity for each queue: > > 25: 1 103 0 2989138 Hyper-V PCIe MSI 4138200989697-edge mana_q0@pci:7870:00:00.0 > 26: 0 1 4005360 0 Hyper-V PCIe MSI 4138200989698-edge mana_q1@pci:7870:00:00.0 > 27: 0 0 1 2997584 Hyper-V PCIe MSI 4138200989699-edge mana_q2@pci:7870:00:00.0 > 28: 3565461 0 0 1 Hyper-V PCIe MSI 4138200989700-edge mana_q3 > @pci:7870:00:00.0 > > As seen the CPU-queue mapping is not 1:1, Queue 0 and Queue 2 are both mapped > to cpu3. From this knowledge we can figure out the total RX stats processed by > each CPU by adding the values of mana_q0 and mana_q2 stats for cpu3. But if > this data changes dynamically using irqbalance or smp_affinity file edits, the > above assumption fails. > > Interrupt affinity for mana_q2 changes and the affinity table looks as follows > 25: 1 103 0 3038084 Hyper-V PCIe MSI 4138200989697-edge mana_q0@pci:7870:00:00.0 > 26: 0 1 4012447 0 Hyper-V PCIe MSI 4138200989698-edge mana_q1@pci:7870:00:00.0 > 27: 157181 10 1 3007990 Hyper-V PCIe MSI 4138200989699-edge mana_q2@pci:7870:00:00.0 > 28: 3593858 0 0 1 Hyper-V PCIe MSI 4138200989700-edge mana_q3@pci:7870:00:00.0 > > And during this time we might end up calculating the per-CPU stats incorrectly, > messing up the understanding of CPU usage by MANA driver that is consumed by > monitoring services. Like Stephen said, forget about irqbalance for networking. Assume that the IRQs are affinitized and XPS set, correctly. Now, presumably you can use your pcpu stats to "trade queues", e.g. 4 CPUs / 4 queues, if CPU 0 insists on using queue 1 instead of queue 0, you can swap the 0 <> 1 assignment. That's just an example of an "algorithm", maybe you have other use cases. But if the problem is "user runs broken irqbalance" the solution is not in the kernel...