From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACADD410D19 for ; Tue, 4 Aug 2026 21:27:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785878829; cv=none; b=pAeyBGgMCn3ipIAG66gmxe2pRr8GQndNkuw6RpZRyBaPrV2ehedabQFNjiG88b1l88V2U49hYZNezNqU5+zUbCxy9MuY9xxieawp0E1J+8MHM7mHaOlgWtqQ+URetiIVSgUu3m0zuetwwmV6XEcBsFq62+REXXSBljB5BLvqsjw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785878829; c=relaxed/simple; bh=y1hbng64taIZo53uqzaaEbFupMKeH70emJ6qMiD6xWo=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=VRiHiTWNDdGIFb30O/pnSfohQtAtey8NgA6/j/FleU73bE+tOYtN28DwBV5tpNkLBGT3uTCpS9TbiN+lRxa7+JkR1i6fZdYTDXw3lVtrVxnDbK8u5zzJ8/V0xgSEqT2X1UweQYTxy0If1tS9aFYA5pSJBRPEwlNCoR6y/aMDJVA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=JNon9AfG; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=J9lqVlWG; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="JNon9AfG"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="J9lqVlWG" From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1785878825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=98NWwjeQ47AEV9OvSXwGo5rgdF6MuG1JybkZ0BHf7RM=; b=JNon9AfG0zedJyWZ9/MWOusWV0UkxB499OZWE7YDc6tcF5oeQmTX4UHmdyM3vafqrA7txp NBeqnUhXnxr8LKsfsai6Vj9+3SGMAa+PKM6HuLQ21GANet0rSRA/Pi8EOHCCFnPs/PFe6U 8ga6aceBBC7AaOD3Zy2f/NPXvVwqJtYfm3VLExNncUuY+qv3xNV0z4fZZWd1rT9Z4ETe04 7HlEWLCiPpNLLT/z9HPexBVp8by6FEHrPEljoMZ/xDVHXnWMObuMgHvxVqQt6ysrA2ZarG 67lHPiBEGv0GHtpRCtQZu4KFj/KED2cGGDyYev5viphu2PL/jK6ugvPpV0qaeg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1785878825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=98NWwjeQ47AEV9OvSXwGo5rgdF6MuG1JybkZ0BHf7RM=; b=J9lqVlWGs89TbrHWkva9kiWM7CFSIAOleX961HdkbqU0MiajAVyy6gzpbDtpUR6XFiDLrS nBau6D6FuYeV2wCw== To: Keith Busch , Christoph Hellwig Cc: santhosh kumar , linux-kernel@vger.kernel.org, Ming Lei , x86@kernel.org Subject: Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system In-Reply-To: References: <87cxw6hgli.ffs@fw13> <87ldasfwht.ffs@fw13> <87ik5wfeju.ffs@fw13> <20260804141023.GA9171@lst.de> Date: Tue, 04 Aug 2026 23:27:04 +0200 Message-ID: <871pcdfufr.ffs@fw13> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain On Tue, Aug 04 2026 at 04:56, Keith Busch wrote: > On Tue, Aug 04, 2026 at 04:10:23PM +0200, Christoph Hellwig wrote: >> On Mon, Aug 03, 2026 at 05:43:59AM -1000, Keith Busch wrote: >> > On Thu, Jul 30, 2026 at 09:44:21PM +0200, Thomas Gleixner wrote: >> > > On Thu, Jul 30 2026 at 07:28, Keith Busch wrote: >> > > > We can introduce a module parameter to throttle down the maximum number >> > > > of IO queues to allocate per controller. I don't think the driver can >> > > > automatically reason out what the correct number should be because it >> > > > doesn't know how many devices it's going to see. >> > > >> > > I might be missing something here as usual, but the number of io queues >> > > per controller does not affect that at all as I explained before. >> > >> > Oh, you did explain that. I skipped to the end as I had too many >> > distractions last week. Sorry about that, and thanks for the >> > re-explanation. >> > >> > I mistakenly was thinking the effective_affinity was the only hardware >> > resource used. Back to the drawing board. >> >> But given that we don't use multiple tag_set by default this can't >> be the issue of the reported unless they didnt report enabling >> multiple tag sets. > > I believe the use of additional sets makes the problem worse, but I > think we still have a problem even if we have only the one default set. > This is the quote from Thomas that I'm reading into: > > "the per CPU vector consumption is always _one_ per set." > > So just the default set with only 1 NVMe IO queue managed IRQ will still > consume 384 vectors for the reported system even though the effective > affinity is pinned to 1 CPU. Correct. We don't treat the one I/O queue case differently than the multi I/O queue case. We could come up with some less restrictive mechanism, but that would at the end run into the vector limitation on hotplug/hibernate because you can't fit more than ~200 vectors into the last online CPU. OTOH. With 1000 devices which consume also one non-managed interrupt for their management queues, i.e. a total of 1000, that's not going to work anyway. The restriction given by the x86 vector limitations when ignoring that there might be network cards and other devices which consume interrupts as well for the current managed model is: nr_sets nr_nonmanaged device limit 1 1 200/2 = 100 2 1 200/3 = 66 Subtract the interrupts consumed by other devices and your number might become significantly smaller. FRED will lift that restriction in the non-foreseeable future, but until then you can only create a mode which effectively prevents hotplug either upfront or in the hotplug case (see below). The spreading mechanism would be basically the same as for managed mode, so you can spread out several queues per device and let the vector allocation pick one CPU out of the provided mask, but without guarantees that in the hotplug case there is a free vector on another online CPU within the mask. Right now there is no mechanism to figure this out, so lapic_can_unplug_cpu() would have to walk the vectors, look at the affinity mask in each 'light managed' descriptor and validate that _all_ of them can be moved within their masks to prevent hotplug if not. The hackery needed for that will be disgusting at best. The only simple case are devices with one I/O queue per set. There you could just do what Santhosh did: make the interrupts not be managed at all and hotplug will fail at some point when the still online CPUs do not have the capacity to take over the ones of the outgoing CPU. That needs no change and works already today. If you would use two sets with one queue each that works too, but then hotplug will simply fail earlier. Thanks, tglx