From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1071E399CED for ; Thu, 30 Jul 2026 19:44:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785440672; cv=none; b=Di9DHns33zkWnwl+6TDuaH8dU/mMAX+uWuMAdbw9EWT1+0kcDQTWuacqUqoXa+zAI0tNSN1eHQ+Fa8kyg159x48xN+pwn9sSfQOjiLK8XEMf/iLtz0i9AXlnLQIikLq3Sae0jcGoU0q+WhPHPkSbTWo8DEMyIZeCSLfFhf7EB6I= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785440672; c=relaxed/simple; bh=RL+T6xWBMCa/Lh559uVvRpzBuh6p1CqVxUM5IHwGgTg=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=MFCmZK2TOf22+VpUDHlm8A329GZJ+9OZL7JubCfvPJA52AYFM0y0Tk3aMhZaCHA9rAcCkaEie4+6YaPMxlB9UmaMYen0eH8ku2hpDgb8sRETTXW8WnE+KvECfP89Y+Rlzdlay+XdrQdPAY81+My5zlL2po5uH2CAjapzwHElTBE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=IFA1m7YF; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=XZamnHcB; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="IFA1m7YF"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="XZamnHcB" From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1785440663; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=IMk1vMg+zVRHq0ONDje2B0fdkmgdyHLJCVvy8/x6WVI=; b=IFA1m7YFkt5i/gsBzpzCZWbmQ4WgEkNxJ3iM3KnR8mOMwaAyjNZVQqrcskyjsA9YBK0Y3G 4YMtuC8eljgSqIxhfzHzoYhCPw9wZj9A1cv+keZ8p7YgTOkqCajWIk45HTPWjkJqZBrd62 AsQJlFQ32xDSJoJwc5YI+HJxqUZ076fPNpdS1aol/Iv5+jIprL62Id3Q1L682W4p4x1Dou JugY3bTqv1tHPprfNIvXoEwRePRauvWGFTr/bSynbIJ/4zBEr0pm2bCz8xUbEhfjPEfRBt O+eUOuuiku78qygz7MQBXq7WbenvVPanT1b6CoP6XrKczLEIDf1y55rCwSWjyg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1785440663; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=IMk1vMg+zVRHq0ONDje2B0fdkmgdyHLJCVvy8/x6WVI=; b=XZamnHcBc8JWxWwYYOpWUar4QoCtoHhmLVVQxHnQNzqChs2xPIxGC5ZS87LL/Na/m+HQlQ Fku+FKWg70jByhBA== To: Keith Busch Cc: santhosh kumar , linux-kernel@vger.kernel.org, Christoph Hellwig , Ming Lei , x86@kernel.org Subject: Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system In-Reply-To: References: <87cxw6hgli.ffs@fw13> <87ldasfwht.ffs@fw13> Date: Thu, 30 Jul 2026 21:44:21 +0200 Message-ID: <87ik5wfeju.ffs@fw13> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain On Thu, Jul 30 2026 at 07:28, Keith Busch wrote: > On Thu, Jul 30, 2026 at 03:16:46PM +0200, Thomas Gleixner wrote: >> As I told you before it's simple math and resource exhaustion due to the >> way how NVME is implemented and the x86 vector space limitations. >> >> If you want that to behave differently, then you have to talk to the >> NVME people as I told you before. > > We can introduce a module parameter to throttle down the maximum number > of IO queues to allocate per controller. I don't think the driver can > automatically reason out what the correct number should be because it > doesn't know how many devices it's going to see. I might be missing something here as usual, but the number of io queues per controller does not affect that at all as I explained before. The number of io queues just influences the spreading, but not the number of vectors required. The queues segment the system into IO zones from 1 up to min(max_queues, number of CPUs). What influences the vector allocation is the number of sets, i.e. when the PCI controller has enough queues to support read queues, then you have one set for the default queues and one set for the read queues. If not it's only the default set. This is the per set resource consumption: nr queues == | nr_masks | CPUs per mask | vectors | vectors nr_interrupts | | | per CPU | total ------------------------------------------------------------------------ 1 | 1 | 64 | 1 | 64 2 | 2 | 32 | 1 | 64 4 | 4 | 16 | 1 | 64 ... 32 | 32 | 2 | 1 | 64 64 | 64 | 1 | 1 | 64 So it does not matter whether you limit the number of IO queues per PCI controller because the per CPU vector consumption is always _one_ per set. The number of interrupts is irrelevant because they are just virtual Linux software interrupt numbers. They are only limited by memory, but if those tiny allocations fail then the NVME device is the least of your worries. The only relevant limiting factor is the x86 vector space, but who needs more than 640K RAM and 256 interrupt vectors :) This all was designed to make multi queue operations simpler in terms of affinity management and also to guarantee that CPU hot-unplug, which is required for suspend/hibernate, does not end up trying to move hundreds of queues to the boot CPU and ends up in vector exhaustion on x86. The whole concept of multi-queue is built around that. If the last CPU in the affinity mask of a queue goes down, then the multi-queue core has also disabled request queueing for that queue, drained the outstanding requests and shut down the queue in the controller. Then the interrupt core deals with the still requested interrupt and instead of moving it to an online CPU it shuts the interrupt down. When the first CPU in the mask comes online again, the interrupt core starts up the interrupt again and the multi-queue core reenables the queue and makes it available for request queueing again. What Christoph and me wanted to avoid was the insanity to have drivers managing all of this, each driver on it's own. Guess how well that would work in general and especially with CPU hotplug... So there is a very good reason why this was implemented that way. Christoph and myself were well aware of the limitations of x86, but neither did we expect such insanities, nor has this come up until now, which is a whopping 10 years later. Santosh clearly pointed to PCI_IRQ_AFFINITY, which is only used in drivers/nvme/host/pci.c: nvme_probe() nvme_setup_io_queues() nvme_setup_irqs() pci_alloc_irq_vectors_affinity(PCI_IRQ_AFFINITY) That means he actually wants to instantiate 1000+ PCIe devices, where each of them consumes one or two vectors per CPU independent of the number of IO queues the device supports, which is according to my understanding of math not possible on x86. So when he removes PCI_IRQ_AFFINITY from the allocation call, then this won't go into the spreading code and therefore neither ends up with managed interrupts nor with separate read queues. It will allocate one for the management queue and one for an io queue, which means each PCI device consumes two vectors total and 384 CPUs will share that IO queue. And that obviously fits into the available vector space of 384 CPUs. So the only knob you can provide is a 'no-multi-queue' knob, which limits the device to one system wide IO queue and skips the managed interrupt allocation. That obviously will fit into the available vector space across all 384 CPUs, but will provide absymal performance and is guaranteed to fail CPU hotplug operations. If you want to support this 'I want a pony' use case, feel free to do so. Thanks, tglx