From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 559624307A2 for ; Thu, 30 Jul 2026 13:16:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785417410; cv=none; b=gA6uZHhyE/aLREHtk1xc4PI5r2z5YET5JcSHBnd/i8Tq11OoJo6Q36XY5RyY8CHJFF0o6PXsynGQ7hhgYaNjnImnhsMyZB9soQqr2QJPl0/BsbaRp5Tz6w7nUcGdB9stZVswiL/5TofdstcqspEhOb5watUuyZlc/mKIPqS5vNU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785417410; c=relaxed/simple; bh=0RXI4a470lNh7IuwhJMItl+ExUf0kcUEnOsZDgkQnU4=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=QWtIpofTuZjYlsec5Ysfea3tsa8UFW4bAyMHvjYOsKadccZAQe7oaWfaH+5gU5n2uF/5AVqIHlOeYKRcB1Zb2MbToESabykLAu95dH0jaDIjp6w8t/qjprBAhRBlB5svxa57dL2Hxq5f0JJ2jhYE6FRHph03XUSugvaHHBrPnoI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=LGlpN1xY; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=MTi/DCRL; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="LGlpN1xY"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="MTi/DCRL" From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1785417407; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Stm/OgJbBwyTUJjUafoGzoEoFzkqXFf4gWznhaatNzY=; b=LGlpN1xY2e552mbq3o+2ruA8mSU3xMdHenygnuD1qPeB26P5C7SMuf+CivSVce3CJmt+QR U7pHEA7+hywdqpYUPcfJ5kczRRHBftRjtcK/xfPip+yEEpn8EgV7LLN9dSD/bVLebb4T/o oQwvm9bDiaLRrMJf6KdApe8r1I83/lf5LVtkeo/w0XNy2+hsymvdW8a20VKyQXhSTj3+4D qvbm0eUEOv/serPlAdelEhOGXABAXk1PlXJfA6zi+Vwq7MXuuugg9xHeUuFT5OC22fDeNW AQlKMu+Lr/8yIUZBThJ1GlQy5CjkHgEn+To2b96CzC/Yqjs097fvIJulbRlabQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1785417407; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Stm/OgJbBwyTUJjUafoGzoEoFzkqXFf4gWznhaatNzY=; b=MTi/DCRLKWU+16pS11dpMRXe1crqNOs5jHZKLdRe1sSk86gb2VBhnMrd/jqb9OTtptmqri w2qIes3wU1G4kxBw== To: santhosh kumar Cc: linux-kernel@vger.kernel.org, Christoph Hellwig , Keith Busch , Ming Lei , x86@kernel.org Subject: Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system In-Reply-To: References: <87cxw6hgli.ffs@fw13> Date: Thu, 30 Jul 2026 15:16:46 +0200 Message-ID: <87ldasfwht.ffs@fw13> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Wed, Jul 29 2026 at 11:59, santhosh kumar wrote: > On Wed, Jul 29, 2026 at 4:22=E2=80=AFAM Thomas Gleixner wrote: >> Nothing to see here. It's simply resource exhaustion. >> >> If you want that odd setup to be supported you have to talk to the NVME >> people and not to a random list of folks which have absolutely nothing >> to do with NVME. I've cleaned up the CC list for you because your AI buddy ignored that request. > I am trying to create 1024 nvme devices , each device requesting 2 msix v= ectors. > Total we need 1024*2=3D2048 msix vectors > Total number of available device vectors: ~ (200 * 384) =3D 76800 > But still seeing following messages: > [ 9166.370948] Interrupt reservation exceeds available resources > [ 9167.455029] Interrupt reservation exceeds available resources > [ 9167.455572] Interrupt reservation exceeds available resources > [ 9167.455888] Interrupt reservation exceeds available resources > > =E2=97=8F Summary: Affinity Mask Generation in Linux 6.13 Copying and pasting the output of your AI buddy is a pretty useless exercise simply because your AI buddy does not understand how all of this works. Neither did you actually validate that his slop makes any sense. Aside of that reports want to be against the latest upstream kernel and not against something which is 8 revisions behind and eventually lacks a ton of updates. But let me explain you why your AI buddy gets it wrong. > The affinity mask is generated in multiple layers, here's the exact flo= w: We all know how that works even without the wisdom of your AI buddy. > The Problem Code: > // lib/group_cpus.c - Creates restricted CPU masks > grp_spread_init_one() > { > cpu =3D cpumask_first(nmsk); // Pick CPU 68 for nvme0q0 > cpumask_set_cpu(cpu, irqmsk); // Mask =3D {68} only! > } See below. > // arch/x86/kernel/apic/vector.c - Can only use that mask > assign_irq_vector_any_locked() > { > affmsk =3D irq_data_get_affinity_mask(irqd); // Gets {68} > assign_vector_locked(irqd, affmsk); // ONLY tries CPU 68! > // =E2=9D=8C No fallback to CPU 200-383 even if they have free vect= ors Again your AI buddy is wrong. assign_irq_vector_any_locked() has a fallback if the mask does not result in an allocatable interrupt. It tries to get a close vector and falls through to the very end of the function, which does: /* Try the full online mask */ return assign_vector_locked(irqd, cpu_online_mask); That's why the function has _any_ in the name. This function has a full fallback. and that function is irrelevant for managed interrupts. It only is invoked for non-managed interrupts. Managed interrupts go through a different code path and that has no fallback by design because that's the fundamental property of managed interrupts. So if group_spread_one() inits the mask for the managed interrupt with only CPU 68 set then there is no fallback because there is no other choice. But that also means that the driver did allocate more than two vectors. Why? If it really only allocates only two, then one is non-managed and the other one is managed. In that case both end up with the a cpumask of 0-383, because spreading _ONE_ interrupt over 384 CPUs simply results in a affinity mask with all CPUs set. Just for illustration with a system with 64 CPUs and two nodes, where CPU 0-15 and 32-47 are on node 0, CPU 16-31 and 48-63 are on node 1, the spreading results in: nr interrupts | nr_masks | masks ----------------------------------------------------------------------- 1 | 1 | [0-63] 2 | 2 | [0-15,32-47] [16-31,48-63] 4 | 4 | [0-7,32-39] [8-15,40-47] [16-23,48-55] ... ... 32 | 32 | [0,32] [1,33] ... [16,48] [17,49] ... 64 | 64 | [0] [1] [2] .... That's the basic principle of the spreading mechanism. It's pretty obvious, no? Can you now explain me how that mechanism ends up creating a affinity mask with a single CPU set if there is only _ONE_ interrupt to be spreaded out? I doubt it, but I can explain to you what happens in principle with NVME and managed interrupts independent of the number of queues per device. Managed interrupts ensure on startup, that for each CPU in the affinity mask of each managed interrupt there is a vector guaranteed available. That means: nr interrupts | nr_masks | CPUs per mask | vectors | vectors | | | per CPU | total ------------------------------------------------------------------------ 1 | 1 | 64 | 1 | 64 2 | 2 | 32 | 1 | 64 4 | 4 | 16 | 1 | 64 ... 32 | 32 | 2 | 1 | 64 64 | 64 | 1 | 1 | 64 So depending on the other interrupts allocated this will fail somewhere around 200 devices for sure. I'm sure it's the same on your side because that's independent of the number of CPUs, even if you failed to provide that information. As I told you before it's simple math and resource exhaustion due to the way how NVME is implemented and the x86 vector space limitations. If you want that to behave differently, then you have to talk to the NVME people as I told you before. Thanks, tglx