From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2B84C3DA7A for ; Thu, 5 Jan 2023 20:23:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=skNHaOIvmCxxM/q9+AeaoyYTLhVcODODs5U+KFg0a2o=; b=uu9MpMNGEEClxD hVrfLHDz4A5JyKMTs7cg+nLCniGOazxe7eeDZ9202sCOyo1klujbeDuIbGMQ+X7ra+ABwiQ3BxbK8 5uokVyMriDWWzRdvbyqRTpUFTycuqH7dgoXSNTtXXaC6EzdpoPEC5LiwqY0hBJUJd5eqbjERlZE+B +KJqDR8nph1M/0AwP4z2T2XAHjXdsCVqJXvO9nGOuCfGFxY8pAgtdBbX3b+UQdNOo0eZnGn1fThfX WqqYB8ktVDzA3CKrVK131jhOg+dQhyvLRXbUx7eVqCSkyhNDcvAMu7ePn8eJso5sUUlb53hN6tQG/ QqZvrjZzWR0OOLdNC7gA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pDWku-00EUvU-JK; Thu, 05 Jan 2023 20:22:29 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pDNxn-00BGO0-Tk for linux-arm-kernel@lists.infradead.org; Thu, 05 Jan 2023 10:59:13 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 54DAEB81716; Thu, 5 Jan 2023 10:59:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F0202C433D2; Thu, 5 Jan 2023 10:59:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672916349; bh=q5sRND00z/nYDdb5Pj9GpICSC+MozgkMP+AsrilBba4=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=InqIGzLxpQsqnsIhJlEw/W4xYY8Bd4fbLw2otBGQWOWUIKIGt5aLyMmQE2TbBYyFF DUj07xb8Us7FAtR7h2JirHfFkCfmso3HkkH7FdM8YH5974LXsd5z4GIkMEEeVSdHgx gf4f15BXEobCjfyuG2cqVuAwk56XL1DTpoLwjBdOXWW+OX8Gp8ULezuHUM+xHvd32b GYFvoSj2PyCTAVhPMq/n4XtVqqgmD3EjGkMTpvRgr8hxI4eyWvRORNnrBxOBAMl9ze u/IEMtgL7YZWbYAMm5yL78jVgmyAWQXQt34dfBBZBsj8sglnUIK/ODR4pgZXjOw4qu 0jb68zqiUIUbQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pDNxi-00Gyfa-KQ; Thu, 05 Jan 2023 10:59:06 +0000 Date: Thu, 05 Jan 2023 10:59:06 +0000 Message-ID: <86k0216ydh.wl-maz@kernel.org> From: Marc Zyngier To: Shanker Donthineni Cc: Catalin Marinas , Will Deacon , James Morse , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] arm64: gic: increase the number of IRQ descriptors In-Reply-To: <2a0116a8-fbd0-d866-ada0-ed50f0523f1d@nvidia.com> References: <20230104023738.1258925-1-sdonthineni@nvidia.com> <86sfgq7jb3.wl-maz@kernel.org> <2a0116a8-fbd0-d866-ada0-ed50f0523f1d@nvidia.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: sdonthineni@nvidia.com, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230105_025912_303567_232374BD X-CRM114-Status: GOOD ( 36.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, 04 Jan 2023 13:47:03 +0000, Shanker Donthineni wrote: > > Hi Marc, > > On 1/4/23 03:14, Marc Zyngier wrote: > > External email: Use caution opening links or attachments > > > > > > On Wed, 04 Jan 2023 02:37:38 +0000, > > Shanker Donthineni wrote: > >> > >> The default value of NR_IRQS is not sufficient to support GICv4.1 > >> features and ~56K LPIs. This parameter would be too small for certain > >> server platforms where it has many IO devices and is capable of > >> direct injection of vSGI and vLPI features. > >> > >> Currently, maximum of 64 + 8192 (IRQ_BITMAP_BITS) IRQ descriptors > >> are allowed. The vCPU creation fails after reaching count ~400 with > >> kvm-arm.vgic_v4_enable=1. > >> > >> This patch increases NR_IRQS to 1^19 to cover 56K LPIs and 262144 > >> vSGIs (16K vPEs x 16). > >> > >> Signed-off-by: Shanker Donthineni > >> --- > >> Changes since v1: > >> -create from v6.2-rc1 and edit commit text > >> > >> arch/arm64/include/asm/irq.h | 4 ++++ > >> 1 file changed, 4 insertions(+) > >> > >> diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h > >> index fac08e18bcd5..3fffc0b8b704 100644 > >> --- a/arch/arm64/include/asm/irq.h > >> +++ b/arch/arm64/include/asm/irq.h > >> @@ -4,6 +4,10 @@ > >> > >> #ifndef __ASSEMBLER__ > >> > >> +#if defined(CONFIG_ARM_GIC_V3_ITS) > >> +#define NR_IRQS (1 << 19) > >> +#endif > >> + > >> #include > >> > >> struct pt_regs; > > > > Sorry, but I don't think this is an acceptable change. This is a large > > overhead that affects *everyone*, and that will eventually be too > > small anyway with larger systems and larger interrupt spaces. > > > > A better way to address this would be to move to a more dynamic > > allocation, converting the irqdesc rb-tree into an xarray, getting rid > > of the bitmaps (the allocation bitmap and the resend one), and track > > everything in the xarray. > > The actual memory allocation for IRQ descriptors is still dynamic for ARM64. > This change increases static memory for variable 'allocated_irqs' by 64KB, > feel not a noticeable overhead. 64kB for each bitmap, so that's already 128kB (you missed the irqs_resend bitmap). And that's for a number of IRQs that is still way below what the GIC architecture supports today. The architecture supports 32bit INTIDs, and that's 1GB worth of bitmaps, only for the physical side. Add the virtual stuff for which we create host-side descriptors, and we can go way beyond that. So what happens next, once you exceed the arbitrary limit that only satisfies your own use case? We will bump it up again, and again, bloating the kernel with useless static data that *nobody* needs. Specially not the VMs that you plan to run. So I'm putting my foot down right now, and saying that it needs to be fixed once and for all. The current scheme was OK for small interrupt spaces, but it isn't fit for purpose anymore, certainly not with things like the GICv4 architecture. I'm happy to help with it, but I'm certainly not willing to accept any sort of new compile-time limit. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel