From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 50596C4708E for ; Thu, 5 Jan 2023 21:24:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:From:References:Cc:To: Subject:Date:Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RnCZs/Eml8CI14WvvW8mn/UBe0/SzW3VePJD7wPETJc=; b=1JjScmRBGvdHc72QR90aIZ+FZz UdJVE/eUL3+gwvXBGxYCjTTVhI2OxxpQltGPpRajcvW7Hbe7do+lVFNbkE5DInNiiX9VEIqdAS+kp g9CV4Vi5qamQB5W/Ry7QbnffZjNgUma8cTwNyn0hHZbz40jm4TZKWcdKmkmv6+6vW3UWAFr2zOxF4 DKPcwqAANN332VuJjVgz7Bf2hHD7VsKfHQ+RHGvKlK//CTEVGSmxA2pBL/pY2pIkmOo+Km9fsD/c3 rbcy2BnQdDvFZwyLrgDoGbcjOziKgE4ROsk/hrUKto2gVHraZGKbSf3CIbHnAwd3co/DFVLeXPTNI wtiK4YOw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pDXhy-00F3fr-Ql; Thu, 05 Jan 2023 21:23:32 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pDUtZ-00DWsH-7q for linux-arm-kernel@bombadil.infradead.org; Thu, 05 Jan 2023 18:23:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Transfer-Encoding :Content-Type:In-Reply-To:From:References:Cc:To:Subject:Date:Message-ID: Sender:Reply-To:Content-ID:Content-Description; bh=HFmCEfY4FkDCnTO+HrIWCDVcfdPe8PhylFCPvOOjNYc=; b=m3KedfTF/8MXwSPZSKRlhtzu+I sPiGfGuh09O7aQXlD+wgXX4cC87bMUw9AAIWAe2RJxoF4WO2qTXInHnJAGVjUsU4GGuhQZ3RG3GxG 3psYdiPlHwBbq2w2fBevDnw/4D6FZCCskR60nj4e4aMjAxctUqft3HHb5opavcKgLH2r4eOQk78oS xRK6LyR8NGb2Y+3fqkNiuUmyhw5ELgFH4V9q/3JmXXGCLQYIuCM8XNS7AhIu7a1fEv4+oTtiQwCKP nFG6oNh2MW7z96KSBsVCRro/1MenvbkUP45seik08rsqz/i6gUOdCdnZ4UsWVPqBs57++QTyTXNaG rutPS3Rg==; Received: from mail-bn7nam10on2047.outbound.protection.outlook.com ([40.107.92.47] helo=NAM10-BN7-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pDRXC-001OtM-2u for linux-arm-kernel@lists.infradead.org; Thu, 05 Jan 2023 14:48:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nUb/E3FpZOXnY+d24CrijlHAXaLSKkOOoqVg0QMgSsxQsdw95RLGIscygBUyzFKAVpw2ybmomkl3SZgsyFDAflvde9PVDjvllEUVpjupWlDkHR7K98szduCw3vba1v8EkPy14IKIRVuDDlkWDOuHrBw1OHzqexHp55GvWOLI6tyvXt0vqs5KGyL5I1fBmCP8D4jGG5erEwEgATlC7c1s9EfQDIMN0IDq+u5qCrCbor+dgUNmV+8LzntVSordLyOU7uKKYaj6ahlsSm5HF87Ph91g1HzmlMJRMQsLlWmDnh4mjiT48JHmnCNkx6Hl/sGG7Ii1RQSggSOEpJETLsgG4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HFmCEfY4FkDCnTO+HrIWCDVcfdPe8PhylFCPvOOjNYc=; b=k9/J2qru6xcfr0+j/zVcpNjcNcXGasQkkI0PuRGuT4hVawocj8DJaAzsDgVo1EIp+U6RjR09cbOzsMsyQyg/MgMShtnTI7hth13iZi+vaDIjvz0swkD07ehSPwIDg5QGDoBL0ce39v9/fdPu1IdQYEPjZmrOk4pGKtJ8YtCZvbgPS3NZrYHiLo7Qi+uz54+7cja5QYnEDqdjNJX7BCI6cL4WambGZs8Ns68EfQ/nPvex9sdP/MS+gzun8BU4hoW1BUuUddm10bzymUpYVY+UYPmllUWi+sQ4RY4XnE0JjFgL1REgldkyb1ivT2UJ7Q4MECxkPeFPK3svCqX1hS/2ow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HFmCEfY4FkDCnTO+HrIWCDVcfdPe8PhylFCPvOOjNYc=; b=WhX8Dt53z/u1wYM4ZPUYtEagUKZ3LY348hDRthFbj35NL+Gh4AU7a4AdFAyhnIAwM0L6hGMsvutWZGVrxBvQkoTbovojtZZN9y6vMOPphfS6i+dsrHSoLVr8fdEt/TOQc2AWtpJGtP1ZtAZ7X8OYfClYIXEFe0YtBDmjK/ScdwyvV9ZIqWPUdl1kzFhivyGgnFgRWeoNlD+iP2Y8ux/NjdqrH689WykbfjerNnNOprHmOUSNsnOOFFFK2koAz2Jxj6jckm9LYb3Gpxf/TWlNo8ITsyHDSU1w3Ctl0b5okKkbRxauItoGoI5/Cce5kRsd7eVeXWVmZde3eY/3+mUaPQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from MN0PR12MB5762.namprd12.prod.outlook.com (2603:10b6:208:375::12) by DM4PR12MB6231.namprd12.prod.outlook.com (2603:10b6:8:a6::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5944.19; Thu, 5 Jan 2023 14:47:53 +0000 Received: from MN0PR12MB5762.namprd12.prod.outlook.com ([fe80::f263:d93d:ba53:25fe]) by MN0PR12MB5762.namprd12.prod.outlook.com ([fe80::f263:d93d:ba53:25fe%5]) with mapi id 15.20.5944.019; Thu, 5 Jan 2023 14:47:53 +0000 Message-ID: <4cc4114d-7fa5-1c23-3504-0ca4dbdd0f62@nvidia.com> Date: Thu, 5 Jan 2023 08:47:44 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.2 Subject: Re: [PATCH v2] arm64: gic: increase the number of IRQ descriptors Content-Language: en-US To: Marc Zyngier Cc: Catalin Marinas , Will Deacon , James Morse , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org References: <20230104023738.1258925-1-sdonthineni@nvidia.com> <86sfgq7jb3.wl-maz@kernel.org> <2a0116a8-fbd0-d866-ada0-ed50f0523f1d@nvidia.com> <86k0216ydh.wl-maz@kernel.org> From: Shanker Donthineni In-Reply-To: <86k0216ydh.wl-maz@kernel.org> X-ClientProxiedBy: SA9PR13CA0059.namprd13.prod.outlook.com (2603:10b6:806:22::34) To MN0PR12MB5762.namprd12.prod.outlook.com (2603:10b6:208:375::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN0PR12MB5762:EE_|DM4PR12MB6231:EE_ X-MS-Office365-Filtering-Correlation-Id: 8d6b6e2a-2be0-48ac-4ffa-08daef2bd300 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BVlyUaovb0Pc63UrYPjAtVKxUPpTYmhqZmHaoPz7rsLxQJkxwZF69SGzC+SKnQ1wRqQKaO7kGH8OrOtqxSPK3xGhYY3slnlz5iAmU59ALOrMaW6WN5xoGpg1gSqXzDnYvBwAoEWvL5UF/TmqzWfpOMMY+lepVc3g5KGVnbeiKyRuJW26YGdiF/P7HMIUAtuLchJDrTNf66ND9vAH74MM2UJnq58CjDjtJBKgkkV28T1vJEWjULTXy3BU2BP7cUHwfVQIC8rWkort6lJncWjc0EIritsPvEDZLeQVax9OnmNdiGzx+1rAJgDbbIi6wPuKpy4u8Eew3G0K+ZVLlSyISRCPdGOcPnIaOvt7JaiE8Xv4s3TUcFNnmh+8eQTqCkXms1v0fByWNef1FHFniQal5f6jngAeIfi6GT7nrW7OjhLEMtziQGoNB83Ii0xx1Hs3bbPyg9jCpo1lxrRijyjw8038MxTtgu56z2Igmsj6mWh4XZt6vWVbqFa3gurg9qa0gsO0gCg6UEKOGjPSByLMTchwBiE9EFHdQbsQZh0qRPK8GhJw70n7GeyH7wjDYdPbgtegCBN+XFUNmALszHb8CuyafIkauyiag7DJDOMpnjJIecnBB7Z/JxjMV1u8nPhVbq9avUsRfpcoyz6Vj5s1DDqrfiTCvjTBYGEeEMBfzoSLhxiBqvR9BlFNiPcyapWlsBDY1LAu7c/iAFct5s7qMfsrOeJ13hoSQQAwBuu9FIs= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN0PR12MB5762.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(39860400002)(376002)(136003)(346002)(366004)(396003)(451199015)(83380400001)(6512007)(26005)(6666004)(6506007)(86362001)(36756003)(31696002)(2616005)(53546011)(38100700002)(186003)(316002)(4326008)(478600001)(41300700001)(8676002)(2906002)(5660300002)(8936002)(6486002)(31686004)(66476007)(54906003)(66556008)(66946007)(6916009)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?M0E4YWh2WUNHdUhNUmN0anU1T1pGQmdkL1NleFlpTWdDM0NuZnNIdHgyTTZE?= =?utf-8?B?YVZSVkNCb04yRU1DNlVMa0wxV3drb2tkaUJaY05zWXpzbmExT3JhVFJiNERD?= =?utf-8?B?bFJPUjVnOGVKSzFCejVaaTlvQTNZd0xoRFlTc1B0VHNTN2ZZL2pIYWRKWkZv?= =?utf-8?B?SHhTUmF0SzlXYTRMdzl6bkdIMklMeUprRkRvc1BuZzZQczlRVUlhRUhOYVlu?= =?utf-8?B?djhiRmlyV0Y0YlRHSDZZbGRIQ3JVSU5UMm9IZGZyN3BzeFVjNk4xSkpiZFZV?= =?utf-8?B?bkFiOHdBcmZTZ3hZSS9vZE41VExPU3o4UDJ6cU03d2ljRTRsWDRyNG5TRTFG?= =?utf-8?B?em01SEIwNWtHY21OZGJzYnFSRldpVFhtbUU5VVhORFRrRE00dFEwbERFSWNu?= =?utf-8?B?TWU3WGNLVE1DOVVjdUczby9FTmEra3Ftd1V6bXRnSWR6elVIVThtODA5Tyt6?= =?utf-8?B?U1NJU016Z3IxN1ExVXllZ29kNm40U2tTUkZtdG9zUWViOTF3dGV1QjRHWFdQ?= =?utf-8?B?b21jSGN0b2tBdmg5Y1p2NTg3S2lFQ3pESTlUU1FVdEhFSWM2R0M2dklYL3pO?= =?utf-8?B?OHNURnVKSC9pdFJVSW5hUVliV3JKWUtiZzZYRlRMcWRNK2RHb2RmVWhQY3pY?= =?utf-8?B?RzZkM2lTVmcvdGViUmFwdUY5MmlPb2dQQWp3TlRhTHNaOWtyV2xLSE1NTldu?= =?utf-8?B?SHlybTNCZzlEOXc3bFNDN0NDUnllMHNYaW84c1RlSFgxNWtaZ2EybkgvNlNj?= =?utf-8?B?cHlIOTRVSFdsRWt2RjNTZVUxdVd3ZXNnemNPSENvRE9ocE5ER0VCVWZ2WTYy?= =?utf-8?B?STZDQmtVUmRXMFZUak5rNUg1T1NPbkI0S0dVNk1SbW81eFVUdFdkRVVGQzd0?= =?utf-8?B?REdyazFVMjJoOHdlOGFTOUxJSjNLM2VsamQ5bmd2ODRXTThLYWFmakNpbVNC?= =?utf-8?B?aXZKcGU0VUVTY1dkRzZqVk5INFd2Umc3aWR1eGVxMnM3ZjArNjNSQlU4NldI?= =?utf-8?B?Qm5uM0w3c0kwaFFDbWdtd3NsWUE2ZFNCcytCNTZEdEhOUXZtUWRESUo2Vm1k?= =?utf-8?B?VzdLZ3ZqRUd4d0tEYmg0cTZITTRJSUFvdVVXUXNuZU1GUzZrZkhaWG52a1pj?= =?utf-8?B?TGtkbjJZSC81MFhIZ0xKd1c0dG9ObjVwWG5vNzRCaElpTXd1K2VKZi9yRXcw?= =?utf-8?B?S2R4Zm9mRktaeStEcDBiRFE5OVVTQ1BNMFVlZ3orSS8yRno4UHBLVlp3UGx5?= =?utf-8?B?OGd5WHdYeHI1NXFkODIxcVA2OHNMRys4NEI4MENQSHVUZjY4N1NHRldVbUtW?= =?utf-8?B?V1ZzNnY3QW5Kb1hrTVMrWUJITmNRWG05eXkxYkVrditoZ2tYMmRON1d4aU1m?= =?utf-8?B?RzhlY210UFJEVnVFbFJPYmhXMEx1RzZuWkdTRXRlb3J4VG4yN2lvV0RxZkhn?= =?utf-8?B?RmR2V3lhM09uNUZnZkFzR1ZYVTQ4LzZwNE1WTlIzalROTkt4RzErdGVWYmQz?= =?utf-8?B?UkcwaFhkSVFaaWowNVA4WFZUYVNGa3VjcTdXcFN0Q3plRmYzdUJUajZWVE9E?= =?utf-8?B?VkNNS1pLNGhtMDRLaXpTUWxFeFRJbjIvWlA1WnFCUlpBVGF3bWpNQ2FQSk9m?= =?utf-8?B?eURYMHZwekVLZXVia0NqWC9FZVpEV2VPRG5IMVVGc1diaHlRYjBsMGtQNk9D?= =?utf-8?B?UENTSnB1MG9IR0ZIVFNra2lZNWRrT0V5WmJaZzdIbEo1SVk5RVp4eHc3TE9F?= =?utf-8?B?V09vMzRtTXh1bEtqZ2JVZCt4bHpVNTdrVVhHMnhmcmVoTk5qSVRwdGFvZFdk?= =?utf-8?B?Y3UxRHBndytVd1dlOE1VYU5yMzhadkJ4UlRLcWgrTWt2WS9RWnJZa2ZIM0dW?= =?utf-8?B?WGZRRmxCdjlGckdkTzNWcy90TU92VTNkZm5FMklYd2tJMVQzWEY0dVk2QjVE?= =?utf-8?B?LzVpRSsxTmFRNXdEWmJwV0FwM3NLMHBNS1ZQUGdmMExBaC9xOXpsK2l0Nmtw?= =?utf-8?B?Yng0am1HRWdlNVJLVjg5cG5kUG02UW1WT3R4dTB0N1BaVUduRVJPaTNxNUJH?= =?utf-8?B?WWRvUUhCdkhOVG5OMFdiQUd4ZU0wZHRxcXlJc1hkUnQwTURWWm9wUkNPcTFN?= =?utf-8?B?Y3IreHJ1REZ0MEJTSERxQUV1ZmtwcjFWa1FuMjFFZ2h4NWh1R3g2aGc3L2ZW?= =?utf-8?B?eXc9PQ==?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8d6b6e2a-2be0-48ac-4ffa-08daef2bd300 X-MS-Exchange-CrossTenant-AuthSource: MN0PR12MB5762.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jan 2023 14:47:53.0710 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: poNWJu4EwYAkRYl3zFB2Ax5XjMH/mTP3VNkTIyQcfid7A+1+RRy5K/Spb57KdEjHIMUF5Qg2FtFppUDxDMKbyw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6231 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230105_144759_293276_16BEC6D0 X-CRM114-Status: GOOD ( 26.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 1/5/23 04:59, Marc Zyngier wrote: > External email: Use caution opening links or attachments > > > On Wed, 04 Jan 2023 13:47:03 +0000, > Shanker Donthineni wrote: >> >> Hi Marc, >> >> On 1/4/23 03:14, Marc Zyngier wrote: >>> External email: Use caution opening links or attachments >>> >>> >>> On Wed, 04 Jan 2023 02:37:38 +0000, >>> Shanker Donthineni wrote: >>>> >>>> The default value of NR_IRQS is not sufficient to support GICv4.1 >>>> features and ~56K LPIs. This parameter would be too small for certain >>>> server platforms where it has many IO devices and is capable of >>>> direct injection of vSGI and vLPI features. >>>> >>>> Currently, maximum of 64 + 8192 (IRQ_BITMAP_BITS) IRQ descriptors >>>> are allowed. The vCPU creation fails after reaching count ~400 with >>>> kvm-arm.vgic_v4_enable=1. >>>> >>>> This patch increases NR_IRQS to 1^19 to cover 56K LPIs and 262144 >>>> vSGIs (16K vPEs x 16). >>>> >>>> Signed-off-by: Shanker Donthineni >>>> --- >>>> Changes since v1: >>>> -create from v6.2-rc1 and edit commit text >>>> >>>> arch/arm64/include/asm/irq.h | 4 ++++ >>>> 1 file changed, 4 insertions(+) >>>> >>>> diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h >>>> index fac08e18bcd5..3fffc0b8b704 100644 >>>> --- a/arch/arm64/include/asm/irq.h >>>> +++ b/arch/arm64/include/asm/irq.h >>>> @@ -4,6 +4,10 @@ >>>> >>>> #ifndef __ASSEMBLER__ >>>> >>>> +#if defined(CONFIG_ARM_GIC_V3_ITS) >>>> +#define NR_IRQS (1 << 19) >>>> +#endif >>>> + >>>> #include >>>> >>>> struct pt_regs; >>> >>> Sorry, but I don't think this is an acceptable change. This is a large >>> overhead that affects *everyone*, and that will eventually be too >>> small anyway with larger systems and larger interrupt spaces. >>> >>> A better way to address this would be to move to a more dynamic >>> allocation, converting the irqdesc rb-tree into an xarray, getting rid >>> of the bitmaps (the allocation bitmap and the resend one), and track >>> everything in the xarray. >> >> The actual memory allocation for IRQ descriptors is still dynamic for ARM64. >> This change increases static memory for variable 'allocated_irqs' by 64KB, >> feel not a noticeable overhead. > > 64kB for each bitmap, so that's already 128kB (you missed the > irqs_resend bitmap). And that's for a number of IRQs that is still way > below what the GIC architecture supports today. > > The architecture supports 32bit INTIDs, and that's 1GB worth of > bitmaps, only for the physical side. Add the virtual stuff for which > we create host-side descriptors, and we can go way beyond that. > > So what happens next, once you exceed the arbitrary limit that only > satisfies your own use case? We will bump it up again, and again, > bloating the kernel with useless static data that *nobody* needs. > Specially not the VMs that you plan to run. > > So I'm putting my foot down right now, and saying that it needs to be > fixed once and for all. The current scheme was OK for small interrupt > spaces, but it isn't fit for purpose anymore, certainly not with > things like the GICv4 architecture. > > I'm happy to help with it, but I'm certainly not willing to accept any > sort of new compile-time limit. Thanks for helping with a scalable solution instead of static allocation. Please include me whenever patches posted to LKML. I'm happy to verify on NVIDIA server platforms and provide test feedback. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel