From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0D63DC3DA4A for ; Thu, 22 Aug 2024 21:21:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:MIME-Version: Message-ID:Date:References:In-Reply-To:Subject:Cc:To:From:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=w8w5KFvwMEkPaR1nlkraOvagGUvNuN8aW94+98RSA6Q=; b=feTbnBjjyJR0iWK6ZfUkQ3g45y 4HYwJZHce5gFukyEwzYcM6yc4HJyLtWbqiiGQ8+0LPW/hZQx/XejQz7agERayOGeL7DTQPlA4TC3e fLGzxDOTHizoGyt16pHLbbFM4LuSfFS6kHFXQ/Sgflej6zrxF/gixA/O/K13cgbhtttHZEX0xm9nY 8L7jRFvoreFe6r3KSfaVqkEuua/SPrD2Lq6niaNcQtpzlA7w30+QEUWkH51dZKtvSo5s4aNhjodX0 lU0ndadxOBBFPTTLzxeWJBXBKTVuyPUyGe8XtucjHRub1g15aYjdxMGlBxep3tV/QgZTT5kedO7Rg XlBwte/g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1shFFY-0000000ELGT-2T1H; Thu, 22 Aug 2024 21:21:44 +0000 Received: from galois.linutronix.de ([193.142.43.55]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1shFEd-0000000EL7i-2ezk for linux-arm-kernel@lists.infradead.org; Thu, 22 Aug 2024 21:20:56 +0000 From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1724361644; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=w8w5KFvwMEkPaR1nlkraOvagGUvNuN8aW94+98RSA6Q=; b=NttPQIj49COj8Q3doWN4BqlRwYJ6oFot5UkRfolWKXvNI6c8gfH7qAUVNUofNftT4uKwTt KrKksbzvD5cs80lhYYo5GPMKh7znPTCAt8bMxth3B+l28SQAV7/LgPHx8PMc7TysDVr3Gv zbk4nWr/FfO2EGpgZ75d5BD4ZEarWeDiUBDvMC59Nm+QWaISu2JCUqLmdG7UIKJ9R9m1s3 K+ZuGwEweuhpoKjwX0AoKvsgS7MmCVl2EqyagvWxk9zRU8gEW+jjWIDqKtEm8hdU8Evrs7 Sc/M429m2F9exiyq/bnLwxlA4GklODeVQDs3vt8AfQB9UNZ4Kh3emVxhJTm4Fg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1724361644; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=w8w5KFvwMEkPaR1nlkraOvagGUvNuN8aW94+98RSA6Q=; b=q/V1sD4C5mziHs7on644hl9X0kv7fJs3fKc/kI2dn1PA5qwVHviMg3KmFCTOoy3XN1f+/n lB+lMODTwRy834Aw== To: Marc Zyngier , Kunkun Jiang Cc: Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , "open list:IRQ SUBSYSTEM" , "moderated list:ARM SMMU DRIVERS" , kvmarm@lists.linux.dev, "wanghaibin.wang@huawei.com" , nizhiqiang1@huawei.com, "tangnianyao@huawei.com" , wangzhou1@hisilicon.com Subject: Re: [bug report] GICv4.1: multiple vpus execute vgic_v4_load at the same time will greatly increase the time consumption In-Reply-To: <864j7cybay.wl-maz@kernel.org> References: <86msl6xhu2.wl-maz@kernel.org> <867cc9x8si.wl-maz@kernel.org> <864j7cybay.wl-maz@kernel.org> Date: Thu, 22 Aug 2024 23:20:43 +0200 Message-ID: <87o75kgspg.ffs@tglx> MIME-Version: 1.0 Content-Type: text/plain X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240822_142047_843385_D52DC1BD X-CRM114-Status: GOOD ( 18.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Aug 22 2024 at 13:47, Marc Zyngier wrote: > On Thu, 22 Aug 2024 11:59:50 +0100, > Kunkun Jiang wrote: >> > but that will eat a significant portion of your stack if your kernel is >> > configured for a large number of CPUs. >> > >> >> Currently CONFIG_NR_CPUS=4096,each `struct cpumask` occupies 512 bytes. > > This seems crazy. Why would you build a kernel with something *that* > big, specially considering that you have a lot less than 1k CPUs? That's why CONFIG_CPUMASK_OFFSTACK exists, but that does not help in that context. :) >> > The removal of this global lock is the only option in my opinion. >> > Either the cpumask becomes a stack variable, or it becomes a static >> > per-CPU variable. Both have drawbacks, but they are not a bottleneck >> > anymore. >> >> I also prefer to remove the global lock. Which variable do you think is >> better? > > Given the number of CPUs your system is configured for, there is no > good answer. An on-stack variable is dangerously large, and a per-CPU > cpumask results in 2MB being allocated, which I find insane. Only if there are actually 4096 CPUs enumerated. The per CPU magic is smart enough to limit the damage to the actual number of possible CPUs which are enumerated at boot time. It still will over-allocate due to NR_CPUS being insanely large but on a 4 CPU machine this boils down to 2k of memory waste unless Aaarg64 is stupid enough to allocate for NR_CPUS instead of num_possible_cpus()... That said, on a real 4k CPU system 2M of memory should be the least of your worries. > You'll have to pick your own poison and convince Thomas of the > validity of your approach. As this is an operation which is really not suitable for on demand or large stack allocations the per CPU approach makes sense. Thanks, tglx