From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F76FC5AD4E for ; Mon, 10 Aug 2026 09:42:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:MIME-Version: References:In-Reply-To:Subject:Cc:To:From:Message-ID:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=YRvZvvHIVpj4XP+Xsbw9vpzce7EuvuetlAXvbhkPlhE=; b=iyN2xNX7O2QI5DemBkCP4t1xH+ F5ogMYO+UtclrOiM/a4gCt1v1NAUHFLFlVzOtFWqFdJ1qfuf2i69zsrh+EB1v0uOMy9Z9WMKM5zJd a5GHSPnvMcwSfHDdXdFKkPxqD6j14uHfHWZBsnT12tcMGCnnXZm+AvUIsiy7kIvy213r2EFbmibOZ oEfYpTHIeZFs68DFYfE7RxlPaT0m7bEfuq3d/ok5kYucvgk3hgIyp7ZLsy3ynKGVbtaJ0TvuOK2Ci ID3LZTSqHvj4CAhLac+4fCVPRmtiC1a2Y39ZZtyvjZzVoIUEtkHwMv5jJFSeoUBL2j4y91SICxXyR J6RDqlTg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wtMWA-0000000BQKu-1Ryl; Mon, 10 Aug 2026 09:42:02 +0000 Received: from sea.source.kernel.org ([172.234.252.31]) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1wtMW9-0000000BQKn-3jgk for linux-arm-kernel@lists.infradead.org; Mon, 10 Aug 2026 09:42:01 +0000 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) by sea.source.kernel.org (Postfix) with ESMTP id 6BA3541310; Mon, 10 Aug 2026 09:42:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 44CB81F000E9; Mon, 10 Aug 2026 09:42:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786354921; bh=YRvZvvHIVpj4XP+Xsbw9vpzce7EuvuetlAXvbhkPlhE=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=lSVpwzjmSuwmAGnmejzEHRj/Qb/ufx9XmA+X48gJ9k9URKIk1fNF0kvhsHA856m9H gR8O3bXZrso2oTm7UMQ9SWgkZ5YvCF9BesZYdBlupKe0qX16HnXDItJEjQ/mYpBhcK h7JTBYXdguDPWaNVRzwXyEkzk9p9TtlMQIxn3OaZzKlVud5+QFAmQJkwPC4rXIOOlO ULjLZ6/q6Dn1NsVinRhzZjiT9cceLzc5YB7YU+RXxPpr2aDEiN0OqdgF9KoLG/1l24 Fsx2VdnAbbEvcG20EMziFhk8HLbdDuSLsnTx745AAiB8CpbH5GkQtFc3yZBg1UEZ+o BM9Km83rcOMJQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1wtMW6-0000000E2IC-3tVB; Mon, 10 Aug 2026 09:41:59 +0000 Date: Mon, 10 Aug 2026 10:41:51 +0100 Message-ID: <86wlty1fds.wl-maz@kernel.org> From: Marc Zyngier To: Kohei Enju Cc: Steven Price , kvm@vger.kernel.org, kvmarm@lists.linux.dev, Catalin Marinas , Will Deacon , James Morse , Oliver Upton , Suzuki K Poulose , Zenghui Yu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Joey Gouly , Alexandru Elisei , Christoffer Dall , Fuad Tabba , linux-coco@lists.linux.dev, Ganapatrao Kulkarni , Gavin Shan , Shanker Donthineni , Alper Gun , "Aneesh Kumar K . V" , Emi Kisanuki , Vishal Annapurve , WeiLin.Chang@arm.com, Lorenzo Pieralisi Subject: Re: [PATCH v16 44/45] KVM: arm64: CCA: Require ICH_HCR_EL2.TDIR for realms In-Reply-To: References: <20260803134403.80630-1-steven.price@arm.com> <20260803134403.80630-45-steven.price@arm.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: enju.kohei@fujitsu.com, steven.price@arm.com, kvm@vger.kernel.org, kvmarm@lists.linux.dev, catalin.marinas@arm.com, will@kernel.org, james.morse@arm.com, oliver.upton@linux.dev, suzuki.poulose@arm.com, yuzenghui@huawei.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, joey.gouly@arm.com, alexandru.elisei@arm.com, christoffer.dall@arm.com, tabba@google.com, linux-coco@lists.linux.dev, gankulkarni@os.amperecomputing.com, gshan@redhat.com, sdonthineni@nvidia.com, alpergun@google.com, aneesh.kumar@kernel.org, fj0570is@fujitsu.com, vannapurve@google.com, WeiLin.Chang@arm.com, lpieralisi@kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, 10 Aug 2026 05:58:10 +0100, Kohei Enju wrote: > > On 08/03 14:44, Steven Price wrote: > > KVM advertises realm support when the RMM is available, and allows > > userspace to create a VM with KVM_VM_TYPE_ARM_REALM on that basis. > > > > On CPUs that lack ICH_HCR_EL2.TDIR, KVM uses ICH_HCR_EL2.TC for > > normal guests so that ICC_DIR_EL1 is still trapped via the common GICv3 > > CPU interface trap. Realms cannot rely on the normal hyp-side trap > > handling for that fallback, so advertising RMI support on such systems > > lets userspace create a realm that cannot safely run. > > > > Require the finalized ARM64_HAS_ICH_HCR_EL2_TDIR capability when > > reporting KVM_CAP_ARM_RMI and when accepting KVM_VM_TYPE_ARM_REALM. > > This leaves normal VM creation unchanged on systems that need the TC > > workaround. > > Hi Steven, > > Thanks for your work on upstreaming CCA. > > In the v15 discussion [0], you asked whether the system I was testing was a > "hacked up test system" or closer to "production hardware", and I said I would > share more when the time came. I can now say that this is not a hacked-up test > system. At Fujitsu, we have real hardware (FUJITSU-MONAKA) which implements CCA > (FEAT_RME) but does not implement FEAT_GICv3_TDIR. The hardware details are as > follows: > > - GICv4.2 compliant implementation > - Supports FEAT_GICv3, FEAT_GICv3p1, FEAT_GICv4, FEAT_GICv4p1, and FEAT_GICv3_NMI > - Does not support FEAT_GICv3_LEGACY (deprecated) > - Does not support FEAT_GICv3_TDIR (ICH_VTR_EL2.TDS == 0) > > For reference, compared with Arm Neoverse V3, the virtual GIC configuration is > largely equivalent. The only missing non-deprecated architectural feature is > FEAT_GICv3_TDIR. A *very* significant difference. Given the cost of trapping between R-EL1 and NS-EL2, something as simple as accesses to ICV_PMR_EL1 result in an extremely expensive trap. > > The issue I see is that the CCA KVM code currently does not support a > configuration (non-TDIR/common-trap) that normal KVM already supports. For > normal guests, KVM handles systems without TDIR by using ICH_HCR_EL2.TC and the > existing GICv3 CPU interface emulation path. However, Realm guests currently > fail because the CCA path bypasses that existing emulation path, as Marc also > pointed out in [1]. Plugging CCA in the emulation code will solve the *functional* aspect. The performance aspect is still there, unfortunately, and there isn't much KVM can do about that. > > Also, this is not limited to systems that actually lack TDIR. The same failure > can be reproduced on a TDIR-capable system by booting with: > kvm-arm.vgic_v3_common_trap=1 This is a *debug* option for broken hardware. ThunderX, for example. You really are in good company when this bit is set. > > So it seems that the current CCA KVM implementation does not yet cover a > configuration that normal KVM already supports today, rather than this being a > limitation of the RMM specification or the underlying hardware. > > I've included a patch below which reuses the existing GICv3 early emulation > path for Realm sysreg exits. This patch does not add any new vGIC emulation > code, and leaves the existing vGIC emulation code unchanged. So I believe this > is in line with Marc's request in [1]. With this patch, Realm guests can run > when the common CPU interface trap path is enabled. > > I tested the exact patch both on our real silicon and on QEMU, and > confirmed that all Realm-related tests in kvm-unit-tests-cca passed. KUTs are unfortunately not something that people run in production. I wonder why... Please run a Linux guest compiled with CONFIG_ARM64_PSEUDO_NMI=y and irqchip.gicv3_pseudo_nmi=1 on the command line. Run any significant workload (hackbench, for example), and report the overhead. This will give you the expected impact introduced by the lack of TDIR. M. -- Without deviation from the norm, progress is not possible.