[PATCH 00/11] KVM: arm64: CCA: Prepare Realm memory management

[PATCH 00/11] KVM: arm64: CCA: Prepare Realm memory management

[Steven Price]

This series adds the memory plumbing for CCA building on the basic
plumbing[1] that has just been merged.

These patches add the internal realm VM state and RMM feature checks,
prepare realm stage-2 setup and teardown, and provide RTT mapping and
invalidation helpers. Realm memslots are now contrained to the lower
half of the IPA space, because faults treat the protected and shared
halves as aliasing. The final patch then connects these helpers to the
KVM stage-2 MMU operations.

There's also an integration branch[2] that builds on this series and
includes the realm descriptor creation, REC entry/exit, activation and
the new uAPI. This series by itself is also available as a git
branch[3]. However the new code isn't generally active without the full
integration branch.

[1] https://lore.kernel.org/all/20261007073537.2454351-1-suzuki.poulose@arm.com/
[2] https://gitlab.arm.com/linux-arm/linux-cca
    branch: cca-host-mm/v1/integration
[3] https://gitlab.arm.com/linux-arm/linux-cca
    branch: cca-host-mm/v1/mm

Steven Price (7):
  KVM: arm64: CCA: Reserve the RMM mode
  KVM: arm64: CCA: Check for RMI support at KVM init
  KVM: arm64: CCA: Check for LPA2 support and SHA-256
  KVM: arm64: CCA: Add basic infrastructure for creating a realm
  KVM: arm64: CCA: Tear down RTTs
  KVM: arm64: CCA: Add Realm RTT mapping and invalidation helpers
  KVM: arm64: CCA: Support runtime faulting of memory

Suzuki K Poulose (4):
  KVM: arm64: CCA: Introduce Realms
  KVM: arm64: Handle attributes in stage2_unmap_range
  KVM: arm64: CCA: Prevent memslot modifications after Realm creation
  KVM: arm64: CCA: Add Realm stage-2 MMU operations

 arch/arm64/include/asm/kvm_emulate.h   |  29 +
 arch/arm64/include/asm/kvm_host.h      |  62 +-
 arch/arm64/include/asm/kvm_rmi.h       | 109 ++++
 arch/arm64/include/asm/virt.h          |   1 +
 arch/arm64/kvm/Kconfig                 |   1 +
 arch/arm64/kvm/Makefile                |   2 +-
 arch/arm64/kvm/arm.c                   |  33 +-
 arch/arm64/kvm/hyp/include/nvhe/pkvm.h |   2 +-
 arch/arm64/kvm/hyp/nvhe/switch.c       |   4 +-
 arch/arm64/kvm/hyp/nvhe/timer-sr.c     |   2 +-
 arch/arm64/kvm/mmu.c                   | 242 +++++--
 arch/arm64/kvm/rmi.c                   | 852 +++++++++++++++++++++++++
 12 files changed, 1275 insertions(+), 64 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_rmi.h
 create mode 100644 arch/arm64/kvm/rmi.c

-- 
2.43.0

[PATCH 01/11] KVM: arm64: CCA: Reserve the RMM mode

[Steven Price]

Reserve KVM_MODE_RMM for CCA realm support. The kernel parameter that
selects this mode will be added when the Linux CCA implementation is
ready.

Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/include/asm/kvm_host.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 97332ffcb130..ccd2dc8f73c3 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -120,6 +120,7 @@ enum kvm_mode {
 	KVM_MODE_DEFAULT,
 	KVM_MODE_PROTECTED,
 	KVM_MODE_NV,
+	KVM_MODE_RMM,
 	KVM_MODE_NONE,
 };
 #ifdef CONFIG_KVM
-- 
2.43.0

[PATCH 02/11] KVM: arm64: CCA: Introduce Realms

[Steven Price]

From: Suzuki K Poulose <suzuki.poulose@arm.com>

Add foundational work for supporting Realms.
 - Add a new VM flavor.
 - At KVM init, check if the KVM can support Realms (though not functional
   yet) and will be advertised by static key kvm_rmi_is_available. This
   will be turned on in a later patch, once we have all the bits and
   pieces ready. For now check if we are blessed with KVM_MODE_RMM.
 - Add realm specific tracking in kvm_arch. Since Realm and protected pKVM
   states are mutually exclusive, move them into a union.
 - Also adjust the memslot preparation checks to reflect the changes.

Please note that we cannot create Realm VMs yet. This requires further
changes to the UABI and core RMI driver support, which will come later.

Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/include/asm/kvm_emulate.h   | 16 +++++++
 arch/arm64/include/asm/kvm_host.h      | 57 +++++++++++++++++-------
 arch/arm64/include/asm/kvm_rmi.h       | 61 ++++++++++++++++++++++++++
 arch/arm64/include/asm/virt.h          |  1 +
 arch/arm64/kvm/Makefile                |  2 +-
 arch/arm64/kvm/arm.c                   | 24 ++++++++--
 arch/arm64/kvm/hyp/include/nvhe/pkvm.h |  2 +-
 arch/arm64/kvm/hyp/nvhe/switch.c       |  4 +-
 arch/arm64/kvm/hyp/nvhe/timer-sr.c     |  2 +-
 arch/arm64/kvm/mmu.c                   | 13 +++---
 arch/arm64/kvm/rmi.c                   | 18 ++++++++
 11 files changed, 169 insertions(+), 31 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_rmi.h
 create mode 100644 arch/arm64/kvm/rmi.c

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 334211b8b2ee..32021bdf2610 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -831,4 +831,20 @@ static inline void kvm_reset_vcpu_psci(struct kvm_vcpu *vcpu,
 	vcpu_set_reg(vcpu, 0, reset_state->r0);
 }
 
+static inline enum realm_state kvm_realm_state(struct kvm *kvm)
+{
+	return READ_ONCE(kvm->arch.realm.state);
+}
+
+static inline void kvm_set_realm_state(struct kvm *kvm,
+				       enum realm_state new_state)
+{
+	WRITE_ONCE(kvm->arch.realm.state, new_state);
+}
+
+static inline bool kvm_realm_is_created(struct kvm *kvm)
+{
+	return kvm_vm_is_realm(kvm) && kvm_realm_state(kvm) != REALM_STATE_NONE;
+}
+
 #endif /* __ARM64_KVM_EMULATE_H__ */
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index ccd2dc8f73c3..7d0222047ca1 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -29,6 +29,7 @@
 #include <asm/kvm.h>
 #include <asm/kvm_asm.h>
 #include <asm/kvm_hcall.h>
+#include <asm/kvm_rmi.h>
 #include <asm/vncr_mapping.h>
 
 #define __KVM_HAVE_ARCH_INTC_INITIALIZED
@@ -48,6 +49,13 @@
 #define KVM_VCPU_MAX_FEATURES 10
 #define KVM_VCPU_VALID_FEATURES	(BIT(KVM_VCPU_MAX_FEATURES) - 1)
 
+#define KVM_REALM_VCPU_VALID_FEATURES			\
+	(BIT(KVM_ARM_VCPU_POWER_OFF)		|	\
+	 BIT(KVM_ARM_VCPU_PSCI_0_2)		|	\
+	 BIT(KVM_ARM_VCPU_PTRAUTH_ADDRESS)	|	\
+	 BIT(KVM_ARM_VCPU_SVE)			|	\
+	 BIT(KVM_ARM_VCPU_PTRAUTH_GENERIC))
+
 #define KVM_REQ_SLEEP \
 	KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_IRQ_PENDING		KVM_ARCH_REQ(1)
@@ -409,6 +417,7 @@ enum kvm_arm_vm_flavor {
 	VM_PKVM,		/* Normal guests on pKVM */
 	MARKER(__VM_PROTECTED),
 	VM_PROTECTED_PKVM,	/* Protected VM */
+	VM_REALM,		/* CCA */
 	VM_FLAVOR_MAX
 };
 
@@ -502,11 +511,14 @@ struct kvm_arch {
 	/* Count the number of VNCR_EL2 TLBs */
 	atomic_t vncr_tlb_count;
 
-	/*
-	 * For an untrusted host VM, 'pkvm.handle' is used to lookup
-	 * the associated pKVM instance in the hypervisor.
-	 */
-	struct kvm_protected_vm pkvm;
+	union {
+		/*
+		 * For an untrusted host VM, 'pkvm.handle' is used to lookup
+		 * the associated pKVM instance in the hypervisor.
+		 */
+		struct kvm_protected_vm pkvm;
+		struct realm realm;
+	};
 
 #ifdef CONFIG_PTDUMP_STAGE2_DEBUGFS
 	/* Nested virtualization info */
@@ -1572,30 +1584,43 @@ struct kvm *kvm_arch_alloc_vm(void);
 
 #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
 
-#ifdef __KVM_NVHE_HYPERVISOR__
+#define kvm_vm_is_protected(kvm)	((kvm)->arch.vm_flavor >= __VM_PROTECTED)
 
-#define kvm_vm_is_protected(kvm)			\
-	(is_protected_kvm_enabled() && ((kvm)->arch.vm_flavor == VM_PROTECTED_PKVM))
+#ifdef __KVM_NVHE_HYPERVISOR__
 /*
  * Accessing vcpu->kvm from nVHE hyp stub is tricky, as we need to convert the
  * pointer to the hyp VA. With pKVM, the nVHE code runs with the hyp_vcpu,
- * which is populated correctly and is gated on is_protected_kvm_enabled().
+ * which is populated correctly. Always vcpu_is_protected_pkvm(), which is
+ * gated on is_protected_kvm_enabled().
+ */
+#define vcpu_is_protected(vcpu)		BUILD_BUG_ON(1)
+#else
+#define vcpu_is_protected(vcpu)		kvm_vm_is_protected((vcpu)->kvm)
+#endif
+
+#define kvm_vm_is_protected_pkvm(kvm)					\
+	(is_protected_kvm_enabled() && ((kvm)->arch.vm_flavor == VM_PROTECTED_PKVM))
+
+/*
+ * Rely on is_protected_kvm_enabled() check in kvm_vm_is_protected_pkvm() to
+ * make sure the vcpu->kvm is always valid VA in the context
  */
-#define vcpu_is_protected(vcpu)						\
+#define vcpu_is_protected_pkvm(vcpu)					\
 	({								\
 		struct kvm *__kvm = READ_ONCE((vcpu)->kvm);		\
 									\
-		(__kvm && kvm_vm_is_protected(__kvm));			\
+		(__kvm && kvm_vm_is_protected_pkvm(__kvm));		\
 	})
 
-#else
-
-#define kvm_vm_is_protected(kvm)	((kvm)->arch.vm_flavor >= __VM_PROTECTED)
-#define vcpu_is_protected(vcpu)		kvm_vm_is_protected((vcpu)->kvm)
+#define kvm_vm_is_realm(kvm)		((kvm)->arch.vm_flavor == VM_REALM)
+#define vcpu_is_rec(vcpu)		kvm_vm_is_realm((vcpu)->kvm)
 
 #define kvm_vm_hyp_is_distrusting(kvm)	((kvm)->arch.vm_flavor >= __VM_DISTRUSTING_HYP)
 
-#endif	/* __KVM_NVHE_HYPERVISOR__ */
+#define kvm_arch_has_private_mem(kvm) kvm_vm_is_realm(kvm)
+
+int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature);
+bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
 
 #define kvm_has_mte(kvm)					\
 	(system_supports_mte() &&				\
diff --git a/arch/arm64/include/asm/kvm_rmi.h b/arch/arm64/include/asm/kvm_rmi.h
new file mode 100644
index 000000000000..3194daaacf2b
--- /dev/null
+++ b/arch/arm64/include/asm/kvm_rmi.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2023-2026 ARM Ltd.
+ */
+
+#ifndef __ASM_KVM_RMI_H
+#define __ASM_KVM_RMI_H
+
+/**
+ * enum realm_state - State of a Realm
+ *
+ * Mirrors the RMM's Realm lifecycle states where they are meaningful to KVM,
+ * with REALM_STATE_DYING being a KVM-internal state used to prevent further
+ * requests while teardown is in progress. KVM does not track REALM_SYSTEM_OFF
+ * or REALM_ZOMBIE separately as they naturally lead to teardown.
+ */
+enum realm_state {
+	/**
+	 * @REALM_STATE_NONE:
+	 *      Realm has not yet been created. rmi_realm_create() has not
+	 *      yet been called.
+	 */
+	REALM_STATE_NONE,
+	/**
+	 * @REALM_STATE_NEW:
+	 *      Realm is under construction, Realm Descriptor has been created,
+	 *      but it is not yet activated. Memory may be populated.
+	 */
+	REALM_STATE_NEW,
+	/**
+	 * @REALM_STATE_ACTIVE:
+	 *      Realm has been created and is eligible for execution with
+	 *      rmi_rec_enter(). Memory may no longer be populated with known
+	 *	contents.
+	 */
+	REALM_STATE_ACTIVE,
+	/**
+	 * @REALM_STATE_DYING:
+	 *      Realm is in the process of being destroyed or has already been
+	 *      destroyed.
+	 */
+	REALM_STATE_DYING,
+	/**
+	 * @REALM_STATE_DEAD:
+	 *      Realm has been destroyed.
+	 */
+	REALM_STATE_DEAD
+};
+
+/**
+ * struct realm - Additional per VM data for a Realm
+ *
+ * @state: The lifetime state machine for the realm
+ */
+struct realm {
+	enum realm_state state;
+};
+
+void kvm_init_rmi(void);
+
+#endif /* __ASM_KVM_RMI_H */
diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h
index b546703c3ab9..92cec42952f4 100644
--- a/arch/arm64/include/asm/virt.h
+++ b/arch/arm64/include/asm/virt.h
@@ -87,6 +87,7 @@ void __hyp_reset_vectors(void);
 bool is_kvm_arm_initialised(void);
 
 DECLARE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
+DECLARE_STATIC_KEY_FALSE(kvm_rmi_is_available);
 
 static inline bool is_pkvm_initialized(void)
 {
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 7aaeeb84e788..66a719ab5b6b 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -16,7 +16,7 @@ CFLAGS_handle_exit.o += -Wno-override-init
 kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
 	 inject_fault.o va_layout.o handle_exit.o config.o \
 	 guest.o debug.o reset.o sys_regs.o stacktrace.o \
-	 vgic-sys-reg-v3.o fpsimd.o pkvm.o \
+	 vgic-sys-reg-v3.o fpsimd.o pkvm.o rmi.o \
 	 arch_timer.o trng.o vmid.o emulate-nested.o nested.o at.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 157bf4b81202..c8599fed5902 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -42,6 +42,7 @@
 #include <asm/kvm_nested.h>
 #include <asm/kvm_pkvm.h>
 #include <asm/kvm_ptrauth.h>
+#include <asm/kvm_rmi.h>
 #include <asm/sections.h>
 #include <asm/stacktrace/nvhe.h>
 
@@ -112,6 +113,8 @@ long kvm_get_cap_for_kvm_ioctl(unsigned int ioctl, long *ext)
 	return -EINVAL;
 }
 
+DEFINE_STATIC_KEY_FALSE(kvm_rmi_is_available);
+
 DECLARE_KVM_HYP_PER_CPU(unsigned long, kvm_hyp_vector);
 
 DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_base);
@@ -1732,10 +1735,19 @@ static unsigned long system_supported_vcpu_features(struct kvm_vcpu *vcpu)
 {
 	unsigned long features = KVM_VCPU_VALID_FEATURES;
 
-	if (vcpu->kvm->arch.vm_flavor == VM_PROTECTED_PKVM)
+	switch (vcpu->kvm->arch.vm_flavor) {
+	case VM_PROTECTED_PKVM:
 		kvm_pkvm_vcpu_allowed_features(vcpu->kvm, &features);
+		break;
+	case VM_REALM:
+		features = KVM_REALM_VCPU_VALID_FEATURES;
+		break;
+	default:
+		break;
+	}
 
-	if (!cpus_have_final_cap(ARM64_HAS_32BIT_EL1))
+	if (vcpu_is_protected(vcpu) ||
+	    !cpus_have_final_cap(ARM64_HAS_32BIT_EL1))
 		clear_bit(KVM_ARM_VCPU_EL1_32BIT, &features);
 
 	if (!kvm_supports_guest_pmuv3()) {
@@ -1751,7 +1763,8 @@ static unsigned long system_supported_vcpu_features(struct kvm_vcpu *vcpu)
 		clear_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, &features);
 	}
 
-	if (!cpus_have_final_cap(ARM64_HAS_NESTED_VIRT))
+	if (vcpu_is_protected(vcpu) ||
+	    !cpus_have_final_cap(ARM64_HAS_NESTED_VIRT))
 		clear_bit(KVM_ARM_VCPU_HAS_EL2, &features);
 
 	return features;
@@ -1775,7 +1788,7 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu,
 	if (features & ~system_supported_vcpu_features(vcpu))
 		return -EINVAL;
 
-	/* EL2 implements PSCI 1.1; the host must not dispatch as 0.1. */
+	/* Both pKVM and RMM implements PSCI 1.1; the host must not dispatch as 0.1. */
 	if (vcpu_is_protected(vcpu) &&
 	    !test_bit(KVM_ARM_VCPU_PSCI_0_2, &features))
 		return -EINVAL;
@@ -2312,6 +2325,7 @@ static const struct kvm_vcpu_ops *arm64_vcpu_ops[] = {
 	KVM_VCPU_OPS(VM_VHE, vhe_vcpu_ops),
 	KVM_VCPU_OPS(VM_PKVM, pkvm_vcpu_ops),
 	KVM_VCPU_OPS(VM_PROTECTED_PKVM, pkvm_vcpu_ops),
+	[VM_REALM] = NULL,
 };
 
 static void kvm_init_vcpu_ops(struct kvm_vcpu *vcpu)
@@ -3269,6 +3283,8 @@ static __init int kvm_arm_init(void)
 
 	in_hyp_mode = is_kernel_in_hyp_mode();
 
+	kvm_init_rmi();
+
 	if (cpus_have_final_cap(ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) ||
 	    cpus_have_final_cap(ARM64_WORKAROUND_1508412))
 		kvm_info("Guests without required CPU erratum workarounds can deadlock system!\n" \
diff --git a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
index 0261b2bc681f..ae6ced6d7b9e 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
@@ -84,7 +84,7 @@ pkvm_hyp_vcpu_to_hyp_vm(struct pkvm_hyp_vcpu *hyp_vcpu)
 
 static inline bool pkvm_hyp_vcpu_is_protected(struct pkvm_hyp_vcpu *hyp_vcpu)
 {
-	return vcpu_is_protected(&hyp_vcpu->vcpu);
+	return vcpu_is_protected_pkvm(&hyp_vcpu->vcpu);
 }
 
 static inline bool pkvm_hyp_vm_is_protected(struct pkvm_hyp_vm *hyp_vm)
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 89fb9dd351ad..6c03f48e6be3 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -238,7 +238,7 @@ static const exit_handler_fn pvm_exit_handlers[] = {
 
 static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)
 {
-	if (unlikely(vcpu_is_protected(vcpu)))
+	if (unlikely(vcpu_is_protected_pkvm(vcpu)))
 		return pvm_exit_handlers;
 
 	return hyp_exit_handlers;
@@ -259,7 +259,7 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
 	 * it.  The check below is based on the one in
 	 * kvm_arch_vcpu_ioctl_run().
 	 */
-	if (unlikely(vcpu_is_protected(vcpu) && vcpu_mode_is_32bit(vcpu))) {
+	if (unlikely(vcpu_is_protected_pkvm(vcpu) && vcpu_mode_is_32bit(vcpu))) {
 		/*
 		 * As we have caught the guest red-handed, decide that it isn't
 		 * fit for purpose anymore by making the vcpu invalid. The VMM
diff --git a/arch/arm64/kvm/hyp/nvhe/timer-sr.c b/arch/arm64/kvm/hyp/nvhe/timer-sr.c
index 993065716913..17bd1ff621ab 100644
--- a/arch/arm64/kvm/hyp/nvhe/timer-sr.c
+++ b/arch/arm64/kvm/hyp/nvhe/timer-sr.c
@@ -48,7 +48,7 @@ void __timer_enable_traps(struct kvm_vcpu *vcpu)
 	 * or running a protected VM (we don't offset anything in this case).
 	 */
 	clr = CNTHCTL_EL1PCEN;
-	if (vcpu_is_protected(vcpu) ||
+	if (vcpu_is_protected_pkvm(vcpu) ||
 	    !timer_get_offset(vcpu_ptimer(vcpu)))
 		set |= CNTHCTL_EL1PCTEN;
 	else
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 1db10d8d6db7..34ddcb2f094a 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2814,17 +2814,17 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
 	hva_t hva, reg_end;
 	int ret = 0;
 
-	if (kvm_vm_is_protected(kvm)) {
+	if (kvm_vm_is_protected(kvm) && new &&
+	    (new->flags & (KVM_MEM_LOG_DIRTY_PAGES | KVM_MEM_READONLY)))
+		return -EPERM;
+
+	if (kvm->arch.vm_flavor == VM_PROTECTED_PKVM) {
 		/* Cannot modify memslots once a pVM has run. */
 		if (pkvm_hyp_vm_is_created(kvm) &&
 		    (change == KVM_MR_DELETE || change == KVM_MR_MOVE)) {
 			return -EPERM;
 		}
 
-		if (new &&
-		    new->flags & (KVM_MEM_LOG_DIRTY_PAGES | KVM_MEM_READONLY)) {
-			return -EPERM;
-		}
 	}
 
 	if (change != KVM_MR_CREATE && change != KVM_MR_MOVE &&
@@ -3158,13 +3158,14 @@ static const struct kvm_vm_s2_ops kvm_default_vm_s2_ops = {
 };
 
 #define KVM_VM_S2_OPS(flavor, ops)		\
-		[(flavor)] = &(ops)
+		[flavor] = &(ops)
 
 static const struct kvm_vm_s2_ops *arm64_vm_s2_ops[] = {
 	KVM_VM_S2_OPS(VM_VHE, kvm_default_vm_s2_ops),
 	KVM_VM_S2_OPS(VM_NVHE, kvm_default_vm_s2_ops),
 	KVM_VM_S2_OPS(VM_PKVM, pkvm_vm_s2_ops),
 	KVM_VM_S2_OPS(VM_PROTECTED_PKVM, protected_pkvm_vm_s2_ops),
+	[VM_REALM] = NULL,
 };
 
 static int kvm_vm_init_vm_s2_ops(struct kvm *kvm)
diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
new file mode 100644
index 000000000000..5ecc8b349869
--- /dev/null
+++ b/arch/arm64/kvm/rmi.c
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2023-2026 ARM Ltd.
+ */
+
+#include <linux/kvm_host.h>
+
+#include <asm/virt.h>
+
+void kvm_init_rmi(void)
+{
+	if (kvm_get_mode() != KVM_MODE_RMM)
+		return;
+
+	/* TODO: Check if the RMI is available */
+
+	/* Future patch will enable static branch kvm_rmi_is_available */
+}
-- 
2.43.0

[PATCH 03/11] KVM: arm64: CCA: Check for RMI support at KVM init

[Steven Price]

Check if the RMI support is sufficient for using in KVM. Specifically, we
will only support creating realm VMs when KVM is set to 'rmm' mode.

Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/kvm/Kconfig | 1 +
 arch/arm64/kvm/rmi.c   | 4 +++-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index cecea0820e52..d067063a3686 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -34,6 +34,7 @@ menuconfig KVM
 	select GUEST_PERF_EVENTS if PERF_EVENTS
 	select KVM_GUEST_MEMFD
 	select KVM_GENERIC_PRE_FAULT_MEMORY
+	select ARM_RMM_RMI
 	help
 	  Support hosting virtualized guest machines.
 
diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
index 5ecc8b349869..9d118768550d 100644
--- a/arch/arm64/kvm/rmi.c
+++ b/arch/arm64/kvm/rmi.c
@@ -3,6 +3,7 @@
  * Copyright (C) 2023-2026 ARM Ltd.
  */
 
+#include <linux/arm-rmi-cmds.h>
 #include <linux/kvm_host.h>
 
 #include <asm/virt.h>
@@ -12,7 +13,8 @@ void kvm_init_rmi(void)
 	if (kvm_get_mode() != KVM_MODE_RMM)
 		return;
 
-	/* TODO: Check if the RMI is available */
+	if (!is_rmi_available())
+		return;
 
 	/* Future patch will enable static branch kvm_rmi_is_available */
 }
-- 
2.43.0

[PATCH 04/11] KVM: arm64: CCA: Check for LPA2 support and SHA-256

[Steven Price]

If KVM has enabled LPA2 support then check that the RMM also supports
it. If there is a mismatch then disable support for realm guests as the
VMM may attempt to create a guest which is incompatible with the RMM.

Also check for SHA-256 support in the RMM as to begin with we only
support SHA-256 based measurements. This can be relaxed in the future.

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/kvm/rmi.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
index 9d118768550d..5b07beff8bc9 100644
--- a/arch/arm64/kvm/rmi.c
+++ b/arch/arm64/kvm/rmi.c
@@ -6,8 +6,30 @@
 #include <linux/arm-rmi-cmds.h>
 #include <linux/kvm_host.h>
 
+#include <asm/kvm_pgtable.h>
 #include <asm/virt.h>
 
+static bool rmi_has_feature(int reg, unsigned long feature)
+{
+	return !!u64_get_bits(rmi_feat_reg(reg), feature);
+}
+
+static int rmm_check_features(void)
+{
+	if (kvm_lpa2_is_enabled() &&
+	    !rmi_has_feature(0, RMI_FEATURE_REGISTER_0_LPA2)) {
+		kvm_err("RMM doesn't support LPA2\n");
+		return -ENXIO;
+	}
+
+	if (!rmi_has_feature(1, RMI_FEATURE_REGISTER_1_HASH_SHA_256)) {
+		kvm_err("RMM doesn't support SHA-256 measurements\n");
+		return -ENXIO;
+	}
+
+	return 0;
+}
+
 void kvm_init_rmi(void)
 {
 	if (kvm_get_mode() != KVM_MODE_RMM)
@@ -16,5 +38,8 @@ void kvm_init_rmi(void)
 	if (!is_rmi_available())
 		return;
 
+	if (rmm_check_features())
+		return;
+
 	/* Future patch will enable static branch kvm_rmi_is_available */
 }
-- 
2.43.0

[PATCH 05/11] KVM: arm64: Handle attributes in stage2_unmap_range

[Steven Price]

From: Suzuki K Poulose <suzuki.poulose@arm.com>

With the introduction of private memory support, we need to
handle invalidation requests with filters for a given type.
Add support for passing the attributes down.

Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/include/asm/kvm_host.h |  4 +++-
 arch/arm64/kvm/mmu.c              | 31 +++++++++++++++++++++++--------
 2 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 7d0222047ca1..41e15230263d 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -241,6 +241,7 @@ struct kvm_vcpu_ops {
 };
 
 struct kvm_gfn_range;
+enum kvm_gfn_range_filter;
 struct kvm_s2_fault_desc;
 
 struct kvm_vm_s2_ops {
@@ -251,7 +252,8 @@ struct kvm_vm_s2_ops {
 					  u64 nr_pages);
 	void (*vm_stage2_unmap_range)(struct kvm_s2_mmu *mmu,
 				      phys_addr_t start, u64 size,
-				      bool may_block);
+				      bool may_block,
+				      enum kvm_gfn_range_filter filter);
 	int (*vm_mem_abort)(const struct kvm_s2_fault_desc *s2fd);
 };
 
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 34ddcb2f094a..1301c563d705 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -324,7 +324,8 @@ static int kvm_pgtable_stage2_unmap_tracked(struct kvm_pgtable *pgt,
 
 static void kvm_vm_stage2_unmap_range(struct kvm_s2_mmu *mmu,
 				      phys_addr_t start,
-				      u64 size, bool may_block)
+				      u64 size, bool may_block,
+				      enum kvm_gfn_range_filter filter)
 {
 	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
 	int (*fn)(struct kvm_pgtable *, u64, u64);
@@ -339,14 +340,16 @@ static void kvm_vm_stage2_unmap_range(struct kvm_s2_mmu *mmu,
 
 static void pkvm_stage2_unmap_range(struct kvm_s2_mmu *mmu,
 				    phys_addr_t start,
-				    u64 size, bool may_block)
+				    u64 size, bool may_block,
+				    enum kvm_gfn_range_filter filter)
 {
 	WARN_ON(stage2_apply_range(mmu, start, start + size,
 				   pkvm_pgtable_stage2_unmap, may_block));
 }
 
 static void no_stage2_unmap_range(struct kvm_s2_mmu *mmu,
-				  phys_addr_t start, u64 size, bool may_block)
+				  phys_addr_t start, u64 size, bool may_block,
+				  enum kvm_gfn_range_filter filter)
 {
 }
 
@@ -388,25 +391,36 @@ static int kvm_pgtable_stage2_unmap_tracked(struct kvm_pgtable *pgt, u64 addr, u
 }
 
 /**
- * kvm_stage2_unmap_range -- Clear stage2 page table entries to unmap a range
+ * kvm_stage2_unmap_range_filter -- Clear stage2 page table entries to unmap a
+ * range
  * @mmu:   The KVM stage-2 MMU pointer
  * @start: The intermediate physical base address of the range to unmap
  * @size:  The size of the area to unmap
  * @may_block: Whether or not we are permitted to block
+ * @filter: Filter for the ipa range (private and/or shared)
  *
  * Clear a range of stage-2 mappings, lowering the various ref-counts.  Must
  * be called while holding mmu_lock otherwise another faulting VCPU may
  * come in and mess with things behind our backs.
  */
-void kvm_stage2_unmap_range(struct kvm_s2_mmu *mmu, phys_addr_t start,
-			    u64 size, bool may_block)
+static void kvm_stage2_unmap_range_filter(struct kvm_s2_mmu *mmu, phys_addr_t start,
+					  u64 size, bool may_block,
+					  enum kvm_gfn_range_filter filter)
 {
 	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
 
 	lockdep_assert_held_write(&kvm->mmu_lock);
 	WARN_ON(size & ~PAGE_MASK);
 
-	kvm->arch.vm_s2_ops->vm_stage2_unmap_range(mmu, start, size, may_block);
+	kvm->arch.vm_s2_ops->vm_stage2_unmap_range(mmu, start, size,
+						   may_block, filter);
+}
+
+void kvm_stage2_unmap_range(struct kvm_s2_mmu *mmu, phys_addr_t start,
+			    u64 size, bool may_block)
+{
+	kvm_stage2_unmap_range_filter(mmu, start, size, may_block,
+				      KVM_FILTER_PRIVATE | KVM_FILTER_SHARED);
 }
 
 void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end)
@@ -2597,7 +2611,8 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	if (!kvm->arch.mmu.pgt)
 		return false;
 
-	kvm_stage2_unmap_range(&kvm->arch.mmu, gpa, size, may_block);
+	kvm_stage2_unmap_range_filter(&kvm->arch.mmu, gpa, size, may_block,
+				      range->attr_filter);
 	kvm_nested_unmap_cipa_range(kvm, gpa, size, may_block);
 
 	return false;
-- 
2.43.0

[PATCH 06/11] KVM: arm64: CCA: Add basic infrastructure for creating a realm

[Steven Price]

Introduce the skeleton functions for creating and destroying a realm.
The IPA size requested is checked against what the RMM supports.

The actual work of constructing the realm will be added in future
patches.

Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/include/asm/kvm_rmi.h | 20 ++++++++++++
 arch/arm64/kvm/arm.c             |  9 +++++
 arch/arm64/kvm/mmu.c             | 18 ++++++++--
 arch/arm64/kvm/rmi.c             | 56 ++++++++++++++++++++++++++++++++
 4 files changed, 101 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_rmi.h b/arch/arm64/include/asm/kvm_rmi.h
index 3194daaacf2b..f1a1ed6b91f5 100644
--- a/arch/arm64/include/asm/kvm_rmi.h
+++ b/arch/arm64/include/asm/kvm_rmi.h
@@ -6,6 +6,9 @@
 #ifndef __ASM_KVM_RMI_H
 #define __ASM_KVM_RMI_H
 
+#include <linux/arm-smccc-rmi.h>
+#include <linux/kvm.h>
+
 /**
  * enum realm_state - State of a Realm
  *
@@ -50,12 +53,29 @@ enum realm_state {
 /**
  * struct realm - Additional per VM data for a Realm
  *
+ * @rd: Kernel mapping of the RMM-managed Realm Descriptor (RD) granule
+ * @sro: Preallocated SRO state context for Realm MMU operations
  * @state: The lifetime state machine for the realm
+ * @ia_bits: Number of valid Input Address bits in the IPA
  */
 struct realm {
+	void *rd;
+	/*
+	 * Reused by RTT map/unmap SRO commands. Those commands are only
+	 * issued from Realm stage-2 map/unmap paths while kvm->mmu_lock is
+	 * held for write, including Realm fault handling where
+	 * kvm_fault_lock() takes the write side, so concurrent use is
+	 * serialized.
+	 */
+	struct rmi_sro_state *sro;
 	enum realm_state state;
+	unsigned int ia_bits;
 };
 
 void kvm_init_rmi(void);
+u32 kvm_rmm_ipa_limit(void);
+
+int kvm_init_realm(struct kvm *kvm);
+void kvm_destroy_realm(struct kvm *kvm);
 
 #endif /* __ASM_KVM_RMI_H */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index c8599fed5902..9a7bbfaf9675 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -303,6 +303,13 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
 	bitmap_zero(kvm->arch.vcpu_features, KVM_VCPU_MAX_FEATURES);
 
+	/* Initialise the realm bits after the generic bits are enabled */
+	if (kvm_vm_is_realm(kvm)) {
+		ret = kvm_init_realm(kvm);
+		if (ret)
+			goto err_uninit_mmu;
+	}
+
 	return 0;
 
 err_uninit_mmu:
@@ -367,6 +374,8 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 
 	kvm_destroy_nested(kvm);
 	kvm_arm_teardown_hypercalls(kvm);
+	if (kvm_vm_is_realm(kvm))
+		kvm_destroy_realm(kvm);
 }
 
 static bool kvm_has_full_ptr_auth(void)
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 1301c563d705..b049ea67d9af 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -981,10 +981,14 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = {
 
 static int kvm_init_ipa_range(struct kvm_s2_mmu *mmu, unsigned long type)
 {
+	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
 	u32 kvm_ipa_limit = get_kvm_ipa_limit();
 	u64 mmfr0, mmfr1;
 	u32 phys_shift;
 
+	if (kvm_vm_is_realm(kvm))
+		kvm_ipa_limit = kvm_rmm_ipa_limit();
+
 	phys_shift = KVM_VM_TYPE_ARM_IPA_SIZE(type);
 	if (is_protected_kvm_enabled()) {
 		phys_shift = kvm_ipa_limit;
@@ -1034,9 +1038,18 @@ static void stage2_destroy_range(struct kvm_pgtable *pgt, phys_addr_t addr,
 
 static void kvm_stage2_destroy(struct kvm_pgtable *pgt)
 {
+	struct kvm *kvm = kvm_s2_mmu_to_kvm(pgt->mmu);
 	unsigned int ia_bits = VTCR_EL2_IPA(pgt->mmu->vtcr);
 
-	stage2_destroy_range(pgt, 0, BIT(ia_bits));
+	/*
+	 * Realm RTTs are inaccessible to the host and may contain stale data
+	 * after the RMM has released them. The non-root RTTs are explicitly
+	 * destroyed through RMI before the RD is destroyed; only the root PGD
+	 * pages remain to be freed here.
+	 */
+	if (!kvm_vm_is_realm(kvm))
+		stage2_destroy_range(pgt, 0, BIT(ia_bits));
+
 	KVM_PGT_FN(kvm_pgtable_stage2_destroy_pgd)(pgt);
 }
 
@@ -1084,6 +1097,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 		return -EINVAL;
 	}
 
+	mmu->arch = &kvm->arch;
+
 	err = kvm_init_ipa_range(mmu, type);
 	if (err)
 		return err;
@@ -1092,7 +1107,6 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 	if (!pgt)
 		return -ENOMEM;
 
-	mmu->arch = &kvm->arch;
 	err = KVM_PGT_FN(kvm_pgtable_stage2_init)(pgt, mmu, &kvm_s2_mm_ops);
 	if (err)
 		goto out_free_pgtable;
diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
index 5b07beff8bc9..45ecbaa467d1 100644
--- a/arch/arm64/kvm/rmi.c
+++ b/arch/arm64/kvm/rmi.c
@@ -6,6 +6,8 @@
 #include <linux/arm-rmi-cmds.h>
 #include <linux/kvm_host.h>
 
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_mmu.h>
 #include <asm/kvm_pgtable.h>
 #include <asm/virt.h>
 
@@ -14,6 +16,60 @@ static bool rmi_has_feature(int reg, unsigned long feature)
 	return !!u64_get_bits(rmi_feat_reg(reg), feature);
 }
 
+u32 kvm_rmm_ipa_limit(void)
+{
+	return u64_get_bits(rmi_feat_reg(0), RMI_FEATURE_REGISTER_0_S2SZ);
+}
+
+void kvm_destroy_realm(struct kvm *kvm)
+{
+	struct realm *realm = &kvm->arch.realm;
+	size_t pgd_size = kvm_pgtable_stage2_pgd_size(kvm->arch.mmu.vtcr);
+
+	if (!kvm_realm_is_created(kvm)) {
+		kfree(realm->sro);
+		realm->sro = NULL;
+		return;
+	}
+
+	kvm_set_realm_state(kvm, REALM_STATE_DYING);
+
+	if (realm->rd) {
+		phys_addr_t rd_phys = virt_to_phys(realm->rd);
+
+		if (WARN_ON(rmi_realm_terminate(rd_phys, realm->sro)))
+			return;
+
+		if (WARN_ON(rmi_realm_destroy(rd_phys, realm->sro)))
+			return;
+		rmi_free_delegated_page(rd_phys);
+		realm->rd = NULL;
+	}
+
+	if (WARN_ON(rmi_undelegate_range(kvm->arch.mmu.pgd_phys,
+					 pgd_size)))
+		return;
+
+	kvm_set_realm_state(kvm, REALM_STATE_DEAD);
+
+	/* Now that the realm is destroyed, free the entry-level RTTs. */
+	kvm_free_stage2_pgd(&kvm->arch.mmu);
+
+	kfree(realm->sro);
+	realm->sro = NULL;
+}
+
+int kvm_init_realm(struct kvm *kvm)
+{
+	struct realm *realm = &kvm->arch.realm;
+
+	realm->sro = kmalloc_obj(*realm->sro);
+	if (!realm->sro)
+		return -ENOMEM;
+
+	return 0;
+}
+
 static int rmm_check_features(void)
 {
 	if (kvm_lpa2_is_enabled() &&
-- 
2.43.0

[PATCH 07/11] KVM: arm64: CCA: Tear down RTTs

[Steven Price]

The RMM owns the stage 2 page tables for a realm, and KVM must request
that the RMM creates/destroys entries as necessary. The physical pages
to store the page tables are delegated to the realm as required, and can
be undelegated when no longer used.

Creating new RTTs is the easy part, tearing down is a little more
tricky. The result of realm_rtt_destroy() can be used to effectively
walk the tree and destroy the entries (undelegating pages that were
given to the realm).

Unmap the Realm IPA space once during stage-2 teardown. Defer destruction
of non-root RTTs until final Realm destruction, after REALM_TERMINATE and
before destroying the RD; root RTTs remain until the RD has gone away.

Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/include/asm/kvm_rmi.h |   9 ++
 arch/arm64/kvm/mmu.c             |  17 ++-
 arch/arm64/kvm/rmi.c             | 189 +++++++++++++++++++++++++++++++
 3 files changed, 214 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kvm_rmi.h b/arch/arm64/include/asm/kvm_rmi.h
index f1a1ed6b91f5..58d825e40bdf 100644
--- a/arch/arm64/include/asm/kvm_rmi.h
+++ b/arch/arm64/include/asm/kvm_rmi.h
@@ -57,6 +57,7 @@ enum realm_state {
  * @sro: Preallocated SRO state context for Realm MMU operations
  * @state: The lifetime state machine for the realm
  * @ia_bits: Number of valid Input Address bits in the IPA
+ * @stage2_unmapped: The Realm stage-2 mappings have been removed
  */
 struct realm {
 	void *rd;
@@ -70,6 +71,7 @@ struct realm {
 	struct rmi_sro_state *sro;
 	enum realm_state state;
 	unsigned int ia_bits;
+	bool stage2_unmapped;
 };
 
 void kvm_init_rmi(void);
@@ -77,5 +79,12 @@ u32 kvm_rmm_ipa_limit(void);
 
 int kvm_init_realm(struct kvm *kvm);
 void kvm_destroy_realm(struct kvm *kvm);
+int kvm_realm_teardown_stage2(struct kvm *kvm);
+
+static inline bool kvm_realm_is_private_address(struct realm *realm,
+						unsigned long addr)
+{
+	return !(addr & BIT(realm->ia_bits - 1));
+}
 
 #endif /* __ASM_KVM_RMI_H */
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index b049ea67d9af..51fda7b2ff7f 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1145,9 +1145,24 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 	return err;
 }
 
+static void kvm_realm_uninit_stage2(struct kvm_s2_mmu *mmu)
+{
+	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
+
+	mutex_lock(&kvm->arch.config_lock);
+	if (kvm_realm_state(kvm) == REALM_STATE_ACTIVE) {
+		kvm_set_realm_state(kvm, REALM_STATE_DYING);
+		WARN_ON(kvm_realm_teardown_stage2(kvm));
+	}
+	mutex_unlock(&kvm->arch.config_lock);
+}
+
 void kvm_uninit_stage2_mmu(struct kvm *kvm)
 {
-	kvm_free_stage2_pgd(&kvm->arch.mmu);
+	if (kvm_vm_is_realm(kvm))
+		kvm_realm_uninit_stage2(&kvm->arch.mmu);
+	else
+		kvm_free_stage2_pgd(&kvm->arch.mmu);
 	kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
 }
 
diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
index 45ecbaa467d1..f190143b33ba 100644
--- a/arch/arm64/kvm/rmi.c
+++ b/arch/arm64/kvm/rmi.c
@@ -11,6 +11,14 @@
 #include <asm/kvm_pgtable.h>
 #include <asm/virt.h>
 
+static inline unsigned long rmi_rtt_level_mapsize(int level)
+{
+	if (WARN_ON(level > KVM_PGTABLE_LAST_LEVEL))
+		return PAGE_SIZE;
+
+	return (1UL << ARM64_HW_PGTABLE_LEVEL_SHIFT(level));
+}
+
 static bool rmi_has_feature(int reg, unsigned long feature)
 {
 	return !!u64_get_bits(rmi_feat_reg(reg), feature);
@@ -21,11 +29,180 @@ u32 kvm_rmm_ipa_limit(void)
 	return u64_get_bits(rmi_feat_reg(0), RMI_FEATURE_REGISTER_0_S2SZ);
 }
 
+static int get_start_level(struct realm *realm)
+{
+	return 4 - stage2_pgtable_levels(realm->ia_bits);
+}
+
+static void free_rtt(phys_addr_t phys)
+{
+	if (rmi_free_delegated_page(phys))
+		return;
+
+	kvm_account_pgtable_pages(phys_to_virt(phys), -1);
+}
+
+/*
+ * realm_rtt_destroy - Destroy an RTT at @level for @addr.
+ *
+ * Returns - Result of the RMI_RTT_DESTROY call, and:
+ * @rtt_granule:	RTT granule, if the RTT was destroyed.
+ * @next_addr:		IPA corresponding to the next possible valid entry we
+ *			can target
+ */
+static long realm_rtt_destroy(struct realm *realm, unsigned long addr,
+			      int level, phys_addr_t *rtt_granule,
+			      unsigned long *next_addr)
+{
+	unsigned long out_rtt;
+	long ret;
+
+	ret = rmi_rtt_destroy(virt_to_phys(realm->rd), addr, level,
+			      &out_rtt, next_addr);
+
+	*rtt_granule = out_rtt;
+
+	return ret;
+}
+
+static int realm_tear_down_rtt_level(struct realm *realm, int level,
+				     unsigned long start, unsigned long end)
+{
+	ssize_t map_size;
+	unsigned long addr, next_addr;
+
+	if (WARN_ON(level > KVM_PGTABLE_LAST_LEVEL))
+		return -EINVAL;
+
+	map_size = rmi_rtt_level_mapsize(level - 1);
+
+	for (addr = start; addr < end; addr = next_addr) {
+		phys_addr_t rtt_granule;
+		long ret;
+		unsigned long align_addr = ALIGN(addr, map_size);
+
+		next_addr = ALIGN(addr + 1, map_size);
+
+		if (next_addr > end || align_addr != addr) {
+			/*
+			 * The target range is smaller than what this level
+			 * covers, recurse deeper.
+			 */
+			ret = realm_tear_down_rtt_level(realm,
+							level + 1,
+							addr,
+							min(next_addr, end));
+			if (ret)
+				return ret;
+			continue;
+		}
+
+		ret = realm_rtt_destroy(realm, addr, level,
+					&rtt_granule, &next_addr);
+		if (ret < 0)
+			return ret;
+
+		switch (RMI_RESULT_STATUS(ret)) {
+		case RMI_SUCCESS:
+			free_rtt(rtt_granule);
+			break;
+		case RMI_ERROR_RTT:
+			if (next_addr > addr) {
+				/* Missing RTT, skip */
+				break;
+			}
+			/*
+			 * We tear down the RTT range for the full IPA
+			 * space, after everything is unmapped. Also we
+			 * descend down only if we cannot tear down a
+			 * top level RTT. Thus RMM must be able to walk
+			 * to the requested level. e.g., a block mapping
+			 * exists at L1 or L2.
+			 */
+			if (WARN_ON(RMI_RESULT_DATA_LEVEL(ret) != level))
+				return -EBUSY;
+			if (WARN_ON(level == KVM_PGTABLE_LAST_LEVEL))
+				return -EBUSY;
+
+			/*
+			 * The table has active entries in it, recurse deeper
+			 * and tear down the RTTs.
+			 */
+			next_addr = ALIGN(addr + 1, map_size);
+			ret = realm_tear_down_rtt_level(realm,
+							level + 1,
+							addr,
+							next_addr);
+			if (ret)
+				return ret;
+			/*
+			 * Now that the child RTTs are destroyed,
+			 * retry at this level.
+			 */
+			next_addr = addr;
+			break;
+		default:
+			WARN_ON(1);
+			return -ENXIO;
+		}
+	}
+
+	return 0;
+}
+
+static int realm_tear_down_rtt_range(struct realm *realm,
+				     unsigned long start, unsigned long end)
+{
+	/*
+	 * Root level RTTs can only be destroyed after the RD is destroyed. So
+	 * tear down everything below the root level
+	 */
+	return realm_tear_down_rtt_level(realm, get_start_level(realm) + 1,
+					 start, end);
+}
+
+static int realm_destroy_rtts(struct kvm *kvm)
+{
+	struct realm *realm = &kvm->arch.realm;
+	unsigned int ia_bits = realm->ia_bits;
+
+	lockdep_assert_held(&kvm->arch.config_lock);
+
+	return realm_tear_down_rtt_range(realm, 0, (1UL << ia_bits));
+}
+
+static void realm_unmap_stage2(struct kvm *kvm)
+{
+	struct realm *realm = &kvm->arch.realm;
+
+	lockdep_assert_held(&kvm->arch.config_lock);
+
+	if (realm->stage2_unmapped)
+		return;
+
+	write_lock(&kvm->mmu_lock);
+	kvm_stage2_unmap_range(&kvm->arch.mmu, 0,
+			       BIT(realm->ia_bits - 1), true);
+	write_unlock(&kvm->mmu_lock);
+
+	realm->stage2_unmapped = true;
+}
+
+int kvm_realm_teardown_stage2(struct kvm *kvm)
+{
+	lockdep_assert_held(&kvm->arch.config_lock);
+
+	realm_unmap_stage2(kvm);
+	return 0;
+}
+
 void kvm_destroy_realm(struct kvm *kvm)
 {
 	struct realm *realm = &kvm->arch.realm;
 	size_t pgd_size = kvm_pgtable_stage2_pgd_size(kvm->arch.mmu.vtcr);
 
+	guard(mutex)(&kvm->arch.config_lock);
+
 	if (!kvm_realm_is_created(kvm)) {
 		kfree(realm->sro);
 		realm->sro = NULL;
@@ -34,12 +211,24 @@ void kvm_destroy_realm(struct kvm *kvm)
 
 	kvm_set_realm_state(kvm, REALM_STATE_DYING);
 
+	/*
+	 * REALM_DESTROY requires the realm to be non-live: all RECs must have
+	 * been destroyed and the root RTTs must be empty. Unmap the IPA space
+	 * and destroy any non-root RTTs before tearing down the RD. The root
+	 * RTT pages are still owned by the RMM at this point, so keep the KVM
+	 * pgtable alive until after REALM_DESTROY and undelegation.
+	 */
+	realm_unmap_stage2(kvm);
+
 	if (realm->rd) {
 		phys_addr_t rd_phys = virt_to_phys(realm->rd);
 
 		if (WARN_ON(rmi_realm_terminate(rd_phys, realm->sro)))
 			return;
 
+		if (WARN_ON(realm_destroy_rtts(kvm)))
+			return;
+
 		if (WARN_ON(rmi_realm_destroy(rd_phys, realm->sro)))
 			return;
 		rmi_free_delegated_page(rd_phys);
-- 
2.43.0

[PATCH 08/11] KVM: arm64: CCA: Add Realm RTT mapping and invalidation helpers

[Steven Price]

Add the RMI RTT creation and unmap helpers needed by CCA Realm stage-2
memory operations. REC exit handling and guest requested RIPAS changes
follow in the vCPU series.

Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/include/asm/kvm_rmi.h |  14 ++
 arch/arm64/kvm/rmi.c             | 346 +++++++++++++++++++++++++++++++
 2 files changed, 360 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_rmi.h b/arch/arm64/include/asm/kvm_rmi.h
index 58d825e40bdf..72a9449f00d7 100644
--- a/arch/arm64/include/asm/kvm_rmi.h
+++ b/arch/arm64/include/asm/kvm_rmi.h
@@ -81,6 +81,20 @@ int kvm_init_realm(struct kvm *kvm);
 void kvm_destroy_realm(struct kvm *kvm);
 int kvm_realm_teardown_stage2(struct kvm *kvm);
 
+enum kvm_gfn_range_filter;
+
+void kvm_realm_unmap_range_filter(struct kvm *kvm,
+				  unsigned long ipa,
+				  unsigned long size,
+				  bool may_block,
+				  enum kvm_gfn_range_filter attr_filter);
+
+void kvm_realm_unmap_range(struct kvm *kvm,
+			   unsigned long ipa,
+			   unsigned long size,
+			   bool unmap_private,
+			   bool may_block);
+
 static inline bool kvm_realm_is_private_address(struct realm *realm,
 						unsigned long addr)
 {
diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
index f190143b33ba..1066f767f85d 100644
--- a/arch/arm64/kvm/rmi.c
+++ b/arch/arm64/kvm/rmi.c
@@ -34,6 +34,83 @@ static int get_start_level(struct realm *realm)
 	return 4 - stage2_pgtable_levels(realm->ia_bits);
 }
 
+static int find_map_level(struct realm *realm,
+			  unsigned long start,
+			  unsigned long end)
+{
+	int level = KVM_PGTABLE_LAST_LEVEL;
+
+	while (level > get_start_level(realm)) {
+		unsigned long map_size = rmi_rtt_level_mapsize(level - 1);
+
+		if (!IS_ALIGNED(start, map_size) ||
+		    (start + map_size) > end)
+			break;
+
+		level--;
+	}
+
+	return level;
+}
+
+static unsigned long rmi_range_entry_size(int rmi_size)
+{
+	if (WARN_ON(rmi_size > KVM_PGTABLE_LAST_LEVEL))
+		return 0;
+
+	return kvm_granule_size(KVM_PGTABLE_LAST_LEVEL - rmi_size);
+}
+
+static int undelegate_range_desc(unsigned long desc)
+{
+	unsigned long size = rmi_range_entry_size(RMI_ADDR_RANGE_BLOCK_SIZE(desc));
+	unsigned long count = RMI_ADDR_RANGE_COUNT(desc);
+	unsigned long addr = RMI_ADDR_RANGE_ADDR(desc);
+	unsigned long state = RMI_ADDR_RANGE_STATE(desc);
+
+	if (state == RMI_OP_MEM_UNDELEGATED)
+		return 0;
+
+	if (size * count == 0)
+		return 0;
+
+	return rmi_undelegate_range(addr, size * count);
+}
+
+static phys_addr_t alloc_delegated_granule(struct kvm_mmu_memory_cache *mc)
+{
+	phys_addr_t phys;
+	void *virt;
+
+	if (mc) {
+		virt = kvm_mmu_memory_cache_alloc(mc);
+	} else {
+		virt = (void *)__get_free_page(GFP_ATOMIC | __GFP_ZERO |
+					       __GFP_ACCOUNT);
+	}
+
+	if (!virt)
+		return PHYS_ADDR_MAX;
+
+	phys = virt_to_phys(virt);
+	if (rmi_delegate_page(phys)) {
+		free_page((unsigned long)virt);
+		return PHYS_ADDR_MAX;
+	}
+
+	return phys;
+}
+
+static phys_addr_t alloc_rtt(struct kvm_mmu_memory_cache *mc)
+{
+	phys_addr_t phys = alloc_delegated_granule(mc);
+
+	if (phys != PHYS_ADDR_MAX)
+		kvm_account_pgtable_pages(phys_to_virt(phys), 1);
+
+	return phys;
+}
+
 static void free_rtt(phys_addr_t phys)
 {
 	if (rmi_free_delegated_page(phys))
@@ -42,6 +119,32 @@ static void free_rtt(phys_addr_t phys)
 	kvm_account_pgtable_pages(phys_to_virt(phys), -1);
 }
 
+static long realm_rtt_create(struct realm *realm,
+			     unsigned long addr,
+			     int level,
+			     phys_addr_t phys)
+{
+	addr = ALIGN_DOWN(addr, rmi_rtt_level_mapsize(level - 1));
+	return rmi_rtt_create(virt_to_phys(realm->rd), phys, addr, level);
+}
+
+static long realm_rtt_fold(struct realm *realm,
+			   unsigned long addr,
+			   int level,
+			   phys_addr_t *rtt_granule)
+{
+	unsigned long out_rtt;
+	long ret;
+
+	addr = ALIGN_DOWN(addr, rmi_rtt_level_mapsize(level - 1));
+	ret = rmi_rtt_fold(virt_to_phys(realm->rd), addr, level, &out_rtt);
+
+	if (rtt_granule)
+		*rtt_granule = out_rtt;
+
+	return ret;
+}
+
 /*
  * realm_rtt_destroy - Destroy an RTT at @level for @addr.
  *
@@ -65,6 +168,43 @@ static long realm_rtt_destroy(struct realm *realm, unsigned long addr,
 	return ret;
 }
 
+static int realm_create_rtt_levels(struct realm *realm,
+				   unsigned long ipa,
+				   int level,
+				   int max_level,
+				   struct kvm_mmu_memory_cache *mc)
+{
+	while (level++ < max_level) {
+		phys_addr_t rtt = alloc_rtt(mc);
+		long ret;
+
+		if (rtt == PHYS_ADDR_MAX)
+			return -ENOMEM;
+
+		ret = realm_rtt_create(realm, ipa, level, rtt);
+		if (ret < 0) {
+			free_rtt(rtt);
+			return ret;
+		}
+
+		if (RMI_RESULT_STATUS(ret) == RMI_ERROR_RTT &&
+		    RMI_RESULT_DATA_LEVEL(ret) == level - 1) {
+			/* The RTT already exists, continue */
+			free_rtt(rtt);
+			continue;
+		}
+
+		if (ret) {
+			WARN(1, "Failed to create RTT at level %d: %ld\n",
+			     level, ret);
+			free_rtt(rtt);
+			return -ENXIO;
+		}
+	}
+
+	return 0;
+}
+
 static int realm_tear_down_rtt_level(struct realm *realm, int level,
 				     unsigned long start, unsigned long end)
 {
@@ -161,6 +301,64 @@ static int realm_tear_down_rtt_range(struct realm *realm,
 					 start, end);
 }
 
+/*
+ * Returns 0 on successful fold, a negative value on error, a positive value if
+ * we were not able to fold all tables at this level.
+ */
+static int realm_fold_rtt_level(struct realm *realm, int level,
+				unsigned long start, unsigned long end)
+{
+	int not_folded = 0;
+	ssize_t map_size;
+	unsigned long addr, next_addr;
+
+	if (WARN_ON(level > KVM_PGTABLE_LAST_LEVEL))
+		return -EINVAL;
+
+	map_size = rmi_rtt_level_mapsize(level - 1);
+
+	for (addr = start; addr < end; addr = next_addr) {
+		phys_addr_t rtt_granule;
+		long ret;
+		unsigned long align_addr = ALIGN(addr, map_size);
+
+		next_addr = ALIGN(addr + 1, map_size);
+
+		ret = realm_rtt_fold(realm, align_addr, level, &rtt_granule);
+		if (ret < 0)
+			return ret;
+
+		switch (RMI_RESULT_STATUS(ret)) {
+		case RMI_SUCCESS:
+			free_rtt(rtt_granule);
+			break;
+		case RMI_ERROR_RTT:
+			if (level == KVM_PGTABLE_LAST_LEVEL ||
+			    RMI_RESULT_DATA_LEVEL(ret) < level) {
+				not_folded++;
+				break;
+			}
+			/* Recurse a level deeper */
+			ret = realm_fold_rtt_level(realm,
+						   level + 1,
+						   addr,
+						   next_addr);
+			if (ret < 0) {
+				return ret;
+			} else if (ret == 0) {
+				/* Try again at this level */
+				next_addr = addr;
+			}
+			break;
+		default:
+			WARN_ON(1);
+			return -ENXIO;
+		}
+	}
+
+	return not_folded;
+}
+
 static int realm_destroy_rtts(struct kvm *kvm)
 {
 	struct realm *realm = &kvm->arch.realm;
@@ -196,6 +394,154 @@ int kvm_realm_teardown_stage2(struct kvm *kvm)
 	return 0;
 }
 
+static void realm_unmap_shared_range(struct kvm *kvm,
+				     unsigned long start,
+				     unsigned long end,
+				     bool may_block)
+{
+	struct realm *realm = &kvm->arch.realm;
+	unsigned long rd = virt_to_phys(realm->rd);
+	unsigned long next_addr, addr;
+	unsigned long shared_bit = BIT(realm->ia_bits - 1);
+
+	if (start >= end)
+		return;
+
+	/*
+	 * Callers may pass either an IPA range or its shared alias. Add the
+	 * shared bit only for the former. Use addition so an exclusive end at
+	 * shared_bit advances to the end of the IPA space.
+	 */
+	if (start < shared_bit) {
+		start += shared_bit;
+		end += shared_bit;
+	}
+
+	for (addr = start; addr < end; addr = next_addr) {
+		long ret;
+
+		ret = rmi_rtt_unprot_unmap(rd, addr, end, RMI_ADDR_TYPE_NONE,
+					   0, &next_addr, NULL, NULL);
+		if (WARN_ON(ret < 0))
+			return;
+
+		switch (RMI_RESULT_STATUS(ret)) {
+		case RMI_SUCCESS:
+			break;
+		case RMI_ERROR_RTT: {
+			int err_level = RMI_RESULT_DATA_LEVEL(ret);
+			int level = find_map_level(realm, addr, end);
+
+			if (err_level >= level) {
+				/* Nothing present, so skip */
+				next_addr = addr + rmi_rtt_level_mapsize(err_level);
+				break;
+			}
+
+			ret = realm_create_rtt_levels(realm, addr, err_level,
+						      level, NULL);
+			if (WARN_ON(ret))
+				return;
+			/* Retry with the RTT levels in place */
+			next_addr = addr;
+			break;
+		}
+		default:
+			WARN_ON(1);
+			return;
+		}
+
+		if (may_block)
+			cond_resched_rwlock_write(&kvm->mmu_lock);
+	}
+
+	realm_fold_rtt_level(realm, get_start_level(realm) + 1,
+			     start, end);
+}
+
+static void realm_unmap_private_range(struct kvm *kvm,
+				      unsigned long start,
+				      unsigned long end,
+				      bool may_block)
+{
+	struct realm *realm = &kvm->arch.realm;
+	unsigned long rd = virt_to_phys(realm->rd);
+	unsigned long next_addr, addr;
+	long ret;
+
+	for (addr = start; addr < end; addr = next_addr) {
+		unsigned long out_range;
+		unsigned long flags = RMI_ADDR_TYPE_SINGLE;
+		/* TODO: Optimise using RMI_ADDR_TYPE_LIST */
+
+retry:
+		ret = rmi_rtt_data_unmap(rd, addr, end, flags, 0,
+					 &next_addr, &out_range, NULL);
+		if (WARN_ON(ret < 0))
+			return;
+
+		if (RMI_RESULT_STATUS(ret) == RMI_ERROR_RTT) {
+			phys_addr_t rtt;
+
+			if (next_addr > addr)
+				continue; /* UNASSIGNED */
+
+			rtt = alloc_rtt(NULL);
+			if (WARN_ON(rtt == PHYS_ADDR_MAX))
+				return;
+			ret = realm_rtt_create(realm, addr,
+					       RMI_RESULT_DATA_LEVEL(ret) + 1, rtt);
+			if (WARN_ON(ret)) {
+				free_rtt(rtt);
+				return;
+			}
+			goto retry;
+		} else if (WARN_ON(ret)) {
+			continue;
+		}
+
+		ret = undelegate_range_desc(out_range);
+		if (WARN_ON(ret))
+			break;
+
+		if (may_block)
+			cond_resched_rwlock_write(&kvm->mmu_lock);
+	}
+
+	realm_fold_rtt_level(realm, get_start_level(realm) + 1,
+			     start, end);
+}
+
+void kvm_realm_unmap_range_filter(struct kvm *kvm, unsigned long start,
+				  unsigned long size, bool may_block,
+				  enum kvm_gfn_range_filter attr_filter)
+{
+	unsigned long end = start + size;
+	struct realm *realm = &kvm->arch.realm;
+
+	if (!kvm_realm_is_created(kvm))
+		return;
+
+	end = min(BIT(realm->ia_bits - 1), end);
+
+	if (attr_filter & KVM_FILTER_SHARED)
+		realm_unmap_shared_range(kvm, start, end, may_block);
+	if (attr_filter & KVM_FILTER_PRIVATE)
+		realm_unmap_private_range(kvm, start, end, may_block);
+}
+
+void kvm_realm_unmap_range(struct kvm *kvm, unsigned long start,
+			   unsigned long size, bool unmap_private,
+			   bool may_block)
+{
+	enum kvm_gfn_range_filter attr_filter = KVM_FILTER_SHARED;
+
+	if (unmap_private)
+		attr_filter |= KVM_FILTER_PRIVATE;
+
+	kvm_realm_unmap_range_filter(kvm, start, size, may_block, attr_filter);
+}
+
 void kvm_destroy_realm(struct kvm *kvm)
 {
 	struct realm *realm = &kvm->arch.realm;
-- 
2.43.0

[PATCH 09/11] KVM: arm64: CCA: Prevent memslot modifications after Realm creation

[Steven Price]

From: Suzuki K Poulose <suzuki.poulose@arm.com>

Once the Realm is created, prevent any changes to guest_memfd backed
memslots, as the private memory is provided by slots backed by gmem.
This will also be validated when we handle a fault for Realm Stage2 (in
a later commit).

Additionally, similar to pKVM, we do not support dirty logging or
read-only memslots.

Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/kvm/mmu.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 51fda7b2ff7f..eb8c5f9e9791 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2862,13 +2862,28 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
 	    (new->flags & (KVM_MEM_LOG_DIRTY_PAGES | KVM_MEM_READONLY)))
 		return -EPERM;
 
-	if (kvm->arch.vm_flavor == VM_PROTECTED_PKVM) {
+	switch (kvm->arch.vm_flavor) {
+	case VM_PROTECTED_PKVM:
 		/* Cannot modify memslots once a pVM has run. */
 		if (pkvm_hyp_vm_is_created(kvm) &&
-		    (change == KVM_MR_DELETE || change == KVM_MR_MOVE)) {
+		    (change == KVM_MR_DELETE || change == KVM_MR_MOVE))
 			return -EPERM;
-		}
+		break;
+	case VM_REALM:
+		/*
+		 * Once the Realm is created, we cannot modify any slots that
+		 * could be providing private memory. i.e., guest_memfd backed
+		 * slots.
+		 * TODO: Handle trusted device private memory slots
+		 */
+		if (!kvm_realm_is_created(kvm))
+			break;
 
+		if (kvm_slot_has_gmem(old) || kvm_slot_has_gmem(new))
+			return -EPERM;
+		break;
+	default:
+		break;
 	}
 
 	if (change != KVM_MR_CREATE && change != KVM_MR_MOVE &&
-- 
2.43.0

[PATCH 10/11] KVM: arm64: CCA: Support runtime faulting of memory

[Steven Price]

At runtime if the realm guest accesses memory which hasn't yet been
mapped then KVM needs to either populate the region or fault the guest.

For memory in the lower (protected) region of IPA, verify that the
corresponding guest_memfd page is private before delegating to the RMM.
The RMM then maps the page into the realm and initializes its contents
to zero.

For memory in the upper (shared) IPA region, map the backing page as
non-secure memory. Check that a guest_memfd page is shared when the
fault uses guest_memfd; ordinary userspace-backed memslots follow the
normal user memory fault path.

If the fault IPA and guest_memfd attributes disagree then exit to
userspace with KVM_EXIT_MEMORY_FAULT so the VMM can perform the required
conversion.

Co-developed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/include/asm/kvm_emulate.h |  13 ++
 arch/arm64/include/asm/kvm_rmi.h     |   7 +-
 arch/arm64/kvm/mmu.c                 | 115 +++++++++++---
 arch/arm64/kvm/rmi.c                 | 228 ++++++++++++++++++++++++++-
 4 files changed, 336 insertions(+), 27 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 32021bdf2610..8a71fddfdb85 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -847,4 +847,17 @@ static inline bool kvm_realm_is_created(struct kvm *kvm)
 	return kvm_vm_is_realm(kvm) && kvm_realm_state(kvm) != REALM_STATE_NONE;
 }
 
+static inline gpa_t kvm_gpa_from_fault(struct kvm *kvm, phys_addr_t ipa)
+{
+	if (!kvm_vm_is_realm(kvm))
+		return ipa;
+
+	return ipa & ~BIT(kvm->arch.realm.ia_bits - 1);
+}
+
+static inline gfn_t kvm_gfn_from_fault(struct kvm *kvm, phys_addr_t ipa)
+{
+	return gpa_to_gfn(kvm_gpa_from_fault(kvm, ipa));
+}
+
 #endif /* __ARM64_KVM_EMULATE_H__ */
diff --git a/arch/arm64/include/asm/kvm_rmi.h b/arch/arm64/include/asm/kvm_rmi.h
index 72a9449f00d7..105660d4d56a 100644
--- a/arch/arm64/include/asm/kvm_rmi.h
+++ b/arch/arm64/include/asm/kvm_rmi.h
@@ -6,6 +6,7 @@
 #ifndef __ASM_KVM_RMI_H
 #define __ASM_KVM_RMI_H
 
+#include <asm/kvm_pgtable.h>
 #include <linux/arm-smccc-rmi.h>
 #include <linux/kvm.h>
 
@@ -92,9 +93,13 @@ void kvm_realm_unmap_range_filter(struct kvm *kvm,
 void kvm_realm_unmap_range(struct kvm *kvm,
 			   unsigned long ipa,
 			   unsigned long size,
-			   bool unmap_private,
 			   bool may_block);
 
+int realm_map_ipa(struct kvm *kvm, phys_addr_t ipa,
+		  kvm_pfn_t pfn, unsigned long map_size,
+		  enum kvm_pgtable_prot prot,
+		  struct kvm_mmu_memory_cache *memcache);
+
 static inline bool kvm_realm_is_private_address(struct realm *realm,
 						unsigned long addr)
 {
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index eb8c5f9e9791..20aedbe6ec07 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -434,7 +434,11 @@ static void stage2_flush_memslot(struct kvm *kvm,
 	phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT;
 	phys_addr_t end = addr + PAGE_SIZE * memslot->npages;
 
-	kvm_stage2_flush_range(&kvm->arch.mmu, addr, end);
+	if (kvm_vm_is_realm(kvm))
+		kvm_realm_unmap_range_filter(kvm, addr, end - addr, true,
+					     KVM_FILTER_SHARED);
+	else
+		kvm_stage2_flush_range(&kvm->arch.mmu, addr, end);
 }
 
 /**
@@ -1221,6 +1225,10 @@ void stage2_unmap_vm(struct kvm *kvm)
 	struct kvm_memory_slot *memslot;
 	int idx, bkt;
 
+	/* For realms this is handled by the RMM so nothing to do here */
+	if (kvm_vm_is_realm(kvm))
+		return;
+
 	idx = srcu_read_lock(&kvm->srcu);
 	mmap_read_lock(current->mm);
 	write_lock(&kvm->mmu_lock);
@@ -1790,6 +1798,10 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd,
 	void *memcache = NULL;
 	gfn_t canonical_gfn;
 	int ret;
+	bool is_realm_private = false;
+
+	if (KVM_BUG_ON(!kvm_slot_has_gmem(s2fd->memslot), kvm))
+		return -EINVAL;
 
 	if (!perm_fault) {
 		memcache = get_mmu_memcache(s2fd->vcpu);
@@ -1804,7 +1816,7 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd,
 	}
 
 	s2vi.vma_pagesize = PAGE_SIZE;
-	s2vi.gfn = ALIGN_DOWN(s2fd->fault_ipa, s2vi.vma_pagesize) >> PAGE_SHIFT;
+	s2vi.gfn = gpa_to_gfn(kvm_gpa_from_fault(kvm, s2fd->fault_ipa));
 	canonical_gfn = get_canonical_gfn(s2fd, &s2vi);
 
 	write_fault = kvm_s2_fault_is_write(s2fd);
@@ -1816,13 +1828,28 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd,
 	/* Pairs with the smp_wmb() in kvm_mmu_invalidate_end(). */
 	smp_rmb();
 
+	if (kvm_vm_is_realm(kvm)) {
+		is_realm_private = kvm_realm_is_private_address(&kvm->arch.realm,
+								s2fd->fault_ipa);
+
+		if (is_realm_private != kvm_gmem_is_private_gfn(kvm, canonical_gfn)) {
+			kvm_prepare_memory_fault_exit(s2fd->vcpu,
+						      gfn_to_gpa(canonical_gfn),
+						      PAGE_SIZE,
+						      write_fault, exec_fault,
+						      is_realm_private);
+			kfree(mapping);
+			return -EFAULT;
+		}
+	}
+
 	ret = kvm_gmem_get_pfn(kvm, s2fd->memslot, canonical_gfn, &s2vi.pfn, NULL);
 	if (ret) {
 		/* If result is non-NULL this is a synthetic fault. */
 		if (!result)
 			kvm_prepare_memory_fault_exit(s2fd->vcpu, gfn_to_gpa(canonical_gfn),
 						      s2vi.vma_pagesize, write_fault, exec_fault,
-						      false);
+						      is_realm_private);
 		kfree(mapping);
 		return ret;
 	}
@@ -1845,6 +1872,13 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd,
 		goto out_unlock;
 	}
 
+	if (kvm_vm_is_realm(kvm)) {
+		prot &= ~KVM_PGTABLE_PROT_X;
+		ret = realm_map_ipa(kvm, s2fd->fault_ipa, s2vi.pfn,
+				    PAGE_SIZE, prot, memcache);
+		goto out_unlock;
+	}
+
 	if (perm_fault) {
 		/*
 		 * Drop the SW bits in favour of those stored in the
@@ -2046,7 +2080,8 @@ static int kvm_s2_fault_get_vma_info(const struct kvm_s2_fault_desc *s2fd,
 	 * mapping size to ensure we find the right PFN and lay down the
 	 * mapping in the right place.
 	 */
-	s2vi->gfn = ALIGN_DOWN(s2fd->fault_ipa, s2vi->vma_pagesize) >> PAGE_SHIFT;
+	s2vi->gfn = kvm_gfn_from_fault(kvm, ALIGN_DOWN(s2fd->fault_ipa,
+						       s2vi->vma_pagesize));
 
 	s2vi->mte_allowed = kvm_vma_mte_allowed(vma);
 
@@ -2241,12 +2276,15 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 	if (!perm_fault_granule && !s2vi->map_non_cacheable && kvm_has_mte(kvm))
 		sanitise_mte_tags(kvm, pfn, mapping_size);
 
-	/*
-	 * Under the premise of getting a FSC_PERM fault, we just need to relax
-	 * permissions only if mapping_size equals perm_fault_granule. Otherwise,
-	 * kvm_pgtable_stage2_map() should be called to change block size.
-	 */
-	if (mapping_size == perm_fault_granule) {
+	if (kvm_vm_is_realm(kvm)) {
+		ret = realm_map_ipa(kvm, s2fd->fault_ipa, pfn, mapping_size,
+				    prot, memcache);
+	} else if (mapping_size == perm_fault_granule) {
+		/*
+		 * Under the premise of getting a FSC_PERM fault, we just need to relax
+		 * permissions only if mapping_size equals perm_fault_granule. Otherwise,
+		 * kvm_pgtable_stage2_map() should be called to change block size.
+		 */
 		/*
 		 * Drop the SW bits in favour of those stored in the
 		 * PTE, which will be preserved.
@@ -2339,18 +2377,43 @@ static int user_mem_abort(const struct kvm_s2_fault_desc *s2fd,
 	return kvm_s2_fault_map(s2fd, &s2vi, prot, memcache, result);
 }
 
+static bool private_ipa_fault(struct kvm *kvm, phys_addr_t fault_ipa);
+static bool kvm_fault_is_gmem_abort(struct kvm *kvm,
+				    const struct kvm_s2_fault_desc *s2fd);
+
 static int kvm_vm_mem_abort(const struct kvm_s2_fault_desc *s2fd)
 {
 	VM_WARN_ON_ONCE(kvm_s2_fault_is_perm(s2fd) &&
 			!esr_abt_is_write_fault(s2fd->esr) &&
 			!kvm_s2_fault_is_exec(s2fd));
 
-	if (kvm_slot_has_gmem(s2fd->memslot))
+	if (private_ipa_fault(s2fd->vcpu->kvm, s2fd->fault_ipa) &&
+	    !kvm_slot_has_gmem(s2fd->memslot))
+		return -EINVAL;
+
+	if (kvm_fault_is_gmem_abort(s2fd->vcpu->kvm, s2fd))
 		return gmem_abort(s2fd, NULL);
 	else
 		return user_mem_abort(s2fd, NULL);
 }
 
+/* Decide if an abort should be handled by the gmem */
+static bool kvm_fault_is_gmem_abort(struct kvm *kvm,
+				    const struct kvm_s2_fault_desc *s2fd)
+{
+	if (!kvm_slot_has_gmem(s2fd->memslot))
+		return false;
+	if (kvm_memslot_is_gmem_only(s2fd->memslot))
+		return true;
+	/*
+	 * For Realms, all private faults must be backed by GMEM.
+	 * TODO: Handle Trusted device private memory mappings.
+	 */
+	if (private_ipa_fault(kvm, s2fd->fault_ipa))
+		return true;
+	return false;
+}
+
 /* Resolve the access fault by making the page young again. */
 static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
 {
@@ -2444,6 +2507,12 @@ int kvm_handle_guest_sea(struct kvm_vcpu *vcpu)
 	return 0;
 }
 
+static bool private_ipa_fault(struct kvm *kvm, phys_addr_t fault_ipa)
+{
+	return kvm_vm_is_realm(kvm) &&
+		kvm_realm_is_private_address(&kvm->arch.realm, fault_ipa);
+}
+
 /**
  * kvm_handle_guest_abort - handles all 2nd stage aborts
  * @vcpu:	the VCPU pointer
@@ -2457,10 +2526,10 @@ int kvm_handle_guest_sea(struct kvm_vcpu *vcpu)
  */
 int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 {
+	struct kvm *kvm = vcpu->kvm;
 	struct kvm_s2_trans nested_trans, *nested = NULL;
 	unsigned long esr = kvm_vcpu_get_esr(vcpu);
 	struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu;
-	struct kvm *kvm = vcpu->kvm;
 	phys_addr_t fault_ipa; /* The address we faulted on */
 	phys_addr_t ipa; /* Always the IPA in the L1 guest phys space */
 	struct kvm_memory_slot *memslot;
@@ -2553,8 +2622,9 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 		nested = &nested_trans;
 	}
 
-	gfn = ipa >> PAGE_SHIFT;
+	gfn = kvm_gfn_from_fault(kvm, ipa);
 	memslot = gfn_to_memslot(kvm, gfn);
+
 	hva = gfn_to_hva_memslot_prot(memslot, gfn, &writable);
 	write_fault = esr_abt_is_write_fault(esr);
 	if (kvm_is_error_hva(hva) || (write_fault && !writable)) {
@@ -2597,7 +2667,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 		 * of the page size.
 		 */
 		ipa |= FAR_TO_FIPA_OFFSET(kvm_vcpu_get_hfar(vcpu));
-		ret = io_mem_abort(vcpu, ipa);
+		ret = io_mem_abort(vcpu, kvm_gpa_from_fault(kvm, ipa));
 		goto out_unlock;
 	}
 
@@ -2856,6 +2926,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
 				   enum kvm_mr_change change)
 {
 	hva_t hva, reg_end;
+	gfn_t ipa_limit;
 	int ret = 0;
 
 	if (kvm_vm_is_protected(kvm) && new &&
@@ -2891,17 +2962,21 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
 		return 0;
 
 	/*
-	 * Prevent userspace from creating a memory region outside of the IPA
-	 * space addressable by the KVM guest IPA space.
+	 * Realm shared IPAs alias the corresponding private IPAs, so memslots
+	 * must reside entirely in the canonical lower half of the IPA space.
 	 */
-	if ((new->base_gfn + new->npages) > (kvm_phys_size(&kvm->arch.mmu) >> PAGE_SHIFT))
+	ipa_limit = kvm_phys_size(&kvm->arch.mmu) >> PAGE_SHIFT;
+	if (kvm_vm_is_realm(kvm))
+		ipa_limit >>= 1;
+	if (new->base_gfn + new->npages > ipa_limit)
 		return -EFAULT;
 
 	/*
-	 * Only support guest_memfd backed memslots with mappable memory, since
-	 * there aren't any CoCo VMs that support only private memory on arm64.
+	 * Only support guest_memfd backed memslots with mappable memory,
+	 * unless the guest is a CCA realm guest.
 	 */
-	if (kvm_slot_has_gmem(new) && !kvm_memslot_is_gmem_only(new))
+	if (kvm_slot_has_gmem(new) && !kvm_memslot_is_gmem_only(new) &&
+	    !kvm_vm_is_realm(kvm))
 		return -EINVAL;
 
 	/* guest_memfd is incompatible with MTE. */
diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
index 1066f767f85d..f89460d8adcb 100644
--- a/arch/arm64/kvm/rmi.c
+++ b/arch/arm64/kvm/rmi.c
@@ -531,15 +531,231 @@ void kvm_realm_unmap_range_filter(struct kvm *kvm, unsigned long start,
 }
 
 void kvm_realm_unmap_range(struct kvm *kvm, unsigned long start,
-			   unsigned long size, bool unmap_private,
-			   bool may_block)
+			   unsigned long size, bool may_block)
 {
-	enum kvm_gfn_range_filter attr_filter = KVM_FILTER_SHARED;
+	kvm_realm_unmap_range_filter(kvm, start, size, may_block,
+				     KVM_FILTER_SHARED | KVM_FILTER_PRIVATE);
+}
+
+static unsigned long addr_range_desc(unsigned long phys, unsigned long size,
+				     unsigned long state)
+{
+	unsigned long count = 1;
+	unsigned long out;
+	int rmi_size;
+
+	for (rmi_size = KVM_PGTABLE_LAST_LEVEL; rmi_size >= 0; rmi_size--) {
+		unsigned long entry_size = rmi_range_entry_size(rmi_size);
+
+		if (size == entry_size)
+			break;
+		if (size > entry_size) {
+			rmi_size = -1;
+			break;
+		}
+	}
+
+	if (rmi_size < 0) {
+		/*
+		 * Only support mapping at the page level granularity when
+		 * it's an unusual length. This should get us back onto a larger
+		 * block size for the subsequent mappings.
+		 */
+		rmi_size = 0;
+		count = MIN(size >> PAGE_SHIFT, PTRS_PER_PTE - 1);
+	}
+
+	WARN_ON(phys & ~PAGE_MASK);
+
+	out = FIELD_PREP(RMI_ADDR_RANGE_BLOCK_SIZE_MASK, rmi_size) |
+	      FIELD_PREP(RMI_ADDR_RANGE_COUNT_MASK, count) |
+	      FIELD_PREP(RMI_ADDR_RANGE_STATE_MASK, state);
+	out |= phys & PAGE_MASK;
+
+	return out;
+}
+
+static int realm_map_protected(struct kvm *kvm,
+			       unsigned long ipa,
+			       kvm_pfn_t pfn,
+			       unsigned long map_size,
+			       struct kvm_mmu_memory_cache *memcache)
+{
+	struct realm *realm = &kvm->arch.realm;
+	phys_addr_t phys = __pfn_to_phys(pfn);
+	phys_addr_t base_phys = phys;
+	phys_addr_t rd = virt_to_phys(realm->rd);
+	phys_addr_t delegated_phys;
+	unsigned long base_ipa = ipa;
+	unsigned long ipa_top;
+	long ret = 0;
+
+	if (WARN_ON(!IS_ALIGNED(map_size, PAGE_SIZE) ||
+		    !IS_ALIGNED(ipa, map_size)))
+		return -EINVAL;
+
+	if (rmi_delegate_range(phys, map_size, &delegated_phys)) {
+		if (delegated_phys == phys) {
+			/*
+			 * It's likely we raced with another VCPU on the same
+			 * fault. Assume the other VCPU has handled the fault
+			 * and return to the guest.
+			 */
+			return 0;
+		}
+		/* Partial delegation - map as much as we can */
+		map_size = delegated_phys - phys;
+	}
+
+	ipa_top = ipa + map_size;
+
+	while (ipa < ipa_top) {
+		unsigned long flags = RMI_ADDR_TYPE_SINGLE;
+		unsigned long range_desc = addr_range_desc(phys, ipa_top - ipa,
+							   RMI_OP_MEM_DELEGATED);
+		unsigned long out_top;
+
+		ret = rmi_rtt_data_map(rd, ipa, ipa_top, flags, range_desc,
+				       &out_top);
+		if (ret < 0)
+			goto err_undelegate;
+
+		if (RMI_RESULT_STATUS(ret) == RMI_ERROR_RTT) {
+			/* Create missing RTTs and retry */
+			int level = RMI_RESULT_DATA_LEVEL(ret);
+
+			if (WARN_ON(level >= KVM_PGTABLE_LAST_LEVEL))
+				goto err_undelegate;
+			ret = realm_create_rtt_levels(realm, ipa, level,
+						      level + 1,
+						      memcache);
+			if (ret)
+				goto err_undelegate;
 
-	if (unmap_private)
-		attr_filter |= KVM_FILTER_PRIVATE;
+			continue;
+		}
+
+		if (WARN_ON(ret))
+			goto err_undelegate;
+
+		phys += out_top - ipa;
+		ipa = out_top;
+	}
+
+	return 0;
+
+err_undelegate:
+	realm_unmap_private_range(kvm, base_ipa, ipa, true);
+	/* A range which cannot be undelegated remains owned by the RMM. */
+	WARN_ON(rmi_undelegate_range(base_phys, map_size));
+	return ret < 0 ? ret : -ENXIO;
+}
+
+static int realm_map_non_secure(struct kvm *kvm,
+				unsigned long ipa,
+				kvm_pfn_t pfn,
+				unsigned long size,
+				enum kvm_pgtable_prot prot,
+				struct kvm_mmu_memory_cache *memcache)
+{
+	struct realm *realm = &kvm->arch.realm;
+	unsigned long attr, flags = 0;
+	phys_addr_t rd = virt_to_phys(realm->rd);
+	phys_addr_t phys = __pfn_to_phys(pfn);
+	unsigned long ipa_top = ipa + size;
+	long ret;
+
+	if (WARN_ON(!IS_ALIGNED(size, PAGE_SIZE)))
+		return -EINVAL;
+
+	switch (prot & (KVM_PGTABLE_PROT_DEVICE | KVM_PGTABLE_PROT_NORMAL_NC)) {
+	case KVM_PGTABLE_PROT_DEVICE | KVM_PGTABLE_PROT_NORMAL_NC:
+		return -EINVAL;
+	case KVM_PGTABLE_PROT_DEVICE:
+		attr = MT_S2_FWB_DEVICE_nGnRE;
+		break;
+	case KVM_PGTABLE_PROT_NORMAL_NC:
+		attr = MT_S2_FWB_NORMAL_NC;
+		break;
+	default:
+		attr = MT_S2_FWB_NORMAL;
+	}
+
+	flags |= FIELD_PREP(RMI_RTT_UNPROT_MAP_FLAGS_MEMATTR_MASK, attr);
+
+	if (prot & KVM_PGTABLE_PROT_R)
+		flags |= FIELD_PREP(RMI_RTT_UNPROT_MAP_FLAGS_S2AP_MASK, RMI_S2AP_DIRECT_READ);
+	if (prot & KVM_PGTABLE_PROT_W)
+		flags |= FIELD_PREP(RMI_RTT_UNPROT_MAP_FLAGS_S2AP_MASK, RMI_S2AP_DIRECT_WRITE);
+
+	flags |= RMI_ADDR_TYPE_SINGLE;
+
+	while (ipa < ipa_top) {
+		unsigned long range_desc = addr_range_desc(phys, ipa_top - ipa,
+							   RMI_OP_MEM_UNDELEGATED);
+		unsigned long out_top;
+
+		ret = rmi_rtt_unprot_map(rd, ipa, ipa_top, flags, range_desc,
+					 &out_top);
+		if (ret < 0)
+			return ret;
+
+		if (RMI_RESULT_STATUS(ret) == RMI_ERROR_RTT) {
+			/* Create missing RTTs and retry */
+			int level = RMI_RESULT_DATA_LEVEL(ret);
+			int req_level = find_map_level(realm, ipa, ipa_top);
+
+			/*
+			 * There already exists a mapping at the level. May be
+			 * we are relaxing a permission for the given range ?
+			 */
+			if (level >= req_level) {
+				realm_unmap_shared_range(kvm, ipa, ipa_top, true);
+				continue;
+			}
+
+			ret = realm_create_rtt_levels(realm, ipa, level,
+						      req_level,
+						      memcache);
+			if (ret)
+				return ret;
+
+			ret = rmi_rtt_unprot_map(rd, ipa, ipa_top, flags,
+						 range_desc, &out_top);
+			if (ret < 0)
+				return ret;
+		}
+
+		if (ret && RMI_RESULT_STATUS(ret) != RMI_ERROR_RTT)
+			return ret;
+
+		phys += out_top - ipa;
+		ipa = out_top;
+	}
+
+	return 0;
+}
+
+int realm_map_ipa(struct kvm *kvm, phys_addr_t ipa,
+		  kvm_pfn_t pfn, unsigned long map_size,
+		  enum kvm_pgtable_prot prot,
+		  struct kvm_mmu_memory_cache *memcache)
+{
+	struct realm *realm = &kvm->arch.realm;
+
+	if (kvm_realm_state(kvm) != REALM_STATE_ACTIVE)
+		return -EBUSY;
+
+	ipa = ALIGN_DOWN(ipa, map_size);
+	if (!kvm_realm_is_private_address(realm, ipa)) {
+		return realm_map_non_secure(kvm, ipa, pfn, map_size, prot,
+					    memcache);
+	}
 
-	kvm_realm_unmap_range_filter(kvm, start, size, may_block, attr_filter);
+	/* It's impossible to map protected pages read-only. */
+	if (WARN_ON(!(prot & KVM_PGTABLE_PROT_W)))
+		return -EFAULT;
+	return realm_map_protected(kvm, ipa, pfn, map_size, memcache);
 }
 
 void kvm_destroy_realm(struct kvm *kvm)
-- 
2.43.0

[PATCH 11/11] KVM: arm64: CCA: Add Realm stage-2 MMU operations

[Steven Price]

From: Suzuki K Poulose <suzuki.poulose@arm.com>

Register the Realm stage-2 MMU operations now that the RMM mapping
and unmapping helpers are available. Route unmapping through the
RMM, and use the existing fault handler for Realm memory faults.

The RMM handles TLB invalidation when stage-2 mappings change, so
the host has no separate TLB flush to perform. Realm stage-2 does
not support access flags, including for shared IPAs.

Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Steven Price <steven.price@arm.com>
---
 arch/arm64/kvm/mmu.c | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 20aedbe6ec07..bb33c07569e3 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -191,6 +191,12 @@ static int kvm_vm_flush_remote_tlbs(struct kvm *kvm)
 	return 0;
 }
 
+static int realm_flush_remote_tlbs(struct kvm *kvm)
+{
+	/* Nothing to do here, RMM does the job */
+	return 0;
+}
+
 /**
  * kvm_arch_flush_remote_tlbs() - flush all VM TLB entries for v7/8
  * @kvm:	pointer to kvm structure.
@@ -218,6 +224,13 @@ static int kvm_vm_flush_remote_tlbs_range(struct kvm *kvm,
 	return 0;
 }
 
+static int realm_flush_remote_tlbs_range(struct kvm *kvm,
+					 gfn_t gfn, u64 nr_pages)
+{
+	/* Nothing to do here, RMM does the job */
+	return 0;
+}
+
 int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm,
 				     gfn_t gfn, u64 nr_pages)
 {
@@ -353,6 +366,15 @@ static void no_stage2_unmap_range(struct kvm_s2_mmu *mmu,
 {
 }
 
+static void realm_stage2_unmap_range(struct kvm_s2_mmu *mmu,
+				     phys_addr_t start, u64 size, bool may_block,
+				     enum kvm_gfn_range_filter filter)
+{
+	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
+
+	kvm_realm_unmap_range_filter(kvm, start, size, may_block, filter);
+}
+
 /*
  * Unmapping vs dcache management:
  *
@@ -3291,6 +3313,15 @@ static const struct kvm_vm_s2_ops kvm_default_vm_s2_ops = {
 	.vm_mem_abort			= kvm_vm_mem_abort,
 };
 
+static const struct kvm_vm_s2_ops realm_vm_s2_ops = {
+	.vm_flush_remote_tlbs		= realm_flush_remote_tlbs,
+	.vm_flush_remote_tlbs_range	= realm_flush_remote_tlbs_range,
+	.vm_age_gfn			= no_age_gfn,
+	.vm_test_age_gfn		= no_age_gfn,
+	.vm_stage2_unmap_range		= realm_stage2_unmap_range,
+	.vm_mem_abort			= kvm_vm_mem_abort,
+};
+
 #define KVM_VM_S2_OPS(flavor, ops)		\
 		[flavor] = &(ops)
 
@@ -3299,7 +3330,7 @@ static const struct kvm_vm_s2_ops *arm64_vm_s2_ops[] = {
 	KVM_VM_S2_OPS(VM_NVHE, kvm_default_vm_s2_ops),
 	KVM_VM_S2_OPS(VM_PKVM, pkvm_vm_s2_ops),
 	KVM_VM_S2_OPS(VM_PROTECTED_PKVM, protected_pkvm_vm_s2_ops),
-	[VM_REALM] = NULL,
+	KVM_VM_S2_OPS(VM_REALM, realm_vm_s2_ops),
 };
 
 static int kvm_vm_init_vm_s2_ops(struct kvm *kvm)
-- 
2.43.0

Re: [PATCH 02/11] KVM: arm64: CCA: Introduce Realms

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] KVM: arm64: premature kvm_arch_has_private_mem() breaks gmem checks and caps
- [Low] KVM: arm64: duplicate kvm_arm_vcpu_finalize() declarations in kvm_host.h
--

commit d7e08ee0dbb02a4cf114f13c026edc843145fe6e
Author: Suzuki K Poulose <suzuki.poulose@arm.com>

KVM: arm64: CCA: Introduce Realms

This commit adds foundational infrastructure for Arm CCA Realm VMs,
including the VM_REALM flavor, a static key for RMI availability,
per-VM Realm state tracking in struct kvm_arch, and initial feature and
memslot checks.

> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index ccd2dc8f73c3a..7d0222047ca1e 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
[ ... ]
> @@ -1572,30 +1584,43 @@ struct kvm *kvm_arch_alloc_vm(void);
[ ... ]
> +#define kvm_vm_is_realm(kvm)		((kvm)->arch.vm_flavor == VM_REALM)
> +#define vcpu_is_rec(vcpu)		kvm_vm_is_realm((vcpu)->kvm)
>  
>  #define kvm_vm_hyp_is_distrusting(kvm)	((kvm)->arch.vm_flavor >= __VM_DISTRUSTING_HYP)
>  
> -#endif	/* __KVM_NVHE_HYPERVISOR__ */
> +#define kvm_arch_has_private_mem(kvm) kvm_vm_is_realm(kvm)

[Severity: High]
Does defining kvm_arch_has_private_mem(kvm) here cause a regression in
guest_memfd memslot validation and capability reporting across arm64?

Because CONFIG_KVM_PRIVATE_MEMATTR_VM is not selected on arm64, defining
kvm_arch_has_private_mem activates the preprocessor branch in
include/linux/kvm_host.h that sets gmem_in_place_conversion to true at
compile time:

include/linux/kvm_host.h:
    #elif defined kvm_arch_has_private_mem
    #define gmem_in_place_conversion true
    #define kvm_is_private_gfn kvm_gmem_is_private_gfn

With gmem_in_place_conversion unconditionally true,
kvm_gmem_prepare_memory_region() in virt/kvm/guest_memfd.c sets
KVM_MEMSLOT_GMEM_ONLY on every guest_memfd memslot even when
GUEST_MEMFD_FLAG_MMAP is not set:

virt/kvm/guest_memfd.c:kvm_gmem_prepare_memory_region() {
    ...
    if (gmem_in_place_conversion || kvm_gmem_supports_mmap(inode))
        slot->flags |= KVM_MEMSLOT_GMEM_ONLY;
}

As a result, the check in kvm_arch_prepare_memory_region() in
arch/arm64/kvm/mmu.c that rejects non-mappable guest_memfd memslots:

arch/arm64/kvm/mmu.c:kvm_arch_prepare_memory_region() {
    ...
    if (kvm_slot_has_gmem(new) && !kvm_memslot_is_gmem_only(new))
        return -EINVAL;
}

is bypassed for non-Realm VMs, allowing userspace to bind a private
guest_memfd created with flags == 0 via KVM_SET_USER_MEMORY_REGION2 and
fault those pages into non-Realm stage-2 page tables through gmem_abort().

In addition, when userspace queries
KVM_CHECK_EXTENSION(KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES) on the /dev/kvm
fd (where kvm is NULL), kvm_vm_ioctl_check_extension_generic() in
virt/kvm/kvm_main.c calls kvm_supports_private_mem(NULL):

virt/kvm/kvm_main.c:kvm_supports_private_mem() {
    return !kvm || kvm_arch_has_private_mem(kvm);
}

virt/kvm/kvm_main.c:kvm_vm_ioctl_check_extension_generic() {
    ...
    case KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES:
        if (!gmem_in_place_conversion || !kvm_supports_private_mem(kvm))
            return 0;

        return KVM_MEMORY_ATTRIBUTE_PRIVATE;
}

This advertises KVM_MEMORY_ATTRIBUTE_PRIVATE on all arm64 hosts even when
KVM_MODE_RMM is not active and while kvm_gmem_set_attributes() in
virt/kvm/guest_memfd.c rejects KVM_SET_MEMORY_ATTRIBUTES2 with -EINVAL on
every creatable VM.

> +
> +int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature);
> +bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);

[Severity: Low]
This isn't a bug, but commit df6a70516adec8 already moved the declarations
of kvm_arm_vcpu_finalize() and kvm_arm_vcpu_is_finalized() earlier in
arch/arm64/include/asm/kvm_host.h right below VCPU_PKVM_FINALIZED:

arch/arm64/include/asm/kvm_host.h:
    #define VCPU_PKVM_FINALIZED	__vcpu_single_flag(cflags, BIT(2))

    int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature);
    bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);

Can these duplicate declarations be dropped?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261102141304.126883-1-steven.price@arm.com?part=2

Re: [PATCH 04/11] KVM: arm64: CCA: Check for LPA2 support and SHA-256

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] KVM: arm64: rmi_has_feature() passes non-constant mask to u64_get_bits()
--

commit a9400749495394b73c9cccd919996b48771556fe
Author: Steven Price <steven.price@arm.com>

KVM: arm64: CCA: Check for LPA2 support and SHA-256

This commit adds feature checks during KVM RMI initialization to verify that
the RMM supports LPA2 when KVM has LPA2 enabled and that the RMM supports
SHA-256 measurements.

> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
> index 9d118768550d6..5b07beff8bc91 100644
> --- a/arch/arm64/kvm/rmi.c
> +++ b/arch/arm64/kvm/rmi.c
> @@ -6,8 +6,30 @@
>  #include <linux/arm-rmi-cmds.h>
>  #include <linux/kvm_host.h>
>  
> +#include <asm/kvm_pgtable.h>
>  #include <asm/virt.h>
>  
> +static bool rmi_has_feature(int reg, unsigned long feature)
> +{
> +	return !!u64_get_bits(rmi_feat_reg(reg), feature);
> +}

[Severity: Low]
Can this fail to build if the compiler decides not to inline
rmi_has_feature()?

In include/linux/bitfield-fix-width.h, u64_get_bits() calls
field_multiplier(field):

static __always_inline
u64 field_multiplier(u64 field)
{
	if ((field | (field - 1)) & ((field | (field - 1)) + 1))
		__bad_mask();
	return field & -field;
}

Because rmi_has_feature() is not marked __always_inline, if the compiler does
not inline it (for example under CONFIG_CC_OPTIMIZE_FOR_SIZE or compiler
instrumentation), the feature parameter is not a compile-time constant inside
field_multiplier(), preventing constant folding from eliminating the call to
__bad_mask().

Should rmi_has_feature() be marked __always_inline, or use field_get() or a
direct bitwise test instead?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261102141304.126883-1-steven.price@arm.com?part=4

Re: [PATCH 06/11] KVM: arm64: CCA: Add basic infrastructure for creating a realm

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] KVM: arm64: missing realm->ia_bits initialization in kvm_init_realm()
- [Medium] KVM: arm64: cca: clamp Realm IPA limit to host limit in kvm_init_ipa_range()
--

commit e03ea9b753f8211e9e93f2e5edff9ba91b09d112
Author: Steven Price <steven.price@arm.com>

KVM: arm64: CCA: Add basic infrastructure for creating a realm

This commit introduces the initial functions and state structures for
creating and destroying a Realm VM, and checks the requested IPA size
against the RMM's supported Stage-2 address size.

> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index 1301c563d7052..b049ea67d9af3 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -981,10 +981,14 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = {
>  
>  static int kvm_init_ipa_range(struct kvm_s2_mmu *mmu, unsigned long type)
>  {
> +	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
>  	u32 kvm_ipa_limit = get_kvm_ipa_limit();
>  	u64 mmfr0, mmfr1;
>  	u32 phys_shift;
>  
> +	if (kvm_vm_is_realm(kvm))
> +		kvm_ipa_limit = kvm_rmm_ipa_limit();

[Severity: Medium]
Should kvm_init_ipa_range() clamp kvm_ipa_limit with min(kvm_ipa_limit,
kvm_rmm_ipa_limit()) instead of overwriting it?

In arch/arm64/kvm/rmi.c, rmm_check_features() only rejects the case where
kvm_lpa2_is_enabled() is true and the RMM lacks RMI_FEATURE_REGISTER_0_LPA2:

	if (kvm_lpa2_is_enabled() &&
	    !rmi_has_feature(0, RMI_FEATURE_REGISTER_0_LPA2)) {
		kvm_err("RMM doesn't support LPA2\n");
		return -ENXIO;
	}

It does not prevent running with !kvm_lpa2_is_enabled() when the RMM supports
LPA2 and reports a 52-bit S2SZ in RMI_FEATURE_REGISTER_0_S2SZ, or when S2SZ
exceeds the host's sanitised ID_AA64MMFR0_EL1.PARange. Meanwhile,
kvm_set_ipa_limit() caps get_kvm_ipa_limit() to 48 bits when
!kvm_lpa2_is_enabled() && PAGE_SIZE != SZ_64K.

If kvm_rmm_ipa_limit() is larger than get_kvm_ipa_limit(), overwriting
kvm_ipa_limit allows userspace to create a Realm via
kvm_arch_init_vm()->kvm_init_stage2_mmu()->kvm_init_ipa_range() with
phys_shift > get_kvm_ipa_limit(). That causes kvm_get_vtcr() to configure
VTCR_EL2_T0SZ for 52 bits without VTCR_EL2_DS, kvm_pgtable_stage2_init() to
attempt allocating 256 concatenated PGD pages, and kvm_handle_guest_abort()
to inject an address size fault for any fault_ipa >=
BIT_ULL(get_kvm_ipa_limit()):

		/* Beyond sanitised PARange (which is the IPA limit) */
		if (fault_ipa >= BIT_ULL(get_kvm_ipa_limit())) {
			kvm_inject_size_fault(vcpu);
			return 1;
		}

[ ... ]

> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
> index 5b07beff8bc91..45ecbaa467d1d 100644
> --- a/arch/arm64/kvm/rmi.c
> +++ b/arch/arm64/kvm/rmi.c

[ ... ]

> +int kvm_init_realm(struct kvm *kvm)
> +{
> +	struct realm *realm = &kvm->arch.realm;
> +
> +	realm->sro = kmalloc_obj(*realm->sro);
> +	if (!realm->sro)
> +		return -ENOMEM;
> +
> +	return 0;
> +}

[Severity: High]
Where does realm->ia_bits get initialized?

This patch adds unsigned int ia_bits to struct realm in
arch/arm64/include/asm/kvm_rmi.h and calls kvm_init_realm() from
kvm_arch_init_vm() after kvm_init_stage2_mmu() has configured
kvm->arch.mmu.vtcr, but neither kvm_init_realm() nor kvm_init_ipa_range()
sets realm->ia_bits (for example to VTCR_EL2_IPA(kvm->arch.mmu.vtcr)).

Later patches in the series ("KVM: arm64: CCA: Tear down RTTs", "KVM: arm64:
CCA: Add Realm RTT mapping and invalidation helpers", and "KVM: arm64: CCA:
Support runtime faulting of memory") read realm->ia_bits in
get_start_level(), realm_destroy_rtts(), realm_unmap_stage2(),
realm_unmap_shared_range(), kvm_realm_unmap_range_filter(),
kvm_realm_is_private_address(), and kvm_gpa_from_fault(). With realm->ia_bits
left at 0, expressions such as stage2_pgtable_levels(realm->ia_bits),
(1UL << ia_bits), and BIT(realm->ia_bits - 1) will compute wrong RTT start
levels, wrong IPA bounds, and undefined shifts.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261102141304.126883-1-steven.price@arm.com?part=6

Re: [PATCH 07/11] KVM: arm64: CCA: Tear down RTTs

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] KVM: arm64: stage-2 PGD leak for uncreated Realms in kvm_destroy_realm()
- [High] KVM: arm64: unclamped stage-2 levels in get_start_level() breaks 64K RTT teardown
- [High] KVM: arm64: free_rtt() accounts page table pages after freeing the page
- [Medium] KVM: arm64: kvm_realm_uninit_stage2() ignores REALM_STATE_NEW
--

commit 5954ac2ccf8825a659e2f39dc0db05d96c40be84
Author: Steven Price <steven.price@arm.com>

KVM: arm64: CCA: Tear down RTTs

This commit adds stage-2 page table and Realm Translation Table (RTT)
teardown for arm64 CCA realms. It unmaps the Realm IPA space during stage-2
teardown and walks non-root RTTs during final Realm destruction to destroy
and undelegate them before destroying the Realm Descriptor.

> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index b049ea67d9af3..51fda7b2ff7f3 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -1145,9 +1145,24 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
>  	return err;
>  }
>  
> +static void kvm_realm_uninit_stage2(struct kvm_s2_mmu *mmu)
> +{
> +	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
> +
> +	mutex_lock(&kvm->arch.config_lock);
> +	if (kvm_realm_state(kvm) == REALM_STATE_ACTIVE) {
> +		kvm_set_realm_state(kvm, REALM_STATE_DYING);
> +		WARN_ON(kvm_realm_teardown_stage2(kvm));
> +	}
> +	mutex_unlock(&kvm->arch.config_lock);
> +}

[Severity: Medium]
What happens if kvm_uninit_stage2_mmu() is called via
kvm_mmu_notifier_release() -> kvm_flush_shadow_all() ->
kvm_arch_flush_shadow_all() while the Realm is in REALM_STATE_NEW?

According to enum realm_state in arch/arm64/include/asm/kvm_rmi.h, the Realm
Descriptor has already been created and stage-2 memory may already be
populated in REALM_STATE_NEW. Unlike kvm_destroy_realm(), which checks
kvm_realm_is_created(kvm), sets REALM_STATE_DYING, and unmaps stage-2 for
both REALM_STATE_NEW and REALM_STATE_ACTIVE, kvm_realm_uninit_stage2() only
checks for REALM_STATE_ACTIVE.

Should kvm_realm_uninit_stage2() also handle REALM_STATE_NEW (for example by
checking kvm_realm_is_created(kvm)) so that stage-2 mappings are unmapped
before exit_mmap() finishes and the Realm transitions to REALM_STATE_DYING?

>  void kvm_uninit_stage2_mmu(struct kvm *kvm)
>  {
> -	kvm_free_stage2_pgd(&kvm->arch.mmu);
> +	if (kvm_vm_is_realm(kvm))
> +		kvm_realm_uninit_stage2(&kvm->arch.mmu);
> +	else
> +		kvm_free_stage2_pgd(&kvm->arch.mmu);
>  	kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
>  }

[Severity: High]
Does skipping kvm_free_stage2_pgd() here leak mmu->pgt, the root stage-2 PGD
pages (mmu->pgt->pgd), and the per-CPU mmu->last_vcpu_ran allocation when a
Realm VM is destroyed before the Realm is created in the RMM?

During VM creation, kvm_arch_init_vm() calls kvm_init_stage2_mmu() while the
Realm state is still REALM_STATE_NONE. If userspace closes the VM file
descriptor without creating the Realm in the RMM, kvm_destroy_realm() takes
its early return before reaching kvm_free_stage2_pgd(&kvm->arch.mmu):

arch/arm64/kvm/rmi.c:kvm_destroy_realm() {
	...
	if (!kvm_realm_is_created(kvm)) {
		kfree(realm->sro);
		realm->sro = NULL;
		return;
	}
	...
	kvm_free_stage2_pgd(&kvm->arch.mmu);
}

Similarly, if kvm_init_nested() or kvm_init_realm() fails in
kvm_arch_init_vm() after kvm_init_stage2_mmu() succeeds:

arch/arm64/kvm/arm.c:kvm_arch_init_vm() {
	...
	ret = kvm_init_nested(kvm);
	if (ret)
		goto err_uninit_mmu;

	if (kvm_vm_is_realm(kvm)) {
		ret = kvm_init_realm(kvm);
		if (ret)
			goto err_uninit_mmu;
	}

	return 0;

err_uninit_mmu:
	kvm_uninit_stage2_mmu(kvm);
	return ret;
}

the error path only calls kvm_uninit_stage2_mmu(kvm) and never calls
kvm_destroy_realm(), leaking the stage-2 PGD allocations there as well.

> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
> index 45ecbaa467d1d..f190143b33ba5 100644
> --- a/arch/arm64/kvm/rmi.c
> +++ b/arch/arm64/kvm/rmi.c

[ ... ]

> @@ -21,11 +29,180 @@ u32 kvm_rmm_ipa_limit(void)
>  	return u64_get_bits(rmi_feat_reg(0), RMI_FEATURE_REGISTER_0_S2SZ);
>  }
>  
> +static int get_start_level(struct realm *realm)
> +{
> +	return 4 - stage2_pgtable_levels(realm->ia_bits);
> +}

[Severity: High]
Can get_start_level() return a level that mismatches the stage-2 page table
configuration on 64KB page kernels and cause realm_destroy_rtts() to fail?

When kvm_init_ipa_range() configures mmu->vtcr via kvm_get_vtcr(), the
number of stage-2 levels is clamped to at least 2:

arch/arm64/kvm/hyp/pgtable.c:kvm_get_vtcr() {
	...
	lvls = stage2_pgtable_levels(phys_shift);
	if (lvls < 2)
		lvls = 2;
	...
}

Under CONFIG_ARM64_64K_PAGES with a 32-bit or 33-bit IPA,
stage2_pgtable_levels(realm->ia_bits) evaluates to 1, so get_start_level()
returns 3 (KVM_PGTABLE_LAST_LEVEL) instead of 2.

When kvm_destroy_realm() -> realm_destroy_rtts() ->
realm_tear_down_rtt_range() then calls:

	realm_tear_down_rtt_level(realm, get_start_level(realm) + 1,
				  start, end);

it passes level 4 into realm_tear_down_rtt_level(), hitting
WARN_ON(level > KVM_PGTABLE_LAST_LEVEL) and returning -EINVAL. That in turn
causes kvm_destroy_realm() to warn and return early, leaking the delegated
RD, RTT granules, root stage-2 PGD pages, and realm->sro.

> +
> +static void free_rtt(phys_addr_t phys)
> +{
> +	if (rmi_free_delegated_page(phys))
> +		return;
> +
> +	kvm_account_pgtable_pages(phys_to_virt(phys), -1);
> +}

[Severity: High]
Does calling kvm_account_pgtable_pages() after rmi_free_delegated_page()
access the page after it has already been freed and uncharged from its
memcg?

In drivers/firmware/arm_rmm/rmi.c, rmi_free_delegated_page() undelegates the
granule and immediately frees the page back to the page allocator:

drivers/firmware/arm_rmm/rmi.c:rmi_free_delegated_page() {
	if (WARN_ON_ONCE(rmi_undelegate_page(phys))) {
		/* Undelegate failed: leak the page */
		return -EBUSY;
	}

	free_page((unsigned long)phys_to_virt(phys));

	return 0;
}

Because free_page() clears page->memcg_data in __free_pages_prepare() and
releases the page for reallocation, the subsequent call to
kvm_account_pgtable_pages(phys_to_virt(phys), -1) ->
mod_lruvec_page_state() operates on a freed struct page and sees a NULL
folio_memcg(), failing to decrement the memory cgroup's
NR_SECONDARY_PAGETABLE counter (or racing with a concurrent reallocation of
that page).

Should kvm_account_pgtable_pages(phys_to_virt(phys), -1) be called before
freeing the page, matching kvm_s2_free_pages_exact() and kvm_s2_put_page()
in arch/arm64/kvm/mmu.c?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261102141304.126883-1-steven.price@arm.com?part=7

Re: [PATCH 08/11] KVM: arm64: CCA: Add Realm RTT mapping and invalidation helpers

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] KVM: arm64: uninitialized next_addr in realm_unmap_private_range()
- [High] KVM: arm64: broken address alignment and recursion in realm_fold_rtt_level()
- [Medium] KVM: arm64: page accounting after free_page() in free_rtt()
--

commit cc0b24728b73f60b1dd39eb4cc60fe5eb12322c6
Author: Steven Price <steven.price@arm.com>

KVM: arm64: CCA: Add Realm RTT mapping and invalidation helpers

This commit adds Realm Management Interface (RMI) Realm Translation Table
(RTT) creation, folding, and unmap helpers needed for CCA Realm stage-2
memory operations.

> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
> index f190143b33ba5..1066f767f85d9 100644
> --- a/arch/arm64/kvm/rmi.c
> +++ b/arch/arm64/kvm/rmi.c
> @@ -34,6 +34,83 @@ static int get_start_level(struct realm *realm)

[ ... ]

> +static phys_addr_t alloc_delegated_granule(struct kvm_mmu_memory_cache *mc)
> +{
> +	phys_addr_t phys;
> +	void *virt;
> +
> +	if (mc) {
> +		virt = kvm_mmu_memory_cache_alloc(mc);
> +	} else {
> +		virt = (void *)__get_free_page(GFP_ATOMIC | __GFP_ZERO |
> +					       __GFP_ACCOUNT);
> +	}
> +
> +	if (!virt)
> +		return PHYS_ADDR_MAX;
> +
> +	phys = virt_to_phys(virt);
> +	if (rmi_delegate_page(phys)) {
> +		free_page((unsigned long)virt);
> +		return PHYS_ADDR_MAX;
> +	}
> +
> +	return phys;
> +}
> +
> +static phys_addr_t alloc_rtt(struct kvm_mmu_memory_cache *mc)
> +{
> +	phys_addr_t phys = alloc_delegated_granule(mc);
> +
> +	if (phys != PHYS_ADDR_MAX)
> +		kvm_account_pgtable_pages(phys_to_virt(phys), 1);
> +
> +	return phys;
> +}
> +
>  static void free_rtt(phys_addr_t phys)
>  {
>  	if (rmi_free_delegated_page(phys))
> @@ -42,6 +119,32 @@ static void free_rtt(phys_addr_t phys)
>  	kvm_account_pgtable_pages(phys_to_virt(phys), -1);
>  }

[Severity: Medium]
Does free_rtt() leak the NR_SECONDARY_PAGETABLE counter in the memory cgroup
lruvec (or touch a reallocated page) by calling kvm_account_pgtable_pages()
after rmi_free_delegated_page()?

In alloc_rtt(), RTT pages are allocated via alloc_delegated_granule() with
__GFP_ACCOUNT and accounted with kvm_account_pgtable_pages(..., 1). However,
when free_rtt() frees an RTT, rmi_free_delegated_page() already calls
free_page() before returning:

drivers/firmware/arm_rmm/rmi.c:rmi_free_delegated_page() {
	if (WARN_ON_ONCE(rmi_undelegate_page(phys))) {
		/* Undelegate failed: leak the page */
		return -EBUSY;
	}

	free_page((unsigned long)phys_to_virt(phys));

	return 0;
}

Because free_page() uncharges the page and clears page->memcg_data (and
returns the page to the allocator where another CPU can immediately
reallocate it), the subsequent kvm_account_pgtable_pages(phys_to_virt(phys),
-1) in free_rtt() operates on a freed page whose memcg association has been
cleared.

[ ... ]

> @@ -161,6 +301,64 @@ static int realm_tear_down_rtt_range(struct realm *realm,
>  					 start, end);
>  }
>  
> +/*
> + * Returns 0 on successful fold, a negative value on error, a positive value if
> + * we were not able to fold all tables at this level.
> + */
> +static int realm_fold_rtt_level(struct realm *realm, int level,
> +				unsigned long start, unsigned long end)
> +{
> +	int not_folded = 0;
> +	ssize_t map_size;
> +	unsigned long addr, next_addr;
> +
> +	if (WARN_ON(level > KVM_PGTABLE_LAST_LEVEL))
> +		return -EINVAL;
> +
> +	map_size = rmi_rtt_level_mapsize(level - 1);
> +
> +	for (addr = start; addr < end; addr = next_addr) {
> +		phys_addr_t rtt_granule;
> +		long ret;
> +		unsigned long align_addr = ALIGN(addr, map_size);
> +
> +		next_addr = ALIGN(addr + 1, map_size);
> +
> +		ret = realm_rtt_fold(realm, align_addr, level, &rtt_granule);
> +		if (ret < 0)
> +			return ret;
> +
> +		switch (RMI_RESULT_STATUS(ret)) {
> +		case RMI_SUCCESS:\
> +			free_rtt(rtt_granule);
> +			break;
> +		case RMI_ERROR_RTT:
> +			if (level == KVM_PGTABLE_LAST_LEVEL ||
> +			    RMI_RESULT_DATA_LEVEL(ret) < level) {
> +				not_folded++;
> +				break;
> +			}
> +			/* Recurse a level deeper */
> +			ret = realm_fold_rtt_level(realm,
> +						   level + 1,
> +						   addr,
> +						   next_addr);
> +			if (ret < 0) {
> +				return ret;
> +			} else if (ret == 0) {
> +				/* Try again at this level */
> +				next_addr = addr;
> +			}
> +			break;
> +		default:
> +			WARN_ON(1);
> +			return -ENXIO;
> +		}
> +	}
> +
> +	return not_folded;
> +}

[Severity: High]
In realm_fold_rtt_level(), align_addr is computed using ALIGN(addr, map_size)
before calling realm_rtt_fold(realm, align_addr, level, &rtt_granule).

When addr is not aligned to map_size, ALIGN(addr, map_size) rounds addr up
to the next RTT boundary (equal to next_addr), defeating the ALIGN_DOWN()
inside realm_rtt_fold(). As a result, realm_rtt_fold() folds the next RTT
instead of the RTT covering addr, and if addr lies in the final map_size
block below BIT(realm->ia_bits), align_addr rounds up to BIT(realm->ia_bits),
which is out of bounds for rmi_rtt_fold() and triggers the default WARN_ON(1)
path.

Additionally, in the RMI_ERROR_RTT case:
1. The recursive call passes next_addr instead of min(next_addr, end),
   scanning up to a full parent map_size beyond end while holding
   kvm->mmu_lock for write.
2. When the recursive call returns a positive value (some child tables were
   not folded), not_folded is not incremented at the current level. If this
   level returns 0 to its caller, the caller sees ret == 0, sets
   next_addr = addr to retry, and can loop indefinitely.
3. Conversely, when recursing to level + 1 across [addr, next_addr), any
   entry at level that is already a leaf entry (such as UNASSIGNED) has no
   table at level + 1, so rmi_rtt_fold() returns RMI_ERROR_RTT with
   RMI_RESULT_DATA_LEVEL(ret) < level + 1 and increments not_folded,
   preventing the parent table at level from being retried and folded even if
   all of its child RTTs were folded. And if ret == 0 is returned when the
   table at level still cannot be folded (because entries in [addr, next_addr)
   are non-homogeneous or align_addr targeted the next block), setting
   next_addr = addr will retry and recurse on the same range in an infinite
   loop.

Should realm_fold_rtt_level() pass addr directly to realm_rtt_fold() (which
already aligns down to map_size), clamp the recursive end to
min(next_addr, end), and avoid retrying in a loop when the parent table
itself cannot be folded?

> @@ -196,6 +394,154 @@ int kvm_realm_teardown_stage2(struct kvm *kvm)

[ ... ]

> +static void realm_unmap_private_range(struct kvm *kvm,
> +				      unsigned long start,
> +				      unsigned long end,
> +				      bool may_block)
> +{
> +	struct realm *realm = &kvm->arch.realm;
> +	unsigned long rd = virt_to_phys(realm->rd);
> +	unsigned long next_addr, addr;
> +	long ret;
> +
> +	for (addr = start; addr < end; addr = next_addr) {
> +		unsigned long out_range;
> +		unsigned long flags = RMI_ADDR_TYPE_SINGLE;
> +		/* TODO: Optimise using RMI_ADDR_TYPE_LIST */
> +
> +retry:
> +		ret = rmi_rtt_data_unmap(rd, addr, end, flags, 0,
> +					 &next_addr, &out_range, NULL);
> +		if (WARN_ON(ret < 0))
> +			return;
> +
> +		if (RMI_RESULT_STATUS(ret) == RMI_ERROR_RTT) {
> +			phys_addr_t rtt;
> +
> +			if (next_addr > addr)
> +				continue; /* UNASSIGNED */
> +
> +			rtt = alloc_rtt(NULL);
> +			if (WARN_ON(rtt == PHYS_ADDR_MAX))
> +				return;
> +			ret = realm_rtt_create(realm, addr,
> +					       RMI_RESULT_DATA_LEVEL(ret) + 1, rtt);
> +			if (WARN_ON(ret)) {
> +				free_rtt(rtt);
> +				return;
> +			}
> +			goto retry;
> +		} else if (WARN_ON(ret)) {
> +			continue;
> +		}

[Severity: High]
What happens in realm_unmap_private_range() when rmi_rtt_data_unmap() returns
RMI_ERROR_RTT or another non-zero status?

Looking at rmi_rtt_data_unmap() in include/linux/arm-rmi-cmds.h:

	ret = rmi_sro_execute(&regs);

	if (ret != RMI_SUCCESS)
		return ret;

	if (out_top)
		*out_top = regs.a1;

*out_top (&next_addr) is only written when ret == RMI_SUCCESS.

When rmi_rtt_data_unmap() returns RMI_ERROR_RTT on the first iteration
(addr == start), next_addr is uninitialized on the stack when evaluated by:

	if (next_addr > addr)
		continue; /* UNASSIGNED */

If the uninitialized stack value happens to be greater than start, the loop
advances addr to that garbage address and skips unmapping the range.

If next_addr <= addr on the first iteration (or on any subsequent iteration
where addr == next_addr from the loop header), next_addr > addr is false even
when the entry is UNASSIGNED. The code then allocates RTTs and calls
realm_rtt_create(realm, addr, RMI_RESULT_DATA_LEVEL(ret) + 1, rtt) in a retry
loop until RMI_RESULT_DATA_LEVEL(ret) reaches KVM_PGTABLE_LAST_LEVEL (3),
where it attempts to create an RTT at level 4, hits WARN_ON(ret), and returns
early without unmapping the remaining range.

Similarly, in the else if (WARN_ON(ret)) continue; branch, next_addr was not
updated by rmi_rtt_data_unmap(), so continuing either jumps to uninitialized
stack memory on the first iteration or repeats the same addr in an infinite
loop while holding kvm->mmu_lock for write.

Should this handle RMI_ERROR_RTT in the same way as
realm_unmap_shared_range(), using find_map_level(realm, addr, end) and
advancing next_addr by rmi_rtt_level_mapsize(err_level) when err_level >=
level?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261102141304.126883-1-steven.price@arm.com?part=8

Re: [PATCH 02/11] KVM: arm64: CCA: Introduce Realms

[Steven Price]

On 02/11/2026 14:30, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
> - [High] KVM: arm64: premature kvm_arch_has_private_mem() breaks gmem checks and caps
> - [Low] KVM: arm64: duplicate kvm_arm_vcpu_finalize() declarations in kvm_host.h
> --
> 
> commit d7e08ee0dbb02a4cf114f13c026edc843145fe6e
> Author: Suzuki K Poulose <suzuki.poulose@arm.com>
> 
> KVM: arm64: CCA: Introduce Realms
> 
> This commit adds foundational infrastructure for Arm CCA Realm VMs,
> including the VM_REALM flavor, a static key for RMI availability,
> per-VM Realm state tracking in struct kvm_arch, and initial feature and
> memslot checks.
> 
>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>> index ccd2dc8f73c3a..7d0222047ca1e 100644
>> --- a/arch/arm64/include/asm/kvm_host.h
>> +++ b/arch/arm64/include/asm/kvm_host.h
> [ ... ]
>> @@ -1572,30 +1584,43 @@ struct kvm *kvm_arch_alloc_vm(void);
> [ ... ]
>> +#define kvm_vm_is_realm(kvm)		((kvm)->arch.vm_flavor == VM_REALM)
>> +#define vcpu_is_rec(vcpu)		kvm_vm_is_realm((vcpu)->kvm)
>>  
>>  #define kvm_vm_hyp_is_distrusting(kvm)	((kvm)->arch.vm_flavor >= __VM_DISTRUSTING_HYP)
>>  
>> -#endif	/* __KVM_NVHE_HYPERVISOR__ */
>> +#define kvm_arch_has_private_mem(kvm) kvm_vm_is_realm(kvm)
> 
> [Severity: High]
> Does defining kvm_arch_has_private_mem(kvm) here cause a regression in
> guest_memfd memslot validation and capability reporting across arm64?
> 
> Because CONFIG_KVM_PRIVATE_MEMATTR_VM is not selected on arm64, defining
> kvm_arch_has_private_mem activates the preprocessor branch in
> include/linux/kvm_host.h that sets gmem_in_place_conversion to true at
> compile time:
> 
> include/linux/kvm_host.h:
>     #elif defined kvm_arch_has_private_mem
>     #define gmem_in_place_conversion true
>     #define kvm_is_private_gfn kvm_gmem_is_private_gfn
> 
> With gmem_in_place_conversion unconditionally true,
> kvm_gmem_prepare_memory_region() in virt/kvm/guest_memfd.c sets
> KVM_MEMSLOT_GMEM_ONLY on every guest_memfd memslot even when
> GUEST_MEMFD_FLAG_MMAP is not set:
> 
> virt/kvm/guest_memfd.c:kvm_gmem_prepare_memory_region() {
>     ...
>     if (gmem_in_place_conversion || kvm_gmem_supports_mmap(inode))
>         slot->flags |= KVM_MEMSLOT_GMEM_ONLY;
> }
> 
> As a result, the check in kvm_arch_prepare_memory_region() in
> arch/arm64/kvm/mmu.c that rejects non-mappable guest_memfd memslots:
> 
> arch/arm64/kvm/mmu.c:kvm_arch_prepare_memory_region() {
>     ...
>     if (kvm_slot_has_gmem(new) && !kvm_memslot_is_gmem_only(new))
>         return -EINVAL;
> }
> 
> is bypassed for non-Realm VMs, allowing userspace to bind a private
> guest_memfd created with flags == 0 via KVM_SET_USER_MEMORY_REGION2 and
> fault those pages into non-Realm stage-2 page tables through gmem_abort().
> 
> In addition, when userspace queries
> KVM_CHECK_EXTENSION(KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES) on the /dev/kvm
> fd (where kvm is NULL), kvm_vm_ioctl_check_extension_generic() in
> virt/kvm/kvm_main.c calls kvm_supports_private_mem(NULL):
> 
> virt/kvm/kvm_main.c:kvm_supports_private_mem() {
>     return !kvm || kvm_arch_has_private_mem(kvm);
> }
> 
> virt/kvm/kvm_main.c:kvm_vm_ioctl_check_extension_generic() {
>     ...
>     case KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES:
>         if (!gmem_in_place_conversion || !kvm_supports_private_mem(kvm))
>             return 0;
> 
>         return KVM_MEMORY_ATTRIBUTE_PRIVATE;
> }
> 
> This advertises KVM_MEMORY_ATTRIBUTE_PRIVATE on all arm64 hosts even when
> KVM_MODE_RMM is not active and while kvm_gmem_set_attributes() in
> virt/kvm/guest_memfd.c rejects KVM_SET_MEMORY_ATTRIBUTES2 with -EINVAL on
> every creatable VM.

Dang, yes that's broken. The later commit "KVM: arm64: CCA: Support
runtime faulting of memory" changes the condition in
kvm_arch_prepare_memory_region() to check "!kvm_vm_is_realm()" but
clearly I've broken the bisect here. That check also isn't quite right.
I think rather than using kvm_memslot_is_gmem_only() we should check the
gmem file with kvm_gmem_supports_mmap(). So we end up with the check:

       if (kvm_slot_has_gmem(new) && !kvm_vm_is_realm(kvm) &&
           !kvm_gmem_slot_supports_mmap(new))

and a new function:

 bool kvm_gmem_slot_supports_mmap(const struct kvm_memory_slot *slot)
 {
        /* The guest_memfd file is pinned while preparing a new memslot. */
        return kvm_gmem_supports_mmap(file_inode(slot->gmem.file));
 }


I'm not sure what we can do about the KVM extension when !kvm. x86
presumably already has the situation where !kvm can expose
KVM_MEMORY_ATTRIBUTE_PRIVATE even if the host doesn't support it. Is
this a problem?

>> +
>> +int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature);
>> +bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
> 
> [Severity: Low]
> This isn't a bug, but commit df6a70516adec8 already moved the declarations
> of kvm_arm_vcpu_finalize() and kvm_arm_vcpu_is_finalized() earlier in
> arch/arm64/include/asm/kvm_host.h right below VCPU_PKVM_FINALIZED:
> 
> arch/arm64/include/asm/kvm_host.h:
>     #define VCPU_PKVM_FINALIZED	__vcpu_single_flag(cflags, BIT(2))
> 
>     int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature);
>     bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
> 
> Can these duplicate declarations be dropped?

Indeed they can.

Steve

Re: [PATCH 09/11] KVM: arm64: CCA: Prevent memslot modifications after Realm creation

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] KVM: arm64: CCA: guest_memfd memslot delete rejected after Stage-2 unmap
--

commit 5106402397c36eec1e5fa59a5bb2c5adf4ee2aa8
Author: Suzuki K Poulose <suzuki.poulose@arm.com>

KVM: arm64: CCA: Prevent memslot modifications after Realm creation

This commit updates kvm_arch_prepare_memory_region() to disallow creating,
modifying, or deleting guest_memfd-backed memslots once a Realm has been
created.

> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index 51fda7b2ff7f3..eb8c5f9e9791d 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -2862,13 +2862,28 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
>  	    (new->flags & (KVM_MEM_LOG_DIRTY_PAGES | KVM_MEM_READONLY)))
>  		return -EPERM;
>  
> -	if (kvm->arch.vm_flavor == VM_PROTECTED_PKVM) {
> +	switch (kvm->arch.vm_flavor) {
> +	case VM_PROTECTED_PKVM:
>  		/* Cannot modify memslots once a pVM has run. */
>  		if (pkvm_hyp_vm_is_created(kvm) &&
> -		    (change == KVM_MR_DELETE || change == KVM_MR_MOVE)) {
> +		    (change == KVM_MR_DELETE || change == KVM_MR_MOVE))
>  			return -EPERM;
> -		}
> +		break;
> +	case VM_REALM:
> +		/*
> +		 * Once the Realm is created, we cannot modify any slots that
> +		 * could be providing private memory. i.e., guest_memfd backed
> +		 * slots.
> +		 * TODO: Handle trusted device private memory slots
> +		 */
> +		if (!kvm_realm_is_created(kvm))
> +			break;
>  
> +		if (kvm_slot_has_gmem(old) || kvm_slot_has_gmem(new))
> +			return -EPERM;

[Severity: High]
Does checking kvm_slot_has_gmem(old) in kvm_arch_prepare_memory_region() run
too late to prevent Stage-2 unmapping when deleting a guest_memfd-backed
memslot on a created Realm?

In virt/kvm/kvm_main.c:kvm_set_memory_region(), non-delete changes
(KVM_MR_MOVE and KVM_MR_FLAGS_ONLY) on KVM_MEM_GUEST_MEMFD memslots are
already rejected upfront with -EINVAL, so kvm_slot_has_gmem(old) is reached
here when change == KVM_MR_DELETE.

However, in virt/kvm/kvm_main.c:kvm_set_memslot(), KVM invalidates and
flushes the old memslot before calling kvm_prepare_memory_region():

virt/kvm/kvm_main.c:kvm_set_memslot() {
	if (change == KVM_MR_DELETE || change == KVM_MR_MOVE) {
		invalid_slot = kzalloc_obj(*invalid_slot, GFP_KERNEL_ACCOUNT);
		if (!invalid_slot) {
			mutex_unlock(&kvm->slots_arch_lock);
			return -ENOMEM;
		}
		kvm_invalidate_memslot(kvm, old, invalid_slot);
	}

	r = kvm_prepare_memory_region(kvm, old, new, change);
	if (r)
		goto err;
}

kvm_invalidate_memslot() marks the active slot as KVM_MEMSLOT_INVALID and
calls arch/arm64/kvm/mmu.c:kvm_arch_flush_shadow_memslot():

arch/arm64/kvm/mmu.c:kvm_arch_flush_shadow_memslot() {
	write_lock(&kvm->mmu_lock);
	kvm_stage2_unmap_range(&kvm->arch.mmu, gpa, size, true);
	kvm_nested_unmap_cipa_range(kvm, gpa, size, true);
	write_unlock(&kvm->mmu_lock);
}

For a Realm VM, kvm_stage2_unmap_range() invokes
arch/arm64/kvm/rmi.c:kvm_realm_unmap_range_filter() with
KVM_FILTER_PRIVATE | KVM_FILTER_SHARED:

arch/arm64/kvm/rmi.c:kvm_realm_unmap_range_filter() {
	if (attr_filter & KVM_FILTER_SHARED)
		realm_unmap_shared_range(kvm, start, end, may_block);
	if (attr_filter & KVM_FILTER_PRIVATE)
		realm_unmap_private_range(kvm, start, end, may_block);
}

By the time kvm_arch_prepare_memory_region() returns -EPERM and
kvm_set_memslot() restores the old memslot in the error path, hasn't
realm_unmap_private_range() already unmapped and undelegated the Realm's
private Stage-2 DATA granules across the memslot's GPA range?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261102141304.126883-1-steven.price@arm.com?part=9

Re: [PATCH 04/11] KVM: arm64: CCA: Check for LPA2 support and SHA-256

[Steven Price]

On 02/11/2026 14:38, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Low] KVM: arm64: rmi_has_feature() passes non-constant mask to u64_get_bits()
> --
> 
> commit a9400749495394b73c9cccd919996b48771556fe
> Author: Steven Price <steven.price@arm.com>
> 
> KVM: arm64: CCA: Check for LPA2 support and SHA-256
> 
> This commit adds feature checks during KVM RMI initialization to verify that
> the RMM supports LPA2 when KVM has LPA2 enabled and that the RMM supports
> SHA-256 measurements.
> 
>> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
>> index 9d118768550d6..5b07beff8bc91 100644
>> --- a/arch/arm64/kvm/rmi.c
>> +++ b/arch/arm64/kvm/rmi.c
>> @@ -6,8 +6,30 @@
>>  #include <linux/arm-rmi-cmds.h>
>>  #include <linux/kvm_host.h>
>>  
>> +#include <asm/kvm_pgtable.h>
>>  #include <asm/virt.h>
>>  
>> +static bool rmi_has_feature(int reg, unsigned long feature)
>> +{
>> +	return !!u64_get_bits(rmi_feat_reg(reg), feature);
>> +}
> 
> [Severity: Low]
> Can this fail to build if the compiler decides not to inline
> rmi_has_feature()?
> 
> In include/linux/bitfield-fix-width.h, u64_get_bits() calls
> field_multiplier(field):
> 
> static __always_inline
> u64 field_multiplier(u64 field)
> {
> 	if ((field | (field - 1)) & ((field | (field - 1)) + 1))
> 		__bad_mask();
> 	return field & -field;
> }
> 
> Because rmi_has_feature() is not marked __always_inline, if the compiler does
> not inline it (for example under CONFIG_CC_OPTIMIZE_FOR_SIZE or compiler
> instrumentation), the feature parameter is not a compile-time constant inside
> field_multiplier(), preventing constant folding from eliminating the call to
> __bad_mask().
> 
> Should rmi_has_feature() be marked __always_inline, or use field_get() or a
> direct bitwise test instead?
> 

I'm not sure how likely this was to fail. But there's an even easier way
of writing this function:

       return !!(rmi_feat_reg(reg) & feature);

So lets go with that. I'm not sure why I'd used get_bits here in the
first place.

Steve

Re: [PATCH 06/11] KVM: arm64: CCA: Add basic infrastructure for creating a realm

[Steven Price]

On 02/11/2026 15:05, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
> - [High] KVM: arm64: missing realm->ia_bits initialization in kvm_init_realm()
> - [Medium] KVM: arm64: cca: clamp Realm IPA limit to host limit in kvm_init_ipa_range()
> --
> 
> commit e03ea9b753f8211e9e93f2e5edff9ba91b09d112
> Author: Steven Price <steven.price@arm.com>
> 
> KVM: arm64: CCA: Add basic infrastructure for creating a realm
> 
> This commit introduces the initial functions and state structures for
> creating and destroying a Realm VM, and checks the requested IPA size
> against the RMM's supported Stage-2 address size.
> 
>> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
>> index 1301c563d7052..b049ea67d9af3 100644
>> --- a/arch/arm64/kvm/mmu.c
>> +++ b/arch/arm64/kvm/mmu.c
>> @@ -981,10 +981,14 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = {
>>  
>>  static int kvm_init_ipa_range(struct kvm_s2_mmu *mmu, unsigned long type)
>>  {
>> +	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
>>  	u32 kvm_ipa_limit = get_kvm_ipa_limit();
>>  	u64 mmfr0, mmfr1;
>>  	u32 phys_shift;
>>  
>> +	if (kvm_vm_is_realm(kvm))
>> +		kvm_ipa_limit = kvm_rmm_ipa_limit();
> 
> [Severity: Medium]
> Should kvm_init_ipa_range() clamp kvm_ipa_limit with min(kvm_ipa_limit,
> kvm_rmm_ipa_limit()) instead of overwriting it?
> 
> In arch/arm64/kvm/rmi.c, rmm_check_features() only rejects the case where
> kvm_lpa2_is_enabled() is true and the RMM lacks RMI_FEATURE_REGISTER_0_LPA2:
> 
> 	if (kvm_lpa2_is_enabled() &&
> 	    !rmi_has_feature(0, RMI_FEATURE_REGISTER_0_LPA2)) {
> 		kvm_err("RMM doesn't support LPA2\n");
> 		return -ENXIO;
> 	}
> 
> It does not prevent running with !kvm_lpa2_is_enabled() when the RMM supports
> LPA2 and reports a 52-bit S2SZ in RMI_FEATURE_REGISTER_0_S2SZ, or when S2SZ
> exceeds the host's sanitised ID_AA64MMFR0_EL1.PARange. Meanwhile,
> kvm_set_ipa_limit() caps get_kvm_ipa_limit() to 48 bits when
> !kvm_lpa2_is_enabled() && PAGE_SIZE != SZ_64K.
> 
> If kvm_rmm_ipa_limit() is larger than get_kvm_ipa_limit(), overwriting
> kvm_ipa_limit allows userspace to create a Realm via
> kvm_arch_init_vm()->kvm_init_stage2_mmu()->kvm_init_ipa_range() with
> phys_shift > get_kvm_ipa_limit(). That causes kvm_get_vtcr() to configure
> VTCR_EL2_T0SZ for 52 bits without VTCR_EL2_DS, kvm_pgtable_stage2_init() to
> attempt allocating 256 concatenated PGD pages, and kvm_handle_guest_abort()
> to inject an address size fault for any fault_ipa >=
> BIT_ULL(get_kvm_ipa_limit()):
> 
> 		/* Beyond sanitised PARange (which is the IPA limit) */
> 		if (fault_ipa >= BIT_ULL(get_kvm_ipa_limit())) {
> 			kvm_inject_size_fault(vcpu);
> 			return 1;
> 		}
> 
> [ ... ]

Valid point - I'm not really convinced the situation will happen in
reality but if for some reason Linux has capped the IPA size it supports
then we don't want to override that just because the RMM supports a
larger size.

>> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
>> index 5b07beff8bc91..45ecbaa467d1d 100644
>> --- a/arch/arm64/kvm/rmi.c
>> +++ b/arch/arm64/kvm/rmi.c
> 
> [ ... ]
> 
>> +int kvm_init_realm(struct kvm *kvm)
>> +{
>> +	struct realm *realm = &kvm->arch.realm;
>> +
>> +	realm->sro = kmalloc_obj(*realm->sro);
>> +	if (!realm->sro)
>> +		return -ENOMEM;
>> +
>> +	return 0;
>> +}
> 
> [Severity: High]
> Where does realm->ia_bits get initialized?
> 
> This patch adds unsigned int ia_bits to struct realm in
> arch/arm64/include/asm/kvm_rmi.h and calls kvm_init_realm() from
> kvm_arch_init_vm() after kvm_init_stage2_mmu() has configured
> kvm->arch.mmu.vtcr, but neither kvm_init_realm() nor kvm_init_ipa_range()
> sets realm->ia_bits (for example to VTCR_EL2_IPA(kvm->arch.mmu.vtcr)).
> 
> Later patches in the series ("KVM: arm64: CCA: Tear down RTTs", "KVM: arm64:
> CCA: Add Realm RTT mapping and invalidation helpers", and "KVM: arm64: CCA:
> Support runtime faulting of memory") read realm->ia_bits in
> get_start_level(), realm_destroy_rtts(), realm_unmap_stage2(),
> realm_unmap_shared_range(), kvm_realm_unmap_range_filter(),
> kvm_realm_is_private_address(), and kvm_gpa_from_fault(). With realm->ia_bits
> left at 0, expressions such as stage2_pgtable_levels(realm->ia_bits),
> (1UL << ia_bits), and BIT(realm->ia_bits - 1) will compute wrong RTT start
> levels, wrong IPA bounds, and undefined shifts.

The assignment of ia_bits is in a later commit (not in this posted
series) as part of realm_create_rd(). But actually it probably makes
more sense done here, so I'll fix that up. Imagine kvm_init_realm() has
the following:

	realm->ia_bits = VTCR_EL2_IPA(kvm->arch.mmu.vtcr);

Thanks,
Steve

Re: [PATCH 10/11] KVM: arm64: CCA: Support runtime faulting of memory

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 11 potential issue(s) to consider:
- [Critical] KVM: arm64: kvm_fault_lock() takes read_lock on mmu_lock for Realm VMs
- [Critical] KVM: arm64: realm_map_non_secure() drops mmu_lock after mmu_invalidate_retry()
- [High] KVM: arm64: unaligned PFN passed to realm_map_ipa() on THP perm fault
- [High] KVM: arm64: missing memcache topup on Realm permission faults
- [High] KVM: arm64: uninitialized out_top and positive error return in realm_map_non_secure()
- [High] KVM: arm64: upward ALIGN() in realm_fold_rtt_level() prevents RTT folding
- [High] KVM: arm64: uninitialized next_addr read in realm_unmap_private_range()
- [High] KVM: arm64: broken pre-fault handling for Realms in gmem_abort() and mmu
- [Medium] KVM: arm64: missing alignment check and bad page count in addr_range_desc()
- [Medium] KVM: arm64: kvm_memslot_is_gmem_only() is always true for gmem slots
- [Low] KVM: arm64: unhandled exec fault on shared IPA in gmem_abort()
--

commit fc65c5faff3855065d9684753b8a03a5284ae490
Author: Steven Price <steven.price@arm.com>

KVM: arm64: CCA: Support runtime faulting of memory

This commit adds runtime Stage-2 fault handling for CCA Realm guests,
delegating and mapping protected pages via the RMM for lower-half IPAs,
mapping non-secure shared pages for upper-half IPAs, and exiting with
KVM_EXIT_MEMORY_FAULT on guest_memfd attribute mismatches.

> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index eb8c5f9e9791d..20aedbe6ec072 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c

[ ... ]

> @@ -1790,6 +1798,10 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd,
>  	void *memcache = NULL;
>  	gfn_t canonical_gfn;
>  	int ret;
> +	bool is_realm_private = false;
> +
> +	if (KVM_BUG_ON(!kvm_slot_has_gmem(s2fd->memslot), kvm))
> +		return -EINVAL;
>  
>  	if (!perm_fault) {
>  		memcache = get_mmu_memcache(s2fd->vcpu);
> @@ -1804,7 +1816,7 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd,
>  	}

[Severity: High]
Should gmem_abort() and user_mem_abort() also top up memcache on permission
faults when kvm_vm_is_realm(kvm) is true?

In gmem_abort(), memcache is left NULL when perm_fault is true, and in
user_mem_abort(), topup_mmu_memcache() is skipped on permission faults
because both memslot_is_logging() and is_protected_kvm_enabled() are false
for a Realm VM:

arch/arm64/kvm/mmu.c:user_mem_abort() {
	memcache = get_mmu_memcache(s2fd->vcpu);
	if (!perm_fault || memslot_is_logging(s2fd->memslot) ||
	    is_protected_kvm_enabled()) {
		ret = topup_mmu_memcache(s2fd->mmu, memcache);
		if (ret)
			return ret;
	}
}

However, unlike normal VMs that relax permissions in place, both paths call
realm_map_ipa(..., memcache) -> realm_map_non_secure(..., memcache). When
relaxing permissions on an existing shared mapping, realm_map_non_secure()
calls realm_unmap_shared_range(), which unmaps the entry and folds empty RTT
levels via realm_fold_rtt_level(), and the subsequent iteration calls
realm_create_rtt_levels(..., memcache) -> alloc_rtt(memcache) ->
alloc_delegated_granule(memcache).

In user_mem_abort(), passing an un-topped-up memcache to
kvm_mmu_memory_cache_alloc() can trigger WARN_ON(!mc->nobjs) (and BUG_ON(!p)
if the atomic fallback fails), while in gmem_abort() passing a NULL memcache
forces alloc_delegated_granule() to fall back to GFP_ATOMIC page allocations
under kvm->mmu_lock.

>  	s2vi.vma_pagesize = PAGE_SIZE;
> -	s2vi.gfn = ALIGN_DOWN(s2fd->fault_ipa, s2vi.vma_pagesize) >> PAGE_SHIFT;
> +	s2vi.gfn = gpa_to_gfn(kvm_gpa_from_fault(kvm, s2fd->fault_ipa));
>  	canonical_gfn = get_canonical_gfn(s2fd, &s2vi);
>  
>  	write_fault = kvm_s2_fault_is_write(s2fd);
> @@ -1816,13 +1828,28 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd,
>  	/* Pairs with the smp_wmb() in kvm_mmu_invalidate_end(). */
>  	smp_rmb();
>  
> +	if (kvm_vm_is_realm(kvm)) {
> +		is_realm_private = kvm_realm_is_private_address(&kvm->arch.realm,
> +								s2fd->fault_ipa);
> +
> +		if (is_realm_private != kvm_gmem_is_private_gfn(kvm, canonical_gfn)) {
> +			kvm_prepare_memory_fault_exit(s2fd->vcpu,
> +						      gfn_to_gpa(canonical_gfn),
> +						      PAGE_SIZE,
> +						      write_fault, exec_fault,
> +						      is_realm_private);
> +			kfree(mapping);
> +			return -EFAULT;
> +		}
> +	}

[Severity: High]
How is KVM_PRE_FAULT_MEMORY supposed to interact with Realm VMs here and in
kvm_arch_pre_fault_allowed()?

In gmem_abort(), the kvm_gmem_get_pfn() error path below guards
kvm_prepare_memory_fault_exit() with if (!result) so that synthetic faults
from __pre_fault_s2() do not overwrite vcpu->run, whereas this attribute
mismatch path calls kvm_prepare_memory_fault_exit() unconditionally.

More broadly, kvm_arch_pre_fault_allowed() only checks
is_protected_kvm_enabled():

arch/arm64/kvm/mmu.c:kvm_arch_pre_fault_allowed() {
	if (is_protected_kvm_enabled())
		return -EOPNOTSUPP;

	return 0;
}

Because is_protected_kvm_enabled() is false in KVM_MODE_RMM, userspace can
invoke KVM_PRE_FAULT_MEMORY on a Realm vCPU:

1. In pre_fault_s2(), kvm_walk_s2(mmu->pgt, gpa, &level) walks the stage-2
   page tables directly from the host CPU, which will trigger a Granule
   Protection Fault once the root PGD pages have been delegated to the RMM.

2. If called before the Realm is created while realm.ia_bits is still 0,
   kvm_gpa_from_fault() and kvm_realm_is_private_address() evaluate
   BIT(realm->ia_bits - 1), causing a negative shift.

3. __pre_fault_s2() dispatches solely on kvm_slot_has_gmem(memslot),
   bypassing the private_ipa_fault() and kvm_fault_is_gmem_abort() checks in
   kvm_vm_mem_abort().

Should kvm_arch_pre_fault_allowed() reject Realm VMs (or support them via
RMM-aware walks), and should the attribute mismatch path in gmem_abort()
also check if (!result)?

> +
>  	ret = kvm_gmem_get_pfn(kvm, s2fd->memslot, canonical_gfn, &s2vi.pfn, NULL);
>  	if (ret) {
>  		/* If result is non-NULL this is a synthetic fault. */
>  		if (!result)
>  			kvm_prepare_memory_fault_exit(s2fd->vcpu, gfn_to_gpa(canonical_gfn),
>  						      s2vi.vma_pagesize, write_fault, exec_fault,
> -						      false);
> +						      is_realm_private);
>  		kfree(mapping);
>  		return ret;
>  	}
> @@ -1845,6 +1872,13 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd,
>  		goto out_unlock;
>  	}
>  
> +	if (kvm_vm_is_realm(kvm)) {
> +		prot &= ~KVM_PGTABLE_PROT_X;
> +		ret = realm_map_ipa(kvm, s2fd->fault_ipa, s2vi.pfn,
> +				    PAGE_SIZE, prot, memcache);
> +		goto out_unlock;
> +	}

[Severity: Critical]
Does kvm_fault_lock() acquire the wrong lock mode on kvm->mmu_lock for Realm
VMs?

Both gmem_abort() and kvm_s2_fault_map() call realm_map_ipa() while holding
kvm_fault_lock(kvm). In arch/arm64/include/asm/kvm_rmi.h, struct realm notes
that Realm stage-2 map/unmap paths require kvm->mmu_lock to be held for
write, and assumes kvm_fault_lock() takes the write lock:

arch/arm64/include/asm/kvm_mmu.h:kvm_fault_lock() {
	if (is_protected_kvm_enabled())
		write_lock(&kvm->mmu_lock);
	else
		read_lock(&kvm->mmu_lock);
}

However, is_protected_kvm_enabled() only checks ARM64_KVM_PROTECTED_MODE
(pKVM) and is false in KVM_MODE_RMM, so kvm_fault_lock() only takes
read_lock(&kvm->mmu_lock) for a Realm VM.

Inside realm_map_ipa(), realm_map_protected() calls
realm_unmap_private_range(kvm, base_ipa, ipa, true) on error and
realm_map_non_secure() calls realm_unmap_shared_range(kvm, ipa, ipa_top,
true) when level >= req_level. Both unmap helpers execute
cond_resched_rwlock_write(&kvm->mmu_lock) when may_block is true, performing
a write_unlock() and write_lock() on a read-locked rwlock before
kvm_fault_unlock() calls read_unlock(). Holding only the read lock also
allows concurrent vCPUs to race on RTT creation/folding, page delegation,
and realm->sro usage.

Should kvm_fault_lock() and kvm_fault_unlock() take kvm->mmu_lock for write
when kvm_vm_is_protected(kvm) (or kvm_vm_is_realm(kvm)) is true?

[Severity: Low]
This question rests on an ARM CCA / RMM architecture detail that I could not
confirm from the kernel source: can a Realm guest instruction fetch from an
unprotected (shared) IPA cause a stage-2 instruction abort exit to the host
that reaches kvm_handle_guest_abort(), or does the RMM always handle or
reject unprotected instruction fetches internally?

If the RMM can exit to the host with a stage-2 instruction abort on a shared
IPA (where kvm_s2_fault_is_exec(s2fd) is true), unprotected mappings only
support RMI_S2AP_DIRECT_READ and RMI_S2AP_DIRECT_WRITE in
realm_map_non_secure(), and gmem_abort() strips KVM_PGTABLE_PROT_X. Neither
gmem_abort() nor user_mem_abort() returns -ENOEXEC to inject an instruction
abort into the guest, so realm_map_non_secure() would install a
non-executable mapping and repeatedly unmap and remap it on subsequent
instruction permission faults.

[ ... ]

> @@ -2241,12 +2276,15 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
>  	if (!perm_fault_granule && !s2vi->map_non_cacheable && kvm_has_mte(kvm))
>  		sanitise_mte_tags(kvm, pfn, mapping_size);
>  
> -	/*
> -	 * Under the premise of getting a FSC_PERM fault, we just need to relax
> -	 * permissions only if mapping_size equals perm_fault_granule. Otherwise,
> -	 * kvm_pgtable_stage2_map() should be called to change block size.
> -	 */
> -	if (mapping_size == perm_fault_granule) {
> +	if (kvm_vm_is_realm(kvm)) {
> +		ret = realm_map_ipa(kvm, s2fd->fault_ipa, pfn, mapping_size,
> +				    prot, memcache);
> +	} else if (mapping_size == perm_fault_granule) {

[Severity: High]
Can kvm_s2_fault_map() pass an unaligned 4KB subpage pfn with a 2MB
mapping_size to realm_map_ipa() on a Transparent Huge Page permission fault?

When a shared IPA is backed by a non-hugetlbfs VMA using a 2MB THP,
kvm_s2_fault_get_vma_info() sets s2vi->vma_pagesize = PAGE_SIZE and resolves
s2vi->pfn to the specific 4KB page for s2fd->fault_ipa. If the 2MB block was
initially mapped read-only and the guest later writes to a non-zero 4KB page
offset within that block, perm_fault_granule is PMD_SIZE.

Earlier in kvm_s2_fault_map():

arch/arm64/kvm/mmu.c:kvm_s2_fault_map() {
	if (mapping_size == PAGE_SIZE && s2vi->max_map_size != PAGE_SIZE) {
		if (perm_fault_granule > PAGE_SIZE) {
			mapping_size = perm_fault_granule;
		} else {
			mapping_size = transparent_hugepage_adjust(s2fd, &pfn, &gfn);
...
		}
		canonical_ipa = ALIGN_DOWN(canonical_ipa, mapping_size);
	}
}

When perm_fault_granule > PAGE_SIZE, mapping_size is set to
perm_fault_granule without calling transparent_hugepage_adjust() to align
pfn down to the block boundary (or verify that the host stage-1 mapping is
still a huge page after FOLL_WRITE). Non-Realm VMs take the
kvm_pgtable_stage2_relax_perms() branch which does not use pfn, but Realm
VMs call realm_map_ipa(), which aligns ipa down to map_size while leaving
pfn at the unaligned 4KB subpage physical address.

[ ... ]

> @@ -2339,18 +2377,43 @@ static int user_mem_abort(const struct kvm_s2_fault_desc *s2fd,
>  	return kvm_s2_fault_map(s2fd, &s2vi, prot, memcache, result);
>  }
>  
> +static bool private_ipa_fault(struct kvm *kvm, phys_addr_t fault_ipa);
> +static bool kvm_fault_is_gmem_abort(struct kvm *kvm,
> +				    const struct kvm_s2_fault_desc *s2fd);
> +
>  static int kvm_vm_mem_abort(const struct kvm_s2_fault_desc *s2fd)
>  {
>  	VM_WARN_ON_ONCE(kvm_s2_fault_is_perm(s2fd) &&
>  			!esr_abt_is_write_fault(s2fd->esr) &&
>  			!kvm_s2_fault_is_exec(s2fd));
>  
> -	if (kvm_slot_has_gmem(s2fd->memslot))
> +	if (private_ipa_fault(s2fd->vcpu->kvm, s2fd->fault_ipa) &&
> +	    !kvm_slot_has_gmem(s2fd->memslot))
> +		return -EINVAL;
> +
> +	if (kvm_fault_is_gmem_abort(s2fd->vcpu->kvm, s2fd))
>  		return gmem_abort(s2fd, NULL);
>  	else
>  		return user_mem_abort(s2fd, NULL);
>  }
>  
> +/* Decide if an abort should be handled by the gmem */
> +static bool kvm_fault_is_gmem_abort(struct kvm *kvm,
> +				    const struct kvm_s2_fault_desc *s2fd)
> +{
> +	if (!kvm_slot_has_gmem(s2fd->memslot))
> +		return false;
> +	if (kvm_memslot_is_gmem_only(s2fd->memslot))
> +		return true;
> +	/*
> +	 * For Realms, all private faults must be backed by GMEM.
> +	 * TODO: Handle Trusted device private memory mappings.
> +	 */
> +	if (private_ipa_fault(kvm, s2fd->fault_ipa))
> +		return true;
> +	return false;
> +}

[Severity: Medium]
Isn't kvm_memslot_is_gmem_only() unconditionally true for every guest_memfd
memslot on arm64 once kvm_arch_has_private_mem() is defined?

In include/linux/kvm_host.h, defining kvm_arch_has_private_mem without
CONFIG_KVM_PRIVATE_MEMATTR_VM sets gmem_in_place_conversion to true, which
causes kvm_gmem_prepare_memory_region() to set KVM_MEMSLOT_GMEM_ONLY on
every guest_memfd memslot regardless of whether GUEST_MEMFD_FLAG_MMAP was
requested:

virt/kvm/guest_memfd.c:kvm_gmem_prepare_memory_region() {
	if (gmem_in_place_conversion || kvm_gmem_supports_mmap(inode))
		slot->flags |= KVM_MEMSLOT_GMEM_ONLY;
}

Because kvm_memslot_is_gmem_only() is always true whenever
kvm_slot_has_gmem() is true:

1. In kvm_fault_is_gmem_abort(), the private_ipa_fault() check and final
   return false become unreachable, and shared IPA faults on non-MMAP
   guest_memfd memslots are routed to gmem_abort() instead of
   user_mem_abort().

2. In kvm_arch_prepare_memory_region() below, the check
   if (kvm_slot_has_gmem(new) && !kvm_memslot_is_gmem_only(new) &&
   !kvm_vm_is_realm(kvm)) never triggers, so non-Realm VMs are no longer
   prevented from binding guest_memfd instances created without
   GUEST_MEMFD_FLAG_MMAP.

> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
> index 1066f767f85d9..f89460d8adcba 100644
> --- a/arch/arm64/kvm/rmi.c
> +++ b/arch/arm64/kvm/rmi.c
> @@ -531,15 +531,231 @@ void kvm_realm_unmap_range_filter(struct kvm *kvm, unsigned long start,
>  }
>  
>  void kvm_realm_unmap_range(struct kvm *kvm, unsigned long start,
> -			   unsigned long size, bool unmap_private,
> -			   bool may_block)
> +			   unsigned long size, bool may_block)
>  {
> -	enum kvm_gfn_range_filter attr_filter = KVM_FILTER_SHARED;
> +	kvm_realm_unmap_range_filter(kvm, start, size, may_block,
> +				     KVM_FILTER_SHARED | KVM_FILTER_PRIVATE);
> +}
> +
> +static unsigned long addr_range_desc(unsigned long phys, unsigned long size,
> +				     unsigned long state)
> +{
> +	unsigned long count = 1;
> +	unsigned long out;
> +	int rmi_size;
> +
> +	for (rmi_size = KVM_PGTABLE_LAST_LEVEL; rmi_size >= 0; rmi_size--) {
> +		unsigned long entry_size = rmi_range_entry_size(rmi_size);
> +
> +		if (size == entry_size)
> +			break;
> +		if (size > entry_size) {
> +			rmi_size = -1;
> +			break;
> +		}
> +	}
> +
> +	if (rmi_size < 0) {
> +		/*
> +		 * Only support mapping at the page level granularity when
> +		 * it's an unusual length. This should get us back onto a larger
> +		 * block size for the subsequent mappings.
> +		 */
> +		rmi_size = 0;
> +		count = MIN(size >> PAGE_SHIFT, PTRS_PER_PTE - 1);
> +	}

[Severity: Medium]
Should addr_range_desc() also verify that phys is aligned to entry_size
before selecting a block rmi_size?

When rmi_size < 0 (for example, after a partial delegation in
realm_map_protected()), mapping PTRS_PER_PTE - 1 (511) pages from a
block-aligned address advances ipa and phys to one page short of the next
2MB block boundary. If the remaining size (ipa_top - ipa) on a subsequent
iteration happens to equal a block entry_size, the loop will select that
block rmi_size even though ipa and phys are only page-aligned, causing the
RMM map call to fail with RMI_ERROR_INPUT.

[ ... ]

> +err_undelegate:
> +	realm_unmap_private_range(kvm, base_ipa, ipa, true);
> +	/* A range which cannot be undelegated remains owned by the RMM. */
> +	WARN_ON(rmi_undelegate_range(base_phys, map_size));
> +	return ret < 0 ? ret : -ENXIO;
> +}

[Severity: High]
When realm_map_protected() (or kvm_realm_unmap_range()) calls
realm_unmap_private_range(), can realm_unmap_private_range() read an
uninitialized or stale next_addr when rmi_rtt_data_unmap() returns
RMI_ERROR_RTT?

In realm_unmap_private_range():

arch/arm64/kvm/rmi.c:realm_unmap_private_range() {
	unsigned long next_addr, addr;
	long ret;

	for (addr = start; addr < end; addr = next_addr) {
...
retry:
		ret = rmi_rtt_data_unmap(rd, addr, end, flags, 0,
					 &next_addr, &out_range, NULL);
		if (WARN_ON(ret < 0))
			return;

		if (RMI_RESULT_STATUS(ret) == RMI_ERROR_RTT) {
			phys_addr_t rtt;

			if (next_addr > addr)
				continue; /* UNASSIGNED */
}

In include/linux/arm-rmi-cmds.h, rmi_rtt_data_unmap() returns early when
ret != RMI_SUCCESS without writing *out_top (&next_addr). On the first loop
iteration, next_addr is uninitialized on the stack; on later iterations
after a successful unmap, next_addr equals addr, so next_addr > addr is
false even for an unassigned entry at the leaf level, causing
realm_unmap_private_range() to attempt to create a level-4 RTT and trigger
WARN_ON(ret).

[ ... ]

> +	while (ipa < ipa_top) {
> +		unsigned long range_desc = addr_range_desc(phys, ipa_top - ipa,
> +							   RMI_OP_MEM_UNDELEGATED);
> +		unsigned long out_top;
> +
> +		ret = rmi_rtt_unprot_map(rd, ipa, ipa_top, flags, range_desc,
> +					 &out_top);
> +		if (ret < 0)
> +			return ret;
> +
> +		if (RMI_RESULT_STATUS(ret) == RMI_ERROR_RTT) {
> +			/* Create missing RTTs and retry */
> +			int level = RMI_RESULT_DATA_LEVEL(ret);
> +			int req_level = find_map_level(realm, ipa, ipa_top);
> +
> +			/*
> +			 * There already exists a mapping at the level. May be
> +			 * we are relaxing a permission for the given range ?
> +			 */
> +			if (level >= req_level) {
> +				realm_unmap_shared_range(kvm, ipa, ipa_top, true);
> +				continue;
> +			}

[Severity: Critical]
Is it safe to pass may_block = true to realm_unmap_shared_range() here while
mapping a PFN resolved before kvm_fault_lock()?

In gmem_abort() and kvm_s2_fault_map(), mmu_invalidate_retry() is checked
once after acquiring kvm_fault_lock(kvm). Passing may_block = true to
realm_unmap_shared_range() calls cond_resched_rwlock_write(&kvm->mmu_lock),
which can drop and re-acquire kvm->mmu_lock. If a concurrent MMU notifier or
guest_memfd invalidation runs while the lock is dropped and frees or
converts the backing page, continuing the loop will map the stale phys into
the Realm without re-checking mmu_invalidate_retry().

[Severity: High]
Can this level >= req_level path also get stuck in an infinite loop when
replacing 4KB page mappings with a 2MB block mapping at an IPA that is not
aligned to the top-level RTT map_size?

realm_unmap_shared_range() calls realm_fold_rtt_level(), which computes
align_addr using ALIGN() instead of ALIGN_DOWN():

arch/arm64/kvm/rmi.c:realm_fold_rtt_level() {
	for (addr = start; addr < end; addr = next_addr) {
		phys_addr_t rtt_granule;
		long ret;
		unsigned long align_addr = ALIGN(addr, map_size);

		next_addr = ALIGN(addr + 1, map_size);

		ret = realm_rtt_fold(realm, align_addr, level, &rtt_granule);
}

When start is not aligned to map_size at the starting level (for example, a
2MB range at a non-1GB-aligned IPA), ALIGN(addr, map_size) rounds align_addr
up to the next table boundary (or to BIT(realm->ia_bits) if addr is in the
last top-level entry, triggering WARN_ON(1) on RMI_ERROR_INPUT). As a
result, realm_fold_rtt_level() does not fold the emptied child RTT covering
ipa, and the next iteration of realm_map_non_secure() sees RMI_ERROR_RTT
with level >= req_level again and loops indefinitely.

> +
> +			ret = realm_create_rtt_levels(realm, ipa, level,
> +						      req_level,
> +						      memcache);
> +			if (ret)
> +				return ret;
> +
> +			ret = rmi_rtt_unprot_map(rd, ipa, ipa_top, flags,
> +						 range_desc, &out_top);
> +			if (ret < 0)
> +				return ret;
> +		}
> +
> +		if (ret && RMI_RESULT_STATUS(ret) != RMI_ERROR_RTT)
> +			return ret;
> +
> +		phys += out_top - ipa;
> +		ipa = out_top;
> +	}

[Severity: High]
What happens in realm_map_non_secure() if the second rmi_rtt_unprot_map()
call returns RMI_ERROR_RTT, or if rmi_rtt_unprot_map() returns a positive
RMI error code?

1. out_top is declared uninitialized at the top of the loop, and
   rmi_rtt_unprot_map() only writes *out_top when ret == RMI_SUCCESS. If the
   second rmi_rtt_unprot_map() call after realm_create_rtt_levels() also
   returns RMI_ERROR_RTT (for example, if addr_range_desc() fell back to
   page granularity while req_level < KVM_PGTABLE_LAST_LEVEL, or if a deeper
   RTT exists in part of the range), the check
   if (ret && RMI_RESULT_STATUS(ret) != RMI_ERROR_RTT) is false and
   execution falls through to phys += out_top - ipa; ipa = out_top; using an
   uninitialized out_top. Why not use continue after
   realm_create_rtt_levels() like realm_map_protected() does?

2. When rmi_rtt_unprot_map() returns a non-zero positive RMI status code
   (such as RMI_ERROR_INPUT), returning ret directly propagates a positive
   integer back through realm_map_ipa() -> kvm_s2_fault_map() / gmem_abort()
   -> kvm_vm_mem_abort() -> kvm_handle_guest_abort(), where a positive
   return value is treated as a successfully handled fault and re-enters the
   guest. Should this return -ENXIO on positive RMI errors like
   realm_map_protected() does?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261102141304.126883-1-steven.price@arm.com?part=10

Re: [PATCH 07/11] KVM: arm64: CCA: Tear down RTTs

[Steven Price]

On 02/11/2026 15:18, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
> - [High] KVM: arm64: stage-2 PGD leak for uncreated Realms in kvm_destroy_realm()
> - [High] KVM: arm64: unclamped stage-2 levels in get_start_level() breaks 64K RTT teardown
> - [High] KVM: arm64: free_rtt() accounts page table pages after freeing the page
> - [Medium] KVM: arm64: kvm_realm_uninit_stage2() ignores REALM_STATE_NEW
> --
> 
> commit 5954ac2ccf8825a659e2f39dc0db05d96c40be84
> Author: Steven Price <steven.price@arm.com>
> 
> KVM: arm64: CCA: Tear down RTTs
> 
> This commit adds stage-2 page table and Realm Translation Table (RTT)
> teardown for arm64 CCA realms. It unmaps the Realm IPA space during stage-2
> teardown and walks non-root RTTs during final Realm destruction to destroy
> and undelegate them before destroying the Realm Descriptor.
> 
>> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
>> index b049ea67d9af3..51fda7b2ff7f3 100644
>> --- a/arch/arm64/kvm/mmu.c
>> +++ b/arch/arm64/kvm/mmu.c
>> @@ -1145,9 +1145,24 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
>>  	return err;
>>  }
>>  
>> +static void kvm_realm_uninit_stage2(struct kvm_s2_mmu *mmu)
>> +{
>> +	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
>> +
>> +	mutex_lock(&kvm->arch.config_lock);
>> +	if (kvm_realm_state(kvm) == REALM_STATE_ACTIVE) {
>> +		kvm_set_realm_state(kvm, REALM_STATE_DYING);
>> +		WARN_ON(kvm_realm_teardown_stage2(kvm));
>> +	}
>> +	mutex_unlock(&kvm->arch.config_lock);
>> +}
> 
> [Severity: Medium]
> What happens if kvm_uninit_stage2_mmu() is called via
> kvm_mmu_notifier_release() -> kvm_flush_shadow_all() ->
> kvm_arch_flush_shadow_all() while the Realm is in REALM_STATE_NEW?
> 
> According to enum realm_state in arch/arm64/include/asm/kvm_rmi.h, the Realm
> Descriptor has already been created and stage-2 memory may already be
> populated in REALM_STATE_NEW. Unlike kvm_destroy_realm(), which checks
> kvm_realm_is_created(kvm), sets REALM_STATE_DYING, and unmaps stage-2 for
> both REALM_STATE_NEW and REALM_STATE_ACTIVE, kvm_realm_uninit_stage2() only
> checks for REALM_STATE_ACTIVE.
> 
> Should kvm_realm_uninit_stage2() also handle REALM_STATE_NEW (for example by
> checking kvm_realm_is_created(kvm)) so that stage-2 mappings are unmapped
> before exit_mmap() finishes and the Realm transitions to REALM_STATE_DYING?

Yes we should handle REALM_STATE_NEW. I'm not convinced about using
kvm_realm_is_created(). That would include REALM_STATE_DYING.

>>  void kvm_uninit_stage2_mmu(struct kvm *kvm)
>>  {
>> -	kvm_free_stage2_pgd(&kvm->arch.mmu);
>> +	if (kvm_vm_is_realm(kvm))
>> +		kvm_realm_uninit_stage2(&kvm->arch.mmu);
>> +	else
>> +		kvm_free_stage2_pgd(&kvm->arch.mmu);
>>  	kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
>>  }
> 
> [Severity: High]
> Does skipping kvm_free_stage2_pgd() here leak mmu->pgt, the root stage-2 PGD
> pages (mmu->pgt->pgd), and the per-CPU mmu->last_vcpu_ran allocation when a
> Realm VM is destroyed before the Realm is created in the RMM?
> 
> During VM creation, kvm_arch_init_vm() calls kvm_init_stage2_mmu() while the
> Realm state is still REALM_STATE_NONE. If userspace closes the VM file
> descriptor without creating the Realm in the RMM, kvm_destroy_realm() takes
> its early return before reaching kvm_free_stage2_pgd(&kvm->arch.mmu):
> 
> arch/arm64/kvm/rmi.c:kvm_destroy_realm() {
> 	...
> 	if (!kvm_realm_is_created(kvm)) {
> 		kfree(realm->sro);
> 		realm->sro = NULL;
> 		return;
> 	}
> 	...
> 	kvm_free_stage2_pgd(&kvm->arch.mmu);
> }
> 
> Similarly, if kvm_init_nested() or kvm_init_realm() fails in
> kvm_arch_init_vm() after kvm_init_stage2_mmu() succeeds:
> 
> arch/arm64/kvm/arm.c:kvm_arch_init_vm() {
> 	...
> 	ret = kvm_init_nested(kvm);
> 	if (ret)
> 		goto err_uninit_mmu;
> 
> 	if (kvm_vm_is_realm(kvm)) {
> 		ret = kvm_init_realm(kvm);
> 		if (ret)
> 			goto err_uninit_mmu;
> 	}
> 
> 	return 0;
> 
> err_uninit_mmu:
> 	kvm_uninit_stage2_mmu(kvm);
> 	return ret;
> }
> 
> the error path only calls kvm_uninit_stage2_mmu(kvm) and never calls
> kvm_destroy_realm(), leaking the stage-2 PGD allocations there as well.

Here I think using kvm_realm_is_created() make sense. Once the realm is
up we want to "skip" kvm_free_stage2_pgd() because we need the RMM to
tear things down first.

>> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
>> index 45ecbaa467d1d..f190143b33ba5 100644
>> --- a/arch/arm64/kvm/rmi.c
>> +++ b/arch/arm64/kvm/rmi.c
> 
> [ ... ]
> 
>> @@ -21,11 +29,180 @@ u32 kvm_rmm_ipa_limit(void)
>>  	return u64_get_bits(rmi_feat_reg(0), RMI_FEATURE_REGISTER_0_S2SZ);
>>  }
>>  
>> +static int get_start_level(struct realm *realm)
>> +{
>> +	return 4 - stage2_pgtable_levels(realm->ia_bits);
>> +}
> 
> [Severity: High]
> Can get_start_level() return a level that mismatches the stage-2 page table
> configuration on 64KB page kernels and cause realm_destroy_rtts() to fail?
> 
> When kvm_init_ipa_range() configures mmu->vtcr via kvm_get_vtcr(), the
> number of stage-2 levels is clamped to at least 2:
> 
> arch/arm64/kvm/hyp/pgtable.c:kvm_get_vtcr() {
> 	...
> 	lvls = stage2_pgtable_levels(phys_shift);
> 	if (lvls < 2)
> 		lvls = 2;
> 	...
> }
> 
> Under CONFIG_ARM64_64K_PAGES with a 32-bit or 33-bit IPA,
> stage2_pgtable_levels(realm->ia_bits) evaluates to 1, so get_start_level()
> returns 3 (KVM_PGTABLE_LAST_LEVEL) instead of 2.
> 
> When kvm_destroy_realm() -> realm_destroy_rtts() ->
> realm_tear_down_rtt_range() then calls:
> 
> 	realm_tear_down_rtt_level(realm, get_start_level(realm) + 1,
> 				  start, end);
> 
> it passes level 4 into realm_tear_down_rtt_level(), hitting
> WARN_ON(level > KVM_PGTABLE_LAST_LEVEL) and returning -EINVAL. That in turn
> causes kvm_destroy_realm() to warn and return early, leaking the delegated
> RD, RTT granules, root stage-2 PGD pages, and realm->sro.

Good spot - yes get_start_level() should do the same clamping to a min of 2.

>> +
>> +static void free_rtt(phys_addr_t phys)
>> +{
>> +	if (rmi_free_delegated_page(phys))
>> +		return;
>> +
>> +	kvm_account_pgtable_pages(phys_to_virt(phys), -1);
>> +}
> 
> [Severity: High]
> Does calling kvm_account_pgtable_pages() after rmi_free_delegated_page()
> access the page after it has already been freed and uncharged from its
> memcg?
> 
> In drivers/firmware/arm_rmm/rmi.c, rmi_free_delegated_page() undelegates the
> granule and immediately frees the page back to the page allocator:
> 
> drivers/firmware/arm_rmm/rmi.c:rmi_free_delegated_page() {
> 	if (WARN_ON_ONCE(rmi_undelegate_page(phys))) {
> 		/* Undelegate failed: leak the page */
> 		return -EBUSY;
> 	}
> 
> 	free_page((unsigned long)phys_to_virt(phys));
> 
> 	return 0;
> }
> 
> Because free_page() clears page->memcg_data in __free_pages_prepare() and
> releases the page for reallocation, the subsequent call to
> kvm_account_pgtable_pages(phys_to_virt(phys), -1) ->
> mod_lruvec_page_state() operates on a freed struct page and sees a NULL
> folio_memcg(), failing to decrement the memory cgroup's
> NR_SECONDARY_PAGETABLE counter (or racing with a concurrent reallocation of
> that page).
> 
> Should kvm_account_pgtable_pages(phys_to_virt(phys), -1) be called before
> freeing the page, matching kvm_s2_free_pages_exact() and kvm_s2_put_page()
> in arch/arm64/kvm/mmu.c?
> 

Yes. The issue here is that rmi_free_delegated_page() could fail and
then the accounting is wrong. I guess we can decrement the account,
attempt the free and increment the account again if it fails.

        kvm_account_pgtable_pages(phys_to_virt(phys), -1);
        if (rmi_free_delegated_page(phys))
                kvm_account_pgtable_pages(phys_to_virt(phys), 1);

Thanks,
Steve

Re: [PATCH 11/11] KVM: arm64: CCA: Add Realm stage-2 MMU operations

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 7 potential issue(s) to consider:
- [Critical] KVM: arm64: read_lock() used instead of write_lock() in kvm_fault_lock()
- [Critical] KVM: arm64: unaligned PFN and empty memcache on Realm permission faults
- [High] KVM: arm64: uninitialized next_addr in realm_unmap_private_range()
- [High] KVM: arm64: page accounting after page free in free_rtt()
- [High] KVM: arm64: stage-2 PGD leak in kvm_uninit_stage2_mmu() for Realms
- [High] KVM: arm64: leaked delegated pages on error in realm_map_protected()
- [High] KVM: arm64: uninitialized out_top and positive error in realm_map_non_secure()
--

commit c7041c45fed830c50c8f37ff21d6036da29c1302
Author: Suzuki K Poulose <suzuki.poulose@arm.com>

KVM: arm64: CCA: Add Realm stage-2 MMU operations

This commit defines and registers realm_vm_s2_ops for VM_REALM, routing
stage-2 unmapping through kvm_realm_unmap_range_filter() and stage-2 memory
aborts through kvm_vm_mem_abort(). It also provides no-op TLB flush and GFN
aging callbacks since the RMM manages stage-2 TLB invalidation and does not
support stage-2 access flags.

> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index 20aedbe6ec072..bb33c07569e3b 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c

[ ... ]

> @@ -353,6 +366,15 @@ static void no_stage2_unmap_range(struct kvm_s2_mmu *mmu,
>  {
>  }
>  
> +static void realm_stage2_unmap_range(struct kvm_s2_mmu *mmu,
> +				     phys_addr_t start, u64 size, bool may_block,
> +				     enum kvm_gfn_range_filter filter)
> +{
> +	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
> +
> +	kvm_realm_unmap_range_filter(kvm, start, size, may_block, filter);
> +}

[Severity: High]
When realm_stage2_unmap_range() calls
kvm_realm_unmap_range_filter()->realm_unmap_private_range(), can next_addr be
read uninitialized or left stale when rmi_rtt_data_unmap() returns a non-zero
status?

In arch/arm64/kvm/rmi.c, realm_unmap_private_range() loops with next_addr:

	unsigned long next_addr, addr;
	long ret;

	for (addr = start; addr < end; addr = next_addr) {
		...
retry:
		ret = rmi_rtt_data_unmap(rd, addr, end, flags, 0,
					 &next_addr, &out_range, NULL);
		if (WARN_ON(ret < 0))
			return;

		if (RMI_RESULT_STATUS(ret) == RMI_ERROR_RTT) {
			phys_addr_t rtt;

			if (next_addr > addr)
				continue; /* UNASSIGNED */
			...
		} else if (WARN_ON(ret)) {
			continue;
		}

However, in include/linux/arm-rmi-cmds.h, rmi_rtt_data_unmap() returns early
before writing *out_top when ret != RMI_SUCCESS:

	ret = rmi_sro_execute(&regs);

	if (ret != RMI_SUCCESS)
		return ret;

	if (out_top)
		*out_top = regs.a1;

If rmi_rtt_data_unmap() returns RMI_ERROR_RTT on an unpopulated private IPA
range, next_addr is either uninitialized on the first iteration or equal to
addr from the previous iteration. As a result, next_addr > addr is not
reliably true for unassigned entries, causing realm_unmap_private_range() to
allocate and create RTT levels down to KVM_PGTABLE_LAST_LEVEL + 1 where
realm_rtt_create() triggers WARN_ON(ret) and aborts the unmap early (or jumps
to a garbage address if stack contents happen to be greater than addr).

[Severity: High]
In the same unmap path, when realm_unmap_shared_range() or
realm_unmap_private_range() folds RTT levels via
realm_fold_rtt_level()->free_rtt(), does free_rtt() access the page after it
has already been freed?

In arch/arm64/kvm/rmi.c:

static void free_rtt(phys_addr_t phys)
{
	if (rmi_free_delegated_page(phys))
		return;

	kvm_account_pgtable_pages(phys_to_virt(phys), -1);
}

And in drivers/firmware/arm_rmm/rmi.c:

int rmi_free_delegated_page(phys_addr_t phys)
{
	if (WARN_ON_ONCE(rmi_undelegate_page(phys))) {
		/* Undelegate failed: leak the page */
		return -EBUSY;
	}

	free_page((unsigned long)phys_to_virt(phys));

	return 0;
}

Because rmi_free_delegated_page() calls free_page() first (which uncharges the
folio and clears its memcg association before returning the page to the
allocator), calling kvm_account_pgtable_pages() afterwards touches a freed
page and fails to decrement NR_SECONDARY_PAGETABLE on the original memcg.
Should kvm_account_pgtable_pages(phys_to_virt(phys), -1) be called before
free_page() (or before rmi_free_delegated_page())?

[ ... ]

> @@ -3291,6 +3313,15 @@ static const struct kvm_vm_s2_ops kvm_default_vm_s2_ops = {
>  	.vm_mem_abort			= kvm_vm_mem_abort,
>  };
>  
> +static const struct kvm_vm_s2_ops realm_vm_s2_ops = {
> +	.vm_flush_remote_tlbs		= realm_flush_remote_tlbs,
> +	.vm_flush_remote_tlbs_range	= realm_flush_remote_tlbs_range,
> +	.vm_age_gfn			= no_age_gfn,
> +	.vm_test_age_gfn		= no_age_gfn,
> +	.vm_stage2_unmap_range		= realm_stage2_unmap_range,
> +	.vm_mem_abort			= kvm_vm_mem_abort,

[Severity: Critical]
When kvm_vm_mem_abort() handles stage-2 faults for a Realm VM via
gmem_abort() or user_mem_abort()->kvm_s2_fault_map(), it acquires
kvm->mmu_lock using kvm_fault_lock(kvm). Does kvm_fault_lock() take a read
lock instead of a write lock for Realm VMs?

In arch/arm64/include/asm/kvm_mmu.h:

static inline void kvm_fault_lock(struct kvm *kvm)
{
	if (is_protected_kvm_enabled())
		write_lock(&kvm->mmu_lock);
	else
		read_lock(&kvm->mmu_lock);
}

is_protected_kvm_enabled() checks ARM64_KVM_PROTECTED_MODE, which is false in
KVM_MODE_RMM. However, struct realm in arch/arm64/include/asm/kvm_rmi.h notes
that RTT map/unmap SRO commands require kvm->mmu_lock to be held for write
during fault handling:

	 * Reused by RTT map/unmap SRO commands. Those commands are only
	 * issued from Realm stage-2 map/unmap paths while kvm->mmu_lock is
	 * held for write, including Realm fault handling where
	 * kvm_fault_lock() takes the write side, so concurrent use is
	 * serialized.

Furthermore, during fault handling, realm_map_non_secure() calls
realm_unmap_shared_range(kvm, ipa, ipa_top, true) and realm_map_protected()
calls realm_unmap_private_range(kvm, base_ipa, ipa, true) on error. Both pass
may_block = true and execute cond_resched_rwlock_write(&kvm->mmu_lock) while
only a read lock is held, which triggers lockdep_assert_held_write() and calls
write_unlock() followed by write_lock() on a read-locked rwlock, corrupting
the lock state and self-deadlocking on the caller's own reader count.

In addition, calling realm_unmap_shared_range() with may_block = true inside
realm_map_non_secure() can drop kvm->mmu_lock after mmu_invalidate_retry() has
already been checked in gmem_abort() or kvm_s2_fault_map(), allowing a
concurrent mmu_notifier invalidation to unmap and free the page before
rmi_rtt_unprot_map() installs the mapping. Should kvm_fault_lock() and
kvm_fault_unlock() also take the write lock when kvm_vm_is_realm(kvm) is true,
and should realm_map_non_secure() pass may_block = false to
realm_unmap_shared_range()?

[Severity: Critical]
In kvm_vm_mem_abort()->user_mem_abort()->kvm_s2_fault_map(), how are stage-2
permission faults on shared IPAs handled for Realm VMs?

First, in user_mem_abort():

	memcache = get_mmu_memcache(s2fd->vcpu);
	if (!perm_fault || memslot_is_logging(s2fd->memslot) ||
	    is_protected_kvm_enabled()) {
		ret = topup_mmu_memcache(s2fd->mmu, memcache);
		if (ret)
			return ret;
	}

Because is_protected_kvm_enabled() is false for Realm VMs, memcache is not
topped up on permission faults, even though realm_map_non_secure() unmaps and
folds the existing RTT entry via realm_unmap_shared_range() and then allocates
RTTs from memcache via realm_create_rtt_levels().

Second, in kvm_s2_fault_map():

	if (mapping_size == PAGE_SIZE && s2vi->max_map_size != PAGE_SIZE) {
		if (perm_fault_granule > PAGE_SIZE) {
			mapping_size = perm_fault_granule;
		} else {
			mapping_size = transparent_hugepage_adjust(s2fd, &pfn, &gfn);
			if (mapping_size < 0) {
				ret = mapping_size;
				goto out_unlock;
			}
		}
		canonical_ipa = ALIGN_DOWN(canonical_ipa, mapping_size);
	}

	if (!perm_fault_granule && !s2vi->map_non_cacheable && kvm_has_mte(kvm))
		sanitise_mte_tags(kvm, pfn, mapping_size);

	if (kvm_vm_is_realm(kvm)) {
		ret = realm_map_ipa(kvm, s2fd->fault_ipa, pfn, mapping_size,
				    prot, memcache);

When perm_fault_granule > PAGE_SIZE on a THP VMA (where s2vi->vma_pagesize is
PAGE_SIZE), kvm_s2_fault_map() sets mapping_size = perm_fault_granule without
calling transparent_hugepage_adjust() to verify that the host VMA is still
backed by a huge page (for example, if a write fault split the THP via CoW) or
to align pfn down to mapping_size. For normal VMs this is safe because
kvm_pgtable_stage2_relax_perms() does not use pfn, but for a Realm VM
realm_map_ipa() aligns fault_ipa down to mapping_size while mapping from the
unaligned 4KB pfn for mapping_size bytes. Can this map adjacent host physical
pages beyond the end of the folio (or from a split THP) into the Realm guest?

[Severity: High]
When kvm_vm_mem_abort() calls realm_map_ipa()->realm_map_protected(), can the
err_undelegate path fail to undelegate the remaining unmapped pages if an
error occurs after part of the range was already mapped?

In arch/arm64/kvm/rmi.c:

err_undelegate:
	realm_unmap_private_range(kvm, base_ipa, ipa, true);
	/* A range which cannot be undelegated remains owned by the RMM. */
	WARN_ON(rmi_undelegate_range(base_phys, map_size));
	return ret < 0 ? ret : -ENXIO;

If the loop mapped [base_ipa, ipa) before failing, realm_unmap_private_range()
unmaps [base_ipa, ipa) and undelegates [base_phys, phys). Calling
rmi_undelegate_range(base_phys, map_size) then starts at base_phys, which is
already undelegated, so rmi_granule_range_undelegate() fails on the first page
and aborts the loop, triggering the WARN_ON and leaving [phys,
base_phys + map_size) permanently delegated. Should this call
rmi_undelegate_range(phys, ipa_top - ipa) instead?

[Severity: High]
In kvm_vm_mem_abort()->...->realm_map_ipa()->realm_map_non_secure(), what
happens if rmi_rtt_unprot_map() returns a positive RMI error code?

In arch/arm64/kvm/rmi.c:

		if (RMI_RESULT_STATUS(ret) == RMI_ERROR_RTT) {
			...
			ret = realm_create_rtt_levels(realm, ipa, level,
						      req_level,
						      memcache);
			if (ret)
				return ret;

			ret = rmi_rtt_unprot_map(rd, ipa, ipa_top, flags,
						 range_desc, &out_top);
			if (ret < 0)
				return ret;
		}

		if (ret && RMI_RESULT_STATUS(ret) != RMI_ERROR_RTT)
			return ret;

		phys += out_top - ipa;
		ipa = out_top;

If the retry of rmi_rtt_unprot_map() after realm_create_rtt_levels() returns
RMI_ERROR_RTT, out_top is never written by rmi_rtt_unprot_map(), and execution
falls through to phys += out_top - ipa; ipa = out_top; using an uninitialized
out_top. And if rmi_rtt_unprot_map() returns any other positive RMI status
code, realm_map_non_secure() returns that positive value up through
kvm_vm_mem_abort() to kvm_handle_guest_abort(), which treats positive return
values as a successfully handled fault and re-enters the guest.

> +};
> +
>  #define KVM_VM_S2_OPS(flavor, ops)		\
>  		[flavor] = &(ops)
>  
> @@ -3299,7 +3330,7 @@ static const struct kvm_vm_s2_ops *arm64_vm_s2_ops[] = {
>  	KVM_VM_S2_OPS(VM_NVHE, kvm_default_vm_s2_ops),
>  	KVM_VM_S2_OPS(VM_PKVM, pkvm_vm_s2_ops),
>  	KVM_VM_S2_OPS(VM_PROTECTED_PKVM, protected_pkvm_vm_s2_ops),
> -	[VM_REALM] = NULL,
> +	KVM_VM_S2_OPS(VM_REALM, realm_vm_s2_ops),

[Severity: High]
Now that arm64_vm_s2_ops[VM_REALM] is populated, kvm_init_stage2_mmu()
succeeds for Realm VMs and allocates mmu->pgt, its PGD pages, and
mmu->last_vcpu_ran. Does kvm_uninit_stage2_mmu() leak those allocations if
kvm_arch_init_vm() fails at err_uninit_mmu or if the VM is destroyed before
the Realm is created?

In arch/arm64/kvm/mmu.c:

void kvm_uninit_stage2_mmu(struct kvm *kvm)
{
	if (kvm_vm_is_realm(kvm))
		kvm_realm_uninit_stage2(&kvm->arch.mmu);
	else
		kvm_free_stage2_pgd(&kvm->arch.mmu);
	kvm_mmu_free_memory_cache(&kvm->arch.mmu.split_page_cache);
}

And in arch/arm64/kvm/rmi.c:

void kvm_destroy_realm(struct kvm *kvm)
{
	struct realm *realm = &kvm->arch.realm;
	int i;

	if (!kvm_realm_is_created(kvm)) {
		kfree(realm->sro);
		realm->sro = NULL;
		return;
	}
	...
	kvm_free_stage2_pgd(&kvm->arch.mmu);
}

When realm->state is REALM_STATE_NONE, kvm_realm_uninit_stage2() is a no-op
and kvm_destroy_realm() returns early before calling
kvm_free_stage2_pgd(&kvm->arch.mmu) (or is not called at all on the
err_uninit_mmu failure path of kvm_arch_init_vm()), leaving mmu->pgt, the
stage-2 PGD pages, and mmu->last_vcpu_ran allocated.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261102141304.126883-1-steven.price@arm.com?part=11