[PATCH v1 00/26] KVM: arm64 on s390 System Register Handling

[PATCH v1 00/26] KVM: arm64 on s390 System Register Handling

[Steffen Eiden]

Add system register handling for KVM/arm64 on s390. Restructure and share
KVM/arm64 code, introduce ARM guest management functions for s390 hosts,
and implement host sysreg & exception handling.

Changes in detail:

arm64:

Refactor arm64 feature detection and ID register handling to make it
generic and reusable across architectures.

Restructure ID register storage and core register handling. Refactor core
registers to use functions instead of direct memory access.

Move arm64-specific definitions (CPU types, cache, KVM features, ID
registers, system registers) to shared locations for reuse by other
architectures.

s390:

Add s390 instruction support for ARM guest management: easr/sasr for
system register access, QAAF for feature queries, and ptff extensions
for guest time handling.

Implement complete sysreg handling for s390 including feature
sanitisation, register enumeration and access, exception injection,
and finalized page fault handling.

The series builds upon the foundation established in the first series and
requires the first series v3[1] as base.

	Steffen

[1] https://lore.kernel.org/lkml/20260529155050.2902245-1-seiden@linux.ibm.com/

Andreas Grapentin (1):
  KVM: arm64: Fix set_oslsr_el1 to write to OSLAR_EL1

Steffen Eiden (25):
  KVM: arm64: Extract some feature related changes to kvm_feature.h
  KVM: arm64: Remove __expand_field_sign_(un)signed
  KVM: arm64: Generalize get_idreg_field_*()
  KVM: arm64: Generalize kvm_cmp_feat_*()
  KVM: arm64: Generalize kvm_has_feat_*
  KVM: arm64: Remove get_idreg_field_*() and kvm_cmp_feat_*()
  KVM: arm64: Remove kvm_has_feat_range
  KVM: arm64: Split up feature sysreg sanitisation
  KVM: arm64: Refactor idreg caching into dedicated structure
  KVM: arm64: Move definitions from sys_regs.c to sys_regs.h
  KVM: arm64: Add PVM_ prefix to avoid name collisions
  s390: Introduce read/write ARM sysreg instructions
  s390: Introduce Query Available Arm features
  s390: Add functions to query arm guest time
  KVM: s390: arm64: Add sysreg related functions and definitions
  arm64: Extract cputype definitions.
  arm64: Extract cache definitions
  KVM: arm64: Share KVM feature detection macros
  KVM: arm64: Share ID reg handling
  KVM: arm64: Share sys-reg handling
  KVM: arm64: Refactor core reg handling
  KVM: s390: arm64: Implement feature sanitisation
  KVM: s390: arm64: Implement sysreg handling
  KVM: s390: arm64: Implement exception injection
  KVM: s390: arm64: Finalize page fault handling

 arch/arm64/include/asm/cache.h                |   19 +-
 arch/arm64/include/asm/cputype.h              |  246 +---
 arch/arm64/include/asm/kvm_emulate.h          |    1 +
 arch/arm64/include/asm/kvm_feature.h          |   27 +
 arch/arm64/include/asm/kvm_host.h             |  137 +-
 arch/arm64/include/asm/kvm_nested.h           |    1 +
 arch/arm64/kvm/arm.c                          |    2 +-
 arch/arm64/kvm/at.c                           |    1 +
 arch/arm64/kvm/config.c                       |    3 +-
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h    |    1 +
 arch/arm64/kvm/hyp/nvhe/hyp-main.c            |    1 +
 arch/arm64/kvm/hyp/nvhe/pkvm.c                |    7 +-
 arch/arm64/kvm/hyp/nvhe/sys_regs.c            |   55 +-
 arch/arm64/kvm/nested.c                       |    2 +-
 arch/arm64/kvm/sys_regs.c                     | 1307 +----------------
 arch/arm64/kvm/sys_regs.h                     |  276 ----
 arch/arm64/kvm/trace_handle_exit.h            |   36 +-
 arch/arm64/kvm/vgic-sys-reg-v3.c              |    2 +-
 arch/arm64/kvm/vgic/vgic-init.c               |    1 +
 arch/arm64/kvm/vgic/vgic.h                    |    1 +
 arch/s390/include/asm/kvm_emulate.h           |   34 +
 arch/s390/include/asm/kvm_feature.h           |  111 ++
 arch/s390/include/asm/kvm_host_arm64.h        |  168 ++-
 arch/s390/include/asm/kvm_host_arm64_types.h  |   97 ++
 arch/s390/include/asm/kvm_nested.h            |    5 +
 arch/s390/include/asm/sae-asm.h               |   48 +
 arch/s390/include/asm/sae.h                   |   86 ++
 arch/s390/include/asm/timex.h                 |   49 +
 arch/s390/kernel/dis.c                        |    1 +
 arch/s390/kernel/time.c                       |    1 +
 arch/s390/kvm/arm64/Makefile                  |    3 +
 arch/s390/kvm/arm64/arm.c                     |   44 +-
 arch/s390/kvm/arm64/exception.c               |  105 ++
 arch/s390/kvm/arm64/feature.c                 |  170 +++
 arch/s390/kvm/arm64/guest.c                   |   20 +-
 arch/s390/kvm/arm64/handle_exit.c             |    1 +
 arch/s390/kvm/arm64/inject_fault.c            |   72 +-
 arch/s390/kvm/arm64/mmu.c                     |   62 +-
 arch/s390/kvm/arm64/reset.c                   |    5 +
 arch/s390/kvm/arm64/sys_regs.c                |  769 ++++++++++
 arch/s390/kvm/arm64/trace.h                   |   33 +
 arch/s390/tools/opcodes.txt                   |    3 +
 include/arch/arm64/asm/cache-defs.h           |   22 +
 .../arch/arm64/asm/cputype-defs.h             |   92 +-
 include/arch/arm64/asm/sysreg-defs.h          |    9 +
 include/kvm/arm64/kvm_feature.h               |   68 +
 include/kvm/arm64/kvm_host.h                  |   52 +
 include/kvm/arm64/sys_regs.h                  |  548 +++++++
 virt/kvm/arm64/Makefile.kvm                   |    1 +
 virt/kvm/arm64/guest.c                        |  100 +-
 virt/kvm/arm64/mmio.c                         |    1 +
 virt/kvm/arm64/sys_regs.c                     | 1039 +++++++++++++
 virt/kvm/arm64/trace.h                        |   34 +
 53 files changed, 3838 insertions(+), 2141 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_feature.h
 delete mode 100644 arch/arm64/kvm/sys_regs.h
 create mode 100644 arch/s390/include/asm/kvm_feature.h
 create mode 100644 arch/s390/include/asm/sae-asm.h
 create mode 100644 arch/s390/kvm/arm64/exception.c
 create mode 100644 arch/s390/kvm/arm64/feature.c
 create mode 100644 arch/s390/kvm/arm64/sys_regs.c
 create mode 100644 arch/s390/kvm/arm64/trace.h
 create mode 100644 include/arch/arm64/asm/cache-defs.h
 copy arch/arm64/include/asm/cputype.h => include/arch/arm64/asm/cputype-defs.h (85%)
 create mode 100644 include/kvm/arm64/kvm_feature.h
 create mode 100644 include/kvm/arm64/sys_regs.h
 create mode 100644 virt/kvm/arm64/sys_regs.c


base-commit: 4095afb932d1a98a6fcb3f4f490964949d0de338
-- 
2.53.0

[PATCH v1 01/26] KVM: arm64: Extract some feature related changes to kvm_feature.h

[Steffen Eiden]

Move definitions related to kvm feature handling to a separate header.
Add flexibility through separation of concerns and allow sharing those
definitions in the future.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_emulate.h       |  1 +
 arch/arm64/include/asm/kvm_feature.h       | 93 ++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h          | 84 -------------------
 arch/arm64/include/asm/kvm_nested.h        |  1 +
 arch/arm64/kvm/at.c                        |  1 +
 arch/arm64/kvm/config.c                    |  1 +
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h |  1 +
 arch/arm64/kvm/hyp/nvhe/hyp-main.c         |  1 +
 arch/arm64/kvm/hyp/nvhe/sys_regs.c         |  1 +
 arch/arm64/kvm/vgic/vgic.h                 |  1 +
 10 files changed, 101 insertions(+), 84 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_feature.h

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 41eac2b5de14..2cb8511baddc 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -16,6 +16,7 @@
 
 #include <asm/debug-monitors.h>
 #include <asm/esr.h>
+#include <asm/kvm_feature.h>
 #include <asm/kvm_hyp.h>
 #include <asm/kvm_nested.h>
 #include <asm/ptrace.h>
diff --git a/arch/arm64/include/asm/kvm_feature.h b/arch/arm64/include/asm/kvm_feature.h
new file mode 100644
index 000000000000..8d0c65246aa0
--- /dev/null
+++ b/arch/arm64/include/asm/kvm_feature.h
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ARM64_KVM_FEATURE_H__
+#define __ARM64_KVM_FEATURE_H__
+
+#include <linux/types.h>
+#include <linux/bitfield.h>
+#include <asm/sysreg-defs.h>
+
+#define __expand_field_sign_unsigned(id, fld, val)			\
+	((u64)SYS_FIELD_VALUE(id, fld, val))
+
+#define __expand_field_sign_signed(id, fld, val)			\
+	({								\
+		u64 __val = SYS_FIELD_VALUE(id, fld, val);		\
+		sign_extend64(__val, id##_##fld##_WIDTH - 1);		\
+	})
+
+#define get_idreg_field_unsigned(kvm, id, fld)				\
+	({								\
+		u64 __val = kvm_read_vm_id_reg((kvm), SYS_##id);	\
+		FIELD_GET(id##_##fld##_MASK, __val);			\
+	})
+
+#define get_idreg_field_signed(kvm, id, fld)				\
+	({								\
+		u64 __val = get_idreg_field_unsigned(kvm, id, fld);	\
+		sign_extend64(__val, id##_##fld##_WIDTH - 1);		\
+	})
+
+#define get_idreg_field_enum(kvm, id, fld)				\
+	get_idreg_field_unsigned(kvm, id, fld)
+
+#define kvm_cmp_feat_signed(kvm, id, fld, op, limit)			\
+	(get_idreg_field_signed((kvm), id, fld) op __expand_field_sign_signed(id, fld, limit))
+
+#define kvm_cmp_feat_unsigned(kvm, id, fld, op, limit)			\
+	(get_idreg_field_unsigned((kvm), id, fld) op __expand_field_sign_unsigned(id, fld, limit))
+
+#define kvm_cmp_feat(kvm, id, fld, op, limit)				\
+	(id##_##fld##_SIGNED ?						\
+	 kvm_cmp_feat_signed(kvm, id, fld, op, limit) :			\
+	 kvm_cmp_feat_unsigned(kvm, id, fld, op, limit))
+
+#define __kvm_has_feat(kvm, id, fld, limit)				\
+	kvm_cmp_feat(kvm, id, fld, >=, limit)
+
+#define kvm_has_feat(kvm, ...) __kvm_has_feat(kvm, __VA_ARGS__)
+
+#define __kvm_has_feat_enum(kvm, id, fld, val)				\
+	kvm_cmp_feat_unsigned(kvm, id, fld, ==, val)
+
+#define kvm_has_feat_enum(kvm, ...) __kvm_has_feat_enum(kvm, __VA_ARGS__)
+
+#define kvm_has_feat_range(kvm, id, fld, min, max)			\
+	(kvm_cmp_feat(kvm, id, fld, >=, min) &&				\
+	kvm_cmp_feat(kvm, id, fld, <=, max))
+
+/* Check for a given level of PAuth support */
+#define kvm_has_pauth(k, l)						\
+	({								\
+		bool pa, pi, pa3;					\
+									\
+		pa  = kvm_has_feat((k), ID_AA64ISAR1_EL1, APA, l);	\
+		pa &= kvm_has_feat((k), ID_AA64ISAR1_EL1, GPA, IMP);	\
+		pi  = kvm_has_feat((k), ID_AA64ISAR1_EL1, API, l);	\
+		pi &= kvm_has_feat((k), ID_AA64ISAR1_EL1, GPI, IMP);	\
+		pa3  = kvm_has_feat((k), ID_AA64ISAR2_EL1, APA3, l);	\
+		pa3 &= kvm_has_feat((k), ID_AA64ISAR2_EL1, GPA3, IMP);	\
+									\
+		(pa + pi + pa3) == 1;					\
+	})
+
+#define kvm_has_fpmr(k)					\
+	(system_supports_fpmr() &&			\
+	 kvm_has_feat((k), ID_AA64PFR2_EL1, FPMR, IMP))
+
+#define kvm_has_tcr2(k)				\
+	(kvm_has_feat((k), ID_AA64MMFR3_EL1, TCRX, IMP))
+
+#define kvm_has_s1pie(k)				\
+	(kvm_has_feat((k), ID_AA64MMFR3_EL1, S1PIE, IMP))
+
+#define kvm_has_s1poe(k)				\
+	(system_supports_poe() &&			\
+	 kvm_has_feat((k), ID_AA64MMFR3_EL1, S1POE, IMP))
+
+#define kvm_has_ras(k)					\
+	(kvm_has_feat((k), ID_AA64PFR0_EL1, RAS, IMP))
+
+#define kvm_has_sctlr2(k)				\
+	(kvm_has_feat((k), ID_AA64MMFR3_EL1, SCTLRX, IMP))
+
+#endif /* __ARM64_KVM_FEATURE_H__*/
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 2db51746b4d8..4c2c62b8b506 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -1423,90 +1423,6 @@ static inline u64 *__vm_id_reg(struct kvm_arch *ka, u32 reg)
 
 void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val);
 
-#define __expand_field_sign_unsigned(id, fld, val)			\
-	((u64)SYS_FIELD_VALUE(id, fld, val))
-
-#define __expand_field_sign_signed(id, fld, val)			\
-	({								\
-		u64 __val = SYS_FIELD_VALUE(id, fld, val);		\
-		sign_extend64(__val, id##_##fld##_WIDTH - 1);		\
-	})
-
-#define get_idreg_field_unsigned(kvm, id, fld)				\
-	({								\
-		u64 __val = kvm_read_vm_id_reg((kvm), SYS_##id);	\
-		FIELD_GET(id##_##fld##_MASK, __val);			\
-	})
-
-#define get_idreg_field_signed(kvm, id, fld)				\
-	({								\
-		u64 __val = get_idreg_field_unsigned(kvm, id, fld);	\
-		sign_extend64(__val, id##_##fld##_WIDTH - 1);		\
-	})
-
-#define get_idreg_field_enum(kvm, id, fld)				\
-	get_idreg_field_unsigned(kvm, id, fld)
-
-#define kvm_cmp_feat_signed(kvm, id, fld, op, limit)			\
-	(get_idreg_field_signed((kvm), id, fld) op __expand_field_sign_signed(id, fld, limit))
-
-#define kvm_cmp_feat_unsigned(kvm, id, fld, op, limit)			\
-	(get_idreg_field_unsigned((kvm), id, fld) op __expand_field_sign_unsigned(id, fld, limit))
-
-#define kvm_cmp_feat(kvm, id, fld, op, limit)				\
-	(id##_##fld##_SIGNED ?						\
-	 kvm_cmp_feat_signed(kvm, id, fld, op, limit) :			\
-	 kvm_cmp_feat_unsigned(kvm, id, fld, op, limit))
-
-#define __kvm_has_feat(kvm, id, fld, limit)				\
-	kvm_cmp_feat(kvm, id, fld, >=, limit)
-
-#define kvm_has_feat(kvm, ...) __kvm_has_feat(kvm, __VA_ARGS__)
-
-#define __kvm_has_feat_enum(kvm, id, fld, val)				\
-	kvm_cmp_feat_unsigned(kvm, id, fld, ==, val)
-
-#define kvm_has_feat_enum(kvm, ...) __kvm_has_feat_enum(kvm, __VA_ARGS__)
-
-#define kvm_has_feat_range(kvm, id, fld, min, max)			\
-	(kvm_cmp_feat(kvm, id, fld, >=, min) &&				\
-	kvm_cmp_feat(kvm, id, fld, <=, max))
-
-/* Check for a given level of PAuth support */
-#define kvm_has_pauth(k, l)						\
-	({								\
-		bool pa, pi, pa3;					\
-									\
-		pa  = kvm_has_feat((k), ID_AA64ISAR1_EL1, APA, l);	\
-		pa &= kvm_has_feat((k), ID_AA64ISAR1_EL1, GPA, IMP);	\
-		pi  = kvm_has_feat((k), ID_AA64ISAR1_EL1, API, l);	\
-		pi &= kvm_has_feat((k), ID_AA64ISAR1_EL1, GPI, IMP);	\
-		pa3  = kvm_has_feat((k), ID_AA64ISAR2_EL1, APA3, l);	\
-		pa3 &= kvm_has_feat((k), ID_AA64ISAR2_EL1, GPA3, IMP);	\
-									\
-		(pa + pi + pa3) == 1;					\
-	})
-
-#define kvm_has_fpmr(k)					\
-	(system_supports_fpmr() &&			\
-	 kvm_has_feat((k), ID_AA64PFR2_EL1, FPMR, IMP))
-
-#define kvm_has_tcr2(k)				\
-	(kvm_has_feat((k), ID_AA64MMFR3_EL1, TCRX, IMP))
-
-#define kvm_has_s1pie(k)				\
-	(kvm_has_feat((k), ID_AA64MMFR3_EL1, S1PIE, IMP))
-
-#define kvm_has_s1poe(k)				\
-	(system_supports_poe() &&			\
-	 kvm_has_feat((k), ID_AA64MMFR3_EL1, S1POE, IMP))
-
-#define kvm_has_ras(k)					\
-	(kvm_has_feat((k), ID_AA64PFR0_EL1, RAS, IMP))
-
-#define kvm_has_sctlr2(k)				\
-	(kvm_has_feat((k), ID_AA64MMFR3_EL1, SCTLRX, IMP))
-
 static inline bool kvm_arch_has_irq_bypass(void)
 {
 	return true;
diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h
index dc2957662ff2..5da9ffae4f73 100644
--- a/arch/arm64/include/asm/kvm_nested.h
+++ b/arch/arm64/include/asm/kvm_nested.h
@@ -5,6 +5,7 @@
 #include <linux/bitfield.h>
 #include <linux/kvm_host.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_feature.h>
 #include <asm/kvm_pgtable.h>
 
 static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu)
diff --git a/arch/arm64/kvm/at.c b/arch/arm64/kvm/at.c
index 9f8f0ae8e86e..ae91734dde37 100644
--- a/arch/arm64/kvm/at.c
+++ b/arch/arm64/kvm/at.c
@@ -7,6 +7,7 @@
 #include <linux/kvm_host.h>
 
 #include <asm/esr.h>
+#include <asm/kvm_feature.h>
 #include <asm/kvm_hyp.h>
 #include <asm/kvm_mmu.h>
 #include <asm/lsui.h>
diff --git a/arch/arm64/kvm/config.c b/arch/arm64/kvm/config.c
index 0622162b089e..014fe04daabf 100644
--- a/arch/arm64/kvm/config.c
+++ b/arch/arm64/kvm/config.c
@@ -6,6 +6,7 @@
 
 #include <linux/kvm_host.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_feature.h>
 #include <asm/kvm_nested.h>
 #include <asm/sysreg.h>
 
diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
index a17cbe7582de..dd824096dfc1 100644
--- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
+++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
@@ -13,6 +13,7 @@
 #include <asm/kprobes.h>
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_feature.h>
 #include <asm/kvm_hyp.h>
 #include <asm/kvm_mmu.h>
 
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 06db299c37a8..3e5c9107d78f 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -10,6 +10,7 @@
 #include <asm/pgtable-types.h>
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_feature.h>
 #include <asm/kvm_host.h>
 #include <asm/kvm_hyp.h>
 #include <asm/kvm_hypevents.h>
diff --git a/arch/arm64/kvm/hyp/nvhe/sys_regs.c b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
index 8c3fbb413a06..b5a0de84ce01 100644
--- a/arch/arm64/kvm/hyp/nvhe/sys_regs.c
+++ b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
@@ -7,6 +7,7 @@
 #include <linux/irqchip/arm-gic-v3.h>
 
 #include <asm/kvm_asm.h>
+#include <asm/kvm_feature.h>
 #include <asm/kvm_mmu.h>
 
 #include <hyp/adjust_pc.h>
diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h
index 9d941241c8a2..b275d3cabe21 100644
--- a/arch/arm64/kvm/vgic/vgic.h
+++ b/arch/arm64/kvm/vgic/vgic.h
@@ -6,6 +6,7 @@
 #define __KVM_ARM_VGIC_NEW_H__
 
 #include <linux/irqchip/arm-gic-common.h>
+#include <asm/kvm_feature.h>
 #include <asm/kvm_mmu.h>
 
 #define PRODUCT_ID_KVM		0x4b	/* ASCII code K */
-- 
2.53.0

[PATCH v1 02/26] KVM: arm64: Remove __expand_field_sign_(un)signed

[Steffen Eiden]

__expand_field_sign_unsigned is a very small abstraction that makes it
harder to see what's happening when looking at the caller. Just inline
it. Create a macro S64_SYS_FIELD_VALUE that is a sign extended
SYS_FIELD_VALUE. Then also get rid of __expand_field_sign_signed.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_feature.h | 13 ++-----------
 include/arch/arm64/asm/sysreg-defs.h |  8 ++++++++
 2 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_feature.h b/arch/arm64/include/asm/kvm_feature.h
index 8d0c65246aa0..d580f4ffab34 100644
--- a/arch/arm64/include/asm/kvm_feature.h
+++ b/arch/arm64/include/asm/kvm_feature.h
@@ -6,15 +6,6 @@
 #include <linux/bitfield.h>
 #include <asm/sysreg-defs.h>
 
-#define __expand_field_sign_unsigned(id, fld, val)			\
-	((u64)SYS_FIELD_VALUE(id, fld, val))
-
-#define __expand_field_sign_signed(id, fld, val)			\
-	({								\
-		u64 __val = SYS_FIELD_VALUE(id, fld, val);		\
-		sign_extend64(__val, id##_##fld##_WIDTH - 1);		\
-	})
-
 #define get_idreg_field_unsigned(kvm, id, fld)				\
 	({								\
 		u64 __val = kvm_read_vm_id_reg((kvm), SYS_##id);	\
@@ -31,10 +22,10 @@
 	get_idreg_field_unsigned(kvm, id, fld)
 
 #define kvm_cmp_feat_signed(kvm, id, fld, op, limit)			\
-	(get_idreg_field_signed((kvm), id, fld) op __expand_field_sign_signed(id, fld, limit))
+	(get_idreg_field_signed((kvm), id, fld) op S64_SYS_FIELD_VALUE(id, fld, limit))
 
 #define kvm_cmp_feat_unsigned(kvm, id, fld, op, limit)			\
-	(get_idreg_field_unsigned((kvm), id, fld) op __expand_field_sign_unsigned(id, fld, limit))
+	(get_idreg_field_unsigned((kvm), id, fld) op (u64)SYS_FIELD_VALUE(id, fld, limit))
 
 #define kvm_cmp_feat(kvm, id, fld, op, limit)				\
 	(id##_##fld##_SIGNED ?						\
diff --git a/include/arch/arm64/asm/sysreg-defs.h b/include/arch/arm64/asm/sysreg-defs.h
index 27646c91e15c..3e280d4156ce 100644
--- a/include/arch/arm64/asm/sysreg-defs.h
+++ b/include/arch/arm64/asm/sysreg-defs.h
@@ -998,9 +998,17 @@
 
 #ifndef __ASSEMBLER__
 #include <linux/bitfield.h>
+#include <linux/bitops.h>
+#include <linux/types.h>
 
 #define SYS_FIELD_VALUE(reg, field, val)	reg##_##field##_##val
 
+#define S64_SYS_FIELD_VALUE(id, fld, val)				\
+	({								\
+		u64 __val = SYS_FIELD_VALUE(id, fld, val);		\
+		sign_extend64(__val, id##_##fld##_WIDTH - 1);		\
+	})
+
 #define SYS_FIELD_GET(reg, field, val)		\
 		 FIELD_GET(reg##_##field##_MASK, val)
 
-- 
2.53.0

[PATCH v1 03/26] KVM: arm64: Generalize get_idreg_field_*()

[Steffen Eiden]

Introduce intermediate macros that extract the value from a passed
parameter instead of reading the VM's ID register. Allow using other
sources of ID register values, i.e. read directly from the hardware or
during a sequence of sanitization steps.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_feature.h | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_feature.h b/arch/arm64/include/asm/kvm_feature.h
index d580f4ffab34..067550d5b208 100644
--- a/arch/arm64/include/asm/kvm_feature.h
+++ b/arch/arm64/include/asm/kvm_feature.h
@@ -6,20 +6,23 @@
 #include <linux/bitfield.h>
 #include <asm/sysreg-defs.h>
 
-#define get_idreg_field_unsigned(kvm, id, fld)				\
-	({								\
-		u64 __val = kvm_read_vm_id_reg((kvm), SYS_##id);	\
-		FIELD_GET(id##_##fld##_MASK, __val);			\
-	})
+#define extract_id_field_unsigned(val, id, fld)				\
+	(FIELD_GET(id##_##fld##_MASK, (val)))
 
-#define get_idreg_field_signed(kvm, id, fld)				\
+#define extract_id_field_signed(val, id, fld)				\
 	({								\
-		u64 __val = get_idreg_field_unsigned(kvm, id, fld);	\
+		u64 __val = extract_id_field_unsigned((val), id, fld);	\
 		sign_extend64(__val, id##_##fld##_WIDTH - 1);		\
 	})
 
+#define get_idreg_field_unsigned(kvm, id, fld)				\
+	extract_id_field_unsigned(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld)
+
+#define get_idreg_field_signed(kvm, id, fld)				\
+	extract_id_field_signed(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld)
+
 #define get_idreg_field_enum(kvm, id, fld)				\
-	get_idreg_field_unsigned(kvm, id, fld)
+	extract_id_field_unsigned(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld)
 
 #define kvm_cmp_feat_signed(kvm, id, fld, op, limit)			\
 	(get_idreg_field_signed((kvm), id, fld) op S64_SYS_FIELD_VALUE(id, fld, limit))
-- 
2.53.0

[PATCH v1 04/26] KVM: arm64: Generalize kvm_cmp_feat_*()

[Steffen Eiden]

Introduce an intermediate macro that extracts the value from a passed
parameter instead of reading the VM's ID register. Allow using other
sources of ID register values, i.e. read directly from the hardware or
during a sequence of sanitization steps.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_feature.h | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_feature.h b/arch/arm64/include/asm/kvm_feature.h
index 067550d5b208..6dd7b4a4929c 100644
--- a/arch/arm64/include/asm/kvm_feature.h
+++ b/arch/arm64/include/asm/kvm_feature.h
@@ -15,6 +15,17 @@
 		sign_extend64(__val, id##_##fld##_WIDTH - 1);		\
 	})
 
+#define cmp_id_feat_signed(val, id, fld, op, limit)			\
+	(extract_id_field_signed((val), id, fld) op S64_SYS_FIELD_VALUE(id, fld, limit))
+
+#define cmp_id_feat_unsigned(val, id, fld, op, limit)			\
+	(extract_id_field_unsigned((val), id, fld) op (u64)SYS_FIELD_VALUE(id, fld, limit))
+
+#define cmp_id_feat(val, id, fld, op, limit)				\
+	(id##_##fld##_SIGNED ?						\
+	 cmp_id_feat_signed(val, id, fld, op, limit) :			\
+	 cmp_id_feat_unsigned(val, id, fld, op, limit))
+
 #define get_idreg_field_unsigned(kvm, id, fld)				\
 	extract_id_field_unsigned(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld)
 
@@ -25,15 +36,13 @@
 	extract_id_field_unsigned(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld)
 
 #define kvm_cmp_feat_signed(kvm, id, fld, op, limit)			\
-	(get_idreg_field_signed((kvm), id, fld) op S64_SYS_FIELD_VALUE(id, fld, limit))
+	cmp_id_feat_signed(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, op, limit)
 
 #define kvm_cmp_feat_unsigned(kvm, id, fld, op, limit)			\
-	(get_idreg_field_unsigned((kvm), id, fld) op (u64)SYS_FIELD_VALUE(id, fld, limit))
+	cmp_id_feat_unsigned(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, op, limit)
 
 #define kvm_cmp_feat(kvm, id, fld, op, limit)				\
-	(id##_##fld##_SIGNED ?						\
-	 kvm_cmp_feat_signed(kvm, id, fld, op, limit) :			\
-	 kvm_cmp_feat_unsigned(kvm, id, fld, op, limit))
+	cmp_id_feat(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, op, limit)
 
 #define __kvm_has_feat(kvm, id, fld, limit)				\
 	kvm_cmp_feat(kvm, id, fld, >=, limit)
-- 
2.53.0

[PATCH v1 05/26] KVM: arm64: Generalize kvm_has_feat_*

[Steffen Eiden]

Introduce an intermediate macro that extracts the value from a passed
parameter instead of reading the VM's ID register. Allow using other
sources of ID register values, i.e. read directly from the hardware or
during a sequence of sanitization steps.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_feature.h | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_feature.h b/arch/arm64/include/asm/kvm_feature.h
index 6dd7b4a4929c..b627696ac648 100644
--- a/arch/arm64/include/asm/kvm_feature.h
+++ b/arch/arm64/include/asm/kvm_feature.h
@@ -26,6 +26,16 @@
 	 cmp_id_feat_signed(val, id, fld, op, limit) :			\
 	 cmp_id_feat_unsigned(val, id, fld, op, limit))
 
+#define id_has_feat(val, id, fld, limit)				\
+	cmp_id_feat(val, id, fld, >=, limit)
+
+#define id_has_feat_enum(val, id, fld, variant)				\
+	cmp_id_feat_unsigned(val, id, fld, ==, variant)
+
+#define id_has_feat_range(val, id, fld, min, max)			\
+	(cmp_id_feat(val, id, fld, >=, min) &&				\
+	cmp_id_feat(val, id, fld, <=, max))
+
 #define get_idreg_field_unsigned(kvm, id, fld)				\
 	extract_id_field_unsigned(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld)
 
@@ -45,18 +55,17 @@
 	cmp_id_feat(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, op, limit)
 
 #define __kvm_has_feat(kvm, id, fld, limit)				\
-	kvm_cmp_feat(kvm, id, fld, >=, limit)
+	id_has_feat(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, limit)
 
 #define kvm_has_feat(kvm, ...) __kvm_has_feat(kvm, __VA_ARGS__)
 
 #define __kvm_has_feat_enum(kvm, id, fld, val)				\
-	kvm_cmp_feat_unsigned(kvm, id, fld, ==, val)
+	id_has_feat_enum(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, val)
 
 #define kvm_has_feat_enum(kvm, ...) __kvm_has_feat_enum(kvm, __VA_ARGS__)
 
 #define kvm_has_feat_range(kvm, id, fld, min, max)			\
-	(kvm_cmp_feat(kvm, id, fld, >=, min) &&				\
-	kvm_cmp_feat(kvm, id, fld, <=, max))
+	id_has_feat_range(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, min, max)
 
 /* Check for a given level of PAuth support */
 #define kvm_has_pauth(k, l)						\
-- 
2.53.0

[PATCH v1 06/26] KVM: arm64: Remove get_idreg_field_*() and kvm_cmp_feat_*()

[Steffen Eiden]

These macros are now unused after generalizing the feature detection
code in the parent commit. The functionality is preserved through the
new generalized macros that operate on register values directly.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_feature.h | 18 ------------------
 1 file changed, 18 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_feature.h b/arch/arm64/include/asm/kvm_feature.h
index b627696ac648..da9ba5041f44 100644
--- a/arch/arm64/include/asm/kvm_feature.h
+++ b/arch/arm64/include/asm/kvm_feature.h
@@ -36,24 +36,6 @@
 	(cmp_id_feat(val, id, fld, >=, min) &&				\
 	cmp_id_feat(val, id, fld, <=, max))
 
-#define get_idreg_field_unsigned(kvm, id, fld)				\
-	extract_id_field_unsigned(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld)
-
-#define get_idreg_field_signed(kvm, id, fld)				\
-	extract_id_field_signed(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld)
-
-#define get_idreg_field_enum(kvm, id, fld)				\
-	extract_id_field_unsigned(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld)
-
-#define kvm_cmp_feat_signed(kvm, id, fld, op, limit)			\
-	cmp_id_feat_signed(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, op, limit)
-
-#define kvm_cmp_feat_unsigned(kvm, id, fld, op, limit)			\
-	cmp_id_feat_unsigned(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, op, limit)
-
-#define kvm_cmp_feat(kvm, id, fld, op, limit)				\
-	cmp_id_feat(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, op, limit)
-
 #define __kvm_has_feat(kvm, id, fld, limit)				\
 	id_has_feat(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, limit)
 
-- 
2.53.0

[PATCH v1 07/26] KVM: arm64: Remove kvm_has_feat_range

[Steffen Eiden]

It is (and has been) unused.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_feature.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_feature.h b/arch/arm64/include/asm/kvm_feature.h
index da9ba5041f44..27a472d2343e 100644
--- a/arch/arm64/include/asm/kvm_feature.h
+++ b/arch/arm64/include/asm/kvm_feature.h
@@ -46,8 +46,6 @@
 
 #define kvm_has_feat_enum(kvm, ...) __kvm_has_feat_enum(kvm, __VA_ARGS__)
 
-#define kvm_has_feat_range(kvm, id, fld, min, max)			\
-	id_has_feat_range(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, min, max)
 
 /* Check for a given level of PAuth support */
 #define kvm_has_pauth(k, l)						\
-- 
2.53.0

[PATCH v1 08/26] KVM: arm64: Split up feature sysreg sanitisation

[Steffen Eiden]

Split ID register sanitisation into distinct stages:

1) static KVM limits (kvm_max_possible_guest_ftr_reg)
2) host-specific (kvm_sanitised_host_ftr_reg)
3) per-vcpu configuration (kvm_sanitise_vcpu_ftr_reg)

This refactoring improves code organization by separating concerns.
Static limits apply regardless of host or guest configuration. Host
capability checks handle features like GIC, GCIE, and Spectre
mitigations. Per-vcpu feature configuration manages SVE, MTE, PMU, and
similar guest-specific features. Additionally, this enables other
architectures to add different host-implementation-based sanitisation in
the future.

Remove helper functions sanitise_id_aa64{pfr0,pfr1,dfr0}_el1
in favor of organized logic.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/kvm/sys_regs.c | 291 ++++++++++++++++++++------------------
 1 file changed, 153 insertions(+), 138 deletions(-)

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 2434bcc2d50d..b9aa892616ab 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -1841,54 +1841,86 @@ static u8 pmuver_to_perfmon(u8 pmuver)
 	}
 }
 
-static u64 sanitise_id_aa64pfr0_el1(const struct kvm_vcpu *vcpu, u64 val);
-static u64 sanitise_id_aa64pfr1_el1(const struct kvm_vcpu *vcpu, u64 val);
-static u64 sanitise_id_aa64pfr2_el1(const struct kvm_vcpu *vcpu, u64 val);
-static u64 sanitise_id_aa64dfr0_el1(const struct kvm_vcpu *vcpu, u64 val);
-
-/* Read a sanitised cpufeature ID register by sys_reg_desc */
-static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu,
-				       const struct sys_reg_desc *r)
+/*
+ * Sanitise based on the host implementation.
+ */
+static u64 kvm_sanitised_host_ftr_reg(u32 id)
 {
-	u32 id = reg_to_encoding(r);
-	u64 val;
-
-	if (sysreg_visible_as_raz(vcpu, r))
-		return 0;
-
-	val = read_sanitised_ftr_reg(id);
+	u64 val = read_sanitised_ftr_reg(id);
 
 	switch (id) {
-	case SYS_ID_AA64DFR0_EL1:
-		val = sanitise_id_aa64dfr0_el1(vcpu, val);
+	case SYS_ID_AA64ISAR2_EL1:
+		if (!cpus_have_final_cap(ARM64_HAS_WFXT) ||
+		    has_broken_cntvoff())
+			val &= ~ID_AA64ISAR2_EL1_WFxT;
 		break;
 	case SYS_ID_AA64PFR0_EL1:
-		val = sanitise_id_aa64pfr0_el1(vcpu, val);
+		/*
+		 * The default is to expose CSV2 == 1 if the HW isn't affected.
+		 * Although this is a per-CPU feature, we make it global because
+		 * asymmetric systems are just a nuisance.
+		 *
+		 * Userspace can override this as long as it doesn't promise
+		 * the impossible.
+		 */
+		if (arm64_get_spectre_v2_state() == SPECTRE_UNAFFECTED) {
+			val &= ~ID_AA64PFR0_EL1_CSV2_MASK;
+			val |= SYS_FIELD_PREP_ENUM(ID_AA64PFR0_EL1, CSV2, IMP);
+		}
+		if (arm64_get_meltdown_state() == SPECTRE_UNAFFECTED) {
+			val &= ~ID_AA64PFR0_EL1_CSV3_MASK;
+			val |= SYS_FIELD_PREP_ENUM(ID_AA64PFR0_EL1, CSV3, IMP);
+		}
+		if (vgic_host_has_gicv3()) {
+			val &= ~ID_AA64PFR0_EL1_GIC_MASK;
+			val |= SYS_FIELD_PREP_ENUM(ID_AA64PFR0_EL1, GIC, IMP);
+		}
 		break;
-	case SYS_ID_AA64PFR1_EL1:
-		val = sanitise_id_aa64pfr1_el1(vcpu, val);
+	case SYS_ID_AA64PFR1_EL1: {
+		u64 pfr0_host = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
+
+		if (!(cpus_have_final_cap(ARM64_HAS_RASV1P1_EXTN) &&
+		      SYS_FIELD_GET(ID_AA64PFR0_EL1, RAS, pfr0_host) == ID_AA64PFR0_EL1_RAS_IMP))
+			val &= ~ID_AA64PFR1_EL1_RAS_frac;
 		break;
+	}
 	case SYS_ID_AA64PFR2_EL1:
-		val = sanitise_id_aa64pfr2_el1(vcpu, val);
+		if (vgic_host_has_gicv5())
+			val |= SYS_FIELD_PREP_ENUM(ID_AA64PFR2_EL1, GCIE, IMP);
 		break;
-	case SYS_ID_AA64ISAR1_EL1:
-		if (!vcpu_has_ptrauth(vcpu))
-			val &= ~(ID_AA64ISAR1_EL1_APA |
-				 ID_AA64ISAR1_EL1_API |
-				 ID_AA64ISAR1_EL1_GPA |
-				 ID_AA64ISAR1_EL1_GPI);
+	case SYS_ID_AA64MMFR3_EL1:
+		if (!system_supports_poe())
+			val &= ~ID_AA64MMFR3_EL1_S1POE;
+		break;
+	}
+
+	return val;
+}
+
+/*
+ * Statically sanitise the host's feature register, independent of the guest's
+ * configuration and host implementation.
+ */
+static u64 kvm_max_possible_guest_ftr_reg(u32 id, u64 val)
+{
+	switch (id) {
+	case SYS_ID_AA64DFR0_EL1:
+		val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64DFR0_EL1, DebugVer, V8P8);
+
+		/* Hide SPE from guests */
+		val &= ~ID_AA64DFR0_EL1_PMSVer_MASK;
+
+		/* Hide BRBE from guests */
+		val &= ~ID_AA64DFR0_EL1_BRBE_MASK;
 		break;
 	case SYS_ID_AA64ISAR2_EL1:
-		if (!vcpu_has_ptrauth(vcpu))
-			val &= ~(ID_AA64ISAR2_EL1_APA3 |
-				 ID_AA64ISAR2_EL1_GPA3);
-		if (!cpus_have_final_cap(ARM64_HAS_WFXT) ||
-		    has_broken_cntvoff())
+		/* Mask WFxT field unless *both* WFET & WFIT are present. */
+		if (!id_has_feat(val, ID_AA64ISAR2_EL1, WFxT, IMP))
 			val &= ~ID_AA64ISAR2_EL1_WFxT;
 		break;
 	case SYS_ID_AA64ISAR3_EL1:
 		val &= ID_AA64ISAR3_EL1_FPRCVT | ID_AA64ISAR3_EL1_LSFE |
-			ID_AA64ISAR3_EL1_FAMINMAX | ID_AA64ISAR3_EL1_LSUI;
+		       ID_AA64ISAR3_EL1_FAMINMAX | ID_AA64ISAR3_EL1_LSUI;
 		break;
 	case SYS_ID_AA64MMFR2_EL1:
 		val &= ~ID_AA64MMFR2_EL1_CCIDX_MASK;
@@ -1899,13 +1931,81 @@ static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu,
 		       ID_AA64MMFR3_EL1_SCTLRX |
 		       ID_AA64MMFR3_EL1_S1POE |
 		       ID_AA64MMFR3_EL1_S1PIE;
-
-		if (!system_supports_poe())
-			val &= ~ID_AA64MMFR3_EL1_S1POE;
 		break;
 	case SYS_ID_MMFR4_EL1:
 		val &= ~ID_MMFR4_EL1_CCIDX;
 		break;
+	case SYS_ID_AA64PFR0_EL1:
+		val &= ~ID_AA64PFR0_EL1_AMU_MASK;
+		/*
+		 * MPAM is disabled by default as KVM also needs a set of PARTID to
+		 * program the MPAMVPMx_EL2 PARTID remapping registers with. But some
+		 * older kernels let the guest see the ID bit.
+		 */
+		val &= ~ID_AA64PFR0_EL1_MPAM_MASK;
+		break;
+	case SYS_ID_AA64PFR1_EL1:
+		val &= ~ID_AA64PFR1_EL1_SME;
+		val &= ~ID_AA64PFR1_EL1_RNDR_trap;
+		val &= ~ID_AA64PFR1_EL1_NMI;
+		val &= ~ID_AA64PFR1_EL1_GCS;
+		val &= ~ID_AA64PFR1_EL1_THE;
+		val &= ~ID_AA64PFR1_EL1_MTEX;
+		val &= ~ID_AA64PFR1_EL1_PFAR;
+		val &= ~ID_AA64PFR1_EL1_MPAM_frac;
+		break;
+	case SYS_ID_AA64PFR2_EL1:
+		val &= ID_AA64PFR2_EL1_FPMR |
+		       ID_AA64PFR2_EL1_MTEFAR |
+		       ID_AA64PFR2_EL1_MTESTOREONLY;
+		break;
+	}
+
+	return val;
+}
+
+/*
+ * Sanitise based on vCPU configuration.
+ */
+static u64 kvm_sanitise_vcpu_ftr_reg(const struct kvm_vcpu *vcpu, u32 id, u64 val)
+{
+	switch (id) {
+	case SYS_ID_AA64DFR0_EL1:
+		/*
+		 * Only initialize the PMU version if the vCPU was configured with one.
+		 */
+		val &= ~ID_AA64DFR0_EL1_PMUVer_MASK;
+		if (kvm_vcpu_has_pmu(vcpu))
+			val |= SYS_FIELD_PREP(ID_AA64DFR0_EL1, PMUVer,
+					      kvm_arm_pmu_get_pmuver_limit());
+		break;
+	case SYS_ID_AA64PFR0_EL1:
+		if (!vcpu_has_sve(vcpu))
+			val &= ~ID_AA64PFR0_EL1_SVE_MASK;
+		break;
+	case SYS_ID_AA64PFR1_EL1:
+		if (!kvm_has_mte(vcpu->kvm)) {
+			val &= ~ID_AA64PFR1_EL1_MTE;
+			val &= ~ID_AA64PFR1_EL1_MTE_frac;
+		}
+		break;
+	case SYS_ID_AA64PFR2_EL1:
+		if (!kvm_has_mte(vcpu->kvm)) {
+			val &= ~ID_AA64PFR2_EL1_MTEFAR;
+			val &= ~ID_AA64PFR2_EL1_MTESTOREONLY;
+		}
+		break;
+	case SYS_ID_AA64ISAR1_EL1:
+		if (!vcpu_has_ptrauth(vcpu))
+			val &= ~(ID_AA64ISAR1_EL1_APA |
+				 ID_AA64ISAR1_EL1_API |
+				 ID_AA64ISAR1_EL1_GPA |
+				 ID_AA64ISAR1_EL1_GPI);
+		break;
+	case SYS_ID_AA64ISAR2_EL1:
+		if (!vcpu_has_ptrauth(vcpu))
+			val &= ~(ID_AA64ISAR2_EL1_APA3 |
+				 ID_AA64ISAR2_EL1_GPA3);
 	}
 
 	if (vcpu_has_nv(vcpu))
@@ -1914,6 +2014,23 @@ static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu,
 	return val;
 }
 
+/* Read a sanitised cpufeature ID register by sys_reg_desc */
+static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu,
+				       const struct sys_reg_desc *r)
+{
+	u32 id = reg_to_encoding(r);
+	u64 val;
+
+	if (sysreg_visible_as_raz(vcpu, r))
+		return 0;
+
+	val = kvm_sanitised_host_ftr_reg(id);
+	val = kvm_max_possible_guest_ftr_reg(id, val);
+	val = kvm_sanitise_vcpu_ftr_reg(vcpu, id, val);
+
+	return val;
+}
+
 static u64 kvm_read_sanitised_id_reg(struct kvm_vcpu *vcpu,
 				     const struct sys_reg_desc *r)
 {
@@ -2046,108 +2163,6 @@ static unsigned int fp8_visibility(const struct kvm_vcpu *vcpu,
 	return REG_HIDDEN;
 }
 
-static u64 sanitise_id_aa64pfr0_el1(const struct kvm_vcpu *vcpu, u64 val)
-{
-	if (!vcpu_has_sve(vcpu))
-		val &= ~ID_AA64PFR0_EL1_SVE_MASK;
-
-	/*
-	 * The default is to expose CSV2 == 1 if the HW isn't affected.
-	 * Although this is a per-CPU feature, we make it global because
-	 * asymmetric systems are just a nuisance.
-	 *
-	 * Userspace can override this as long as it doesn't promise
-	 * the impossible.
-	 */
-	if (arm64_get_spectre_v2_state() == SPECTRE_UNAFFECTED) {
-		val &= ~ID_AA64PFR0_EL1_CSV2_MASK;
-		val |= SYS_FIELD_PREP_ENUM(ID_AA64PFR0_EL1, CSV2, IMP);
-	}
-	if (arm64_get_meltdown_state() == SPECTRE_UNAFFECTED) {
-		val &= ~ID_AA64PFR0_EL1_CSV3_MASK;
-		val |= SYS_FIELD_PREP_ENUM(ID_AA64PFR0_EL1, CSV3, IMP);
-	}
-
-	if (vgic_host_has_gicv3()) {
-		val &= ~ID_AA64PFR0_EL1_GIC_MASK;
-		val |= SYS_FIELD_PREP_ENUM(ID_AA64PFR0_EL1, GIC, IMP);
-	}
-
-	val &= ~ID_AA64PFR0_EL1_AMU_MASK;
-
-	/*
-	 * MPAM is disabled by default as KVM also needs a set of PARTID to
-	 * program the MPAMVPMx_EL2 PARTID remapping registers with. But some
-	 * older kernels let the guest see the ID bit.
-	 */
-	val &= ~ID_AA64PFR0_EL1_MPAM_MASK;
-
-	return val;
-}
-
-static u64 sanitise_id_aa64pfr1_el1(const struct kvm_vcpu *vcpu, u64 val)
-{
-	u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
-
-	if (!kvm_has_mte(vcpu->kvm)) {
-		val &= ~ID_AA64PFR1_EL1_MTE;
-		val &= ~ID_AA64PFR1_EL1_MTE_frac;
-	}
-
-	if (!(cpus_have_final_cap(ARM64_HAS_RASV1P1_EXTN) &&
-	      SYS_FIELD_GET(ID_AA64PFR0_EL1, RAS, pfr0) == ID_AA64PFR0_EL1_RAS_IMP))
-		val &= ~ID_AA64PFR1_EL1_RAS_frac;
-
-	val &= ~ID_AA64PFR1_EL1_SME;
-	val &= ~ID_AA64PFR1_EL1_RNDR_trap;
-	val &= ~ID_AA64PFR1_EL1_NMI;
-	val &= ~ID_AA64PFR1_EL1_GCS;
-	val &= ~ID_AA64PFR1_EL1_THE;
-	val &= ~ID_AA64PFR1_EL1_MTEX;
-	val &= ~ID_AA64PFR1_EL1_PFAR;
-	val &= ~ID_AA64PFR1_EL1_MPAM_frac;
-
-	return val;
-}
-
-static u64 sanitise_id_aa64pfr2_el1(const struct kvm_vcpu *vcpu, u64 val)
-{
-	val &= ID_AA64PFR2_EL1_FPMR |
-	       ID_AA64PFR2_EL1_MTEFAR |
-	       ID_AA64PFR2_EL1_MTESTOREONLY;
-
-	if (!kvm_has_mte(vcpu->kvm)) {
-		val &= ~ID_AA64PFR2_EL1_MTEFAR;
-		val &= ~ID_AA64PFR2_EL1_MTESTOREONLY;
-	}
-
-	if (vgic_host_has_gicv5())
-		val |= SYS_FIELD_PREP_ENUM(ID_AA64PFR2_EL1, GCIE, IMP);
-
-	return val;
-}
-
-static u64 sanitise_id_aa64dfr0_el1(const struct kvm_vcpu *vcpu, u64 val)
-{
-	val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64DFR0_EL1, DebugVer, V8P8);
-
-	/*
-	 * Only initialize the PMU version if the vCPU was configured with one.
-	 */
-	val &= ~ID_AA64DFR0_EL1_PMUVer_MASK;
-	if (kvm_vcpu_has_pmu(vcpu))
-		val |= SYS_FIELD_PREP(ID_AA64DFR0_EL1, PMUVer,
-				      kvm_arm_pmu_get_pmuver_limit());
-
-	/* Hide SPE from guests */
-	val &= ~ID_AA64DFR0_EL1_PMSVer_MASK;
-
-	/* Hide BRBE from guests */
-	val &= ~ID_AA64DFR0_EL1_BRBE_MASK;
-
-	return val;
-}
-
 /*
  * Older versions of KVM erroneously claim support for FEAT_DoubleLock with
  * NV-enabled VMs on unsupporting hardware. Silently ignore the incorrect
-- 
2.53.0

[PATCH v1 09/26] KVM: arm64: Refactor idreg caching into dedicated structure

[Steffen Eiden]

Move VM-wide ID register emulation fields from struct kvm_arch into a
new struct kvm_vm_id_regs to prepare future sharing of these fields and
functions using them. Update all users to use the new structure. No
functional changes.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_host.h  | 50 ++++++++++++++++--------------
 arch/arm64/kvm/config.c            |  2 +-
 arch/arm64/kvm/hyp/nvhe/pkvm.c     |  7 +++--
 arch/arm64/kvm/hyp/nvhe/sys_regs.c |  4 +--
 arch/arm64/kvm/sys_regs.c          |  2 +-
 5 files changed, 35 insertions(+), 30 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 4c2c62b8b506..a8efff6ea01d 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -352,22 +352,7 @@ struct kvm_arch {
 	struct kvm_smccc_features smccc_feat;
 	struct maple_tree smccc_filter;
 
-	/*
-	 * Emulated CPU ID registers per VM
-	 * (Op0, Op1, CRn, CRm, Op2) of the ID registers to be saved in it
-	 * is (3, 0, 0, crm, op2), where 1<=crm<8, 0<=op2<8.
-	 *
-	 * These emulated idregs are VM-wide, but accessed from the context of a vCPU.
-	 * Atomic access to multiple idregs are guarded by kvm_arch.config_lock.
-	 */
-#define IDREG_IDX(id)		(((sys_reg_CRm(id) - 1) << 3) | sys_reg_Op2(id))
-#define KVM_ARM_ID_REG_NUM	(IDREG_IDX(sys_reg(3, 0, 0, 7, 7)) + 1)
-	u64 id_regs[KVM_ARM_ID_REG_NUM];
-
-	u64 midr_el1;
-	u64 revidr_el1;
-	u64 aidr_el1;
-	u64 ctr_el0;
+	struct kvm_vm_id_regs id_regs;
 
 	/* Masks for VNCR-backed and general EL2 sysregs */
 	struct kvm_sysreg_masks	*sysreg_masks;
@@ -1399,19 +1384,38 @@ static inline void kvm_hyp_reserve(void) { }
 void kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu);
 bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu);
 
-static inline u64 *__vm_id_reg(struct kvm_arch *ka, u32 reg)
+struct kvm_vm_id_regs {
+	/*
+	 * Emulated CPU ID registers per VM
+	 * (Op0, Op1, CRn, CRm, Op2) of the ID registers to be saved in it
+	 * is (3, 0, 0, crm, op2), where 1<=crm<8, 0<=op2<8.
+	 *
+	 * These emulated idregs are VM-wide, but accessed from the context of a vCPU.
+	 * Atomic access to multiple idregs are guarded by kvm_arch.config_lock.
+	 */
+#define IDREG_IDX(id)		(((sys_reg_CRm(id) - 1) << 3) | sys_reg_Op2(id))
+#define KVM_ARM_ID_REG_NUM	(IDREG_IDX(sys_reg(3, 0, 0, 7, 7)) + 1)
+	u64 normal[KVM_ARM_ID_REG_NUM];
+
+	u64 midr_el1;
+	u64 revidr_el1;
+	u64 aidr_el1;
+	u64 ctr_el0;
+};
+
+static inline u64 *__vm_id_reg(struct kvm_vm_id_regs *id_regs, u32 reg)
 {
 	switch (reg) {
 	case sys_reg(3, 0, 0, 1, 0) ... sys_reg(3, 0, 0, 7, 7):
-		return &ka->id_regs[IDREG_IDX(reg)];
+		return &id_regs->normal[IDREG_IDX(reg)];
 	case SYS_CTR_EL0:
-		return &ka->ctr_el0;
+		return &id_regs->ctr_el0;
 	case SYS_MIDR_EL1:
-		return &ka->midr_el1;
+		return &id_regs->midr_el1;
 	case SYS_REVIDR_EL1:
-		return &ka->revidr_el1;
+		return &id_regs->revidr_el1;
 	case SYS_AIDR_EL1:
-		return &ka->aidr_el1;
+		return &id_regs->aidr_el1;
 	default:
 		WARN_ON_ONCE(1);
 		return NULL;
@@ -1419,7 +1423,7 @@ static inline u64 *__vm_id_reg(struct kvm_arch *ka, u32 reg)
 }
 
 #define kvm_read_vm_id_reg(kvm, reg)					\
-	({ u64 __val = *__vm_id_reg(&(kvm)->arch, reg); __val; })
+	({ u64 __val = *__vm_id_reg(&(kvm)->arch.id_regs, reg); __val; })
 
 void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val);
 
diff --git a/arch/arm64/kvm/config.c b/arch/arm64/kvm/config.c
index 014fe04daabf..58a439c3ab9c 100644
--- a/arch/arm64/kvm/config.c
+++ b/arch/arm64/kvm/config.c
@@ -1398,7 +1398,7 @@ void __init check_feature_map(void)
 
 static bool idreg_feat_match(struct kvm *kvm, const struct reg_bits_to_feat_map *map)
 {
-	u64 regval = kvm->arch.id_regs[map->regidx];
+	u64 regval = kvm->arch.id_regs.normal[map->regidx];
 	u64 regfld = (regval >> map->shift) & GENMASK(map->width - 1, 0);
 
 	if (map->sign) {
diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
index eb1c10120f9f..94620f142f42 100644
--- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
+++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c
@@ -343,7 +343,7 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
 	DECLARE_BITMAP(allowed_features, KVM_VCPU_MAX_FEATURES);
 
 	/* CTR_EL0 is always under host control, even for protected VMs. */
-	hyp_vm->kvm.arch.ctr_el0 = host_kvm->arch.ctr_el0;
+	hyp_vm->kvm.arch.id_regs.ctr_el0 = host_kvm->arch.id_regs.ctr_el0;
 
 	/* Preserve the vgic model so that GICv3 emulation works */
 	hyp_vm->kvm.arch.vgic.vgic_model = host_kvm->arch.vgic.vgic_model;
@@ -358,7 +358,7 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
 			    KVM_VCPU_MAX_FEATURES);
 
 		if (test_bit(KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, &host_arch_flags))
-			hyp_vm->kvm.arch.midr_el1 = host_kvm->arch.midr_el1;
+			hyp_vm->kvm.arch.id_regs.midr_el1 = host_kvm->arch.id_regs.midr_el1;
 
 		return;
 	}
@@ -493,7 +493,8 @@ static int vm_copy_id_regs(struct pkvm_hyp_vcpu *hyp_vcpu)
 	if (test_and_set_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags))
 		return 0;
 
-	memcpy(kvm->arch.id_regs, host_kvm->arch.id_regs, sizeof(kvm->arch.id_regs));
+	memcpy(kvm->arch.id_regs.normal, host_kvm->arch.id_regs.normal,
+	       sizeof(kvm->arch.id_regs.normal));
 
 	return 0;
 }
diff --git a/arch/arm64/kvm/hyp/nvhe/sys_regs.c b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
index b5a0de84ce01..e8d773d38905 100644
--- a/arch/arm64/kvm/hyp/nvhe/sys_regs.c
+++ b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
@@ -292,7 +292,7 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu,
 		return 0;
 
 	if (reg >= sys_reg(3, 0, 0, 1, 0) && reg <= sys_reg(3, 0, 0, 7, 7))
-		return kvm->arch.id_regs[IDREG_IDX(reg)];
+		return kvm->arch.id_regs.normal[IDREG_IDX(reg)];
 
 	return 0;
 }
@@ -543,7 +543,7 @@ void kvm_init_pvm_id_regs(struct kvm_vcpu *vcpu)
 	 * for protected VMs.
 	 */
 	for (r = sys_reg(3, 0, 0, 4, 0); r <= sys_reg(3, 0, 0, 7, 7); r += sys_reg(0, 0, 0, 0, 1))
-		ka->id_regs[IDREG_IDX(r)] = pvm_calc_id_reg(vcpu, r);
+		ka->id_regs.normal[IDREG_IDX(r)] = pvm_calc_id_reg(vcpu, r);
 
 	set_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags);
 }
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index b9aa892616ab..195ecdac7bd6 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -2477,7 +2477,7 @@ static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
 
 void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val)
 {
-	u64 *p = __vm_id_reg(&kvm->arch, reg);
+	u64 *p = __vm_id_reg(&kvm->arch.id_regs, reg);
 
 	lockdep_assert_held(&kvm->arch.config_lock);
 
-- 
2.53.0

[PATCH v1 10/26] KVM: arm64: Fix set_oslsr_el1 to write to OSLAR_EL1

[Steffen Eiden]

From: Andreas Grapentin <gra@linux.ibm.com>

The set_oslsr_el1() function was incorrectly writing directly to the
OSLSR_EL1 register, which is architecturally a read-only status register
that reflects the state of the OS Lock.

Fix this by extracting the OSLK bit from the user-provided value and
writing it to OSLAR_EL1 (OS Lock Access Register) instead, which is the
proper control register for managing the OS Lock state. OSLSR_EL1 will
then reflect this state when read.

This ensures the implementation follows the ARM architecture
specification where OSLAR_EL1 controls the lock and OSLSR_EL1 provides
status information.

Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_host.h    |  1 +
 arch/arm64/kvm/sys_regs.c            | 10 +++++++++-
 include/arch/arm64/asm/sysreg-defs.h |  1 +
 3 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index a8efff6ea01d..5734e93cad57 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -408,6 +408,7 @@ enum vcpu_sysreg {
 	PAR_EL1,	/* Physical Address Register */
 	MDCCINT_EL1,	/* Monitor Debug Comms Channel Interrupt Enable Reg */
 	OSLSR_EL1,	/* OS Lock Status Register */
+	OSLAR_EL1,	/* OS Lock Access Register */
 	DISR_EL1,	/* Deferred Interrupt Status Register */
 
 	/* Performance Monitors Registers */
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 195ecdac7bd6..6522f9302967 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -822,6 +822,8 @@ static bool trap_oslsr_el1(struct kvm_vcpu *vcpu,
 static int set_oslsr_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
 			 u64 val)
 {
+	u64 oslk;
+
 	/*
 	 * The only modifiable bit is the OSLK bit. Refuse the write if
 	 * userspace attempts to change any other bit in the register.
@@ -829,7 +831,13 @@ static int set_oslsr_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
 	if ((val ^ rd->val) & ~OSLSR_EL1_OSLK)
 		return -EINVAL;
 
-	__vcpu_assign_sys_reg(vcpu, rd->reg, val);
+	/*
+	 * Redirect the write to the proper control register.
+	 * OSLSR is read-only
+	 */
+	oslk = SYS_FIELD_GET(OSLSR_EL1, OSLK, val);
+	__vcpu_assign_sys_reg(vcpu, OSLAR_EL1,
+			      SYS_FIELD_PREP(OSLAR_EL1, OSLK, oslk));
 	return 0;
 }
 
diff --git a/include/arch/arm64/asm/sysreg-defs.h b/include/arch/arm64/asm/sysreg-defs.h
index 3e280d4156ce..c6bdb0f11e1b 100644
--- a/include/arch/arm64/asm/sysreg-defs.h
+++ b/include/arch/arm64/asm/sysreg-defs.h
@@ -129,6 +129,7 @@
 #define OSLSR_EL1_OSLM_NI		0
 #define OSLSR_EL1_OSLM_IMPLEMENTED	BIT(3)
 #define OSLSR_EL1_OSLK			BIT(1)
+#define OSLSR_EL1_OSLK_MASK             BIT(1)
 
 #define SYS_OSDLR_EL1			sys_reg(2, 0, 1, 3, 4)
 #define SYS_DBGPRCR_EL1			sys_reg(2, 0, 1, 4, 4)
-- 
2.53.0

[PATCH v1 11/26] KVM: arm64: Move definitions from sys_regs.c to sys_regs.h

[Steffen Eiden]

Make kvm_sanitised_host_ftr_reg() and kvm_read_sanitised_id_reg()
available to enable code sharing with s390. Move some helper and ID
register macro definitions to the header file. No functional changes.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/kvm/sys_regs.c | 12 ++----
 arch/arm64/kvm/sys_regs.h | 87 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 6522f9302967..46b24529ec70 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -1852,7 +1852,7 @@ static u8 pmuver_to_perfmon(u8 pmuver)
 /*
  * Sanitise based on the host implementation.
  */
-static u64 kvm_sanitised_host_ftr_reg(u32 id)
+u64 kvm_sanitised_host_ftr_reg(u32 id)
 {
 	u64 val = read_sanitised_ftr_reg(id);
 
@@ -2039,8 +2039,8 @@ static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu,
 	return val;
 }
 
-static u64 kvm_read_sanitised_id_reg(struct kvm_vcpu *vcpu,
-				     const struct sys_reg_desc *r)
+u64 kvm_read_sanitised_id_reg(struct kvm_vcpu *vcpu,
+			      const struct sys_reg_desc *r)
 {
 	return __kvm_read_sanitised_id_reg(vcpu, r);
 }
@@ -2123,12 +2123,6 @@ static unsigned int aa32_id_visibility(const struct kvm_vcpu *vcpu,
 	return id_visibility(vcpu, r);
 }
 
-static unsigned int raz_visibility(const struct kvm_vcpu *vcpu,
-				   const struct sys_reg_desc *r)
-{
-	return REG_RAZ;
-}
-
 /* cpufeature ID register access trap handlers */
 
 static bool access_id_reg(struct kvm_vcpu *vcpu,
diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h
index 2a983664220c..75d581050b09 100644
--- a/arch/arm64/kvm/sys_regs.h
+++ b/arch/arm64/kvm/sys_regs.h
@@ -108,6 +108,12 @@ struct sys_reg_desc {
 #define REG_RAZ			(1 << 1) /* RAZ from userspace and guest */
 #define REG_USER_WI		(1 << 2) /* WI from userspace only */
 
+static inline unsigned int raz_visibility(const struct kvm_vcpu *vcpu,
+				   const struct sys_reg_desc *r)
+{
+	return REG_RAZ;
+}
+
 static __printf(2, 3)
 inline void print_sys_reg_msg(const struct sys_reg_params *p,
 				       char *fmt, ...)
@@ -237,6 +243,12 @@ bool triage_sysreg_trap(struct kvm_vcpu *vcpu, int *sr_index);
 
 int kvm_finalize_sys_regs(struct kvm_vcpu *vcpu);
 
+u64 kvm_read_sanitised_id_reg(struct kvm_vcpu *vcpu,
+			      const struct sys_reg_desc *r);
+
+/* Implemented by each architecture */
+u64 kvm_sanitised_host_ftr_reg(u32 id);
+
 #define AA32(_x)	.aarch32_map = AA32_##_x
 #define Op0(_x) 	.Op0 = _x
 #define Op1(_x) 	.Op1 = _x
@@ -257,6 +269,81 @@ int kvm_finalize_sys_regs(struct kvm_vcpu *vcpu);
 	CRn(sys_reg_CRn(reg)), CRm(sys_reg_CRm(reg)),	\
 	Op2(sys_reg_Op2(reg))
 
+/*
+ * Since reset() callback and field val are not used for idregs, they will be
+ * used for specific purposes for idregs.
+ * The reset() would return KVM sanitised register value. The value would be the
+ * same as the host kernel sanitised value if there is no KVM sanitisation.
+ * The val would be used as a mask indicating writable fields for the idreg.
+ * Only bits with 1 are writable from userspace. This mask might not be
+ * necessary in the future whenever all ID registers are enabled as writable
+ * from userspace.
+ */
+
+#define ID_DESC_DEFAULT_CALLBACKS		\
+	.access	= access_id_reg,		\
+	.get_user = get_id_reg,			\
+	.set_user = set_id_reg,			\
+	.visibility = id_visibility,		\
+	.reset = kvm_read_sanitised_id_reg
+
+#define ID_DESC(name)				\
+	SYS_DESC(SYS_##name),			\
+	ID_DESC_DEFAULT_CALLBACKS
+
+/* sys_reg_desc initialiser for known cpufeature ID registers */
+#define ID_SANITISED(name) {			\
+	ID_DESC(name),				\
+	.val = 0,				\
+}
+
+/* sys_reg_desc initialiser for writable ID registers */
+#define ID_WRITABLE(name, mask) {		\
+	ID_DESC(name),				\
+	.val = mask,				\
+}
+
+/*
+ * 32bit ID regs are fully writable when the guest is 32bit
+ * capable. Nothing in the KVM code should rely on 32bit features
+ * anyway, only 64bit, so let the VMM do its worse.
+ */
+#define AA32_ID_WRITABLE(name) {		\
+	ID_DESC(name),				\
+	.visibility = aa32_id_visibility,	\
+	.val = GENMASK(31, 0),			\
+}
+
+/* sys_reg_desc initialiser for cpufeature ID registers that need filtering */
+#define ID_FILTERED(sysreg, name, mask) {	\
+	ID_DESC(sysreg),				\
+	.set_user = set_##name,				\
+	.val = (mask),					\
+}
+
+/*
+ * sys_reg_desc initialiser for architecturally unallocated cpufeature ID
+ * register with encoding Op0=3, Op1=0, CRn=0, CRm=crm, Op2=op2
+ * (1 <= crm < 8, 0 <= Op2 < 8).
+ */
+#define ID_UNALLOCATED(crm, op2) {			\
+	.name = "S3_0_0_" #crm "_" #op2,		\
+	Op0(3), Op1(0), CRn(0), CRm(crm), Op2(op2),	\
+	ID_DESC_DEFAULT_CALLBACKS,			\
+	.visibility = raz_visibility,			\
+	.val = 0,					\
+}
+
+/*
+ * sys_reg_desc initialiser for known ID registers that we hide from guests.
+ * For now, these are exposed just like unallocated ID regs: they appear
+ * RAZ for the guest.
+ */
+#define ID_HIDDEN(name) {			\
+	ID_DESC(name),				\
+	.visibility = raz_visibility,		\
+	.val = 0,				\
+}
 #define ID_REG_LIMIT_FIELD_ENUM(val, reg, field, limit)			       \
 ({									       \
 	u64 __f_val = FIELD_GET(reg##_##field##_MASK, val);		       \
-- 
2.53.0

[PATCH v1 12/26] KVM: arm64: Add PVM_ prefix to avoid name collisions

[Steffen Eiden]

Rename ID_UNALLOCATED to PVM_ID_UNALLOCATED and read_id_reg to
pvm_read_id_reg to prevent future name collisions with other subsystems.
While at it, fix whitespace issues in the macro invocations

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/kvm/hyp/nvhe/sys_regs.c | 48 +++++++++++++++---------------
 1 file changed, 24 insertions(+), 24 deletions(-)

diff --git a/arch/arm64/kvm/hyp/nvhe/sys_regs.c b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
index e8d773d38905..08b14053568b 100644
--- a/arch/arm64/kvm/hyp/nvhe/sys_regs.c
+++ b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
@@ -282,8 +282,8 @@ static void inject_undef64(struct kvm_vcpu *vcpu)
 	inject_sync64(vcpu, (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT));
 }
 
-static u64 read_id_reg(const struct kvm_vcpu *vcpu,
-		       struct sys_reg_desc const *r)
+static u64 pvm_read_id_reg(const struct kvm_vcpu *vcpu,
+			   struct sys_reg_desc const *r)
 {
 	struct kvm *kvm = vcpu->kvm;
 	u32 reg = reg_to_encoding(r);
@@ -341,7 +341,7 @@ static bool pvm_access_id_aarch64(struct kvm_vcpu *vcpu,
 		return false;
 	}
 
-	p->regval = read_id_reg(vcpu, r);
+	p->regval = pvm_read_id_reg(vcpu, r);
 	return true;
 }
 
@@ -379,7 +379,7 @@ static bool pvm_idst_access(struct kvm_vcpu *vcpu,
  * register with encoding Op0=3, Op1=0, CRn=0, CRm=crm, Op2=op2
  * (1 <= crm < 8, 0 <= Op2 < 8).
  */
-#define ID_UNALLOCATED(crm, op2) {			\
+#define PVM_ID_UNALLOCATED(crm, op2) {			\
 	Op0(3), Op1(0), CRn(0), CRm(crm), Op2(op2),	\
 	.access = pvm_access_id_aarch64,		\
 }
@@ -438,46 +438,46 @@ static const struct sys_reg_desc pvm_sys_reg_descs[] = {
 	AARCH32(SYS_MVFR0_EL1),
 	AARCH32(SYS_MVFR1_EL1),
 	AARCH32(SYS_MVFR2_EL1),
-	ID_UNALLOCATED(3,3),
+	PVM_ID_UNALLOCATED(3, 3),
 	AARCH32(SYS_ID_PFR2_EL1),
 	AARCH32(SYS_ID_DFR1_EL1),
 	AARCH32(SYS_ID_MMFR5_EL1),
-	ID_UNALLOCATED(3,7),
+	PVM_ID_UNALLOCATED(3, 7),
 
 	/* AArch64 ID registers */
 	/* CRm=4 */
 	AARCH64(SYS_ID_AA64PFR0_EL1),
 	AARCH64(SYS_ID_AA64PFR1_EL1),
 	AARCH64(SYS_ID_AA64PFR2_EL1),
-	ID_UNALLOCATED(4,3),
+	PVM_ID_UNALLOCATED(4, 3),
 	AARCH64(SYS_ID_AA64ZFR0_EL1),
-	ID_UNALLOCATED(4,5),
-	ID_UNALLOCATED(4,6),
-	ID_UNALLOCATED(4,7),
+	PVM_ID_UNALLOCATED(4, 5),
+	PVM_ID_UNALLOCATED(4, 6),
+	PVM_ID_UNALLOCATED(4, 7),
 	AARCH64(SYS_ID_AA64DFR0_EL1),
 	AARCH64(SYS_ID_AA64DFR1_EL1),
-	ID_UNALLOCATED(5,2),
-	ID_UNALLOCATED(5,3),
+	PVM_ID_UNALLOCATED(5, 2),
+	PVM_ID_UNALLOCATED(5, 3),
 	AARCH64(SYS_ID_AA64AFR0_EL1),
 	AARCH64(SYS_ID_AA64AFR1_EL1),
-	ID_UNALLOCATED(5,6),
-	ID_UNALLOCATED(5,7),
+	PVM_ID_UNALLOCATED(5, 6),
+	PVM_ID_UNALLOCATED(5, 7),
 	AARCH64(SYS_ID_AA64ISAR0_EL1),
 	AARCH64(SYS_ID_AA64ISAR1_EL1),
 	AARCH64(SYS_ID_AA64ISAR2_EL1),
-	ID_UNALLOCATED(6,3),
-	ID_UNALLOCATED(6,4),
-	ID_UNALLOCATED(6,5),
-	ID_UNALLOCATED(6,6),
-	ID_UNALLOCATED(6,7),
+	PVM_ID_UNALLOCATED(6, 3),
+	PVM_ID_UNALLOCATED(6, 4),
+	PVM_ID_UNALLOCATED(6, 5),
+	PVM_ID_UNALLOCATED(6, 6),
+	PVM_ID_UNALLOCATED(6, 7),
 	AARCH64(SYS_ID_AA64MMFR0_EL1),
 	AARCH64(SYS_ID_AA64MMFR1_EL1),
 	AARCH64(SYS_ID_AA64MMFR2_EL1),
-	ID_UNALLOCATED(7,3),
-	ID_UNALLOCATED(7,4),
-	ID_UNALLOCATED(7,5),
-	ID_UNALLOCATED(7,6),
-	ID_UNALLOCATED(7,7),
+	PVM_ID_UNALLOCATED(7, 3),
+	PVM_ID_UNALLOCATED(7, 4),
+	PVM_ID_UNALLOCATED(7, 5),
+	PVM_ID_UNALLOCATED(7, 6),
+	PVM_ID_UNALLOCATED(7, 7),
 
 	/* Scalable Vector Registers are restricted. */
 
-- 
2.53.0

[PATCH v1 13/26] s390: Introduce read/write ARM sysreg instructions

[Steffen Eiden]

Introduce Extract Arm System Register and Store Arm System Register to
enable s390 hosts to read and write system registers for arm64 guests.
The new instructions use the new RIE_H instruction format. Add assembler
macros to create instructions in RIE_H format manually. Add Support for
disassembling the new instructions.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/sae-asm.h | 48 +++++++++++++++++++++++++++
 arch/s390/include/asm/sae.h     | 58 +++++++++++++++++++++++++++++++++
 arch/s390/kernel/dis.c          |  1 +
 arch/s390/tools/opcodes.txt     |  2 ++
 4 files changed, 109 insertions(+)
 create mode 100644 arch/s390/include/asm/sae-asm.h

diff --git a/arch/s390/include/asm/sae-asm.h b/arch/s390/include/asm/sae-asm.h
new file mode 100644
index 000000000000..d81ed89eb4ed
--- /dev/null
+++ b/arch/s390/include/asm/sae-asm.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_S390_SAE_ASM_H
+#define __ASM_S390_SAE_ASM_H
+
+#ifdef __ASSEMBLER__
+
+.macro	GPR_NUM	opd gr
+	\opd = 255
+	.irp rs,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
+	.ifc \gr,%r\rs
+		\opd = \rs
+	.endif
+	.endr
+	.if \opd == 255
+		\opd = \gr
+	.endif
+.endm
+
+/*
+ * RIE_H - RIE-h instruction format
+ *
+ * RIE-h format: <insn> R1, R3, I2, M4
+ *    +--------+----+----+----+-----------------+----+--------+
+ *    | OpCode | R1 |////| R3 |        I2       | M4 | Opcode |
+ *    +--------+----+----+----+-----------------+----+--------+
+ *    0        8    12   16   20                36   40      47
+ */
+.macro RIE_H	opc, gr1, gr3, imm2, m4
+	GPR_NUM	r1, \gr1
+	GPR_NUM	r3, \gr3
+	.byte	(\opc & 0xff00) >> 8
+	.byte	r1 << 4
+	.byte	(r3 << 4) | ((\imm2 & 0xf000) >> 12)
+	.byte	((\imm2 & 0x0ff0) >> 4)
+	.byte	((\imm2 & 0x000f) << 4) | (\m4 & 0xf)
+	.byte	\opc & 0xff
+.endm
+
+.macro SASR r1, r3, i2, m4
+	RIE_H 0xed99, \r1, \r3, \i2, \m4,
+.endm
+
+.macro EASR r1, r3, i2, m4
+	RIE_H 0xed9b, \r1, \r3, \i2, \m4,
+.endm
+
+#endif /* __ASSEMBLER__ */
+#endif /* __ASM_S390_SAE_ASM_H */
diff --git a/arch/s390/include/asm/sae.h b/arch/s390/include/asm/sae.h
index fe010a1a7729..1d9a16b91b23 100644
--- a/arch/s390/include/asm/sae.h
+++ b/arch/s390/include/asm/sae.h
@@ -4,6 +4,7 @@
 
 #include "linux/linkage.h"
 #include <linux/types.h>
+#include <asm/sae-asm.h>
 
 /* defined in arch/s390/kernel/entry.S */
 asmlinkage int __sae64a(phys_addr_t sae_block_phys);
@@ -12,6 +13,12 @@ asmlinkage int __sae64a(phys_addr_t sae_block_phys);
 #include <linux/io.h>
 #include <asm/kvm_host_arm64_types.h>
 
+asm(".include \"asm/sae-asm.h\"\n");
+
+#define _SAE_ASR_REG_SHIFT	5
+#define SASR_FLAG_INITIALIZED  0x8
+#define EASR_FLAG_SA           0x8
+
 /**
  * __sae64a() - Start Arm Execution
  */
@@ -20,6 +27,57 @@ static inline void sae64a(struct kvm_sae_block *sae_block)
 	__sae64a(virt_to_phys(sae_block));
 }
 
+/**
+ * sasr() - Set Arm System Register
+ * @arm_reg: ARM system register identifier; compile-time constant
+ * @val: Value to set
+ * @save_area: Pointer to SAE save area
+ * @flags: Operation flags; compile-time constant
+ *
+ * Sets an ARM system register value.
+ */
+static __always_inline void sasr(unsigned int arm_reg, u64 val,
+				 struct kvm_sae_save_area *save_area,
+				 u64 flags)
+{
+	struct kvm_sae_save_area *sdo = (void *)save_area->sdo;
+	u16 reg = arm_reg >> _SAE_ASR_REG_SHIFT;
+
+	asm volatile (
+		"	SASR	%[r1],%[r3],%[i2],%[m4]\n"
+		: "+m" (*save_area), "+m" (*sdo)
+		: [r1] "d" (val),
+		  [r3] "a" (save_area), [i2] "K" (reg), [m4] "I" (flags)
+	);
+}
+
+/**
+ * easr() - Extract Arm System Register
+ * @arm_reg: ARM system register identifier; compile-time constant
+ * @save_area: Pointer to SAE save area
+ * @flags: Operation flags; compile-time constant
+ *
+ * Reads an ARM system register value.
+ *
+ * Return: Register value
+ */
+static __always_inline u64 easr(unsigned int arm_reg,
+				const struct kvm_sae_save_area *save_area,
+				u64 flags)
+{
+	struct kvm_sae_save_area *sdo = (void *)save_area->sdo;
+	u16 reg = arm_reg >> _SAE_ASR_REG_SHIFT;
+	u64 val;
+
+	asm volatile(
+		"	EASR	%[r1],%[r3],%[i2],%[m4]\n"
+		: [r1] "=d"(val)
+		: "m"(*save_area),
+		  "m"(*sdo), [r3] "a"(save_area), [i2] "K"(reg), [m4] "I"(flags)
+	);
+	return val;
+}
+
 /**
  * stiasrm() - STore and Invalidate Arm System Register Multiple
  * @save_area: Pointer to SAE save area
diff --git a/arch/s390/kernel/dis.c b/arch/s390/kernel/dis.c
index 1cec93895b3a..6ff8fd09d5bc 100644
--- a/arch/s390/kernel/dis.c
+++ b/arch/s390/kernel/dis.c
@@ -208,6 +208,7 @@ static const unsigned char formats[][6] = {
 	[INSTR_RIE_RUI0]     = { R_8, I16_16, U4_12, 0, 0, 0 },
 	[INSTR_RIE_RUPI]     = { R_8, I8_32, U4_12, J16_16, 0, 0 },
 	[INSTR_RIE_RUPU]     = { R_8, U8_32, U4_12, J16_16, 0, 0 },
+	[INSTR_RIE_R0RIU]    = { R_8, R_16, U16_20, U4_36, 0, 0 },
 	[INSTR_RIL_RI]	     = { R_8, I32_16, 0, 0, 0, 0 },
 	[INSTR_RIL_RP]	     = { R_8, J32_16, 0, 0, 0, 0 },
 	[INSTR_RIL_RU]	     = { R_8, U32_16, 0, 0, 0, 0 },
diff --git a/arch/s390/tools/opcodes.txt b/arch/s390/tools/opcodes.txt
index 0e4773c94af0..18af14071290 100644
--- a/arch/s390/tools/opcodes.txt
+++ b/arch/s390/tools/opcodes.txt
@@ -1255,6 +1255,8 @@ ed64	ley	RXY_FRRD
 ed65	ldy	RXY_FRRD
 ed66	stey	RXY_FRRD
 ed67	stdy	RXY_FRRD
+ed99	sasr	RIE_R0RIU
+ed9b	easr	RIE_R0RIU
 eda8	czdt	RSL_LRDFU
 eda9	czxt	RSL_LRDFU
 edaa	cdzt	RSL_LRDFU
-- 
2.53.0

[PATCH v1 14/26] s390: Introduce Query Available Arm features

[Steffen Eiden]

QAAF enables s390 hosts to gain information about the support
and handling of various arm features supported by the machine.

Function code 1 provides general information about available formats,
machine defined content of system/id register, and other various
information for running arm guests.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host_arm64_types.h | 96 ++++++++++++++++++++
 arch/s390/include/asm/sae.h                  | 28 ++++++
 arch/s390/tools/opcodes.txt                  |  1 +
 3 files changed, 125 insertions(+)

diff --git a/arch/s390/include/asm/kvm_host_arm64_types.h b/arch/s390/include/asm/kvm_host_arm64_types.h
index 3882d5462a05..16f7018a1714 100644
--- a/arch/s390/include/asm/kvm_host_arm64_types.h
+++ b/arch/s390/include/asm/kvm_host_arm64_types.h
@@ -126,4 +126,100 @@ struct kvm_sae_save_area {
 } __packed __aligned(PAGE_SIZE);
 static_assert(sizeof(struct kvm_sae_save_area) == PAGE_SIZE);
 
+#define QAAF_FC_QMC	1
+
+/* QAAF Query Model Capabilities */
+struct qaaf_qmc_block {
+	u64	_0000;			/* 0x0000 */
+	u8	ssdf;			/* 0x0008 */
+	u8	_0009;			/* 0x0009 */
+	u8	ssaf;			/* 0x000a */
+	u8	_000b[3];		/* 0x000b */
+	u16	maxncpu;		/* 0x000e */
+	u64	regs[0x1fe];		/* 0x0010 */
+} __aligned(PAGE_SIZE);
+static_assert(sizeof(struct qaaf_qmc_block) == PAGE_SIZE);
+
+union qaaf_block {
+	struct qaaf_qmc_block qmc;
+} __aligned(PAGE_SIZE);
+static_assert(sizeof(union qaaf_block) == PAGE_SIZE);
+
+/*
+ * Keep in sync with mapping from SYS_* to QAAF_* in feature.c!
+ */
+enum {
+	QAAF_REG_MIDR_EL1		= 0x02,
+	/* 0x03 -0x06 reserved */
+	QAAF_REG_MPIDR_EL1		= 0x07,
+	QAAF_REG_REVIDR_EL1		= 0x08,
+	/* 0x09 reserved */
+	QAAF_REG_ID_PFR0_EL1		= 0x0a,
+	QAAF_REG_ID_PFR1_EL1		= 0x0b,
+	QAAF_REG_ID_DFR0_EL1		= 0x0c,
+	QAAF_REG_ID_AFR0_EL1		= 0x0d,
+	QAAF_REG_ID_MMFR0_EL1		= 0x0e,
+	QAAF_REG_ID_MMFR1_EL1		= 0x0f,
+	QAAF_REG_ID_MMFR2_EL1		= 0x10,
+	QAAF_REG_ID_MMFR3_EL1		= 0x11,
+	QAAF_REG_ID_ISAR0_EL1		= 0x12,
+	QAAF_REG_ID_ISAR1_EL1		= 0x13,
+	QAAF_REG_ID_ISAR2_EL1		= 0x14,
+	QAAF_REG_ID_ISAR3_EL1		= 0x15,
+	QAAF_REG_ID_ISAR4_EL1		= 0x16,
+	QAAF_REG_ID_ISAR5_EL1		= 0x17,
+	QAAF_REG_ID_MMFR4_EL1		= 0x18,
+	QAAF_REG_ID_ISAR6_EL1		= 0x19,
+	QAAF_REG_MVFR0_EL1		= 0x1a,
+	QAAF_REG_MVFR1_EL1		= 0x1b,
+	QAAF_REG_MVFR2_EL1		= 0x1c,
+	/* 0x1d reserved */
+	QAAF_REG_ID_PFR2_EL1		= 0x1e,
+	QAAF_REG_ID_DFR1_EL1		= 0x1f,
+	QAAF_REG_ID_MMFR5_EL1		= 0x20,
+	/* 0x21 reserved */
+	QAAF_REG_ID_AA64PFR0_EL1	= 0x22,
+	QAAF_REG_ID_AA64PFR1_EL1	= 0x23,
+	QAAF_REG_ID_AA64PFR2_EL1	= 0x24,
+	/* 0x25 reserved */
+	QAAF_REG_ID_AA64ZFR0_EL1	= 0x26,
+	QAAF_REG_ID_AA64SMFR0_EL1	= 0x27,
+	/* 0x28 reserved */
+	QAAF_REG_ID_AA64FPFR0_EL1	= 0x29,
+	QAAF_REG_ID_AA64DFR0_EL1	= 0x2a,
+	QAAF_REG_ID_AA64DFR1_EL1	= 0x2b,
+	QAAF_REG_ID_AA64DFR2_EL1	= 0x2c,
+	/* 0x2d reserved */
+	QAAF_REG_ID_AA64AFR0_EL1	= 0x2e,
+	QAAF_REG_ID_AA64AFR1_EL1	= 0x2f,
+	/* 0x30,0x31 reserved */
+	QAAF_REG_ID_AA64ISAR0_EL1	= 0x32,
+	QAAF_REG_ID_AA64ISAR1_EL1	= 0x33,
+	QAAF_REG_ID_AA64ISAR2_EL1	= 0x34,
+	QAAF_REG_ID_AA64ISAR3_EL1	= 0x35,
+	/* 0x36-0x39 reserved */
+	QAAF_REG_ID_AA64MMFR0_EL1	= 0x3a,
+	QAAF_REG_ID_AA64MMFR1_EL1	= 0x3b,
+	QAAF_REG_ID_AA64MMFR2_EL1	= 0x3c,
+	QAAF_REG_ID_AA64MMFR3_EL1	= 0x3d,
+	QAAF_REG_ID_AA64MMFR4_EL1	= 0x3e,
+	/* 0x3f-0x41 reserved */
+	QAAF_REG_CNTFRQ_EL0		= 0x42,
+	QAAF_REG_CTR_EL0		= 0x43,
+	/* 0x44-0x49 reserved */
+	QAAF_IRPTC			= 0x4a,
+	/* 0x4b reserved */
+	QAAF_REG_ICH_VTR_EL2		= 0x4c,
+	QAAF_GIC_ATTR			= 0x4d,
+	/* 0x4E-0x51 reserved */
+	QAAF_REG_PMMIR_EL1		= 0x52,
+	QAAF_REG_PMCR_EL0		= 0x53,
+	QAAF_REG_PMCEID0_EL0		= 0x54,
+	QAAF_REG_PMCEID1_EL0		= 0x55,
+	/* 0x56-0x1ff reserved */
+	_QAAF_MAX
+};
+
+static_assert(sizeof(struct qaaf_qmc_block) / 8 + 1 >= _QAAF_MAX);
+
 #endif /* ASM_KVM_HOST_ARM64_TYPES_H */
diff --git a/arch/s390/include/asm/sae.h b/arch/s390/include/asm/sae.h
index 1d9a16b91b23..f6f79443d6ce 100644
--- a/arch/s390/include/asm/sae.h
+++ b/arch/s390/include/asm/sae.h
@@ -110,5 +110,33 @@ static __always_inline void lasrm(struct kvm_sae_save_area *save_area)
 	);
 }
 
+/**
+ * qaaf() - Query Available Arm Features
+ * @gr0: QAAF function code, placed in greg 0
+ * @qaaf_block: Pointer to the page for the output
+ *
+ * Perform QAAF. The result ins written to qaaf_block.
+ */
+static __always_inline void qaaf(u64 gr0, union qaaf_block *qaaf_block)
+{
+	asm volatile(
+		"	lgr	0,%[r0]\n"
+		"	.insn	rre,0xb9ad0000,%[r1],0"
+		: "=m"(*qaaf_block)
+		: [r1] "a"(qaaf_block), [r0] "d"(gr0)
+		: "r0"
+	);
+}
+
+/**
+ * qaaf_qmc() - Query Available Arm Features for Model Capabilities
+ * @qmc: Pointer to qaaf_qmc_block structure to receive model capabilities
+ *
+ */
+static __always_inline void qaaf_qmc(struct qaaf_qmc_block *qmc)
+{
+	qaaf(QAAF_FC_QMC, (union qaaf_block *)qmc);
+}
+
 #endif /* !__ASSEMBLER__ */
 #endif /* __ASM_S390_SAE_H */
diff --git a/arch/s390/tools/opcodes.txt b/arch/s390/tools/opcodes.txt
index 18af14071290..fd5483107961 100644
--- a/arch/s390/tools/opcodes.txt
+++ b/arch/s390/tools/opcodes.txt
@@ -600,6 +600,7 @@ b9a7	stiasrm	RRE_R0
 b9aa	lptea	RRF_RURR2
 b9ab	essa	RRF_U0RR
 b9ac	irbm	RRE_RR
+b9ad	qaaf	RRE_R0
 b9ae	rrbm	RRE_RR
 b9af	pfmf	RRE_RR
 b9b0	cu14	RRF_U0RR
-- 
2.53.0

[PATCH v1 15/26] s390: Add functions to query arm guest time

[Steffen Eiden]

Add functions to convert between ARM guest time (LSB0) and s390 host
time (MSB0) using new ptff function codes.

Co-developed-by: Nico Boehr <nrb@linux.ibm.com>
Signed-off-by: Nico Boehr <nrb@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/timex.h | 49 +++++++++++++++++++++++++++++++++++
 arch/s390/kernel/time.c       |  1 +
 arch/s390/kvm/arm64/arm.c     |  9 ++++++-
 3 files changed, 58 insertions(+), 1 deletion(-)

diff --git a/arch/s390/include/asm/timex.h b/arch/s390/include/asm/timex.h
index 49447b40f038..9ec22a28bbda 100644
--- a/arch/s390/include/asm/timex.h
+++ b/arch/s390/include/asm/timex.h
@@ -99,6 +99,8 @@ extern unsigned char ptff_function_mask[16];
 #define PTFF_QSI	0x02	/* query steering information */
 #define PTFF_QPT	0x03	/* query physical clock */
 #define PTFF_QUI	0x04	/* query UTC information */
+#define PTFF_QAGTO	0x10	/* query arm guest time offset */
+#define PTFF_QAGPT	0x11	/* query arm guest physical time offset */
 #define PTFF_ATO	0x40	/* adjust tod offset */
 #define PTFF_STO	0x41	/* set tod offset */
 #define PTFF_SFS	0x42	/* set fine steering rate */
@@ -136,6 +138,17 @@ struct ptff_qui {
 	unsigned int pad_0x5c[41];
 } __packed;
 
+/*
+ * Query Arm Guest Time
+ * used for:
+ *	- Query Arm Guest Time Offset
+ *	- Query Arm Guest Physical Time
+ */
+struct ptff_qagt {
+	u64 in;
+	u64 out;
+};
+
 /*
  * ptff - Perform timing facility function
  * @ptff_block: Pointer to ptff parameter block
@@ -286,4 +299,40 @@ static inline int tod_after_eq(unsigned long a, unsigned long b)
 	return a >= b;
 }
 
+/*
+ * ptff_qagto() -  Query Arm Guest Time Offset
+ *
+ * @physical_time: Arm guest physical time in MSb 0
+ *
+ * Converts Arm guest physical time in MSb 0 bit ordering
+ * into the Arm guest offset in LSb 0 bit ordering.
+ *
+ * Return: Arm guest time offset in LSb 0
+ */
+static inline u64 ptff_qagto(u64 physical_time)
+{
+	struct ptff_qagt qagto = { .in = physical_time };
+
+	ptff(&qagto, sizeof(qagto), PTFF_QAGTO);
+	return qagto.out;
+}
+
+/*
+ * ptff_qagpt() - Query Arm Guest Physical Time
+ *
+ * @guest_time_offset: Arm guest time offset in MSb 0
+ *
+ * Converts Arm guest offset in MSb 0 bit ordering
+ * into the Arm guest physical time in LSb 0 bit ordering.
+ *
+ * Return: Arm guest physical time in LSb 0
+ * */
+static inline u64 ptff_qagpt(u64 guest_time_offset)
+{
+	struct ptff_qagt qagpt = { .in = guest_time_offset };
+
+	ptff(&qagpt, sizeof(qagpt), PTFF_QAGPT);
+	return qagpt.out;
+}
+
 #endif
diff --git a/arch/s390/kernel/time.c b/arch/s390/kernel/time.c
index bd0df61d1907..2b989bebd220 100644
--- a/arch/s390/kernel/time.c
+++ b/arch/s390/kernel/time.c
@@ -65,6 +65,7 @@ ATOMIC_NOTIFIER_HEAD(s390_epoch_delta_notifier);
 EXPORT_SYMBOL(s390_epoch_delta_notifier);
 
 unsigned char ptff_function_mask[16];
+EXPORT_SYMBOL(ptff_function_mask);
 
 static unsigned long lpar_offset;
 static unsigned long initial_leap_seconds;
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index bf0866659421..636bbeda98a8 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -692,8 +692,15 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
 
 static int __init kvm_s390_arm64_init(void)
 {
-	if (!sclp.has_aef)
+	if (!sclp.has_aef) {
+		pr_info("SAE is not available\n");
 		return -ENXIO;
+	}
+
+	if (!(ptff_query(PTFF_QAGTO) && ptff_query(PTFF_QAGPT))) {
+		pr_info("PTFF for arm on s390 is not available\n");
+		return -ENXIO;
+	}
 
 	return kvm_init_with_dev(sizeof(struct kvm_vcpu), 0, THIS_MODULE,
 				 KVM_DEV_NAME, MISC_DYNAMIC_MINOR);
-- 
2.53.0

[PATCH v1 16/26] KVM: s390: arm64: Add sysreg related functions and definitions

[Steffen Eiden]

Add guest sysreg access macros in asm/kvm_host_arm64.h using
easr()/sasr()-based helpers for sysregs that map to the SAE save area.

Also add the guest-visible sysreg enum and per-vCPU/per-VM storage for
state that is not directly covered by the save area, such as CLIDR_EL1,
CSSELR_EL1, MPIDR_EL1, and VM-wide ID register state.

This lays out the header-side definitions needed to ensure compilation
success during for the share-code-patches and later sysreg handling
during vCPU setup and runtime.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host_arm64.h | 165 +++++++++++++++++++++++++
 1 file changed, 165 insertions(+)

diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index 8c3214c5b004..d6d9e3ad7a8e 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -44,10 +44,19 @@ struct kvm_vcpu_arch {
 	struct kvm_sae_save_area save_area;
 	struct kvm_cpu_context ctxt;
 
+	/* Guest system registers not part of save area or ID registers */
+	u64 sys_reg_clidr_el1;
+	u64 sys_reg_csselr_el1;
+	/* Per-vcpu CCSIDR override or NULL */
+	u32 *ccsidr;
+
 	u32 host_acrs[NUM_ACRS];
 
 	/* Hypervisor Configuration Register */
 	u64 hcr_elz;
+	u64 hcrx_elz;
+
+	u64 mpidr;
 
 	/* Configuration flags, set once and for all before the vcpu can run */
 	u8 cflags;
@@ -209,4 +218,160 @@ static inline void kvm_arch_async_page_present_queued(struct kvm_vcpu *vcpu)
 
 #define kvm_supports_32bit_el0() false
 
+#define vcpu_read_sys_reg(_v, _r) 0xbad1234bad
+#define vcpu_write_sys_reg(_v, _p, _r) ((void)0)
+
+#define __vcpu_sys_reg(__vcpu, __reg) \
+	vcpu_read_sys_reg(__vcpu, __reg)
+
+#define __vcpu_assign_sys_reg(__vcpu, __reg, __val) \
+	vcpu_write_sys_reg(__vcpu, __val, __reg)
+
+/* Read, modify and write system register
+ *
+ */
+#define _vcpu_rmw_sys_reg(C, V, OP, R) \
+({ \
+	u64 __val = vcpu_read_sys_reg(C, R); \
+	__val OP V; \
+	vcpu_write_sys_reg(C, __val, R); \
+})
+
+/**
+ * _vcpu_read_sys_reg() - read a guest sysreg with easr
+ * - R - sysreg id; must be readable by easr; must be compile time constant
+ *
+ *   if SYSREGS_ON_CPU: proceed with flags = 0
+ *   otherwise:         proceed with either
+ *                         read:  flags = EASR_FLAG_SA
+ *                         write: flags = SASR_FLAG_INITIALIZED
+ *
+ */
+#define _vcpu_read_sys_reg(C, R) \
+	({	BUILD_BUG_ON(!__builtin_constant_p((R))); \
+		BUG_ON(vcpu_is_loaded(C) && smp_processor_id() != (C)->cpu); \
+		(vcpu_is_loaded(C)) \
+			? __vcpu_read_sr((C), (R), 0) \
+			: __vcpu_read_sr((C), (R), EASR_FLAG_SA); })
+
+/**
+ * _vcpu_write_sys_reg() - write a guest sysreg with sasr
+ * - R - sysreg id; must be readable by sasr; must be compile time constant
+
+ *   if SYSREGS_ON_CPU: proceed with flags = 0
+ *   otherwise:         proceed with either
+ *                         read:  flags = EASR_FLAG_SA
+ *                         write: flags = SASR_FLAG_INITIALIZED
+ */
+#define _vcpu_write_sys_reg(C, V, R) \
+	({	BUILD_BUG_ON(!__builtin_constant_p((R))); \
+		BUG_ON(vcpu_is_loaded(C) && smp_processor_id() != (C)->cpu); \
+		(vcpu_is_loaded(C)) \
+			? __vcpu_write_sr((C), (V), (R), 0) \
+			: __vcpu_write_sr((C), (V), (R), SASR_FLAG_INITIALIZED); })
+
+/* Forward to easr / sasr
+ * assert that F and R are constant
+ */
+#define __vcpu_read_sr(C, R, F) \
+	({	BUILD_BUG_ON(!__builtin_constant_p((R))); \
+		BUILD_BUG_ON(!__builtin_constant_p((F))); \
+		easr((R), &(C)->arch.save_area, (F)); })
+
+#define __vcpu_write_sr(C, V, R, F) \
+	({	BUILD_BUG_ON(!__builtin_constant_p((R))); \
+		BUILD_BUG_ON(!__builtin_constant_p((F))); \
+		sasr((R), (V), &(C)->arch.save_area, (F)); })
+
+#define SR_GROUP(NAME, ...)	\
+	__##NAME##_BEGIN__,	\
+	__VA_ARGS__		\
+	__##NAME##_END__
+
+/** enum vcpu_sysreg - available guest sysregs
+ *
+ * Contains all arm64 guest-syregs supported by s390.
+ */
+enum vcpu_sysreg {
+	__INVALID_SYSREG__, /* 0 is reserved as an invalid value */
+
+	/* EL 0,1 Register from state description in order of appearance */
+	SR_GROUP(STATE_DESC,
+	CNTP_CTL_EL0,
+	CNTV_CTL_EL0,
+	CONTEXTIDR_EL1,
+	SP_EL1,
+	),
+
+	/* EL 0,1 Register requiring special handling. */
+	SR_GROUP(SPECIAL,
+	CSSELR_EL1,
+	CLIDR_EL1,
+	MPIDR_EL1,
+	),
+
+	/* EL 0,1 register from save area in order of appearance */
+	SR_GROUP(SAVE_AREA,
+	ACTLR_EL1,
+	AFSR0_EL1,
+	AFSR1_EL1,
+	CNTFRQ_EL0,
+	CNTP_CVAL_EL0,
+	CNTV_CVAL_EL0,
+	DISR_EL1,
+	MIDR_EL1,
+	OSLSR_EL1,
+	PAR_EL1,
+	OSLAR_EL1,
+	SCTLR_EL1,
+	CPACR_EL1,
+	VBAR_EL1,
+	SPSR_EL1,
+	ELR_EL1,
+	ESR_EL1,
+	TCR_EL1,
+	MAIR_EL1,
+	TTBR0_EL1,
+	TTBR1_EL1,
+	FAR_EL1,
+	TPIDR_EL0,
+	TPIDR_EL1,
+	TPIDRRO_EL0,
+	CNTKCTL_EL1,
+	ZCR_EL1,
+	SCXTNUM_EL0,
+	SCXTNUM_EL1,
+	APIBKEYLO_EL1,
+	APIBKEYHI_EL1,
+	APIAKEYLO_EL1,
+	APIAKEYHI_EL1,
+	APGAKEYLO_EL1,
+	APGAKEYHI_EL1,
+	APDBKEYLO_EL1,
+	APDBKEYHI_EL1,
+	APDAKEYLO_EL1,
+	APDAKEYHI_EL1,
+	MDSCR_EL1,
+	),
+
+	NR_SYS_REGS /* Nothing after this line! */
+};
+
+void vcpu_write_host_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg);
+u64 vcpu_read_host_sys_reg(const struct kvm_vcpu *vcpu, int reg);
+
+#define kvm_debug_handle_oslar(_v, _val) /* debug not implemented yet*/
+
+static inline u8 kvm_arm_pmu_get_pmuver_limit(void)
+{
+	return 0;
+}
+
+int __init kvm_sys_reg_table_init(void);
+
+static inline u64 kvm_sanitised_host_ftr_reg(u32 id)
+{
+	return 0xbad1234bad;
+}
+
 #endif /* ASM_KVM_HOST_ARM64_H */
-- 
2.53.0

[PATCH v1 17/26] arm64: Extract cputype definitions.

[Steffen Eiden]

Move CPU type definitions from arch/arm64/include/asm/cputype.h to
include/arch/arm64/asm/cputype-defs.h to prepare sharing with other
architectures. No functional changes.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/cputype.h              | 246 +-----------------
 .../arch/arm64/asm/cputype-defs.h             |  92 +------
 2 files changed, 6 insertions(+), 332 deletions(-)
 copy arch/arm64/include/asm/cputype.h => include/arch/arm64/asm/cputype-defs.h (85%)

diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h
index 7b518e81dd15..67765cdbce84 100644
--- a/arch/arm64/include/asm/cputype.h
+++ b/arch/arm64/include/asm/cputype.h
@@ -5,251 +5,7 @@
 #ifndef __ASM_CPUTYPE_H
 #define __ASM_CPUTYPE_H
 
-#define INVALID_HWID		ULONG_MAX
-
-#define MPIDR_UP_BITMASK	(0x1 << 30)
-#define MPIDR_MT_BITMASK	(0x1 << 24)
-#define MPIDR_HWID_BITMASK	UL(0xff00ffffff)
-
-#define MPIDR_LEVEL_BITS_SHIFT	3
-#define MPIDR_LEVEL_BITS	(1 << MPIDR_LEVEL_BITS_SHIFT)
-#define MPIDR_LEVEL_MASK	((1 << MPIDR_LEVEL_BITS) - 1)
-
-#define MPIDR_LEVEL_SHIFT(level) \
-	(((1 << level) >> 1) << MPIDR_LEVEL_BITS_SHIFT)
-
-#define MPIDR_AFFINITY_LEVEL(mpidr, level) \
-	((mpidr >> MPIDR_LEVEL_SHIFT(level)) & MPIDR_LEVEL_MASK)
-
-#define MIDR_REVISION_MASK	0xf
-#define MIDR_REVISION(midr)	((midr) & MIDR_REVISION_MASK)
-#define MIDR_PARTNUM_SHIFT	4
-#define MIDR_PARTNUM_MASK	(0xfff << MIDR_PARTNUM_SHIFT)
-#define MIDR_PARTNUM(midr)	\
-	(((midr) & MIDR_PARTNUM_MASK) >> MIDR_PARTNUM_SHIFT)
-#define MIDR_ARCHITECTURE_SHIFT	16
-#define MIDR_ARCHITECTURE_MASK	(0xf << MIDR_ARCHITECTURE_SHIFT)
-#define MIDR_ARCHITECTURE(midr)	\
-	(((midr) & MIDR_ARCHITECTURE_MASK) >> MIDR_ARCHITECTURE_SHIFT)
-#define MIDR_VARIANT_SHIFT	20
-#define MIDR_VARIANT_MASK	(0xf << MIDR_VARIANT_SHIFT)
-#define MIDR_VARIANT(midr)	\
-	(((midr) & MIDR_VARIANT_MASK) >> MIDR_VARIANT_SHIFT)
-#define MIDR_IMPLEMENTOR_SHIFT	24
-#define MIDR_IMPLEMENTOR_MASK	(0xffU << MIDR_IMPLEMENTOR_SHIFT)
-#define MIDR_IMPLEMENTOR(midr)	\
-	(((midr) & MIDR_IMPLEMENTOR_MASK) >> MIDR_IMPLEMENTOR_SHIFT)
-
-#define MIDR_CPU_MODEL(imp, partnum) \
-	((_AT(u32, imp)		<< MIDR_IMPLEMENTOR_SHIFT) | \
-	(0xf			<< MIDR_ARCHITECTURE_SHIFT) | \
-	((partnum)		<< MIDR_PARTNUM_SHIFT))
-
-#define MIDR_CPU_VAR_REV(var, rev) \
-	(((var)	<< MIDR_VARIANT_SHIFT) | (rev))
-
-#define MIDR_CPU_MODEL_MASK (MIDR_IMPLEMENTOR_MASK | MIDR_PARTNUM_MASK | \
-			     MIDR_ARCHITECTURE_MASK)
-
-#define ARM_CPU_IMP_ARM			0x41
-#define ARM_CPU_IMP_APM			0x50
-#define ARM_CPU_IMP_CAVIUM		0x43
-#define ARM_CPU_IMP_BRCM		0x42
-#define ARM_CPU_IMP_QCOM		0x51
-#define ARM_CPU_IMP_NVIDIA		0x4E
-#define ARM_CPU_IMP_FUJITSU		0x46
-#define ARM_CPU_IMP_HISI		0x48
-#define ARM_CPU_IMP_APPLE		0x61
-#define ARM_CPU_IMP_AMPERE		0xC0
-#define ARM_CPU_IMP_MICROSOFT		0x6D
-
-#define ARM_CPU_PART_AEM_V8		0xD0F
-#define ARM_CPU_PART_FOUNDATION		0xD00
-#define ARM_CPU_PART_CORTEX_A57		0xD07
-#define ARM_CPU_PART_CORTEX_A72		0xD08
-#define ARM_CPU_PART_CORTEX_A53		0xD03
-#define ARM_CPU_PART_CORTEX_A73		0xD09
-#define ARM_CPU_PART_CORTEX_A75		0xD0A
-#define ARM_CPU_PART_CORTEX_A35		0xD04
-#define ARM_CPU_PART_CORTEX_A55		0xD05
-#define ARM_CPU_PART_CORTEX_A76		0xD0B
-#define ARM_CPU_PART_NEOVERSE_N1	0xD0C
-#define ARM_CPU_PART_CORTEX_A77		0xD0D
-#define ARM_CPU_PART_CORTEX_A76AE	0xD0E
-#define ARM_CPU_PART_NEOVERSE_V1	0xD40
-#define ARM_CPU_PART_CORTEX_A78		0xD41
-#define ARM_CPU_PART_CORTEX_A78AE	0xD42
-#define ARM_CPU_PART_CORTEX_X1		0xD44
-#define ARM_CPU_PART_CORTEX_A510	0xD46
-#define ARM_CPU_PART_CORTEX_A520	0xD80
-#define ARM_CPU_PART_CORTEX_A710	0xD47
-#define ARM_CPU_PART_CORTEX_A715	0xD4D
-#define ARM_CPU_PART_CORTEX_X2		0xD48
-#define ARM_CPU_PART_NEOVERSE_N2	0xD49
-#define ARM_CPU_PART_CORTEX_A78C	0xD4B
-#define ARM_CPU_PART_CORTEX_X1C		0xD4C
-#define ARM_CPU_PART_CORTEX_X3		0xD4E
-#define ARM_CPU_PART_NEOVERSE_V2	0xD4F
-#define ARM_CPU_PART_CORTEX_A720	0xD81
-#define ARM_CPU_PART_CORTEX_X4		0xD82
-#define ARM_CPU_PART_NEOVERSE_V3AE	0xD83
-#define ARM_CPU_PART_NEOVERSE_V3	0xD84
-#define ARM_CPU_PART_CORTEX_X925	0xD85
-#define ARM_CPU_PART_CORTEX_A725	0xD87
-#define ARM_CPU_PART_CORTEX_A720AE	0xD89
-#define ARM_CPU_PART_NEOVERSE_N3	0xD8E
-#define ARM_CPU_PART_C1_PRO		0xD8B
-
-#define APM_CPU_PART_XGENE		0x000
-#define APM_CPU_VAR_POTENZA		0x00
-
-#define CAVIUM_CPU_PART_THUNDERX	0x0A1
-#define CAVIUM_CPU_PART_THUNDERX_81XX	0x0A2
-#define CAVIUM_CPU_PART_THUNDERX_83XX	0x0A3
-#define CAVIUM_CPU_PART_THUNDERX2	0x0AF
-/* OcteonTx2 series */
-#define CAVIUM_CPU_PART_OCTX2_98XX	0x0B1
-#define CAVIUM_CPU_PART_OCTX2_96XX	0x0B2
-#define CAVIUM_CPU_PART_OCTX2_95XX	0x0B3
-#define CAVIUM_CPU_PART_OCTX2_95XXN	0x0B4
-#define CAVIUM_CPU_PART_OCTX2_95XXMM	0x0B5
-#define CAVIUM_CPU_PART_OCTX2_95XXO	0x0B6
-
-#define BRCM_CPU_PART_BRAHMA_B53	0x100
-#define BRCM_CPU_PART_VULCAN		0x516
-
-#define QCOM_CPU_PART_FALKOR_V1		0x800
-#define QCOM_CPU_PART_FALKOR		0xC00
-#define QCOM_CPU_PART_KRYO		0x200
-#define QCOM_CPU_PART_KRYO_2XX_GOLD	0x800
-#define QCOM_CPU_PART_KRYO_2XX_SILVER	0x801
-#define QCOM_CPU_PART_KRYO_3XX_GOLD	0x802
-#define QCOM_CPU_PART_KRYO_3XX_SILVER	0x803
-#define QCOM_CPU_PART_KRYO_4XX_GOLD	0x804
-#define QCOM_CPU_PART_KRYO_4XX_SILVER	0x805
-#define QCOM_CPU_PART_ORYON_X1		0x001
-
-#define NVIDIA_CPU_PART_DENVER		0x003
-#define NVIDIA_CPU_PART_CARMEL		0x004
-#define NVIDIA_CPU_PART_OLYMPUS		0x010
-
-#define FUJITSU_CPU_PART_A64FX		0x001
-
-#define HISI_CPU_PART_TSV110		0xD01
-#define HISI_CPU_PART_HIP09			0xD02
-#define HISI_CPU_PART_HIP12		0xD06
-
-#define APPLE_CPU_PART_M1_ICESTORM	0x022
-#define APPLE_CPU_PART_M1_FIRESTORM	0x023
-#define APPLE_CPU_PART_M1_ICESTORM_PRO	0x024
-#define APPLE_CPU_PART_M1_FIRESTORM_PRO	0x025
-#define APPLE_CPU_PART_M1_ICESTORM_MAX	0x028
-#define APPLE_CPU_PART_M1_FIRESTORM_MAX	0x029
-#define APPLE_CPU_PART_M2_BLIZZARD	0x032
-#define APPLE_CPU_PART_M2_AVALANCHE	0x033
-#define APPLE_CPU_PART_M2_BLIZZARD_PRO	0x034
-#define APPLE_CPU_PART_M2_AVALANCHE_PRO	0x035
-#define APPLE_CPU_PART_M2_BLIZZARD_MAX	0x038
-#define APPLE_CPU_PART_M2_AVALANCHE_MAX	0x039
-
-#define AMPERE_CPU_PART_AMPERE1		0xAC3
-#define AMPERE_CPU_PART_AMPERE1A	0xAC4
-
-#define MICROSOFT_CPU_PART_AZURE_COBALT_100	0xD49 /* Based on r0p0 of ARM Neoverse N2 */
-
-#define MIDR_CORTEX_A53 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A53)
-#define MIDR_CORTEX_A57 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A57)
-#define MIDR_CORTEX_A72 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A72)
-#define MIDR_CORTEX_A73 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A73)
-#define MIDR_CORTEX_A75 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A75)
-#define MIDR_CORTEX_A35 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A35)
-#define MIDR_CORTEX_A55 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A55)
-#define MIDR_CORTEX_A76	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A76)
-#define MIDR_NEOVERSE_N1 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_N1)
-#define MIDR_CORTEX_A77	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A77)
-#define MIDR_CORTEX_A76AE	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A76AE)
-#define MIDR_NEOVERSE_V1	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_V1)
-#define MIDR_CORTEX_A78	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A78)
-#define MIDR_CORTEX_A78AE	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A78AE)
-#define MIDR_CORTEX_X1	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_X1)
-#define MIDR_CORTEX_A510 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A510)
-#define MIDR_CORTEX_A520 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A520)
-#define MIDR_CORTEX_A710 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A710)
-#define MIDR_CORTEX_A715 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A715)
-#define MIDR_CORTEX_X2 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_X2)
-#define MIDR_NEOVERSE_N2 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_N2)
-#define MIDR_CORTEX_A78C	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A78C)
-#define MIDR_CORTEX_X1C	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_X1C)
-#define MIDR_CORTEX_X3 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_X3)
-#define MIDR_NEOVERSE_V2 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_V2)
-#define MIDR_CORTEX_A720 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A720)
-#define MIDR_CORTEX_X4 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_X4)
-#define MIDR_NEOVERSE_V3AE	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_V3AE)
-#define MIDR_NEOVERSE_V3 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_V3)
-#define MIDR_CORTEX_X925 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_X925)
-#define MIDR_CORTEX_A725 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A725)
-#define MIDR_CORTEX_A720AE MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A720AE)
-#define MIDR_NEOVERSE_N3 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_N3)
-#define MIDR_C1_PRO MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_C1_PRO)
-#define MIDR_THUNDERX	MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX)
-#define MIDR_THUNDERX_81XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_81XX)
-#define MIDR_THUNDERX_83XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_83XX)
-#define MIDR_OCTX2_98XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_OCTX2_98XX)
-#define MIDR_OCTX2_96XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_OCTX2_96XX)
-#define MIDR_OCTX2_95XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_OCTX2_95XX)
-#define MIDR_OCTX2_95XXN MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_OCTX2_95XXN)
-#define MIDR_OCTX2_95XXMM MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_OCTX2_95XXMM)
-#define MIDR_OCTX2_95XXO MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_OCTX2_95XXO)
-#define MIDR_CAVIUM_THUNDERX2 MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX2)
-#define MIDR_BRAHMA_B53 MIDR_CPU_MODEL(ARM_CPU_IMP_BRCM, BRCM_CPU_PART_BRAHMA_B53)
-#define MIDR_BRCM_VULCAN MIDR_CPU_MODEL(ARM_CPU_IMP_BRCM, BRCM_CPU_PART_VULCAN)
-#define MIDR_QCOM_FALKOR_V1 MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_FALKOR_V1)
-#define MIDR_QCOM_FALKOR MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_FALKOR)
-#define MIDR_QCOM_KRYO MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_KRYO)
-#define MIDR_QCOM_KRYO_2XX_GOLD MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_KRYO_2XX_GOLD)
-#define MIDR_QCOM_KRYO_2XX_SILVER MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_KRYO_2XX_SILVER)
-#define MIDR_QCOM_KRYO_3XX_GOLD MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_KRYO_3XX_GOLD)
-#define MIDR_QCOM_KRYO_3XX_SILVER MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_KRYO_3XX_SILVER)
-#define MIDR_QCOM_KRYO_4XX_GOLD MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_KRYO_4XX_GOLD)
-#define MIDR_QCOM_KRYO_4XX_SILVER MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_KRYO_4XX_SILVER)
-#define MIDR_QCOM_ORYON_X1 MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_ORYON_X1)
-
-/*
- * NOTES:
- * - Qualcomm Kryo 5XX Prime / Gold ID themselves as MIDR_CORTEX_A77
- * - Qualcomm Kryo 5XX Silver IDs itself as MIDR_QCOM_KRYO_4XX_SILVER
- * - Qualcomm Kryo 6XX Prime IDs itself as MIDR_CORTEX_X1
- * - Qualcomm Kryo 6XX Gold IDs itself as ARM_CPU_PART_CORTEX_A78
- * - Qualcomm Kryo 6XX Silver IDs itself as MIDR_CORTEX_A55
- */
-
-#define MIDR_NVIDIA_DENVER MIDR_CPU_MODEL(ARM_CPU_IMP_NVIDIA, NVIDIA_CPU_PART_DENVER)
-#define MIDR_NVIDIA_CARMEL MIDR_CPU_MODEL(ARM_CPU_IMP_NVIDIA, NVIDIA_CPU_PART_CARMEL)
-#define MIDR_NVIDIA_OLYMPUS MIDR_CPU_MODEL(ARM_CPU_IMP_NVIDIA, NVIDIA_CPU_PART_OLYMPUS)
-#define MIDR_FUJITSU_A64FX MIDR_CPU_MODEL(ARM_CPU_IMP_FUJITSU, FUJITSU_CPU_PART_A64FX)
-#define MIDR_HISI_TSV110 MIDR_CPU_MODEL(ARM_CPU_IMP_HISI, HISI_CPU_PART_TSV110)
-#define MIDR_HISI_HIP09 MIDR_CPU_MODEL(ARM_CPU_IMP_HISI, HISI_CPU_PART_HIP09)
-#define MIDR_HISI_HIP12 MIDR_CPU_MODEL(ARM_CPU_IMP_HISI, HISI_CPU_PART_HIP12)
-#define MIDR_APPLE_M1_ICESTORM MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M1_ICESTORM)
-#define MIDR_APPLE_M1_FIRESTORM MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M1_FIRESTORM)
-#define MIDR_APPLE_M1_ICESTORM_PRO MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M1_ICESTORM_PRO)
-#define MIDR_APPLE_M1_FIRESTORM_PRO MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M1_FIRESTORM_PRO)
-#define MIDR_APPLE_M1_ICESTORM_MAX MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M1_ICESTORM_MAX)
-#define MIDR_APPLE_M1_FIRESTORM_MAX MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M1_FIRESTORM_MAX)
-#define MIDR_APPLE_M2_BLIZZARD MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M2_BLIZZARD)
-#define MIDR_APPLE_M2_AVALANCHE MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M2_AVALANCHE)
-#define MIDR_APPLE_M2_BLIZZARD_PRO MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M2_BLIZZARD_PRO)
-#define MIDR_APPLE_M2_AVALANCHE_PRO MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M2_AVALANCHE_PRO)
-#define MIDR_APPLE_M2_BLIZZARD_MAX MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M2_BLIZZARD_MAX)
-#define MIDR_APPLE_M2_AVALANCHE_MAX MIDR_CPU_MODEL(ARM_CPU_IMP_APPLE, APPLE_CPU_PART_M2_AVALANCHE_MAX)
-#define MIDR_AMPERE1 MIDR_CPU_MODEL(ARM_CPU_IMP_AMPERE, AMPERE_CPU_PART_AMPERE1)
-#define MIDR_AMPERE1A MIDR_CPU_MODEL(ARM_CPU_IMP_AMPERE, AMPERE_CPU_PART_AMPERE1A)
-#define MIDR_MICROSOFT_AZURE_COBALT_100 MIDR_CPU_MODEL(ARM_CPU_IMP_MICROSOFT, MICROSOFT_CPU_PART_AZURE_COBALT_100)
-
-/* Fujitsu Erratum 010001 affects A64FX 1.0 and 1.1, (v0r0 and v1r0) */
-#define MIDR_FUJITSU_ERRATUM_010001		MIDR_FUJITSU_A64FX
-#define MIDR_FUJITSU_ERRATUM_010001_MASK	(~MIDR_CPU_VAR_REV(1, 0))
-#define TCR_CLEAR_FUJITSU_ERRATUM_010001	(TCR_EL1_NFD1 | TCR_EL1_NFD0)
+#include <asm/cputype-defs.h>
 
 #ifndef __ASSEMBLER__
 
diff --git a/arch/arm64/include/asm/cputype.h b/include/arch/arm64/asm/cputype-defs.h
similarity index 85%
copy from arch/arm64/include/asm/cputype.h
copy to include/arch/arm64/asm/cputype-defs.h
index 7b518e81dd15..6f5279aaaa19 100644
--- a/arch/arm64/include/asm/cputype.h
+++ b/include/arch/arm64/asm/cputype-defs.h
@@ -1,9 +1,8 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (C) 2012 ARM Ltd.
- */
-#ifndef __ASM_CPUTYPE_H
-#define __ASM_CPUTYPE_H
+#ifndef __ASM_CPUTYPE_DEFS_H
+#define __ASM_CPUTYPE_DEFS_H
+
+#include <linux/types.h>
 
 #define INVALID_HWID		ULONG_MAX
 
@@ -251,85 +250,4 @@
 #define MIDR_FUJITSU_ERRATUM_010001_MASK	(~MIDR_CPU_VAR_REV(1, 0))
 #define TCR_CLEAR_FUJITSU_ERRATUM_010001	(TCR_EL1_NFD1 | TCR_EL1_NFD0)
 
-#ifndef __ASSEMBLER__
-
-#include <asm/sysreg.h>
-
-#define read_cpuid(reg)			read_sysreg_s(SYS_ ## reg)
-
-/*
- * The CPU ID never changes at run time, so we might as well tell the
- * compiler that it's constant.  Use this function to read the CPU ID
- * rather than directly reading processor_id or read_cpuid() directly.
- */
-static inline u32 __attribute_const__ read_cpuid_id(void)
-{
-	return read_cpuid(MIDR_EL1);
-}
-
-/*
- * Represent a range of MIDR values for a given CPU model and a
- * range of variant/revision values.
- *
- * @model	- CPU model as defined by MIDR_CPU_MODEL
- * @rv_min	- Minimum value for the revision/variant as defined by
- *		  MIDR_CPU_VAR_REV
- * @rv_max	- Maximum value for the variant/revision for the range.
- */
-struct midr_range {
-	u32 model;
-	u32 rv_min;
-	u32 rv_max;
-};
-
-#define MIDR_RANGE(m, v_min, r_min, v_max, r_max)		\
-	{							\
-		.model = m,					\
-		.rv_min = MIDR_CPU_VAR_REV(v_min, r_min),	\
-		.rv_max = MIDR_CPU_VAR_REV(v_max, r_max),	\
-	}
-
-#define MIDR_REV_RANGE(m, v, r_min, r_max) MIDR_RANGE(m, v, r_min, v, r_max)
-#define MIDR_REV(m, v, r) MIDR_RANGE(m, v, r, v, r)
-#define MIDR_ALL_VERSIONS(m) MIDR_RANGE(m, 0, 0, 0xf, 0xf)
-
-static inline bool midr_is_cpu_model_range(u32 midr, u32 model, u32 rv_min,
-					   u32 rv_max)
-{
-	u32 _model = midr & MIDR_CPU_MODEL_MASK;
-	u32 rv = midr & (MIDR_REVISION_MASK | MIDR_VARIANT_MASK);
-
-	return _model == model && rv >= rv_min && rv <= rv_max;
-}
-
-struct target_impl_cpu {
-	u64 midr;
-	u64 revidr;
-	u64 aidr;
-};
-
-bool cpu_errata_set_target_impl(u64 num, void *impl_cpus);
-bool is_midr_in_range_list(struct midr_range const *ranges);
-
-static inline u64 __attribute_const__ read_cpuid_mpidr(void)
-{
-	return read_cpuid(MPIDR_EL1);
-}
-
-static inline unsigned int __attribute_const__ read_cpuid_implementor(void)
-{
-	return MIDR_IMPLEMENTOR(read_cpuid_id());
-}
-
-static inline unsigned int __attribute_const__ read_cpuid_part_number(void)
-{
-	return MIDR_PARTNUM(read_cpuid_id());
-}
-
-static inline u32 __attribute_const__ read_cpuid_cachetype(void)
-{
-	return read_cpuid(CTR_EL0);
-}
-#endif /* __ASSEMBLER__ */
-
-#endif
+#endif /* __ASM_CPUTYPE_DEFS_H */
-- 
2.53.0

[PATCH v1 18/26] arm64: Extract cache definitions

[Steffen Eiden]

Move CPU type definitions from arch/arm64/include/asm/cache.h.h to
include/arch/arm64/asm/cache-defs.h to prepare sharing with other
architectures. No functional changes.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/cache.h      | 19 ++-----------------
 include/arch/arm64/asm/cache-defs.h | 22 ++++++++++++++++++++++
 2 files changed, 24 insertions(+), 17 deletions(-)
 create mode 100644 include/arch/arm64/asm/cache-defs.h

diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h
index 10a7ffadee3d..0f67fe470c29 100644
--- a/arch/arm64/include/asm/cache.h
+++ b/arch/arm64/include/asm/cache.h
@@ -5,26 +5,11 @@
 #ifndef __ASM_CACHE_H
 #define __ASM_CACHE_H
 
+#include <asm/cache-defs.h>
+
 #define L1_CACHE_SHIFT		(6)
 #define L1_CACHE_BYTES		(1 << L1_CACHE_SHIFT)
 
-#define CLIDR_LOUU_SHIFT	27
-#define CLIDR_LOC_SHIFT		24
-#define CLIDR_LOUIS_SHIFT	21
-
-#define CLIDR_LOUU(clidr)	(((clidr) >> CLIDR_LOUU_SHIFT) & 0x7)
-#define CLIDR_LOC(clidr)	(((clidr) >> CLIDR_LOC_SHIFT) & 0x7)
-#define CLIDR_LOUIS(clidr)	(((clidr) >> CLIDR_LOUIS_SHIFT) & 0x7)
-
-/* Ctypen, bits[3(n - 1) + 2 : 3(n - 1)], for n = 1 to 7 */
-#define CLIDR_CTYPE_SHIFT(level)	(3 * (level - 1))
-#define CLIDR_CTYPE_MASK(level)		(7 << CLIDR_CTYPE_SHIFT(level))
-#define CLIDR_CTYPE(clidr, level)	\
-	(((clidr) & CLIDR_CTYPE_MASK(level)) >> CLIDR_CTYPE_SHIFT(level))
-
-/* Ttypen, bits [2(n - 1) + 34 : 2(n - 1) + 33], for n = 1 to 7 */
-#define CLIDR_TTYPE_SHIFT(level)	(2 * ((level) - 1) + CLIDR_EL1_Ttypen_SHIFT)
-
 /*
  * Memory returned by kmalloc() may be used for DMA, so we must make
  * sure that all such allocations are cache aligned. Otherwise,
diff --git a/include/arch/arm64/asm/cache-defs.h b/include/arch/arm64/asm/cache-defs.h
new file mode 100644
index 000000000000..bb0ab69a9cd6
--- /dev/null
+++ b/include/arch/arm64/asm/cache-defs.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_CACHE_DEFS_H
+#define __ASM_CACHE_DEFS_H
+
+#define CLIDR_LOUU_SHIFT	27
+#define CLIDR_LOC_SHIFT		24
+#define CLIDR_LOUIS_SHIFT	21
+
+#define CLIDR_LOUU(clidr)	(((clidr) >> CLIDR_LOUU_SHIFT) & 0x7)
+#define CLIDR_LOC(clidr)	(((clidr) >> CLIDR_LOC_SHIFT) & 0x7)
+#define CLIDR_LOUIS(clidr)	(((clidr) >> CLIDR_LOUIS_SHIFT) & 0x7)
+
+/* Ctypen, bits[3(n - 1) + 2 : 3(n - 1)], for n = 1 to 7 */
+#define CLIDR_CTYPE_SHIFT(level)	(3 * (level - 1))
+#define CLIDR_CTYPE_MASK(level)		(7 << CLIDR_CTYPE_SHIFT(level))
+#define CLIDR_CTYPE(clidr, level)	\
+	(((clidr) & CLIDR_CTYPE_MASK(level)) >> CLIDR_CTYPE_SHIFT(level))
+
+/* Ttypen, bits [2(n - 1) + 34 : 2(n - 1) + 33], for n = 1 to 7 */
+#define CLIDR_TTYPE_SHIFT(level)	(2 * ((level) - 1) + CLIDR_EL1_Ttypen_SHIFT)
+
+#endif /* __ASM_CACHE_DEFS_H */
-- 
2.53.0

[PATCH v1 19/26] KVM: arm64: Share KVM feature detection macros

[Steffen Eiden]

Move kvm feature definitions from arch/arm64/include to a shared location
(include/kvm/arm64) to enable reuse by e.g. s390 for KVM/arm64 on s390
support.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_feature.h          | 60 +------------------
 .../asm => include/kvm/arm64}/kvm_feature.h   | 28 ++-------
 2 files changed, 5 insertions(+), 83 deletions(-)
 copy {arch/arm64/include/asm => include/kvm/arm64}/kvm_feature.h (76%)

diff --git a/arch/arm64/include/asm/kvm_feature.h b/arch/arm64/include/asm/kvm_feature.h
index 27a472d2343e..e1fb8b9f5ce2 100644
--- a/arch/arm64/include/asm/kvm_feature.h
+++ b/arch/arm64/include/asm/kvm_feature.h
@@ -2,65 +2,7 @@
 #ifndef __ARM64_KVM_FEATURE_H__
 #define __ARM64_KVM_FEATURE_H__
 
-#include <linux/types.h>
-#include <linux/bitfield.h>
-#include <asm/sysreg-defs.h>
-
-#define extract_id_field_unsigned(val, id, fld)				\
-	(FIELD_GET(id##_##fld##_MASK, (val)))
-
-#define extract_id_field_signed(val, id, fld)				\
-	({								\
-		u64 __val = extract_id_field_unsigned((val), id, fld);	\
-		sign_extend64(__val, id##_##fld##_WIDTH - 1);		\
-	})
-
-#define cmp_id_feat_signed(val, id, fld, op, limit)			\
-	(extract_id_field_signed((val), id, fld) op S64_SYS_FIELD_VALUE(id, fld, limit))
-
-#define cmp_id_feat_unsigned(val, id, fld, op, limit)			\
-	(extract_id_field_unsigned((val), id, fld) op (u64)SYS_FIELD_VALUE(id, fld, limit))
-
-#define cmp_id_feat(val, id, fld, op, limit)				\
-	(id##_##fld##_SIGNED ?						\
-	 cmp_id_feat_signed(val, id, fld, op, limit) :			\
-	 cmp_id_feat_unsigned(val, id, fld, op, limit))
-
-#define id_has_feat(val, id, fld, limit)				\
-	cmp_id_feat(val, id, fld, >=, limit)
-
-#define id_has_feat_enum(val, id, fld, variant)				\
-	cmp_id_feat_unsigned(val, id, fld, ==, variant)
-
-#define id_has_feat_range(val, id, fld, min, max)			\
-	(cmp_id_feat(val, id, fld, >=, min) &&				\
-	cmp_id_feat(val, id, fld, <=, max))
-
-#define __kvm_has_feat(kvm, id, fld, limit)				\
-	id_has_feat(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, limit)
-
-#define kvm_has_feat(kvm, ...) __kvm_has_feat(kvm, __VA_ARGS__)
-
-#define __kvm_has_feat_enum(kvm, id, fld, val)				\
-	id_has_feat_enum(kvm_read_vm_id_reg((kvm), SYS_##id), id, fld, val)
-
-#define kvm_has_feat_enum(kvm, ...) __kvm_has_feat_enum(kvm, __VA_ARGS__)
-
-
-/* Check for a given level of PAuth support */
-#define kvm_has_pauth(k, l)						\
-	({								\
-		bool pa, pi, pa3;					\
-									\
-		pa  = kvm_has_feat((k), ID_AA64ISAR1_EL1, APA, l);	\
-		pa &= kvm_has_feat((k), ID_AA64ISAR1_EL1, GPA, IMP);	\
-		pi  = kvm_has_feat((k), ID_AA64ISAR1_EL1, API, l);	\
-		pi &= kvm_has_feat((k), ID_AA64ISAR1_EL1, GPI, IMP);	\
-		pa3  = kvm_has_feat((k), ID_AA64ISAR2_EL1, APA3, l);	\
-		pa3 &= kvm_has_feat((k), ID_AA64ISAR2_EL1, GPA3, IMP);	\
-									\
-		(pa + pi + pa3) == 1;					\
-	})
+#include <kvm/arm64/kvm_feature.h>
 
 #define kvm_has_fpmr(k)					\
 	(system_supports_fpmr() &&			\
diff --git a/arch/arm64/include/asm/kvm_feature.h b/include/kvm/arm64/kvm_feature.h
similarity index 76%
copy from arch/arm64/include/asm/kvm_feature.h
copy to include/kvm/arm64/kvm_feature.h
index 27a472d2343e..945abbbf1aa8 100644
--- a/arch/arm64/include/asm/kvm_feature.h
+++ b/include/kvm/arm64/kvm_feature.h
@@ -1,6 +1,7 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
-#ifndef __ARM64_KVM_FEATURE_H__
-#define __ARM64_KVM_FEATURE_H__
+
+#ifndef __KVM_ARM64_FEATURE_H__
+#define __KVM_ARM64_FEATURE_H__
 
 #include <linux/types.h>
 #include <linux/bitfield.h>
@@ -46,7 +47,6 @@
 
 #define kvm_has_feat_enum(kvm, ...) __kvm_has_feat_enum(kvm, __VA_ARGS__)
 
-
 /* Check for a given level of PAuth support */
 #define kvm_has_pauth(k, l)						\
 	({								\
@@ -62,24 +62,4 @@
 		(pa + pi + pa3) == 1;					\
 	})
 
-#define kvm_has_fpmr(k)					\
-	(system_supports_fpmr() &&			\
-	 kvm_has_feat((k), ID_AA64PFR2_EL1, FPMR, IMP))
-
-#define kvm_has_tcr2(k)				\
-	(kvm_has_feat((k), ID_AA64MMFR3_EL1, TCRX, IMP))
-
-#define kvm_has_s1pie(k)				\
-	(kvm_has_feat((k), ID_AA64MMFR3_EL1, S1PIE, IMP))
-
-#define kvm_has_s1poe(k)				\
-	(system_supports_poe() &&			\
-	 kvm_has_feat((k), ID_AA64MMFR3_EL1, S1POE, IMP))
-
-#define kvm_has_ras(k)					\
-	(kvm_has_feat((k), ID_AA64PFR0_EL1, RAS, IMP))
-
-#define kvm_has_sctlr2(k)				\
-	(kvm_has_feat((k), ID_AA64MMFR3_EL1, SCTLRX, IMP))
-
-#endif /* __ARM64_KVM_FEATURE_H__*/
+#endif /* __KVM_ARM64_FEATURE_H__*/
-- 
2.53.0

[PATCH v1 20/26] KVM: arm64: Share ID reg handling

[Steffen Eiden]

Move ID register definitions from arch/arm64/include to a shared
location (include/kvm/arm64) to enable reuse by e.g. s390 for KVM/arm64
on s390 support.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_host.h | 41 ------------------------------
 include/kvm/arm64/kvm_host.h      | 42 +++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+), 41 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 5734e93cad57..a5fca468cc4a 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -1385,47 +1385,6 @@ static inline void kvm_hyp_reserve(void) { }
 void kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu);
 bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu);
 
-struct kvm_vm_id_regs {
-	/*
-	 * Emulated CPU ID registers per VM
-	 * (Op0, Op1, CRn, CRm, Op2) of the ID registers to be saved in it
-	 * is (3, 0, 0, crm, op2), where 1<=crm<8, 0<=op2<8.
-	 *
-	 * These emulated idregs are VM-wide, but accessed from the context of a vCPU.
-	 * Atomic access to multiple idregs are guarded by kvm_arch.config_lock.
-	 */
-#define IDREG_IDX(id)		(((sys_reg_CRm(id) - 1) << 3) | sys_reg_Op2(id))
-#define KVM_ARM_ID_REG_NUM	(IDREG_IDX(sys_reg(3, 0, 0, 7, 7)) + 1)
-	u64 normal[KVM_ARM_ID_REG_NUM];
-
-	u64 midr_el1;
-	u64 revidr_el1;
-	u64 aidr_el1;
-	u64 ctr_el0;
-};
-
-static inline u64 *__vm_id_reg(struct kvm_vm_id_regs *id_regs, u32 reg)
-{
-	switch (reg) {
-	case sys_reg(3, 0, 0, 1, 0) ... sys_reg(3, 0, 0, 7, 7):
-		return &id_regs->normal[IDREG_IDX(reg)];
-	case SYS_CTR_EL0:
-		return &id_regs->ctr_el0;
-	case SYS_MIDR_EL1:
-		return &id_regs->midr_el1;
-	case SYS_REVIDR_EL1:
-		return &id_regs->revidr_el1;
-	case SYS_AIDR_EL1:
-		return &id_regs->aidr_el1;
-	default:
-		WARN_ON_ONCE(1);
-		return NULL;
-	}
-}
-
-#define kvm_read_vm_id_reg(kvm, reg)					\
-	({ u64 __val = *__vm_id_reg(&(kvm)->arch.id_regs, reg); __val; })
-
 void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val);
 
 static inline bool kvm_arch_has_irq_bypass(void)
diff --git a/include/kvm/arm64/kvm_host.h b/include/kvm/arm64/kvm_host.h
index 8bf399508757..379942225d5f 100644
--- a/include/kvm/arm64/kvm_host.h
+++ b/include/kvm/arm64/kvm_host.h
@@ -4,6 +4,7 @@
 #define __KVM_ARM64_KVM_HOST_H
 
 #include <linux/types.h>
+#include <asm/sysreg-defs.h>
 
 #define KVM_VCPU_MAX_FEATURES 9
 
@@ -21,6 +22,47 @@
 #define KVM_REQ_MAP_L1_VNCR_EL2		KVM_ARCH_REQ(10)
 #define KVM_REQ_VGIC_PROCESS_UPDATE	KVM_ARCH_REQ(11)
 
+struct kvm_vm_id_regs {
+	/*
+	 * Emulated CPU ID registers per VM
+	 * (Op0, Op1, CRn, CRm, Op2) of the ID registers to be saved in it
+	 * is (3, 0, 0, crm, op2), where 1<=crm<8, 0<=op2<8.
+	 *
+	 * These emulated idregs are VM-wide, but accessed from the context of a vCPU.
+	 * Atomic access to multiple idregs are guarded by kvm_arch.config_lock.
+	 */
+#define IDREG_IDX(id)		(((sys_reg_CRm(id) - 1) << 3) | sys_reg_Op2(id))
+#define KVM_ARM_ID_REG_NUM	(IDREG_IDX(sys_reg(3, 0, 0, 7, 7)) + 1)
+	u64 normal[KVM_ARM_ID_REG_NUM];
+
+	u64 midr_el1;
+	u64 revidr_el1;
+	u64 aidr_el1;
+	u64 ctr_el0;
+};
+
+static inline u64 *__vm_id_reg(struct kvm_vm_id_regs *id_regs, u32 reg)
+{
+	switch (reg) {
+	case sys_reg(3, 0, 0, 1, 0) ... sys_reg(3, 0, 0, 7, 7):
+		return &id_regs->normal[IDREG_IDX(reg)];
+	case SYS_CTR_EL0:
+		return &id_regs->ctr_el0;
+	case SYS_MIDR_EL1:
+		return &id_regs->midr_el1;
+	case SYS_REVIDR_EL1:
+		return &id_regs->revidr_el1;
+	case SYS_AIDR_EL1:
+		return &id_regs->aidr_el1;
+	default:
+		WARN_ON_ONCE(1);
+		return NULL;
+	}
+}
+
+#define kvm_read_vm_id_reg(kvm, reg)					\
+	({ u64 __val = *__vm_id_reg(&(kvm)->arch.id_regs, reg); __val; })
+
 struct vcpu_reset_state {
 	unsigned long	pc;
 	unsigned long	r0;
-- 
2.53.0

[PATCH v1 21/26] KVM: arm64: Share sys-reg handling

[Steffen Eiden]

Move system register handling code from arch/arm64/kvm to shared
locations (virt/kvm/arm64 and include/kvm/arm64) to enable reuse by e.g.
s390 for KVM/arm64 on s390 support.

Stub functions in s390 are added to maintain linking compatibility. No
functional changes.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/include/asm/kvm_host.h      |   13 -
 arch/arm64/kvm/Makefile                |    1 +
 arch/arm64/kvm/arm.c                   |    2 +-
 arch/arm64/kvm/hyp/nvhe/sys_regs.c     |    2 +-
 arch/arm64/kvm/nested.c                |    2 +-
 arch/arm64/kvm/sys_regs.c              | 1250 +-----------------------
 arch/arm64/kvm/sys_regs.h              |  363 -------
 arch/arm64/kvm/trace_handle_exit.h     |   36 +-
 arch/arm64/kvm/vgic-sys-reg-v3.c       |    2 +-
 arch/arm64/kvm/vgic/vgic-init.c        |    1 +
 arch/s390/include/asm/kvm_host_arm64.h |    5 +-
 arch/s390/kvm/arm64/arm.c              |    9 +
 include/kvm/arm64/kvm_host.h           |   10 +
 include/kvm/arm64/sys_regs.h           |  548 +++++++++++
 virt/kvm/arm64/mmio.c                  |    1 +
 virt/kvm/arm64/sys_regs.c              | 1039 ++++++++++++++++++++
 virt/kvm/arm64/trace.h                 |   34 +
 17 files changed, 1684 insertions(+), 1634 deletions(-)
 delete mode 100644 arch/arm64/kvm/sys_regs.h
 create mode 100644 include/kvm/arm64/sys_regs.h
 create mode 100644 virt/kvm/arm64/sys_regs.c

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index a5fca468cc4a..f3113cdcea0e 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -1073,9 +1073,6 @@ u64 kvm_vcpu_apply_reg_masks(const struct kvm_vcpu *, enum vcpu_sysreg, u64);
 		__v;							\
 	})
 
-u64 vcpu_read_sys_reg(const struct kvm_vcpu *, enum vcpu_sysreg);
-void vcpu_write_sys_reg(struct kvm_vcpu *, u64, enum vcpu_sysreg);
-
 struct kvm_vm_stat {
 	struct kvm_vm_stat_generic generic;
 };
@@ -1095,9 +1092,6 @@ unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu);
 int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg);
 int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg);
 
-unsigned long kvm_arm_num_sys_reg_descs(struct kvm_vcpu *vcpu);
-int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices);
-
 int __kvm_arm_vcpu_get_events(struct kvm_vcpu *vcpu,
 			      struct kvm_vcpu_events *events);
 
@@ -1161,11 +1155,9 @@ int kvm_handle_cp14_32(struct kvm_vcpu *vcpu);
 int kvm_handle_cp14_64(struct kvm_vcpu *vcpu);
 int kvm_handle_cp15_32(struct kvm_vcpu *vcpu);
 int kvm_handle_cp15_64(struct kvm_vcpu *vcpu);
-int kvm_handle_sys_reg(struct kvm_vcpu *vcpu);
 int kvm_handle_cp10_id(struct kvm_vcpu *vcpu);
 
 void kvm_sys_regs_create_debugfs(struct kvm *kvm);
-void kvm_reset_sys_regs(struct kvm_vcpu *vcpu);
 
 int __init kvm_sys_reg_table_init(void);
 struct sys_reg_desc;
@@ -1362,9 +1354,6 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
 	(system_supports_32bit_el0() &&				\
 	 !static_branch_unlikely(&arm64_mismatched_32bit_el0))
 
-#define kvm_vm_has_ran_once(kvm)					\
-	(test_bit(KVM_ARCH_FLAG_HAS_RAN_ONCE, &(kvm)->arch.flags))
-
 static inline bool __vcpu_has_feature(const struct kvm_arch *ka, int feature)
 {
 	return test_bit(feature, ka->vcpu_features);
@@ -1385,8 +1374,6 @@ static inline void kvm_hyp_reserve(void) { }
 void kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu);
 bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu);
 
-void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val);
-
 static inline bool kvm_arch_has_irq_bypass(void)
 {
 	return true;
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 5b4a8d002fc9..d55c7245aad9 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -19,6 +19,7 @@ kvm-y += arm.o mmu.o psci.o hypercalls.o pvtime.o \
 	 guest.o debug.o reset.o sys_regs.o stacktrace.o \
 	 vgic-sys-reg-v3.o fpsimd.o pkvm.o \
 	 arch_timer.o trng.o vmid.o emulate-nested.o nested.o at.o \
+	 $(KVM_ARM64)/sys_regs.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 929b8e6eae9a..99044b1e3c9f 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -54,7 +54,7 @@
 
 #include <kvm/arm64/guest.h>
 
-#include "sys_regs.h"
+#include <kvm/arm64/sys_regs.h>
 
 static enum kvm_mode kvm_mode = KVM_MODE_DEFAULT;
 
diff --git a/arch/arm64/kvm/hyp/nvhe/sys_regs.c b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
index 08b14053568b..51fd49d00c95 100644
--- a/arch/arm64/kvm/hyp/nvhe/sys_regs.c
+++ b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
@@ -14,7 +14,7 @@
 
 #include <nvhe/pkvm.h>
 
-#include "../../sys_regs.h"
+#include <kvm/arm64/sys_regs.h>
 
 /*
  * Copies of the host's CPU features registers holding sanitized values at hyp.
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 5321911f19e0..dce896e2a151 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -15,7 +15,7 @@
 #include <asm/kvm_nested.h>
 #include <asm/sysreg.h>
 
-#include "sys_regs.h"
+#include <kvm/arm64/sys_regs.h>
 
 struct vncr_tlb {
 	/* The guest's VNCR_EL2 */
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 46b24529ec70..b68622f93621 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -34,7 +34,7 @@
 
 #include <trace/events/kvm.h>
 
-#include "sys_regs.h"
+#include <kvm/arm64/sys_regs.h>
 #include "vgic/vgic.h"
 
 #include "trace.h"
@@ -45,43 +45,6 @@
  * 64bit interface.
  */
 
-static u64 sys_reg_to_index(const struct sys_reg_desc *reg);
-static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
-		      u64 val);
-
-static bool undef_access(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
-			 const struct sys_reg_desc *r)
-{
-	kvm_inject_undefined(vcpu);
-	return false;
-}
-
-static bool bad_trap(struct kvm_vcpu *vcpu,
-		     struct sys_reg_params *params,
-		     const struct sys_reg_desc *r,
-		     const char *msg)
-{
-	WARN_ONCE(1, "Unexpected %s\n", msg);
-	print_sys_reg_instr(params);
-	return undef_access(vcpu, params, r);
-}
-
-static bool read_from_write_only(struct kvm_vcpu *vcpu,
-				 struct sys_reg_params *params,
-				 const struct sys_reg_desc *r)
-{
-	return bad_trap(vcpu, params, r,
-			"sys_reg read to write-only register");
-}
-
-static bool write_to_read_only(struct kvm_vcpu *vcpu,
-			       struct sys_reg_params *params,
-			       const struct sys_reg_desc *r)
-{
-	return bad_trap(vcpu, params, r,
-			"sys_reg write to read-only register");
-}
-
 enum sr_loc_attr {
 	SR_LOC_MEMORY	= 0,	  /* Register definitely in memory */
 	SR_LOC_LOADED	= BIT(0), /* Register on CPU, unless it cannot */
@@ -391,111 +354,8 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, enum vcpu_sysreg reg)
 	__vcpu_assign_sys_reg(vcpu, reg, val);
 }
 
-/* CSSELR values; used to index KVM_REG_ARM_DEMUX_ID_CCSIDR */
 #define CSSELR_MAX 14
 
-/*
- * Returns the minimum line size for the selected cache, expressed as
- * Log2(bytes).
- */
-static u8 get_min_cache_line_size(bool icache)
-{
-	u64 ctr = read_sanitised_ftr_reg(SYS_CTR_EL0);
-	u8 field;
-
-	if (icache)
-		field = SYS_FIELD_GET(CTR_EL0, IminLine, ctr);
-	else
-		field = SYS_FIELD_GET(CTR_EL0, DminLine, ctr);
-
-	/*
-	 * Cache line size is represented as Log2(words) in CTR_EL0.
-	 * Log2(bytes) can be derived with the following:
-	 *
-	 * Log2(words) + 2 = Log2(bytes / 4) + 2
-	 * 		   = Log2(bytes) - 2 + 2
-	 * 		   = Log2(bytes)
-	 */
-	return field + 2;
-}
-
-/* Which cache CCSIDR represents depends on CSSELR value. */
-static u32 get_ccsidr(struct kvm_vcpu *vcpu, u32 csselr)
-{
-	u8 line_size;
-
-	if (vcpu->arch.ccsidr)
-		return vcpu->arch.ccsidr[csselr];
-
-	line_size = get_min_cache_line_size(csselr & CSSELR_EL1_InD);
-
-	/*
-	 * Fabricate a CCSIDR value as the overriding value does not exist.
-	 * The real CCSIDR value will not be used as it can vary by the
-	 * physical CPU which the vcpu currently resides in.
-	 *
-	 * The line size is determined with get_min_cache_line_size(), which
-	 * should be valid for all CPUs even if they have different cache
-	 * configuration.
-	 *
-	 * The associativity bits are cleared, meaning the geometry of all data
-	 * and unified caches (which are guaranteed to be PIPT and thus
-	 * non-aliasing) are 1 set and 1 way.
-	 * Guests should not be doing cache operations by set/way at all, and
-	 * for this reason, we trap them and attempt to infer the intent, so
-	 * that we can flush the entire guest's address space at the appropriate
-	 * time. The exposed geometry minimizes the number of the traps.
-	 * [If guests should attempt to infer aliasing properties from the
-	 * geometry (which is not permitted by the architecture), they would
-	 * only do so for virtually indexed caches.]
-	 *
-	 * We don't check if the cache level exists as it is allowed to return
-	 * an UNKNOWN value if not.
-	 */
-	return SYS_FIELD_PREP(CCSIDR_EL1, LineSize, line_size - 4);
-}
-
-static int set_ccsidr(struct kvm_vcpu *vcpu, u32 csselr, u32 val)
-{
-	u8 line_size = FIELD_GET(CCSIDR_EL1_LineSize, val) + 4;
-	u32 *ccsidr = vcpu->arch.ccsidr;
-	u32 i;
-
-	if ((val & CCSIDR_EL1_RES0) ||
-	    line_size < get_min_cache_line_size(csselr & CSSELR_EL1_InD))
-		return -EINVAL;
-
-	if (!ccsidr) {
-		if (val == get_ccsidr(vcpu, csselr))
-			return 0;
-
-		ccsidr = kmalloc_array(CSSELR_MAX, sizeof(u32), GFP_KERNEL_ACCOUNT);
-		if (!ccsidr)
-			return -ENOMEM;
-
-		for (i = 0; i < CSSELR_MAX; i++)
-			ccsidr[i] = get_ccsidr(vcpu, i);
-
-		vcpu->arch.ccsidr = ccsidr;
-	}
-
-	ccsidr[csselr] = val;
-
-	return 0;
-}
-
-static bool access_rw(struct kvm_vcpu *vcpu,
-		      struct sys_reg_params *p,
-		      const struct sys_reg_desc *r)
-{
-	if (p->is_write)
-		vcpu_write_sys_reg(vcpu, p->regval, r->reg);
-	else
-		p->regval = vcpu_read_sys_reg(vcpu, r->reg);
-
-	return true;
-}
-
 /*
  * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized).
  */
@@ -530,24 +390,6 @@ static bool access_dcgsw(struct kvm_vcpu *vcpu,
 	return access_dcsw(vcpu, p, r);
 }
 
-static void get_access_mask(const struct sys_reg_desc *r, u64 *mask, u64 *shift)
-{
-	switch (r->aarch32_map) {
-	case AA32_LO:
-		*mask = GENMASK_ULL(31, 0);
-		*shift = 0;
-		break;
-	case AA32_HI:
-		*mask = GENMASK_ULL(63, 32);
-		*shift = 32;
-		break;
-	default:
-		*mask = GENMASK_ULL(63, 0);
-		*shift = 0;
-		break;
-	}
-}
-
 /*
  * Generic accessor for VM registers. Only called as long as HCR_TVM
  * is set. If the guest enables the MMU, we stop trapping the VM
@@ -766,16 +608,6 @@ static bool access_gicv5_ppi_enabler(struct kvm_vcpu *vcpu,
 	return true;
 }
 
-static bool trap_raz_wi(struct kvm_vcpu *vcpu,
-			struct sys_reg_params *p,
-			const struct sys_reg_desc *r)
-{
-	if (p->is_write)
-		return ignore_write(vcpu, p);
-	else
-		return read_zero(vcpu, p);
-}
-
 /*
  * ARMv8.1 mandates at least a trivial LORegion implementation, where all the
  * RW registers are RES0 (which we can implement as RAZ/WI). On an ARMv8.0
@@ -797,50 +629,6 @@ static bool trap_loregion(struct kvm_vcpu *vcpu,
 	return trap_raz_wi(vcpu, p, r);
 }
 
-static bool trap_oslar_el1(struct kvm_vcpu *vcpu,
-			   struct sys_reg_params *p,
-			   const struct sys_reg_desc *r)
-{
-	if (!p->is_write)
-		return read_from_write_only(vcpu, p, r);
-
-	kvm_debug_handle_oslar(vcpu, p->regval);
-	return true;
-}
-
-static bool trap_oslsr_el1(struct kvm_vcpu *vcpu,
-			   struct sys_reg_params *p,
-			   const struct sys_reg_desc *r)
-{
-	if (p->is_write)
-		return write_to_read_only(vcpu, p, r);
-
-	p->regval = __vcpu_sys_reg(vcpu, r->reg);
-	return true;
-}
-
-static int set_oslsr_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
-			 u64 val)
-{
-	u64 oslk;
-
-	/*
-	 * The only modifiable bit is the OSLK bit. Refuse the write if
-	 * userspace attempts to change any other bit in the register.
-	 */
-	if ((val ^ rd->val) & ~OSLSR_EL1_OSLK)
-		return -EINVAL;
-
-	/*
-	 * Redirect the write to the proper control register.
-	 * OSLSR is read-only
-	 */
-	oslk = SYS_FIELD_GET(OSLSR_EL1, OSLK, val);
-	__vcpu_assign_sys_reg(vcpu, OSLAR_EL1,
-			      SYS_FIELD_PREP(OSLAR_EL1, OSLK, oslk));
-	return 0;
-}
-
 static bool trap_dbgauthstatus_el1(struct kvm_vcpu *vcpu,
 				   struct sys_reg_params *p,
 				   const struct sys_reg_desc *r)
@@ -985,26 +773,6 @@ static u64 reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 	return actlr;
 }
 
-static u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
-{
-	u64 mpidr;
-
-	/*
-	 * Map the vcpu_id into the first three affinity level fields of
-	 * the MPIDR. We limit the number of VCPUs in level 0 due to a
-	 * limitation to 16 CPUs in that level in the ICC_SGIxR registers
-	 * of the GICv3 to be able to address each CPU directly when
-	 * sending IPIs.
-	 */
-	mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
-	mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
-	mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
-	mpidr |= (1ULL << 31);
-	vcpu_write_sys_reg(vcpu, mpidr, MPIDR_EL1);
-
-	return mpidr;
-}
-
 static unsigned int hidden_visibility(const struct kvm_vcpu *vcpu,
 				      const struct sys_reg_desc *r)
 {
@@ -1783,7 +1551,7 @@ static s64 kvm_arm64_ftr_safe_value(u32 id, const struct arm64_ftr_bits *ftrp,
  *
  * Return: 0 if all the fields are safe. Otherwise, return negative errno.
  */
-static int arm64_check_features(struct kvm_vcpu *vcpu,
+int arm64_check_features(struct kvm_vcpu *vcpu,
 				const struct sys_reg_desc *rd,
 				u64 val)
 {
@@ -1905,188 +1673,6 @@ u64 kvm_sanitised_host_ftr_reg(u32 id)
 	return val;
 }
 
-/*
- * Statically sanitise the host's feature register, independent of the guest's
- * configuration and host implementation.
- */
-static u64 kvm_max_possible_guest_ftr_reg(u32 id, u64 val)
-{
-	switch (id) {
-	case SYS_ID_AA64DFR0_EL1:
-		val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64DFR0_EL1, DebugVer, V8P8);
-
-		/* Hide SPE from guests */
-		val &= ~ID_AA64DFR0_EL1_PMSVer_MASK;
-
-		/* Hide BRBE from guests */
-		val &= ~ID_AA64DFR0_EL1_BRBE_MASK;
-		break;
-	case SYS_ID_AA64ISAR2_EL1:
-		/* Mask WFxT field unless *both* WFET & WFIT are present. */
-		if (!id_has_feat(val, ID_AA64ISAR2_EL1, WFxT, IMP))
-			val &= ~ID_AA64ISAR2_EL1_WFxT;
-		break;
-	case SYS_ID_AA64ISAR3_EL1:
-		val &= ID_AA64ISAR3_EL1_FPRCVT | ID_AA64ISAR3_EL1_LSFE |
-		       ID_AA64ISAR3_EL1_FAMINMAX | ID_AA64ISAR3_EL1_LSUI;
-		break;
-	case SYS_ID_AA64MMFR2_EL1:
-		val &= ~ID_AA64MMFR2_EL1_CCIDX_MASK;
-		val &= ~ID_AA64MMFR2_EL1_NV;
-		break;
-	case SYS_ID_AA64MMFR3_EL1:
-		val &= ID_AA64MMFR3_EL1_TCRX |
-		       ID_AA64MMFR3_EL1_SCTLRX |
-		       ID_AA64MMFR3_EL1_S1POE |
-		       ID_AA64MMFR3_EL1_S1PIE;
-		break;
-	case SYS_ID_MMFR4_EL1:
-		val &= ~ID_MMFR4_EL1_CCIDX;
-		break;
-	case SYS_ID_AA64PFR0_EL1:
-		val &= ~ID_AA64PFR0_EL1_AMU_MASK;
-		/*
-		 * MPAM is disabled by default as KVM also needs a set of PARTID to
-		 * program the MPAMVPMx_EL2 PARTID remapping registers with. But some
-		 * older kernels let the guest see the ID bit.
-		 */
-		val &= ~ID_AA64PFR0_EL1_MPAM_MASK;
-		break;
-	case SYS_ID_AA64PFR1_EL1:
-		val &= ~ID_AA64PFR1_EL1_SME;
-		val &= ~ID_AA64PFR1_EL1_RNDR_trap;
-		val &= ~ID_AA64PFR1_EL1_NMI;
-		val &= ~ID_AA64PFR1_EL1_GCS;
-		val &= ~ID_AA64PFR1_EL1_THE;
-		val &= ~ID_AA64PFR1_EL1_MTEX;
-		val &= ~ID_AA64PFR1_EL1_PFAR;
-		val &= ~ID_AA64PFR1_EL1_MPAM_frac;
-		break;
-	case SYS_ID_AA64PFR2_EL1:
-		val &= ID_AA64PFR2_EL1_FPMR |
-		       ID_AA64PFR2_EL1_MTEFAR |
-		       ID_AA64PFR2_EL1_MTESTOREONLY;
-		break;
-	}
-
-	return val;
-}
-
-/*
- * Sanitise based on vCPU configuration.
- */
-static u64 kvm_sanitise_vcpu_ftr_reg(const struct kvm_vcpu *vcpu, u32 id, u64 val)
-{
-	switch (id) {
-	case SYS_ID_AA64DFR0_EL1:
-		/*
-		 * Only initialize the PMU version if the vCPU was configured with one.
-		 */
-		val &= ~ID_AA64DFR0_EL1_PMUVer_MASK;
-		if (kvm_vcpu_has_pmu(vcpu))
-			val |= SYS_FIELD_PREP(ID_AA64DFR0_EL1, PMUVer,
-					      kvm_arm_pmu_get_pmuver_limit());
-		break;
-	case SYS_ID_AA64PFR0_EL1:
-		if (!vcpu_has_sve(vcpu))
-			val &= ~ID_AA64PFR0_EL1_SVE_MASK;
-		break;
-	case SYS_ID_AA64PFR1_EL1:
-		if (!kvm_has_mte(vcpu->kvm)) {
-			val &= ~ID_AA64PFR1_EL1_MTE;
-			val &= ~ID_AA64PFR1_EL1_MTE_frac;
-		}
-		break;
-	case SYS_ID_AA64PFR2_EL1:
-		if (!kvm_has_mte(vcpu->kvm)) {
-			val &= ~ID_AA64PFR2_EL1_MTEFAR;
-			val &= ~ID_AA64PFR2_EL1_MTESTOREONLY;
-		}
-		break;
-	case SYS_ID_AA64ISAR1_EL1:
-		if (!vcpu_has_ptrauth(vcpu))
-			val &= ~(ID_AA64ISAR1_EL1_APA |
-				 ID_AA64ISAR1_EL1_API |
-				 ID_AA64ISAR1_EL1_GPA |
-				 ID_AA64ISAR1_EL1_GPI);
-		break;
-	case SYS_ID_AA64ISAR2_EL1:
-		if (!vcpu_has_ptrauth(vcpu))
-			val &= ~(ID_AA64ISAR2_EL1_APA3 |
-				 ID_AA64ISAR2_EL1_GPA3);
-	}
-
-	if (vcpu_has_nv(vcpu))
-		val = limit_nv_id_reg(vcpu->kvm, id, val);
-
-	return val;
-}
-
-/* Read a sanitised cpufeature ID register by sys_reg_desc */
-static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu,
-				       const struct sys_reg_desc *r)
-{
-	u32 id = reg_to_encoding(r);
-	u64 val;
-
-	if (sysreg_visible_as_raz(vcpu, r))
-		return 0;
-
-	val = kvm_sanitised_host_ftr_reg(id);
-	val = kvm_max_possible_guest_ftr_reg(id, val);
-	val = kvm_sanitise_vcpu_ftr_reg(vcpu, id, val);
-
-	return val;
-}
-
-u64 kvm_read_sanitised_id_reg(struct kvm_vcpu *vcpu,
-			      const struct sys_reg_desc *r)
-{
-	return __kvm_read_sanitised_id_reg(vcpu, r);
-}
-
-static u64 read_id_reg(const struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
-{
-	return kvm_read_vm_id_reg(vcpu->kvm, reg_to_encoding(r));
-}
-
-static bool is_feature_id_reg(u32 encoding)
-{
-	return (sys_reg_Op0(encoding) == 3 &&
-		(sys_reg_Op1(encoding) < 2 || sys_reg_Op1(encoding) == 3) &&
-		sys_reg_CRn(encoding) == 0 &&
-		sys_reg_CRm(encoding) <= 7);
-}
-
-/*
- * Return true if the register's (Op0, Op1, CRn, CRm, Op2) is
- * (3, 0, 0, crm, op2), where 1<=crm<8, 0<=op2<8, which is the range of ID
- * registers KVM maintains on a per-VM basis.
- *
- * Additionally, the implementation ID registers and CTR_EL0 are handled as
- * per-VM registers.
- */
-static inline bool is_vm_ftr_id_reg(u32 id)
-{
-	switch (id) {
-	case SYS_CTR_EL0:
-	case SYS_MIDR_EL1:
-	case SYS_REVIDR_EL1:
-	case SYS_AIDR_EL1:
-		return true;
-	default:
-		return (sys_reg_Op0(id) == 3 && sys_reg_Op1(id) == 0 &&
-			sys_reg_CRn(id) == 0 && sys_reg_CRm(id) >= 1 &&
-			sys_reg_CRm(id) < 8);
-
-	}
-}
-
-static inline bool is_vcpu_ftr_id_reg(u32 id)
-{
-	return is_feature_id_reg(id) && !is_vm_ftr_id_reg(id);
-}
-
 static inline bool is_aa32_id_reg(u32 id)
 {
 	return (sys_reg_Op0(id) == 3 && sys_reg_Op1(id) == 0 &&
@@ -2094,49 +1680,6 @@ static inline bool is_aa32_id_reg(u32 id)
 		sys_reg_CRm(id) <= 3);
 }
 
-static unsigned int id_visibility(const struct kvm_vcpu *vcpu,
-				  const struct sys_reg_desc *r)
-{
-	u32 id = reg_to_encoding(r);
-
-	switch (id) {
-	case SYS_ID_AA64ZFR0_EL1:
-		if (!vcpu_has_sve(vcpu))
-			return REG_RAZ;
-		break;
-	}
-
-	return 0;
-}
-
-static unsigned int aa32_id_visibility(const struct kvm_vcpu *vcpu,
-				       const struct sys_reg_desc *r)
-{
-	/*
-	 * AArch32 ID registers are UNKNOWN if AArch32 isn't implemented at any
-	 * EL. Promote to RAZ/WI in order to guarantee consistency between
-	 * systems.
-	 */
-	if (!kvm_supports_32bit_el0())
-		return REG_RAZ | REG_USER_WI;
-
-	return id_visibility(vcpu, r);
-}
-
-/* cpufeature ID register access trap handlers */
-
-static bool access_id_reg(struct kvm_vcpu *vcpu,
-			  struct sys_reg_params *p,
-			  const struct sys_reg_desc *r)
-{
-	if (p->is_write)
-		return write_to_read_only(vcpu, p, r);
-
-	p->regval = read_id_reg(vcpu, r);
-
-	return true;
-}
-
 /* Visibility overrides for SVE-specific control registers */
 static unsigned int sve_visibility(const struct kvm_vcpu *vcpu,
 				   const struct sys_reg_desc *rd)
@@ -2333,290 +1876,48 @@ static int set_id_aa64pfr1_el1(struct kvm_vcpu *vcpu,
 	return set_id_reg(vcpu, rd, user_val);
 }
 
-static int set_id_aa64pfr2_el1(struct kvm_vcpu *vcpu,
-			       const struct sys_reg_desc *rd, u64 user_val)
-{
-	return set_id_reg(vcpu, rd, user_val);
-}
-
-/*
- * Allow userspace to de-feature a stage-2 translation granule but prevent it
- * from claiming the impossible.
- */
-#define tgran2_val_allowed(tg, safe, user)			\
-({								\
-	u8 __s = SYS_FIELD_GET(ID_AA64MMFR0_EL1, tg, safe);	\
-	u8 __u = SYS_FIELD_GET(ID_AA64MMFR0_EL1, tg, user);	\
-								\
-	__s == __u || __u == ID_AA64MMFR0_EL1_##tg##_NI;	\
-})
-
-static int set_id_aa64mmfr0_el1(struct kvm_vcpu *vcpu,
-				const struct sys_reg_desc *rd, u64 user_val)
-{
-	u64 sanitized_val = kvm_read_sanitised_id_reg(vcpu, rd);
-
-	if (!vcpu_has_nv(vcpu))
-		return set_id_reg(vcpu, rd, user_val);
-
-	if (!tgran2_val_allowed(TGRAN4_2, sanitized_val, user_val) ||
-	    !tgran2_val_allowed(TGRAN16_2, sanitized_val, user_val) ||
-	    !tgran2_val_allowed(TGRAN64_2, sanitized_val, user_val))
-		return -EINVAL;
-
-	return set_id_reg(vcpu, rd, user_val);
-}
-
-static int set_id_aa64mmfr2_el1(struct kvm_vcpu *vcpu,
-				const struct sys_reg_desc *rd, u64 user_val)
-{
-	u64 hw_val = read_sanitised_ftr_reg(SYS_ID_AA64MMFR2_EL1);
-	u64 nv_mask = ID_AA64MMFR2_EL1_NV_MASK;
-
-	/*
-	 * We made the mistake to expose the now deprecated NV field,
-	 * so allow userspace to write it, but silently ignore it.
-	 */
-	if ((hw_val & nv_mask) == (user_val & nv_mask))
-		user_val &= ~nv_mask;
-
-	return set_id_reg(vcpu, rd, user_val);
-}
-
-static int set_ctr_el0(struct kvm_vcpu *vcpu,
-		       const struct sys_reg_desc *rd, u64 user_val)
-{
-	u8 user_L1Ip = SYS_FIELD_GET(CTR_EL0, L1Ip, user_val);
-
-	/*
-	 * Both AIVIVT (0b01) and VPIPT (0b00) are documented as reserved.
-	 * Hence only allow to set VIPT(0b10) or PIPT(0b11) for L1Ip based
-	 * on what hardware reports.
-	 *
-	 * Using a VIPT software model on PIPT will lead to over invalidation,
-	 * but still correct. Hence, we can allow downgrading PIPT to VIPT,
-	 * but not the other way around. This is handled via arm64_ftr_safe_value()
-	 * as CTR_EL0 ftr_bits has L1Ip field with type FTR_EXACT and safe value
-	 * set as VIPT.
-	 */
-	switch (user_L1Ip) {
-	case CTR_EL0_L1Ip_RESERVED_VPIPT:
-	case CTR_EL0_L1Ip_RESERVED_AIVIVT:
-		return -EINVAL;
-	case CTR_EL0_L1Ip_VIPT:
-	case CTR_EL0_L1Ip_PIPT:
-		return set_id_reg(vcpu, rd, user_val);
-	default:
-		return -ENOENT;
-	}
-}
-
-/*
- * cpufeature ID register user accessors
- *
- * For now, these registers are immutable for userspace, so no values
- * are stored, and for set_id_reg() we don't allow the effective value
- * to be changed.
- */
-static int get_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
-		      u64 *val)
-{
-	/*
-	 * Avoid locking if the VM has already started, as the ID registers are
-	 * guaranteed to be invariant at that point.
-	 */
-	if (kvm_vm_has_ran_once(vcpu->kvm)) {
-		*val = read_id_reg(vcpu, rd);
-		return 0;
-	}
-
-	mutex_lock(&vcpu->kvm->arch.config_lock);
-	*val = read_id_reg(vcpu, rd);
-	mutex_unlock(&vcpu->kvm->arch.config_lock);
-
-	return 0;
-}
-
-static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
-		      u64 val)
-{
-	u32 id = reg_to_encoding(rd);
-	int ret;
-
-	mutex_lock(&vcpu->kvm->arch.config_lock);
-
-	/*
-	 * Once the VM has started the ID registers are immutable. Reject any
-	 * write that does not match the final register value.
-	 */
-	if (kvm_vm_has_ran_once(vcpu->kvm)) {
-		if (val != read_id_reg(vcpu, rd))
-			ret = -EBUSY;
-		else
-			ret = 0;
-
-		mutex_unlock(&vcpu->kvm->arch.config_lock);
-		return ret;
-	}
-
-	ret = arm64_check_features(vcpu, rd, val);
-	if (!ret)
-		kvm_set_vm_id_reg(vcpu->kvm, id, val);
-
-	mutex_unlock(&vcpu->kvm->arch.config_lock);
-
-	/*
-	 * arm64_check_features() returns -E2BIG to indicate the register's
-	 * feature set is a superset of the maximally-allowed register value.
-	 * While it would be nice to precisely describe this to userspace, the
-	 * existing UAPI for KVM_SET_ONE_REG has it that invalid register
-	 * writes return -EINVAL.
-	 */
-	if (ret == -E2BIG)
-		ret = -EINVAL;
-	return ret;
-}
-
-void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val)
-{
-	u64 *p = __vm_id_reg(&kvm->arch.id_regs, reg);
-
-	lockdep_assert_held(&kvm->arch.config_lock);
-
-	if (KVM_BUG_ON(kvm_vm_has_ran_once(kvm) || !p, kvm))
-		return;
-
-	*p = val;
-}
-
-static int get_raz_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
-		       u64 *val)
-{
-	*val = 0;
-	return 0;
-}
-
-static int set_wi_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
-		      u64 val)
-{
-	return 0;
-}
-
-static bool access_ctr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
-		       const struct sys_reg_desc *r)
-{
-	if (p->is_write)
-		return write_to_read_only(vcpu, p, r);
-
-	p->regval = kvm_read_vm_id_reg(vcpu->kvm, SYS_CTR_EL0);
-	return true;
-}
-
-static bool access_clidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
-			 const struct sys_reg_desc *r)
-{
-	if (p->is_write)
-		return write_to_read_only(vcpu, p, r);
-
-	p->regval = __vcpu_sys_reg(vcpu, r->reg);
-	return true;
-}
-
-/*
- * Fabricate a CLIDR_EL1 value instead of using the real value, which can vary
- * by the physical CPU which the vcpu currently resides in.
- */
-static u64 reset_clidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
-{
-	u64 ctr_el0 = read_sanitised_ftr_reg(SYS_CTR_EL0);
-	u64 clidr;
-	u8 loc;
-
-	if ((ctr_el0 & CTR_EL0_IDC)) {
-		/*
-		 * Data cache clean to the PoU is not required so LoUU and LoUIS
-		 * will not be set and a unified cache, which will be marked as
-		 * LoC, will be added.
-		 *
-		 * If not DIC, let the unified cache L2 so that an instruction
-		 * cache can be added as L1 later.
-		 */
-		loc = (ctr_el0 & CTR_EL0_DIC) ? 1 : 2;
-		clidr = CACHE_TYPE_UNIFIED << CLIDR_CTYPE_SHIFT(loc);
-	} else {
-		/*
-		 * Data cache clean to the PoU is required so let L1 have a data
-		 * cache and mark it as LoUU and LoUIS. As L1 has a data cache,
-		 * it can be marked as LoC too.
-		 */
-		loc = 1;
-		clidr = 1 << CLIDR_LOUU_SHIFT;
-		clidr |= 1 << CLIDR_LOUIS_SHIFT;
-		clidr |= CACHE_TYPE_DATA << CLIDR_CTYPE_SHIFT(1);
-	}
-
-	/*
-	 * Instruction cache invalidation to the PoU is required so let L1 have
-	 * an instruction cache. If L1 already has a data cache, it will be
-	 * CACHE_TYPE_SEPARATE.
-	 */
-	if (!(ctr_el0 & CTR_EL0_DIC))
-		clidr |= CACHE_TYPE_INST << CLIDR_CTYPE_SHIFT(1);
+static int set_id_aa64pfr2_el1(struct kvm_vcpu *vcpu,
+			       const struct sys_reg_desc *rd, u64 user_val)
+{
+	return set_id_reg(vcpu, rd, user_val);
+}
 
-	clidr |= loc << CLIDR_LOC_SHIFT;
+static int set_id_aa64mmfr2_el1(struct kvm_vcpu *vcpu,
+				const struct sys_reg_desc *rd, u64 user_val)
+{
+	u64 hw_val = read_sanitised_ftr_reg(SYS_ID_AA64MMFR2_EL1);
+	u64 nv_mask = ID_AA64MMFR2_EL1_NV_MASK;
 
 	/*
-	 * Add tag cache unified to data cache. Allocation tags and data are
-	 * unified in a cache line so that it looks valid even if there is only
-	 * one cache line.
+	 * We made the mistake to expose the now deprecated NV field,
+	 * so allow userspace to write it, but silently ignore it.
 	 */
-	if (kvm_has_mte(vcpu->kvm))
-		clidr |= 2ULL << CLIDR_TTYPE_SHIFT(loc);
-
-	__vcpu_assign_sys_reg(vcpu, r->reg, clidr);
+	if ((hw_val & nv_mask) == (user_val & nv_mask))
+		user_val &= ~nv_mask;
 
-	return __vcpu_sys_reg(vcpu, r->reg);
+	return set_id_reg(vcpu, rd, user_val);
 }
 
-static int set_clidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
-		      u64 val)
+static int get_raz_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+		       u64 *val)
 {
-	u64 ctr_el0 = read_sanitised_ftr_reg(SYS_CTR_EL0);
-	u64 idc = !CLIDR_LOC(val) || (!CLIDR_LOUIS(val) && !CLIDR_LOUU(val));
-
-	if ((val & CLIDR_EL1_RES0) || (!(ctr_el0 & CTR_EL0_IDC) && idc))
-		return -EINVAL;
-
-	__vcpu_assign_sys_reg(vcpu, rd->reg, val);
-
+	*val = 0;
 	return 0;
 }
 
-static bool access_csselr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
-			  const struct sys_reg_desc *r)
+static int set_wi_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+		      u64 val)
 {
-	int reg = r->reg;
-
-	if (p->is_write)
-		vcpu_write_sys_reg(vcpu, p->regval, reg);
-	else
-		p->regval = vcpu_read_sys_reg(vcpu, reg);
-	return true;
+	return 0;
 }
 
-static bool access_ccsidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
-			  const struct sys_reg_desc *r)
+static bool access_ctr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+		       const struct sys_reg_desc *r)
 {
-	u32 csselr;
-
 	if (p->is_write)
 		return write_to_read_only(vcpu, p, r);
 
-	csselr = vcpu_read_sys_reg(vcpu, CSSELR_EL1);
-	csselr &= CSSELR_EL1_Level | CSSELR_EL1_InD;
-	if (csselr < CSSELR_MAX)
-		p->regval = get_ccsidr(vcpu, csselr);
-
+	p->regval = kvm_read_vm_id_reg(vcpu->kvm, SYS_CTR_EL0);
 	return true;
 }
 
@@ -2700,82 +2001,6 @@ static bool bad_redir_trap(struct kvm_vcpu *vcpu,
 	SYS_REG_USER_FILTER(name, access_arch_timer, reset_val, 0, \
 			    arch_timer_get_user, arch_timer_set_user, vis)
 
-/*
- * Since reset() callback and field val are not used for idregs, they will be
- * used for specific purposes for idregs.
- * The reset() would return KVM sanitised register value. The value would be the
- * same as the host kernel sanitised value if there is no KVM sanitisation.
- * The val would be used as a mask indicating writable fields for the idreg.
- * Only bits with 1 are writable from userspace. This mask might not be
- * necessary in the future whenever all ID registers are enabled as writable
- * from userspace.
- */
-
-#define ID_DESC_DEFAULT_CALLBACKS		\
-	.access	= access_id_reg,		\
-	.get_user = get_id_reg,			\
-	.set_user = set_id_reg,			\
-	.visibility = id_visibility,		\
-	.reset = kvm_read_sanitised_id_reg
-
-#define ID_DESC(name)				\
-	SYS_DESC(SYS_##name),			\
-	ID_DESC_DEFAULT_CALLBACKS
-
-/* sys_reg_desc initialiser for known cpufeature ID registers */
-#define ID_SANITISED(name) {			\
-	ID_DESC(name),				\
-	.val = 0,				\
-}
-
-/* sys_reg_desc initialiser for writable ID registers */
-#define ID_WRITABLE(name, mask) {		\
-	ID_DESC(name),				\
-	.val = mask,				\
-}
-
-/*
- * 32bit ID regs are fully writable when the guest is 32bit
- * capable. Nothing in the KVM code should rely on 32bit features
- * anyway, only 64bit, so let the VMM do its worse.
- */
-#define AA32_ID_WRITABLE(name) {		\
-	ID_DESC(name),				\
-	.visibility = aa32_id_visibility,	\
-	.val = GENMASK(31, 0),			\
-}
-
-/* sys_reg_desc initialiser for cpufeature ID registers that need filtering */
-#define ID_FILTERED(sysreg, name, mask) {	\
-	ID_DESC(sysreg),				\
-	.set_user = set_##name,				\
-	.val = (mask),					\
-}
-
-/*
- * sys_reg_desc initialiser for architecturally unallocated cpufeature ID
- * register with encoding Op0=3, Op1=0, CRn=0, CRm=crm, Op2=op2
- * (1 <= crm < 8, 0 <= Op2 < 8).
- */
-#define ID_UNALLOCATED(crm, op2) {			\
-	.name = "S3_0_0_" #crm "_" #op2,		\
-	Op0(3), Op1(0), CRn(0), CRm(crm), Op2(op2),	\
-	ID_DESC_DEFAULT_CALLBACKS,			\
-	.visibility = raz_visibility,			\
-	.val = 0,					\
-}
-
-/*
- * sys_reg_desc initialiser for known ID registers that we hide from guests.
- * For now, these are exposed just like unallocated ID regs: they appear
- * RAZ for the guest.
- */
-#define ID_HIDDEN(name) {			\
-	ID_DESC(name),				\
-	.visibility = raz_visibility,		\
-	.val = 0,				\
-}
-
 static bool access_sp_el1(struct kvm_vcpu *vcpu,
 			  struct sys_reg_params *p,
 			  const struct sys_reg_desc *r)
@@ -3100,7 +2325,7 @@ static bool access_ras(struct kvm_vcpu *vcpu,
  * follows attempts to give a user / guest view consistent with the existing
  * ABI.
  */
-static bool access_imp_id_reg(struct kvm_vcpu *vcpu,
+bool access_imp_id_reg(struct kvm_vcpu *vcpu,
 			      struct sys_reg_params *p,
 			      const struct sys_reg_desc *r)
 {
@@ -3143,7 +2368,7 @@ static void init_imp_id_regs(void)
 	boot_cpu_aidr_val = read_sysreg(aidr_el1);
 }
 
-static u64 reset_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+ u64 reset_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 {
 	switch (reg_to_encoding(r)) {
 	case SYS_MIDR_EL1:
@@ -3158,48 +2383,6 @@ static u64 reset_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 	}
 }
 
-static int set_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
-			  u64 val)
-{
-	struct kvm *kvm = vcpu->kvm;
-	u64 expected;
-
-	guard(mutex)(&kvm->arch.config_lock);
-
-	expected = read_id_reg(vcpu, r);
-	if (expected == val)
-		return 0;
-
-	if (!test_bit(KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, &kvm->arch.flags))
-		return -EINVAL;
-
-	/*
-	 * Once the VM has started the ID registers are immutable. Reject the
-	 * write if userspace tries to change it.
-	 */
-	if (kvm_vm_has_ran_once(kvm))
-		return -EBUSY;
-
-	/*
-	 * Any value is allowed for the implementation ID registers so long as
-	 * it is within the writable mask.
-	 */
-	if ((val & r->val) != val)
-		return -EINVAL;
-
-	kvm_set_vm_id_reg(kvm, reg_to_encoding(r), val);
-	return 0;
-}
-
-#define IMPLEMENTATION_ID(reg, mask) {			\
-	SYS_DESC(SYS_##reg),				\
-	.access = access_imp_id_reg,			\
-	.get_user = get_id_reg,				\
-	.set_user = set_imp_id_reg,			\
-	.reset = reset_imp_id_reg,			\
-	.val = mask,					\
-	}
-
 static u64 reset_mdcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 {
 	__vcpu_assign_sys_reg(vcpu, r->reg, vcpu->kvm->arch.nr_pmu_counters);
@@ -4747,61 +3930,12 @@ static const struct sys_reg_desc cp15_64_regs[] = {
 	{ SYS_DESC(SYS_AARCH32_CNTVCTSS),     access_arch_timer },
 };
 
-static bool check_sysreg_table(const struct sys_reg_desc *table, unsigned int n,
-			       bool reset_check)
-{
-	unsigned int i;
-
-	for (i = 0; i < n; i++) {
-		if (reset_check && table[i].reg && !table[i].reset) {
-			kvm_err("sys_reg table %pS entry %d (%s) lacks reset\n",
-				&table[i], i, table[i].name);
-			return false;
-		}
-
-		if (i && cmp_sys_reg(&table[i-1], &table[i]) >= 0) {
-			kvm_err("sys_reg table %pS entry %d (%s -> %s) out of order\n",
-				&table[i], i, table[i - 1].name, table[i].name);
-			return false;
-		}
-	}
-
-	return true;
-}
-
 int kvm_handle_cp14_load_store(struct kvm_vcpu *vcpu)
 {
 	kvm_inject_undefined(vcpu);
 	return 1;
 }
 
-static void perform_access(struct kvm_vcpu *vcpu,
-			   struct sys_reg_params *params,
-			   const struct sys_reg_desc *r)
-{
-	trace_kvm_sys_access(*vcpu_pc(vcpu), params, r);
-
-	/* Check for regs disabled by runtime config */
-	if (sysreg_hidden(vcpu, r)) {
-		kvm_inject_undefined(vcpu);
-		return;
-	}
-
-	/*
-	 * Not having an accessor means that we have configured a trap
-	 * that we don't know how to handle. This certainly qualifies
-	 * as a gross bug that should be fixed right away.
-	 */
-	if (!r->access) {
-		bad_trap(vcpu, params, r, "register access");
-		return;
-	}
-
-	/* Skip instruction if instructed so */
-	if (likely(r->access(vcpu, params, r)))
-		kvm_incr_pc(vcpu);
-}
-
 /*
  * emulate_cp --  tries to match a sys_reg access in a handling table, and
  *                call the corresponding trap handler.
@@ -5256,26 +4390,6 @@ void kvm_sys_regs_create_debugfs(struct kvm *kvm)
 			    &sr_resx_fops);
 }
 
-static void reset_vm_ftr_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *reg)
-{
-	u32 id = reg_to_encoding(reg);
-	struct kvm *kvm = vcpu->kvm;
-
-	if (test_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags))
-		return;
-
-	kvm_set_vm_id_reg(kvm, id, reg->reset(vcpu, reg));
-}
-
-static void reset_vcpu_ftr_id_reg(struct kvm_vcpu *vcpu,
-				  const struct sys_reg_desc *reg)
-{
-	if (kvm_vcpu_initialized(vcpu))
-		return;
-
-	reg->reset(vcpu, reg);
-}
-
 /**
  * kvm_reset_sys_regs - sets system registers to reset value
  * @vcpu: The VCPU pointer
@@ -5352,321 +4466,25 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu)
  * Userspace API
  *****************************************************************************/
 
-static bool index_to_params(u64 id, struct sys_reg_params *params)
-{
-	switch (id & KVM_REG_SIZE_MASK) {
-	case KVM_REG_SIZE_U64:
-		/* Any unused index bits means it's not valid. */
-		if (id & ~(KVM_REG_ARCH_MASK | KVM_REG_SIZE_MASK
-			      | KVM_REG_ARM_COPROC_MASK
-			      | KVM_REG_ARM64_SYSREG_OP0_MASK
-			      | KVM_REG_ARM64_SYSREG_OP1_MASK
-			      | KVM_REG_ARM64_SYSREG_CRN_MASK
-			      | KVM_REG_ARM64_SYSREG_CRM_MASK
-			      | KVM_REG_ARM64_SYSREG_OP2_MASK))
-			return false;
-		params->Op0 = ((id & KVM_REG_ARM64_SYSREG_OP0_MASK)
-			       >> KVM_REG_ARM64_SYSREG_OP0_SHIFT);
-		params->Op1 = ((id & KVM_REG_ARM64_SYSREG_OP1_MASK)
-			       >> KVM_REG_ARM64_SYSREG_OP1_SHIFT);
-		params->CRn = ((id & KVM_REG_ARM64_SYSREG_CRN_MASK)
-			       >> KVM_REG_ARM64_SYSREG_CRN_SHIFT);
-		params->CRm = ((id & KVM_REG_ARM64_SYSREG_CRM_MASK)
-			       >> KVM_REG_ARM64_SYSREG_CRM_SHIFT);
-		params->Op2 = ((id & KVM_REG_ARM64_SYSREG_OP2_MASK)
-			       >> KVM_REG_ARM64_SYSREG_OP2_SHIFT);
-		return true;
-	default:
-		return false;
-	}
-}
-
-const struct sys_reg_desc *get_reg_by_id(u64 id,
-					 const struct sys_reg_desc table[],
-					 unsigned int num)
-{
-	struct sys_reg_params params;
-
-	if (!index_to_params(id, &params))
-		return NULL;
-
-	return find_reg(&params, table, num);
-}
-
-/* Decode an index value, and find the sys_reg_desc entry. */
-static const struct sys_reg_desc *
-id_to_sys_reg_desc(struct kvm_vcpu *vcpu, u64 id,
-		   const struct sys_reg_desc table[], unsigned int num)
-
-{
-	const struct sys_reg_desc *r;
-
-	/* We only do sys_reg for now. */
-	if ((id & KVM_REG_ARM_COPROC_MASK) != KVM_REG_ARM64_SYSREG)
-		return NULL;
-
-	r = get_reg_by_id(id, table, num);
-
-	/* Not saved in the sys_reg array and not otherwise accessible? */
-	if (r && (!(r->reg || r->get_user) || sysreg_hidden(vcpu, r)))
-		r = NULL;
-
-	return r;
-}
-
-static int demux_c15_get(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
-{
-	u32 val;
-	u32 __user *uval = uaddr;
-
-	/* Fail if we have unknown bits set. */
-	if (id & ~(KVM_REG_ARCH_MASK|KVM_REG_SIZE_MASK|KVM_REG_ARM_COPROC_MASK
-		   | ((1 << KVM_REG_ARM_COPROC_SHIFT)-1)))
-		return -ENOENT;
-
-	switch (id & KVM_REG_ARM_DEMUX_ID_MASK) {
-	case KVM_REG_ARM_DEMUX_ID_CCSIDR:
-		if (KVM_REG_SIZE(id) != 4)
-			return -ENOENT;
-		val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
-			>> KVM_REG_ARM_DEMUX_VAL_SHIFT;
-		if (val >= CSSELR_MAX)
-			return -ENOENT;
-
-		return put_user(get_ccsidr(vcpu, val), uval);
-	default:
-		return -ENOENT;
-	}
-}
-
-static int demux_c15_set(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
-{
-	u32 val, newval;
-	u32 __user *uval = uaddr;
-
-	/* Fail if we have unknown bits set. */
-	if (id & ~(KVM_REG_ARCH_MASK|KVM_REG_SIZE_MASK|KVM_REG_ARM_COPROC_MASK
-		   | ((1 << KVM_REG_ARM_COPROC_SHIFT)-1)))
-		return -ENOENT;
-
-	switch (id & KVM_REG_ARM_DEMUX_ID_MASK) {
-	case KVM_REG_ARM_DEMUX_ID_CCSIDR:
-		if (KVM_REG_SIZE(id) != 4)
-			return -ENOENT;
-		val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
-			>> KVM_REG_ARM_DEMUX_VAL_SHIFT;
-		if (val >= CSSELR_MAX)
-			return -ENOENT;
-
-		if (get_user(newval, uval))
-			return -EFAULT;
-
-		return set_ccsidr(vcpu, val, newval);
-	default:
-		return -ENOENT;
-	}
-}
-
-static u64 kvm_one_reg_to_id(const struct kvm_one_reg *reg)
-{
-	switch(reg->id) {
-	case KVM_REG_ARM_TIMER_CVAL:
-		return TO_ARM64_SYS_REG(CNTV_CVAL_EL0);
-	case KVM_REG_ARM_TIMER_CNT:
-		return TO_ARM64_SYS_REG(CNTVCT_EL0);
-	default:
-		return reg->id;
-	}
-}
-
-int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
-			 const struct sys_reg_desc table[], unsigned int num)
-{
-	u64 __user *uaddr = (u64 __user *)(unsigned long)reg->addr;
-	const struct sys_reg_desc *r;
-	u64 id = kvm_one_reg_to_id(reg);
-	u64 val;
-	int ret;
-
-	r = id_to_sys_reg_desc(vcpu, id, table, num);
-	if (!r || sysreg_hidden(vcpu, r))
-		return -ENOENT;
-
-	if (r->get_user) {
-		ret = (r->get_user)(vcpu, r, &val);
-	} else {
-		val = __vcpu_sys_reg(vcpu, r->reg);
-		ret = 0;
-	}
-
-	if (!ret)
-		ret = put_user(val, uaddr);
-
-	return ret;
-}
-
 int kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 {
-	void __user *uaddr = (void __user *)(unsigned long)reg->addr;
-
-	if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
-		return demux_c15_get(vcpu, reg->id, uaddr);
-
-	return kvm_sys_reg_get_user(vcpu, reg,
-				    sys_reg_descs, ARRAY_SIZE(sys_reg_descs));
-}
-
-int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
-			 const struct sys_reg_desc table[], unsigned int num)
-{
-	u64 __user *uaddr = (u64 __user *)(unsigned long)reg->addr;
-	const struct sys_reg_desc *r;
-	u64 id = kvm_one_reg_to_id(reg);
-	u64 val;
-	int ret;
-
-	if (get_user(val, uaddr))
-		return -EFAULT;
-
-	r = id_to_sys_reg_desc(vcpu, id, table, num);
-	if (!r || sysreg_hidden(vcpu, r))
-		return -ENOENT;
-
-	if (sysreg_user_write_ignore(vcpu, r))
-		return 0;
-
-	if (r->set_user) {
-		ret = (r->set_user)(vcpu, r, val);
-	} else {
-		__vcpu_assign_sys_reg(vcpu, r->reg, val);
-		ret = 0;
-	}
-
-	return ret;
+	return __kvm_arm_sys_reg_get_reg(vcpu, reg, ARRAY_SIZE(sys_reg_descs), sys_reg_descs);
 }
 
 int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 {
-	void __user *uaddr = (void __user *)(unsigned long)reg->addr;
-
-	if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
-		return demux_c15_set(vcpu, reg->id, uaddr);
-
-	return kvm_sys_reg_set_user(vcpu, reg,
-				    sys_reg_descs, ARRAY_SIZE(sys_reg_descs));
-}
-
-static unsigned int num_demux_regs(void)
-{
-	return CSSELR_MAX;
-}
-
-static int write_demux_regids(u64 __user *uindices)
-{
-	u64 val = KVM_REG_ARM64 | KVM_REG_SIZE_U32 | KVM_REG_ARM_DEMUX;
-	unsigned int i;
-
-	val |= KVM_REG_ARM_DEMUX_ID_CCSIDR;
-	for (i = 0; i < CSSELR_MAX; i++) {
-		if (put_user(val | i, uindices))
-			return -EFAULT;
-		uindices++;
-	}
-	return 0;
-}
-
-static u64 sys_reg_to_index(const struct sys_reg_desc *reg)
-{
-	return (KVM_REG_ARM64 | KVM_REG_SIZE_U64 |
-		KVM_REG_ARM64_SYSREG |
-		(reg->Op0 << KVM_REG_ARM64_SYSREG_OP0_SHIFT) |
-		(reg->Op1 << KVM_REG_ARM64_SYSREG_OP1_SHIFT) |
-		(reg->CRn << KVM_REG_ARM64_SYSREG_CRN_SHIFT) |
-		(reg->CRm << KVM_REG_ARM64_SYSREG_CRM_SHIFT) |
-		(reg->Op2 << KVM_REG_ARM64_SYSREG_OP2_SHIFT));
-}
-
-static bool copy_reg_to_user(const struct sys_reg_desc *reg, u64 __user **uind)
-{
-	u64 idx;
-
-	if (!*uind)
-		return true;
-
-	switch (reg_to_encoding(reg)) {
-	case SYS_CNTV_CVAL_EL0:
-		idx = KVM_REG_ARM_TIMER_CVAL;
-		break;
-	case SYS_CNTVCT_EL0:
-		idx = KVM_REG_ARM_TIMER_CNT;
-		break;
-	default:
-		idx = sys_reg_to_index(reg);
-	}
-
-	if (put_user(idx, *uind))
-		return false;
-
-	(*uind)++;
-	return true;
-}
-
-static int walk_one_sys_reg(const struct kvm_vcpu *vcpu,
-			    const struct sys_reg_desc *rd,
-			    u64 __user **uind,
-			    unsigned int *total)
-{
-	/*
-	 * Ignore registers we trap but don't save,
-	 * and for which no custom user accessor is provided.
-	 */
-	if (!(rd->reg || rd->get_user))
-		return 0;
-
-	if (sysreg_hidden(vcpu, rd))
-		return 0;
-
-	if (!copy_reg_to_user(rd, uind))
-		return -EFAULT;
-
-	(*total)++;
-	return 0;
-}
-
-/* Assumed ordered tables, see kvm_sys_reg_table_init. */
-static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind)
-{
-	const struct sys_reg_desc *i2, *end2;
-	unsigned int total = 0;
-	int err;
-
-	i2 = sys_reg_descs;
-	end2 = sys_reg_descs + ARRAY_SIZE(sys_reg_descs);
 
-	while (i2 != end2) {
-		err = walk_one_sys_reg(vcpu, i2++, &uind, &total);
-		if (err)
-			return err;
-	}
-	return total;
+	return __kvm_arm_sys_reg_set_reg(vcpu, reg, ARRAY_SIZE(sys_reg_descs), sys_reg_descs);
 }
 
 unsigned long kvm_arm_num_sys_reg_descs(struct kvm_vcpu *vcpu)
 {
-	return num_demux_regs()
-		+ walk_sys_regs(vcpu, (u64 __user *)NULL);
-}
+	return __kvm_arm_num_sys_reg_descs(vcpu, ARRAY_SIZE(sys_reg_descs), sys_reg_descs);}
 
-int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
+int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu * vcpu, u64 __user * uindices)
 {
-	int err;
-
-	err = walk_sys_regs(vcpu, uindices);
-	if (err < 0)
-		return err;
-	uindices += err;
-
-	return write_demux_regids(uindices);
+	return __kvm_arm_copy_sys_reg_indices(
+		vcpu, uindices, ARRAY_SIZE(sys_reg_descs), sys_reg_descs);
 }
 
 #define KVM_ARM_FEATURE_ID_RANGE_INDEX(r)			\
diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h
deleted file mode 100644
index 75d581050b09..000000000000
--- a/arch/arm64/kvm/sys_regs.h
+++ /dev/null
@@ -1,363 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (C) 2012,2013 - ARM Ltd
- * Author: Marc Zyngier <marc.zyngier@arm.com>
- *
- * Derived from arch/arm/kvm/coproc.h
- * Copyright (C) 2012 - Virtual Open Systems and Columbia University
- * Authors: Christoffer Dall <c.dall@virtualopensystems.com>
- */
-
-#ifndef __ARM64_KVM_SYS_REGS_LOCAL_H__
-#define __ARM64_KVM_SYS_REGS_LOCAL_H__
-
-#include <linux/bsearch.h>
-
-#define reg_to_encoding(x)						\
-	sys_reg((u32)(x)->Op0, (u32)(x)->Op1,				\
-		(u32)(x)->CRn, (u32)(x)->CRm, (u32)(x)->Op2)
-
-struct sys_reg_params {
-	u8	Op0;
-	u8	Op1;
-	u8	CRn;
-	u8	CRm;
-	u8	Op2;
-	u64	regval;
-	bool	is_write;
-};
-
-#define encoding_to_params(reg)						\
-	((struct sys_reg_params){ .Op0 = sys_reg_Op0(reg),		\
-				  .Op1 = sys_reg_Op1(reg),		\
-				  .CRn = sys_reg_CRn(reg),		\
-				  .CRm = sys_reg_CRm(reg),		\
-				  .Op2 = sys_reg_Op2(reg) })
-
-#define esr_sys64_to_params(esr)                                               \
-	((struct sys_reg_params){ .Op0 = ((esr) >> 20) & 3,                    \
-				  .Op1 = ((esr) >> 14) & 0x7,                  \
-				  .CRn = ((esr) >> 10) & 0xf,                  \
-				  .CRm = ((esr) >> 1) & 0xf,                   \
-				  .Op2 = ((esr) >> 17) & 0x7,                  \
-				  .is_write = !((esr) & 1) })
-
-#define esr_cp1x_32_to_params(esr)						\
-	((struct sys_reg_params){ .Op1 = ((esr) >> 14) & 0x7,			\
-				  .CRn = ((esr) >> 10) & 0xf,			\
-				  .CRm = ((esr) >> 1) & 0xf,			\
-				  .Op2 = ((esr) >> 17) & 0x7,			\
-				  .is_write = !((esr) & 1) })
-
-/*
- * The Feature ID space is defined as the System register space in AArch64
- * with op0==3, op1=={0, 1, 3}, CRn==0, CRm=={0-7}, op2=={0-7}.
- */
-static inline bool in_feat_id_space(struct sys_reg_params *p)
-{
-	return (p->Op0 == 3 && !(p->Op1 & 0b100) && p->Op1 != 2 &&
-		p->CRn == 0 && !(p->CRm & 0b1000));
-}
-
-struct sys_reg_desc {
-	/* Sysreg string for debug */
-	const char *name;
-
-	enum {
-		AA32_DIRECT,
-		AA32_LO,
-		AA32_HI,
-	} aarch32_map;
-
-	/* MRS/MSR instruction which accesses it. */
-	u8	Op0;
-	u8	Op1;
-	u8	CRn;
-	u8	CRm;
-	u8	Op2;
-
-	/* Trapped access from guest, if non-NULL. */
-	bool (*access)(struct kvm_vcpu *,
-		       struct sys_reg_params *,
-		       const struct sys_reg_desc *);
-
-	/*
-	 * Initialization for vcpu. Return initialized value, or KVM
-	 * sanitized value for ID registers.
-	 */
-	u64 (*reset)(struct kvm_vcpu *, const struct sys_reg_desc *);
-
-	/* Index into sys_reg[], or 0 if we don't need to save it. */
-	int reg;
-
-	/* Value (usually reset value), or write mask for idregs */
-	u64 val;
-
-	/* Custom get/set_user functions, fallback to generic if NULL */
-	int (*get_user)(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
-			u64 *val);
-	int (*set_user)(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
-			u64 val);
-
-	/* Return mask of REG_* runtime visibility overrides */
-	unsigned int (*visibility)(const struct kvm_vcpu *vcpu,
-				   const struct sys_reg_desc *rd);
-};
-
-#define REG_HIDDEN		(1 << 0) /* hidden from userspace and guest */
-#define REG_RAZ			(1 << 1) /* RAZ from userspace and guest */
-#define REG_USER_WI		(1 << 2) /* WI from userspace only */
-
-static inline unsigned int raz_visibility(const struct kvm_vcpu *vcpu,
-				   const struct sys_reg_desc *r)
-{
-	return REG_RAZ;
-}
-
-static __printf(2, 3)
-inline void print_sys_reg_msg(const struct sys_reg_params *p,
-				       char *fmt, ...)
-{
-	va_list va;
-
-	va_start(va, fmt);
-	/* Look, we even formatted it for you to paste into the table! */
-	kvm_pr_unimpl("%pV { Op0(%2u), Op1(%2u), CRn(%2u), CRm(%2u), Op2(%2u), func_%s },\n",
-		      &(struct va_format){ fmt, &va },
-		      p->Op0, p->Op1, p->CRn, p->CRm, p->Op2, str_write_read(p->is_write));
-	va_end(va);
-}
-
-static inline void print_sys_reg_instr(const struct sys_reg_params *p)
-{
-	/* GCC warns on an empty format string */
-	print_sys_reg_msg(p, "%s", "");
-}
-
-static inline bool ignore_write(struct kvm_vcpu *vcpu,
-				const struct sys_reg_params *p)
-{
-	return true;
-}
-
-static inline bool read_zero(struct kvm_vcpu *vcpu,
-			     struct sys_reg_params *p)
-{
-	p->regval = 0;
-	return true;
-}
-
-/* Reset functions */
-static inline u64 reset_unknown(struct kvm_vcpu *vcpu,
-				 const struct sys_reg_desc *r)
-{
-	BUG_ON(!r->reg);
-	BUG_ON(r->reg >= NR_SYS_REGS);
-	__vcpu_assign_sys_reg(vcpu, r->reg, 0x1de7ec7edbadc0deULL);
-	return __vcpu_sys_reg(vcpu, r->reg);
-}
-
-static inline u64 reset_val(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
-{
-	BUG_ON(!r->reg);
-	BUG_ON(r->reg >= NR_SYS_REGS);
-	__vcpu_assign_sys_reg(vcpu, r->reg, r->val);
-	return __vcpu_sys_reg(vcpu, r->reg);
-}
-
-static inline unsigned int sysreg_visibility(const struct kvm_vcpu *vcpu,
-					     const struct sys_reg_desc *r)
-{
-	if (likely(!r->visibility))
-		return 0;
-
-	return r->visibility(vcpu, r);
-}
-
-static inline bool sysreg_hidden(const struct kvm_vcpu *vcpu,
-				 const struct sys_reg_desc *r)
-{
-	return sysreg_visibility(vcpu, r) & REG_HIDDEN;
-}
-
-static inline bool sysreg_visible_as_raz(const struct kvm_vcpu *vcpu,
-					 const struct sys_reg_desc *r)
-{
-	return sysreg_visibility(vcpu, r) & REG_RAZ;
-}
-
-static inline bool sysreg_user_write_ignore(const struct kvm_vcpu *vcpu,
-					    const struct sys_reg_desc *r)
-{
-	return sysreg_visibility(vcpu, r) & REG_USER_WI;
-}
-
-static inline int cmp_sys_reg(const struct sys_reg_desc *i1,
-			      const struct sys_reg_desc *i2)
-{
-	BUG_ON(i1 == i2);
-	if (!i1)
-		return 1;
-	else if (!i2)
-		return -1;
-	if (i1->Op0 != i2->Op0)
-		return i1->Op0 - i2->Op0;
-	if (i1->Op1 != i2->Op1)
-		return i1->Op1 - i2->Op1;
-	if (i1->CRn != i2->CRn)
-		return i1->CRn - i2->CRn;
-	if (i1->CRm != i2->CRm)
-		return i1->CRm - i2->CRm;
-	return i1->Op2 - i2->Op2;
-}
-
-static inline int match_sys_reg(const void *key, const void *elt)
-{
-	const unsigned long pval = (unsigned long)key;
-	const struct sys_reg_desc *r = elt;
-
-	return pval - reg_to_encoding(r);
-}
-
-static inline const struct sys_reg_desc *
-find_reg(const struct sys_reg_params *params, const struct sys_reg_desc table[],
-	 unsigned int num)
-{
-	unsigned long pval = reg_to_encoding(params);
-
-	return __inline_bsearch((void *)pval, table, num, sizeof(table[0]), match_sys_reg);
-}
-
-const struct sys_reg_desc *get_reg_by_id(u64 id,
-					 const struct sys_reg_desc table[],
-					 unsigned int num);
-
-int kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *);
-int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *);
-int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
-			 const struct sys_reg_desc table[], unsigned int num);
-int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
-			 const struct sys_reg_desc table[], unsigned int num);
-
-bool triage_sysreg_trap(struct kvm_vcpu *vcpu, int *sr_index);
-
-int kvm_finalize_sys_regs(struct kvm_vcpu *vcpu);
-
-u64 kvm_read_sanitised_id_reg(struct kvm_vcpu *vcpu,
-			      const struct sys_reg_desc *r);
-
-/* Implemented by each architecture */
-u64 kvm_sanitised_host_ftr_reg(u32 id);
-
-#define AA32(_x)	.aarch32_map = AA32_##_x
-#define Op0(_x) 	.Op0 = _x
-#define Op1(_x) 	.Op1 = _x
-#define CRn(_x)		.CRn = _x
-#define CRm(_x) 	.CRm = _x
-#define Op2(_x) 	.Op2 = _x
-
-#define SYS_DESC(reg)					\
-	.name = #reg,					\
-	Op0(sys_reg_Op0(reg)), Op1(sys_reg_Op1(reg)),	\
-	CRn(sys_reg_CRn(reg)), CRm(sys_reg_CRm(reg)),	\
-	Op2(sys_reg_Op2(reg))
-
-#define CP15_SYS_DESC(reg)				\
-	.name = #reg,					\
-	.aarch32_map = AA32_DIRECT,			\
-	Op0(0), Op1(sys_reg_Op1(reg)),			\
-	CRn(sys_reg_CRn(reg)), CRm(sys_reg_CRm(reg)),	\
-	Op2(sys_reg_Op2(reg))
-
-/*
- * Since reset() callback and field val are not used for idregs, they will be
- * used for specific purposes for idregs.
- * The reset() would return KVM sanitised register value. The value would be the
- * same as the host kernel sanitised value if there is no KVM sanitisation.
- * The val would be used as a mask indicating writable fields for the idreg.
- * Only bits with 1 are writable from userspace. This mask might not be
- * necessary in the future whenever all ID registers are enabled as writable
- * from userspace.
- */
-
-#define ID_DESC_DEFAULT_CALLBACKS		\
-	.access	= access_id_reg,		\
-	.get_user = get_id_reg,			\
-	.set_user = set_id_reg,			\
-	.visibility = id_visibility,		\
-	.reset = kvm_read_sanitised_id_reg
-
-#define ID_DESC(name)				\
-	SYS_DESC(SYS_##name),			\
-	ID_DESC_DEFAULT_CALLBACKS
-
-/* sys_reg_desc initialiser for known cpufeature ID registers */
-#define ID_SANITISED(name) {			\
-	ID_DESC(name),				\
-	.val = 0,				\
-}
-
-/* sys_reg_desc initialiser for writable ID registers */
-#define ID_WRITABLE(name, mask) {		\
-	ID_DESC(name),				\
-	.val = mask,				\
-}
-
-/*
- * 32bit ID regs are fully writable when the guest is 32bit
- * capable. Nothing in the KVM code should rely on 32bit features
- * anyway, only 64bit, so let the VMM do its worse.
- */
-#define AA32_ID_WRITABLE(name) {		\
-	ID_DESC(name),				\
-	.visibility = aa32_id_visibility,	\
-	.val = GENMASK(31, 0),			\
-}
-
-/* sys_reg_desc initialiser for cpufeature ID registers that need filtering */
-#define ID_FILTERED(sysreg, name, mask) {	\
-	ID_DESC(sysreg),				\
-	.set_user = set_##name,				\
-	.val = (mask),					\
-}
-
-/*
- * sys_reg_desc initialiser for architecturally unallocated cpufeature ID
- * register with encoding Op0=3, Op1=0, CRn=0, CRm=crm, Op2=op2
- * (1 <= crm < 8, 0 <= Op2 < 8).
- */
-#define ID_UNALLOCATED(crm, op2) {			\
-	.name = "S3_0_0_" #crm "_" #op2,		\
-	Op0(3), Op1(0), CRn(0), CRm(crm), Op2(op2),	\
-	ID_DESC_DEFAULT_CALLBACKS,			\
-	.visibility = raz_visibility,			\
-	.val = 0,					\
-}
-
-/*
- * sys_reg_desc initialiser for known ID registers that we hide from guests.
- * For now, these are exposed just like unallocated ID regs: they appear
- * RAZ for the guest.
- */
-#define ID_HIDDEN(name) {			\
-	ID_DESC(name),				\
-	.visibility = raz_visibility,		\
-	.val = 0,				\
-}
-#define ID_REG_LIMIT_FIELD_ENUM(val, reg, field, limit)			       \
-({									       \
-	u64 __f_val = FIELD_GET(reg##_##field##_MASK, val);		       \
-	(val) &= ~reg##_##field##_MASK;					       \
-	(val) |= FIELD_PREP(reg##_##field##_MASK,			       \
-			    min(__f_val,				       \
-				(u64)SYS_FIELD_VALUE(reg, field, limit)));     \
-	(val);								       \
-})
-
-#define TO_ARM64_SYS_REG(r)	ARM64_SYS_REG(sys_reg_Op0(SYS_ ## r),	\
-					      sys_reg_Op1(SYS_ ## r),	\
-					      sys_reg_CRn(SYS_ ## r),	\
-					      sys_reg_CRm(SYS_ ## r),	\
-					      sys_reg_Op2(SYS_ ## r))
-
-#endif /* __ARM64_KVM_SYS_REGS_LOCAL_H__ */
diff --git a/arch/arm64/kvm/trace_handle_exit.h b/arch/arm64/kvm/trace_handle_exit.h
index a7ab9a3bbed0..280d79469551 100644
--- a/arch/arm64/kvm/trace_handle_exit.h
+++ b/arch/arm64/kvm/trace_handle_exit.h
@@ -3,7 +3,7 @@
 #define _TRACE_HANDLE_EXIT_ARM64_KVM_H
 
 #include <linux/tracepoint.h>
-#include "sys_regs.h"
+#include <kvm/arm64/sys_regs.h>
 
 #undef TRACE_SYSTEM
 #define TRACE_SYSTEM kvm
@@ -82,40 +82,6 @@ TRACE_EVENT(kvm_handle_sys_reg,
 	TP_printk("HSR 0x%08lx", __entry->hsr)
 );
 
-TRACE_EVENT(kvm_sys_access,
-	TP_PROTO(unsigned long vcpu_pc, struct sys_reg_params *params, const struct sys_reg_desc *reg),
-	TP_ARGS(vcpu_pc, params, reg),
-
-	TP_STRUCT__entry(
-		__field(unsigned long,			vcpu_pc)
-		__field(bool,				is_write)
-		__field(const char *,			name)
-		__field(u8,				Op0)
-		__field(u8,				Op1)
-		__field(u8,				CRn)
-		__field(u8,				CRm)
-		__field(u8,				Op2)
-	),
-
-	TP_fast_assign(
-		__entry->vcpu_pc = vcpu_pc;
-		__entry->is_write = params->is_write;
-		__entry->name = reg->name;
-		__entry->Op0 = reg->Op0;
-		__entry->Op0 = reg->Op0;
-		__entry->Op1 = reg->Op1;
-		__entry->CRn = reg->CRn;
-		__entry->CRm = reg->CRm;
-		__entry->Op2 = reg->Op2;
-	),
-
-	TP_printk("PC: %lx %s (%d,%d,%d,%d,%d) %s",
-		  __entry->vcpu_pc, __entry->name ?: "UNKN",
-		  __entry->Op0, __entry->Op1, __entry->CRn,
-		  __entry->CRm, __entry->Op2,
-		  str_write_read(__entry->is_write))
-);
-
 TRACE_EVENT(kvm_set_guest_debug,
 	TP_PROTO(struct kvm_vcpu *vcpu, __u32 guest_debug),
 	TP_ARGS(vcpu, guest_debug),
diff --git a/arch/arm64/kvm/vgic-sys-reg-v3.c b/arch/arm64/kvm/vgic-sys-reg-v3.c
index bdc2d57370b2..b6cbe8ea679e 100644
--- a/arch/arm64/kvm/vgic-sys-reg-v3.c
+++ b/arch/arm64/kvm/vgic-sys-reg-v3.c
@@ -8,7 +8,7 @@
 #include <linux/kvm_host.h>
 #include <asm/kvm_emulate.h>
 #include "vgic/vgic.h"
-#include "sys_regs.h"
+#include <kvm/arm64/sys_regs.h>
 
 static int set_gic_ctlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
 			u64 val)
diff --git a/arch/arm64/kvm/vgic/vgic-init.c b/arch/arm64/kvm/vgic/vgic-init.c
index 933983bb2005..d75c32f6909e 100644
--- a/arch/arm64/kvm/vgic/vgic-init.c
+++ b/arch/arm64/kvm/vgic/vgic-init.c
@@ -10,6 +10,7 @@
 #include <kvm/arm_vgic.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_mmu.h>
+#include <kvm/arm64/sys_regs.h>
 #include "vgic.h"
 
 /*
diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index d6d9e3ad7a8e..fcbe510cb868 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -145,6 +145,8 @@ struct kvm_arch {
 	DECLARE_BITMAP(vcpu_features, KVM_VCPU_MAX_FEATURES);
 
 	unsigned long mem_limit;
+
+	struct kvm_vm_id_regs id_regs;
 };
 
 static inline bool __vcpu_has_feature(const struct kvm_arch *ka, int feature)
@@ -218,9 +220,6 @@ static inline void kvm_arch_async_page_present_queued(struct kvm_vcpu *vcpu)
 
 #define kvm_supports_32bit_el0() false
 
-#define vcpu_read_sys_reg(_v, _r) 0xbad1234bad
-#define vcpu_write_sys_reg(_v, _p, _r) ((void)0)
-
 #define __vcpu_sys_reg(__vcpu, __reg) \
 	vcpu_read_sys_reg(__vcpu, __reg)
 
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index 636bbeda98a8..48418c46e451 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -690,6 +690,15 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
 	return -ENOIOCTLCMD;
 }
 
+void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, enum vcpu_sysreg reg)
+{
+}
+
+u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg)
+{
+	return 0xbad12324bad;
+}
+
 static int __init kvm_s390_arm64_init(void)
 {
 	if (!sclp.has_aef) {
diff --git a/include/kvm/arm64/kvm_host.h b/include/kvm/arm64/kvm_host.h
index 379942225d5f..deb115a737b5 100644
--- a/include/kvm/arm64/kvm_host.h
+++ b/include/kvm/arm64/kvm_host.h
@@ -233,7 +233,17 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa);
 /* Unhandled SEAs are taken to userspace */
 #define KVM_ARCH_FLAG_EXIT_SEA				11
 
+#define kvm_vm_has_ran_once(kvm)					\
+	(test_bit(KVM_ARCH_FLAG_HAS_RAN_ONCE, &(kvm)->arch.flags))
+
 /* Implemented in architecture specific code */
 unsigned long system_supported_vcpu_features(void);
 
+enum vcpu_sysreg;
+void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, enum vcpu_sysreg reg);
+u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg);
+
+int kvm_handle_sys_reg(struct kvm_vcpu *vcpu);
+void kvm_reset_sys_regs(struct kvm_vcpu *vcpu);
+
 #endif /* __KVM_ARM64_KVM_HOST_H */
diff --git a/include/kvm/arm64/sys_regs.h b/include/kvm/arm64/sys_regs.h
new file mode 100644
index 000000000000..ca652274072a
--- /dev/null
+++ b/include/kvm/arm64/sys_regs.h
@@ -0,0 +1,548 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2012,2013 - ARM Ltd
+ * Author: Marc Zyngier <marc.zyngier@arm.com>
+ *
+ * Derived from arch/arm/kvm/coproc.h
+ * Copyright (C) 2012 - Virtual Open Systems and Columbia University
+ * Authors: Christoffer Dall <c.dall@virtualopensystems.com>
+ */
+
+#ifndef __ARM64_KVM_SYS_REGS_LOCAL_H__
+#define __ARM64_KVM_SYS_REGS_LOCAL_H__
+
+#include <linux/bsearch.h>
+
+#include <asm/kvm_emulate.h>
+
+#define reg_to_encoding(x)						\
+	sys_reg((u32)(x)->Op0, (u32)(x)->Op1,				\
+		(u32)(x)->CRn, (u32)(x)->CRm, (u32)(x)->Op2)
+
+struct sys_reg_params {
+	u8	Op0;
+	u8	Op1;
+	u8	CRn;
+	u8	CRm;
+	u8	Op2;
+	u64	regval;
+	bool	is_write;
+};
+
+#define encoding_to_params(reg)						\
+	((struct sys_reg_params){ .Op0 = sys_reg_Op0(reg),		\
+				  .Op1 = sys_reg_Op1(reg),		\
+				  .CRn = sys_reg_CRn(reg),		\
+				  .CRm = sys_reg_CRm(reg),		\
+				  .Op2 = sys_reg_Op2(reg) })
+
+#define esr_sys64_to_params(esr)                                               \
+	((struct sys_reg_params){ .Op0 = ((esr) >> 20) & 3,                    \
+				  .Op1 = ((esr) >> 14) & 0x7,                  \
+				  .CRn = ((esr) >> 10) & 0xf,                  \
+				  .CRm = ((esr) >> 1) & 0xf,                   \
+				  .Op2 = ((esr) >> 17) & 0x7,                  \
+				  .is_write = !((esr) & 1) })
+
+#define esr_cp1x_32_to_params(esr)						\
+	((struct sys_reg_params){ .Op1 = ((esr) >> 14) & 0x7,			\
+				  .CRn = ((esr) >> 10) & 0xf,			\
+				  .CRm = ((esr) >> 1) & 0xf,			\
+				  .Op2 = ((esr) >> 17) & 0x7,			\
+				  .is_write = !((esr) & 1) })
+
+/*
+ * The Feature ID space is defined as the System register space in AArch64
+ * with op0==3, op1=={0, 1, 3}, CRn==0, CRm=={0-7}, op2=={0-7}.
+ */
+static inline bool in_feat_id_space(struct sys_reg_params *p)
+{
+	return (p->Op0 == 3 && !(p->Op1 & 0b100) && p->Op1 != 2 &&
+		p->CRn == 0 && !(p->CRm & 0b1000));
+}
+
+struct sys_reg_desc {
+	/* Sysreg string for debug */
+	const char *name;
+
+	enum {
+		AA32_DIRECT,
+		AA32_LO,
+		AA32_HI,
+	} aarch32_map;
+
+	/* MRS/MSR instruction which accesses it. */
+	u8	Op0;
+	u8	Op1;
+	u8	CRn;
+	u8	CRm;
+	u8	Op2;
+
+	/* Trapped access from guest, if non-NULL. */
+	bool (*access)(struct kvm_vcpu *,
+		       struct sys_reg_params *,
+		       const struct sys_reg_desc *);
+
+	/*
+	 * Initialization for vcpu. Return initialized value, or KVM
+	 * sanitized value for ID registers.
+	 */
+	u64 (*reset)(struct kvm_vcpu *, const struct sys_reg_desc *);
+
+	/* Index into sys_reg[], or 0 if we don't need to save it. */
+	int reg;
+
+	/* Value (usually reset value), or write mask for idregs */
+	u64 val;
+
+	/* Custom get/set_user functions, fallback to generic if NULL */
+	int (*get_user)(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+			u64 *val);
+	int (*set_user)(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+			u64 val);
+
+	/* Return mask of REG_* runtime visibility overrides */
+	unsigned int (*visibility)(const struct kvm_vcpu *vcpu,
+				   const struct sys_reg_desc *rd);
+};
+
+#define REG_HIDDEN		(1 << 0) /* hidden from userspace and guest */
+#define REG_RAZ			(1 << 1) /* RAZ from userspace and guest */
+#define REG_USER_WI		(1 << 2) /* WI from userspace only */
+
+static inline unsigned int raz_visibility(const struct kvm_vcpu *vcpu,
+				   const struct sys_reg_desc *r)
+{
+	return REG_RAZ;
+}
+
+static __printf(2, 3)
+inline void print_sys_reg_msg(const struct sys_reg_params *p,
+				       char *fmt, ...)
+{
+	va_list va;
+
+	va_start(va, fmt);
+	/* Look, we even formatted it for you to paste into the table! */
+	kvm_pr_unimpl("%pV { Op0(%2u), Op1(%2u), CRn(%2u), CRm(%2u), Op2(%2u), func_%s },\n",
+		      &(struct va_format){ fmt, &va },
+		      p->Op0, p->Op1, p->CRn, p->CRm, p->Op2, str_write_read(p->is_write));
+	va_end(va);
+}
+
+static inline void print_sys_reg_instr(const struct sys_reg_params *p)
+{
+	/* GCC warns on an empty format string */
+	print_sys_reg_msg(p, "%s", "");
+}
+
+static inline bool ignore_write(struct kvm_vcpu *vcpu,
+				const struct sys_reg_params *p)
+{
+	return true;
+}
+
+static inline bool read_zero(struct kvm_vcpu *vcpu,
+			     struct sys_reg_params *p)
+{
+	p->regval = 0;
+	return true;
+}
+
+/* Reset functions */
+static inline u64 reset_unknown(struct kvm_vcpu *vcpu,
+				 const struct sys_reg_desc *r)
+{
+	BUG_ON(!r->reg);
+	BUG_ON(r->reg >= NR_SYS_REGS);
+	__vcpu_assign_sys_reg(vcpu, r->reg, 0x1de7ec7edbadc0deULL);
+	return __vcpu_sys_reg(vcpu, r->reg);
+}
+
+static inline u64 reset_val(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	BUG_ON(!r->reg);
+	BUG_ON(r->reg >= NR_SYS_REGS);
+	__vcpu_assign_sys_reg(vcpu, r->reg, r->val);
+	return __vcpu_sys_reg(vcpu, r->reg);
+}
+
+static inline unsigned int sysreg_visibility(const struct kvm_vcpu *vcpu,
+					     const struct sys_reg_desc *r)
+{
+	if (likely(!r->visibility))
+		return 0;
+
+	return r->visibility(vcpu, r);
+}
+
+static inline bool sysreg_hidden(const struct kvm_vcpu *vcpu,
+				 const struct sys_reg_desc *r)
+{
+	return sysreg_visibility(vcpu, r) & REG_HIDDEN;
+}
+
+static inline bool sysreg_visible_as_raz(const struct kvm_vcpu *vcpu,
+					 const struct sys_reg_desc *r)
+{
+	return sysreg_visibility(vcpu, r) & REG_RAZ;
+}
+
+static inline bool sysreg_user_write_ignore(const struct kvm_vcpu *vcpu,
+					    const struct sys_reg_desc *r)
+{
+	return sysreg_visibility(vcpu, r) & REG_USER_WI;
+}
+
+static inline int cmp_sys_reg(const struct sys_reg_desc *i1,
+			      const struct sys_reg_desc *i2)
+{
+	BUG_ON(i1 == i2);
+	if (!i1)
+		return 1;
+	else if (!i2)
+		return -1;
+	if (i1->Op0 != i2->Op0)
+		return i1->Op0 - i2->Op0;
+	if (i1->Op1 != i2->Op1)
+		return i1->Op1 - i2->Op1;
+	if (i1->CRn != i2->CRn)
+		return i1->CRn - i2->CRn;
+	if (i1->CRm != i2->CRm)
+		return i1->CRm - i2->CRm;
+	return i1->Op2 - i2->Op2;
+}
+
+static inline int match_sys_reg(const void *key, const void *elt)
+{
+	const unsigned long pval = (unsigned long)key;
+	const struct sys_reg_desc *r = elt;
+
+	return pval - reg_to_encoding(r);
+}
+
+static inline const struct sys_reg_desc *
+find_reg(const struct sys_reg_params *params, const struct sys_reg_desc table[],
+	 unsigned int num)
+{
+	unsigned long pval = reg_to_encoding(params);
+
+	return __inline_bsearch((void *)pval, table, num, sizeof(table[0]), match_sys_reg);
+}
+
+const struct sys_reg_desc *get_reg_by_id(u64 id,
+					 const struct sys_reg_desc table[],
+					 unsigned int num);
+
+int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
+			 const struct sys_reg_desc table[], unsigned int num);
+int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
+			 const struct sys_reg_desc table[], unsigned int num);
+
+bool triage_sysreg_trap(struct kvm_vcpu *vcpu, int *sr_index);
+
+int kvm_finalize_sys_regs(struct kvm_vcpu *vcpu);
+
+
+int __kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
+			      size_t n_descs,
+			      const struct sys_reg_desc sys_reg_descs[n_descs])
+;
+
+int __kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, 
+			      const struct kvm_one_reg *reg,
+			      size_t n_descs,
+			      const struct sys_reg_desc sys_reg_descs[n_descs]);
+unsigned long __kvm_arm_num_sys_reg_descs(struct kvm_vcpu *vcpu, size_t n_descs,
+					  const struct sys_reg_desc sys_reg_descs[n_descs]);
+int __kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu,
+				   u64 __user *uindices,
+				   size_t n_descs,
+				   const struct sys_reg_desc sys_reg_descs[n_descs]);
+int get_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 *val);
+int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 val);
+bool access_id_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+		   const struct sys_reg_desc *r);
+int set_id_aa64mmfr0_el1(struct kvm_vcpu *vcpu,
+			 const struct sys_reg_desc *rd,
+			 u64 user_val);
+unsigned int id_visibility(const struct kvm_vcpu *vcpu,
+			   const struct sys_reg_desc *r);
+unsigned int aa32_id_visibility(const struct kvm_vcpu *vcpu,
+				const struct sys_reg_desc *r);
+int set_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
+		   u64 val);
+void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val);
+int set_ctr_el0(struct kvm_vcpu *vcpu, 
+		const struct sys_reg_desc *rd,
+		u64 user_val);
+bool trap_oslar_el1(struct kvm_vcpu *vcpu,
+		    struct sys_reg_params *p,
+		    const struct sys_reg_desc *r);
+bool trap_oslsr_el1(struct kvm_vcpu *vcpu,
+		    struct sys_reg_params *p,
+		    const struct sys_reg_desc *r);
+int set_oslsr_el1(struct kvm_vcpu *vcpu,
+		  const struct sys_reg_desc *rd,
+		  u64 val);
+
+bool trap_raz_wi(struct kvm_vcpu *vcpu,
+		 struct sys_reg_params *p,
+		 const struct sys_reg_desc *r);
+ u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r);
+ bool access_rw(struct kvm_vcpu *vcpu,
+		struct sys_reg_params *p,
+		const struct sys_reg_desc *r);
+bool access_clidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+		  const struct sys_reg_desc *r);
+u64 reset_clidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r);
+int set_clidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 val);
+bool access_csselr(struct kvm_vcpu *vcpu,
+		   struct sys_reg_params *p,
+		   const struct sys_reg_desc *r);
+u32 get_ccsidr(struct kvm_vcpu *vcpu, u32 csselr);
+int set_ccsidr(struct kvm_vcpu *vcpu, u32 csselr, u32 val);
+bool access_ccsidr(struct kvm_vcpu *vcpu,
+		   struct sys_reg_params *p,
+		   const struct sys_reg_desc *r);
+void perform_access(struct kvm_vcpu *vcpu,
+		    struct sys_reg_params *params,
+		    const struct sys_reg_desc *r);
+
+void reset_vm_ftr_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *reg);
+void reset_vcpu_ftr_id_reg(struct kvm_vcpu *vcpu,
+			   const struct sys_reg_desc *reg);
+bool check_sysreg_table(const struct sys_reg_desc *table, unsigned int n,
+			bool reset_check);
+
+static inline u64 read_id_reg(const struct kvm_vcpu *vcpu,
+			      const struct sys_reg_desc *r)
+{
+	return kvm_read_vm_id_reg(vcpu->kvm, reg_to_encoding(r));
+}
+
+static bool is_feature_id_reg(u32 encoding)
+{
+	return (sys_reg_Op0(encoding) == 3 &&
+		(sys_reg_Op1(encoding) < 2 || sys_reg_Op1(encoding) == 3) &&
+		sys_reg_CRn(encoding) == 0 &&
+		sys_reg_CRm(encoding) <= 7);
+}
+
+static inline bool undef_access(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+				const struct sys_reg_desc *r)
+{
+	kvm_inject_undefined(vcpu);
+	return false;
+}
+
+static inline bool bad_trap(struct kvm_vcpu *vcpu,
+			    struct sys_reg_params *params,
+			    const struct sys_reg_desc *r, const char *msg)
+{
+	WARN_ONCE(1, "Unexpected %s\n", msg);
+	print_sys_reg_instr(params);
+	return undef_access(vcpu, params, r);
+}
+
+static inline bool read_from_write_only(struct kvm_vcpu *vcpu,
+					struct sys_reg_params *params,
+					const struct sys_reg_desc *r)
+{
+	return bad_trap(vcpu, params, r,
+			"sys_reg read to write-only register");
+}
+
+static inline bool write_to_read_only(struct kvm_vcpu *vcpu,
+				      struct sys_reg_params *params,
+				      const struct sys_reg_desc *r)
+{
+	return bad_trap(vcpu, params, r,
+			"sys_reg write to read-only register");
+}
+
+static inline void get_access_mask(const struct sys_reg_desc *r, u64 *mask, u64 *shift)
+{
+	switch (r->aarch32_map) {
+	case AA32_LO:
+		*mask = GENMASK_ULL(31, 0);
+		*shift = 0;
+		break;
+	case AA32_HI:
+		*mask = GENMASK_ULL(63, 32);
+		*shift = 32;
+		break;
+	default:
+		*mask = GENMASK_ULL(63, 0);
+		*shift = 0;
+		break;
+	}
+}
+
+/*
+ * Return true if the register's (Op0, Op1, CRn, CRm, Op2) is
+ * (3, 0, 0, crm, op2), where 1<=crm<8, 0<=op2<8, which is the range of ID
+ * registers KVM maintains on a per-VM basis.
+ *
+ * Additionally, the implementation ID registers and CTR_EL0 are handled as
+ * per-VM registers.
+ */
+static inline bool is_vm_ftr_id_reg(u32 id)
+{
+	switch (id) {
+	case SYS_CTR_EL0:
+	case SYS_MIDR_EL1:
+	case SYS_REVIDR_EL1:
+	case SYS_AIDR_EL1:
+		return true;
+	default:
+		return (sys_reg_Op0(id) == 3 && sys_reg_Op1(id) == 0 &&
+			sys_reg_CRn(id) == 0 && sys_reg_CRm(id) >= 1 &&
+			sys_reg_CRm(id) < 8);
+	}
+}
+
+static inline bool is_vcpu_ftr_id_reg(u32 id)
+{
+	return is_feature_id_reg(id) && !is_vm_ftr_id_reg(id);
+}
+
+#define AA32(_x) .aarch32_map = AA32_##_x
+#define Op0(_x) .Op0 = _x
+#define Op1(_x) .Op1 = _x
+#define CRn(_x) .CRn = _x
+#define CRm(_x) .CRm = _x
+#define Op2(_x) .Op2 = _x
+
+#define SYS_DESC(reg)                                               \
+	 .name = #reg, Op0(sys_reg_Op0(reg)), Op1(sys_reg_Op1(reg)), \
+	 CRn(sys_reg_CRn(reg)), CRm(sys_reg_CRm(reg)), Op2(sys_reg_Op2(reg))
+
+#define CP15_SYS_DESC(reg)                                                    \
+	 .name = #reg, .aarch32_map = AA32_DIRECT, Op0(0),                    \
+	 Op1(sys_reg_Op1(reg)), CRn(sys_reg_CRn(reg)), CRm(sys_reg_CRm(reg)), \
+	 Op2(sys_reg_Op2(reg))
+
+/*
+ * Since reset() callback and field val are not used for idregs, they will be
+ * used for specific purposes for idregs.
+ * The reset() would return KVM sanitised register value. The value would be the
+ * same as the host kernel sanitised value if there is no KVM sanitisation.
+ * The val would be used as a mask indicating writable fields for the idreg.
+ * Only bits with 1 are writable from userspace. This mask might not be
+ * necessary in the future whenever all ID registers are enabled as writable
+ * from userspace.
+ */
+
+#define ID_DESC_DEFAULT_CALLBACKS                             \
+	 .access = access_id_reg, .get_user = get_id_reg,     \
+	 .set_user = set_id_reg, .visibility = id_visibility, \
+	 .reset = kvm_read_sanitised_id_reg
+
+#define ID_DESC(name) SYS_DESC(SYS_##name), ID_DESC_DEFAULT_CALLBACKS
+
+/* sys_reg_desc initialiser for known cpufeature ID registers */
+#define ID_SANITISED(name)      \
+	{                       \
+		 ID_DESC(name), \
+		 .val = 0,      \
+	}
+
+/* sys_reg_desc initialiser for writable ID registers */
+#define ID_WRITABLE(name, mask) \
+	{                       \
+		 ID_DESC(name), \
+		 .val = mask,   \
+	}
+
+/*
+ * 32bit ID regs are fully writable when the guest is 32bit
+ * capable. Nothing in the KVM code should rely on 32bit features
+ * anyway, only 64bit, so let the VMM do its worse.
+ */
+#define AA32_ID_WRITABLE(name)                     \
+	{                                          \
+		 ID_DESC(name),                    \
+		 .visibility = aa32_id_visibility, \
+		 .val = GENMASK(31, 0),            \
+	}
+
+/* sys_reg_desc initialiser for cpufeature ID registers that need filtering */
+#define ID_FILTERED(sysreg, name, mask)  \
+	{                                \
+		 ID_DESC(sysreg),        \
+		 .set_user = set_##name, \
+		 .val = (mask),          \
+	}
+
+/*
+ * sys_reg_desc initialiser for architecturally unallocated cpufeature ID
+ * register with encoding Op0=3, Op1=0, CRn=0, CRm=crm, Op2=op2
+ * (1 <= crm < 8, 0 <= Op2 < 8).
+ */
+#define ID_UNALLOCATED(crm, op2)                 \
+	 {                                        \
+		 .name = "S3_0_0_" #crm "_" #op2, \
+		 Op0(3),                          \
+		 Op1(0),                          \
+		 CRn(0),                          \
+		 CRm(crm),                        \
+		 Op2(op2),                        \
+		 ID_DESC_DEFAULT_CALLBACKS,       \
+		 .visibility = raz_visibility,    \
+		 .val = 0,                        \
+	 }
+
+/*
+ * sys_reg_desc initialiser for known ID registers that we hide from guests.
+ * For now, these are exposed just like unallocated ID regs: they appear
+ * RAZ for the guest.
+ */
+#define ID_HIDDEN(name)                       \
+	 {                                     \
+		 ID_DESC(name),                \
+		 .visibility = raz_visibility, \
+		 .val = 0,                     \
+	}
+#define ID_REG_LIMIT_FIELD_ENUM(val, reg, field, limit)                 \
+	 ({                                                             \
+		u64 __f_val = FIELD_GET(reg##_##field##_MASK, val);     \
+		(val) &= ~reg##_##field##_MASK;                         \
+		(val) |= FIELD_PREP(                                    \
+			 reg##_##field##_MASK,                          \
+			 min(__f_val,                                   \
+			     (u64)SYS_FIELD_VALUE(reg, field, limit))); \
+		 (val);                                                 \
+	 })
+
+#define TO_ARM64_SYS_REG(r)                                       \
+	 ARM64_SYS_REG(sys_reg_Op0(SYS_##r), sys_reg_Op1(SYS_##r), \
+		       sys_reg_CRn(SYS_##r), sys_reg_CRm(SYS_##r), \
+		       sys_reg_Op2(SYS_##r))
+
+#define IMPLEMENTATION_ID(reg, mask) {			\
+	SYS_DESC(SYS_##reg),				\
+	.access = access_imp_id_reg,			\
+	.get_user = get_id_reg,				\
+	.set_user = set_imp_id_reg,			\
+	.reset = reset_imp_id_reg,			\
+	.val = mask,					\
+	}
+
+/* Implemented by each architecture */
+int arm64_check_features(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+			 u64 val);
+u64 kvm_sanitised_host_ftr_reg(u32 id);
+bool access_imp_id_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+		       const struct sys_reg_desc *r);
+u64 reset_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r);
+
+u64 kvm_read_sanitised_id_reg(struct kvm_vcpu *vcpu,
+			      const struct sys_reg_desc *r);
+
+int kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg * reg);
+int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg * reg);
+unsigned long kvm_arm_num_sys_reg_descs(struct kvm_vcpu *vcpu);
+int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices);
+
+#endif /* __ARM64_KVM_SYS_REGS_LOCAL_H__ */
diff --git a/virt/kvm/arm64/mmio.c b/virt/kvm/arm64/mmio.c
index 438a554ec1ed..089ee364df87 100644
--- a/virt/kvm/arm64/mmio.c
+++ b/virt/kvm/arm64/mmio.c
@@ -6,6 +6,7 @@
 
 #include <linux/kvm_host.h>
 #include <asm/kvm_emulate.h>
+#include <kvm/arm64/sys_regs.h>
 #include <trace/events/kvm.h>
 
 #define CREATE_TRACE_POINTS
diff --git a/virt/kvm/arm64/sys_regs.c b/virt/kvm/arm64/sys_regs.c
new file mode 100644
index 000000000000..70664bd7d339
--- /dev/null
+++ b/virt/kvm/arm64/sys_regs.c
@@ -0,0 +1,1039 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/kvm_host.h>
+
+#include <asm/sysreg-defs.h>
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+#include <asm/cputype-defs.h>
+#include <asm/cache-defs.h>
+
+#include <kvm/arm64/sys_regs.h>
+
+#include "trace.h"
+/*
+ * Statically sanitise the host's feature register, independent of the guest's
+ * configuration and host implementation.
+ */
+static u64 kvm_max_possible_guest_ftr_reg(u32 id, u64 val)
+{
+	switch (id) {
+	case SYS_ID_AA64DFR0_EL1:
+		val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64DFR0_EL1, DebugVer, V8P8);
+
+		/* Hide SPE from guests */
+		val &= ~ID_AA64DFR0_EL1_PMSVer_MASK;
+
+		/* Hide BRBE from guests */
+		val &= ~ID_AA64DFR0_EL1_BRBE_MASK;
+		break;
+	case SYS_ID_AA64ISAR2_EL1:
+		/* Mask WFxT field unless *both* WFET & WFIT are present. */
+		if (!id_has_feat(val, ID_AA64ISAR2_EL1, WFxT, IMP))
+			val &= ~ID_AA64ISAR2_EL1_WFxT;
+		break;
+	case SYS_ID_AA64ISAR3_EL1:
+		val &= ID_AA64ISAR3_EL1_FPRCVT | ID_AA64ISAR3_EL1_LSFE |
+		       ID_AA64ISAR3_EL1_FAMINMAX | ID_AA64ISAR3_EL1_LSUI;
+		break;
+	case SYS_ID_AA64MMFR2_EL1:
+		val &= ~ID_AA64MMFR2_EL1_CCIDX_MASK;
+		val &= ~ID_AA64MMFR2_EL1_NV;
+		break;
+	case SYS_ID_AA64MMFR3_EL1:
+		val &= ID_AA64MMFR3_EL1_TCRX |
+		       ID_AA64MMFR3_EL1_SCTLRX |
+		       ID_AA64MMFR3_EL1_S1POE |
+		       ID_AA64MMFR3_EL1_S1PIE;
+		break;
+	case SYS_ID_MMFR4_EL1:
+		val &= ~ID_MMFR4_EL1_CCIDX;
+		break;
+	case SYS_ID_AA64PFR0_EL1:
+		val &= ~ID_AA64PFR0_EL1_AMU_MASK;
+		/*
+		 * MPAM is disabled by default as KVM also needs a set of PARTID to
+		 * program the MPAMVPMx_EL2 PARTID remapping registers with. But some
+		 * older kernels let the guest see the ID bit.
+		 */
+		val &= ~ID_AA64PFR0_EL1_MPAM_MASK;
+		break;
+	case SYS_ID_AA64PFR1_EL1:
+		val &= ~ID_AA64PFR1_EL1_SME;
+		val &= ~ID_AA64PFR1_EL1_RNDR_trap;
+		val &= ~ID_AA64PFR1_EL1_NMI;
+		val &= ~ID_AA64PFR1_EL1_GCS;
+		val &= ~ID_AA64PFR1_EL1_THE;
+		val &= ~ID_AA64PFR1_EL1_MTEX;
+		val &= ~ID_AA64PFR1_EL1_PFAR;
+		val &= ~ID_AA64PFR1_EL1_MPAM_frac;
+		break;
+	case SYS_ID_AA64PFR2_EL1:
+		val &= ID_AA64PFR2_EL1_FPMR |
+		       ID_AA64PFR2_EL1_MTEFAR |
+		       ID_AA64PFR2_EL1_MTESTOREONLY;
+		break;
+	}
+
+	return val;
+}
+
+/*
+ * Sanitise based on vCPU configuration.
+ */
+static u64 kvm_sanitise_vcpu_ftr_reg(const struct kvm_vcpu *vcpu, u32 id, u64 val)
+{
+	switch (id) {
+	case SYS_ID_AA64DFR0_EL1:
+		/*
+		 * Only initialize the PMU version if the vCPU was configured with one.
+		 */
+		val &= ~ID_AA64DFR0_EL1_PMUVer_MASK;
+		if (kvm_vcpu_has_pmu(vcpu))
+			val |= SYS_FIELD_PREP(ID_AA64DFR0_EL1, PMUVer,
+					      kvm_arm_pmu_get_pmuver_limit());
+		break;
+	case SYS_ID_AA64PFR0_EL1:
+		if (!vcpu_has_sve(vcpu))
+			val &= ~ID_AA64PFR0_EL1_SVE_MASK;
+		break;
+	case SYS_ID_AA64PFR1_EL1:
+		if (!kvm_has_mte(vcpu->kvm)) {
+			val &= ~ID_AA64PFR1_EL1_MTE;
+			val &= ~ID_AA64PFR1_EL1_MTE_frac;
+		}
+		break;
+	case SYS_ID_AA64PFR2_EL1:
+		if (!kvm_has_mte(vcpu->kvm))
+			val &= ~(ID_AA64PFR2_EL1_MTEFAR |
+				 ID_AA64PFR2_EL1_MTESTOREONLY);
+		break;
+	case SYS_ID_AA64ISAR1_EL1:
+		if (!vcpu_has_ptrauth(vcpu))
+			val &= ~(ID_AA64ISAR1_EL1_APA |
+				 ID_AA64ISAR1_EL1_API |
+				 ID_AA64ISAR1_EL1_GPA |
+				 ID_AA64ISAR1_EL1_GPI);
+		break;
+	case SYS_ID_AA64ISAR2_EL1:
+		if (!vcpu_has_ptrauth(vcpu))
+			val &= ~(ID_AA64ISAR2_EL1_APA3 |
+				 ID_AA64ISAR2_EL1_GPA3);
+		break;
+	}
+
+	if (vcpu_has_nv(vcpu))
+		val = limit_nv_id_reg(vcpu->kvm, id, val);
+
+	return val;
+}
+
+/* Read a sanitised cpufeature ID register by sys_reg_desc */
+static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu,
+				       const struct sys_reg_desc *r)
+{
+	u32 id = reg_to_encoding(r);
+	u64 val;
+
+	if (sysreg_visible_as_raz(vcpu, r))
+		return 0;
+
+	val = kvm_sanitised_host_ftr_reg(id);
+	val = kvm_max_possible_guest_ftr_reg(id, val);
+	val = kvm_sanitise_vcpu_ftr_reg(vcpu, id, val);
+
+	return val;
+}
+
+u64 kvm_read_sanitised_id_reg(struct kvm_vcpu *vcpu,
+			      const struct sys_reg_desc *r)
+{
+	return __kvm_read_sanitised_id_reg(vcpu, r);
+}
+
+/* cpufeature ID register access trap handlers */
+bool access_id_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+		   const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		return write_to_read_only(vcpu, p, r);
+
+	p->regval = read_id_reg(vcpu, r);
+
+	return true;
+}
+
+unsigned int id_visibility(const struct kvm_vcpu *vcpu,
+			   const struct sys_reg_desc *r)
+{
+	u32 id = reg_to_encoding(r);
+
+	switch (id) {
+	case SYS_ID_AA64ZFR0_EL1:
+		if (!vcpu_has_sve(vcpu))
+			return REG_RAZ;
+		break;
+	}
+
+	return 0;
+}
+
+unsigned int aa32_id_visibility(const struct kvm_vcpu *vcpu,
+				const struct sys_reg_desc *r)
+{
+	/*
+	 * AArch32 ID registers are UNKNOWN if AArch32 isn't implemented at any
+	 * EL. Promote to RAZ/WI in order to guarantee consistency between
+	 * systems.
+	 */
+	if (!kvm_supports_32bit_el0())
+		return REG_RAZ | REG_USER_WI;
+
+	return id_visibility(vcpu, r);
+}
+
+/*
+ * cpufeature ID register user accessors
+ *
+ * For now, these registers are immutable for userspace, so no values
+ * are stored, and for set_id_reg() we don't allow the effective value
+ * to be changed.
+ */
+int get_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 *val)
+{
+	/*
+	 * Avoid locking if the VM has already started, as the ID registers are
+	 * guaranteed to be invariant at that point.
+	 */
+	if (kvm_vm_has_ran_once(vcpu->kvm)) {
+		*val = read_id_reg(vcpu, rd);
+		return 0;
+	}
+
+	mutex_lock(&vcpu->kvm->arch.config_lock);
+	*val = read_id_reg(vcpu, rd);
+	mutex_unlock(&vcpu->kvm->arch.config_lock);
+
+	return 0;
+}
+
+int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 val)
+{
+	u32 id = reg_to_encoding(rd);
+	int ret;
+
+	mutex_lock(&vcpu->kvm->arch.config_lock);
+
+	/*
+	 * Once the VM has started the ID registers are immutable. Reject any
+	 * write that does not match the final register value.
+	 */
+	if (kvm_vm_has_ran_once(vcpu->kvm)) {
+		if (val != read_id_reg(vcpu, rd))
+			ret = -EBUSY;
+		else
+			ret = 0;
+
+		mutex_unlock(&vcpu->kvm->arch.config_lock);
+		return ret;
+	}
+
+	ret = arm64_check_features(vcpu, rd, val);
+	if (!ret)
+		kvm_set_vm_id_reg(vcpu->kvm, id, val);
+
+	mutex_unlock(&vcpu->kvm->arch.config_lock);
+
+	/*
+	 * arm64_check_features() returns -E2BIG to indicate the register's
+	 * feature set is a superset of the maximally-allowed register value.
+	 * While it would be nice to precisely describe this to userspace, the
+	 * existing UAPI for KVM_SET_ONE_REG has it that invalid register
+	 * writes return -EINVAL.
+	 */
+	if (ret == -E2BIG)
+		ret = -EINVAL;
+	return ret;
+}
+
+int set_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, u64 val)
+{
+	struct kvm *kvm = vcpu->kvm;
+	u64 expected;
+
+	guard(mutex)(&kvm->arch.config_lock);
+
+	expected = read_id_reg(vcpu, r);
+	if (expected == val)
+		return 0;
+
+	if (!test_bit(KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, &kvm->arch.flags))
+		return -EINVAL;
+
+	/*
+	 * Once the VM has started the ID registers are immutable. Reject the
+	 * write if userspace tries to change it.
+	 */
+	if (kvm_vm_has_ran_once(kvm))
+		return -EBUSY;
+
+	/*
+	 * Any value is allowed for the implementation ID registers so long as
+	 * it is within the writable mask.
+	 */
+	if ((val & r->val) != val)
+		return -EINVAL;
+
+	kvm_set_vm_id_reg(kvm, reg_to_encoding(r), val);
+	return 0;
+}
+
+void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val)
+{
+	u64 *p = __vm_id_reg(&kvm->arch.id_regs, reg);
+
+	lockdep_assert_held(&kvm->arch.config_lock);
+
+	if (KVM_BUG_ON(kvm_vm_has_ran_once(kvm) || !p, kvm))
+		return;
+
+	*p = val;
+}
+
+int set_ctr_el0(struct kvm_vcpu *vcpu,
+		const struct sys_reg_desc *rd,
+		u64 user_val)
+{
+	u8 user_L1Ip = SYS_FIELD_GET(CTR_EL0, L1Ip, user_val);
+
+	/*
+	 * Both AIVIVT (0b01) and VPIPT (0b00) are documented as reserved.
+	 * Hence only allow to set VIPT(0b10) or PIPT(0b11) for L1Ip based
+	 * on what hardware reports.
+	 *
+	 * Using a VIPT software model on PIPT will lead to over invalidation,
+	 * but still correct. Hence, we can allow downgrading PIPT to VIPT,
+	 * but not the other way around. This is handled via arm64_ftr_safe_value()
+	 * as CTR_EL0 ftr_bits has L1Ip field with type FTR_EXACT and safe value
+	 * set as VIPT.
+	 */
+	switch (user_L1Ip) {
+	case CTR_EL0_L1Ip_RESERVED_VPIPT:
+	case CTR_EL0_L1Ip_RESERVED_AIVIVT:
+		return -EINVAL;
+	case CTR_EL0_L1Ip_VIPT:
+	case CTR_EL0_L1Ip_PIPT:
+		return set_id_reg(vcpu, rd, user_val);
+	default:
+		return -ENOENT;
+	}
+}
+
+bool trap_oslar_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+		    const struct sys_reg_desc *r)
+{
+	if (!p->is_write)
+		return read_from_write_only(vcpu, p, r);
+
+	kvm_debug_handle_oslar(vcpu, p->regval);
+	return true;
+}
+
+bool trap_oslsr_el1(struct kvm_vcpu *vcpu,
+		    struct sys_reg_params *p,
+		    const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		return write_to_read_only(vcpu, p, r);
+
+	p->regval = __vcpu_sys_reg(vcpu, r->reg);
+	return true;
+}
+
+int set_oslsr_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 val)
+{
+	u64 oslk;
+
+	/*
+	 * The only modifiable bit is the OSLK bit. Refuse the write if
+	 * userspace attempts to change any other bit in the register.
+	 */
+	if ((val ^ rd->val) & ~OSLSR_EL1_OSLK)
+		return -EINVAL;
+
+	oslk = SYS_FIELD_GET(OSLSR_EL1, OSLK, val);
+	__vcpu_assign_sys_reg(vcpu, OSLAR_EL1, SYS_FIELD_PREP(OSLAR_EL1, OSLK, oslk));
+	return 0;
+}
+
+bool trap_raz_wi(struct kvm_vcpu *vcpu,
+		 struct sys_reg_params *p,
+		 const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		return ignore_write(vcpu, p);
+	else
+		return read_zero(vcpu, p);
+}
+
+u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	u64 mpidr;
+
+	/*
+	 * Map the vcpu_id into the first three affinity level fields of
+	 * the MPIDR. We limit the number of VCPUs in level 0 due to a
+	 * limitation to 16 CPUs in that level in the ICC_SGIxR registers
+	 * of the GICv3 to be able to address each CPU directly when
+	 * sending IPIs.
+	 */
+	mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
+	mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
+	mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
+	mpidr |= (1ULL << 31);
+	vcpu_write_sys_reg(vcpu, mpidr, MPIDR_EL1);
+
+	return mpidr;
+}
+
+/*
+ * Allow userspace to de-feature a stage-2 translation granule but prevent it
+ * from claiming the impossible.
+ */
+#define tgran2_val_allowed(tg, safe, user)			\
+({								\
+	u8 __s = SYS_FIELD_GET(ID_AA64MMFR0_EL1, tg, safe);	\
+	u8 __u = SYS_FIELD_GET(ID_AA64MMFR0_EL1, tg, user);	\
+								\
+	__s == __u || __u == ID_AA64MMFR0_EL1_##tg##_NI;	\
+})
+
+int set_id_aa64mmfr0_el1(struct kvm_vcpu *vcpu,
+			 const struct sys_reg_desc *rd,
+			 u64 user_val)
+{
+	u64 sanitized_val = kvm_read_sanitised_id_reg(vcpu, rd);
+
+	if (!vcpu_has_nv(vcpu))
+		return set_id_reg(vcpu, rd, user_val);
+
+	if (!tgran2_val_allowed(TGRAN4_2, sanitized_val, user_val) ||
+	    !tgran2_val_allowed(TGRAN16_2, sanitized_val, user_val) ||
+	    !tgran2_val_allowed(TGRAN64_2, sanitized_val, user_val))
+		return -EINVAL;
+
+	return set_id_reg(vcpu, rd, user_val);
+}
+
+bool access_rw(struct kvm_vcpu *vcpu,
+	       struct sys_reg_params *p,
+	       const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		vcpu_write_sys_reg(vcpu, p->regval, r->reg);
+	else
+		p->regval = vcpu_read_sys_reg(vcpu, r->reg);
+
+	return true;
+}
+
+bool access_clidr(struct kvm_vcpu *vcpu,
+		  struct sys_reg_params *p,
+		  const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		return write_to_read_only(vcpu, p, r);
+
+	p->regval = __vcpu_sys_reg(vcpu, r->reg);
+	return true;
+}
+
+/*
+ * Fabricate a CLIDR_EL1 value instead of using the real value, which can vary
+ * by the physical CPU which the vcpu currently resides in.
+ */
+u64 reset_clidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	u64 ctr_el0 = kvm_sanitised_host_ftr_reg(SYS_CTR_EL0);
+	u64 clidr;
+	u8 loc;
+
+	if ((ctr_el0 & CTR_EL0_IDC)) {
+		/*
+		 * Data cache clean to the PoU is not required so LoUU and LoUIS
+		 * will not be set and a unified cache, which will be marked as
+		 * LoC, will be added.
+		 *
+		 * If not DIC, let the unified cache L2 so that an instruction
+		 * cache can be added as L1 later.
+		 */
+		loc = (ctr_el0 & CTR_EL0_DIC) ? 1 : 2;
+		clidr = CACHE_TYPE_UNIFIED << CLIDR_CTYPE_SHIFT(loc);
+	} else {
+		/*
+		 * Data cache clean to the PoU is required so let L1 have a data
+		 * cache and mark it as LoUU and LoUIS. As L1 has a data cache,
+		 * it can be marked as LoC too.
+		 */
+		loc = 1;
+		clidr = 1 << CLIDR_LOUU_SHIFT;
+		clidr |= 1 << CLIDR_LOUIS_SHIFT;
+		clidr |= CACHE_TYPE_DATA << CLIDR_CTYPE_SHIFT(1);
+	}
+
+	/*
+	 * Instruction cache invalidation to the PoU is required so let L1 have
+	 * an instruction cache. If L1 already has a data cache, it will be
+	 * CACHE_TYPE_SEPARATE.
+	 */
+	if (!(ctr_el0 & CTR_EL0_DIC))
+		clidr |= CACHE_TYPE_INST << CLIDR_CTYPE_SHIFT(1);
+
+	clidr |= loc << CLIDR_LOC_SHIFT;
+
+	/*
+	 * Add tag cache unified to data cache. Allocation tags and data are
+	 * unified in a cache line so that it looks valid even if there is only
+	 * one cache line.
+	 */
+	if (kvm_has_mte(vcpu->kvm))
+		clidr |= 2ULL << CLIDR_TTYPE_SHIFT(loc);
+
+	__vcpu_assign_sys_reg(vcpu, r->reg, clidr);
+
+	return __vcpu_sys_reg(vcpu, r->reg);
+}
+
+int set_clidr(struct kvm_vcpu *vcpu,
+	      const struct sys_reg_desc *rd,
+	      u64 val)
+{
+	u64 ctr_el0 = kvm_sanitised_host_ftr_reg(SYS_CTR_EL0);
+	u64 idc = !CLIDR_LOC(val) || (!CLIDR_LOUIS(val) && !CLIDR_LOUU(val));
+
+	if ((val & CLIDR_EL1_RES0) || (!(ctr_el0 & CTR_EL0_IDC) && idc))
+		return -EINVAL;
+
+	__vcpu_assign_sys_reg(vcpu, rd->reg, val);
+
+	return 0;
+}
+
+bool access_csselr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	int reg = r->reg;
+
+	if (p->is_write)
+		vcpu_write_sys_reg(vcpu, p->regval, reg);
+	else
+		p->regval = vcpu_read_sys_reg(vcpu, reg);
+	return true;
+}
+
+/* CSSELR values; used to index KVM_REG_ARM_DEMUX_ID_CCSIDR */
+#define CSSELR_MAX 14
+
+/*
+ * Returns the minimum line size for the selected cache, expressed as
+ * Log2(bytes).
+ */
+static u8 get_min_cache_line_size(bool icache)
+{
+	u64 ctr = kvm_sanitised_host_ftr_reg(SYS_CTR_EL0);
+	u8 field;
+
+	if (icache)
+		field = SYS_FIELD_GET(CTR_EL0, IminLine, ctr);
+	else
+		field = SYS_FIELD_GET(CTR_EL0, DminLine, ctr);
+
+	/*
+	 * Cache line size is represented as Log2(words) in CTR_EL0.
+	 * Log2(bytes) can be derived with the following:
+	 *
+	 * Log2(words) + 2 = Log2(bytes / 4) + 2
+	 *		   = Log2(bytes) - 2 + 2
+	 *		   = Log2(bytes)
+	 */
+	return field + 2;
+}
+
+/* Which cache CCSIDR represents depends on CSSELR value. */
+u32 get_ccsidr(struct kvm_vcpu *vcpu, u32 csselr)
+{
+	u8 line_size;
+
+	if (vcpu->arch.ccsidr)
+		return vcpu->arch.ccsidr[csselr];
+
+	line_size = get_min_cache_line_size(csselr & CSSELR_EL1_InD);
+
+	/*
+	 * Fabricate a CCSIDR value as the overriding value does not exist.
+	 * The real CCSIDR value will not be used as it can vary by the
+	 * physical CPU which the vcpu currently resides in.
+	 *
+	 * The line size is determined with get_min_cache_line_size(), which
+	 * should be valid for all CPUs even if they have different cache
+	 * configuration.
+	 *
+	 * The associativity bits are cleared, meaning the geometry of all data
+	 * and unified caches (which are guaranteed to be PIPT and thus
+	 * non-aliasing) are 1 set and 1 way.
+	 * Guests should not be doing cache operations by set/way at all, and
+	 * for this reason, we trap them and attempt to infer the intent, so
+	 * that we can flush the entire guest's address space at the appropriate
+	 * time. The exposed geometry minimizes the number of the traps.
+	 * [If guests should attempt to infer aliasing properties from the
+	 * geometry (which is not permitted by the architecture), they would
+	 * only do so for virtually indexed caches.]
+	 *
+	 * We don't check if the cache level exists as it is allowed to return
+	 * an UNKNOWN value if not.
+	 */
+	return SYS_FIELD_PREP(CCSIDR_EL1, LineSize, line_size - 4);
+}
+
+int set_ccsidr(struct kvm_vcpu *vcpu, u32 csselr, u32 val)
+{
+	u8 line_size = FIELD_GET(CCSIDR_EL1_LineSize, val) + 4;
+	u32 *ccsidr = vcpu->arch.ccsidr;
+	u32 i;
+
+	if ((val & CCSIDR_EL1_RES0) ||
+	    line_size < get_min_cache_line_size(csselr & CSSELR_EL1_InD))
+		return -EINVAL;
+
+	if (!ccsidr) {
+		if (val == get_ccsidr(vcpu, csselr))
+			return 0;
+
+		ccsidr = kmalloc_array(CSSELR_MAX, sizeof(u32), GFP_KERNEL_ACCOUNT);
+		if (!ccsidr)
+			return -ENOMEM;
+
+		for (i = 0; i < CSSELR_MAX; i++)
+			ccsidr[i] = get_ccsidr(vcpu, i);
+
+		vcpu->arch.ccsidr = ccsidr;
+	}
+
+	ccsidr[csselr] = val;
+
+	return 0;
+}
+
+bool access_ccsidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+		   const struct sys_reg_desc *r)
+{
+	u32 csselr;
+
+	if (p->is_write)
+		return write_to_read_only(vcpu, p, r);
+
+	csselr = vcpu_read_sys_reg(vcpu, CSSELR_EL1);
+	csselr &= CSSELR_EL1_Level | CSSELR_EL1_InD;
+	if (csselr < CSSELR_MAX)
+		p->regval = get_ccsidr(vcpu, csselr);
+
+	return true;
+}
+
+void perform_access(struct kvm_vcpu *vcpu,
+		    struct sys_reg_params *params,
+		    const struct sys_reg_desc *r)
+{
+	trace_kvm_sys_access(*vcpu_pc(vcpu), params, r);
+
+	/* Check for regs disabled by runtime config */
+	if (sysreg_hidden(vcpu, r)) {
+		kvm_inject_undefined(vcpu);
+		return;
+	}
+
+	/*
+	 * Not having an accessor means that we have configured a trap
+	 * that we don't know how to handle. This certainly qualifies
+	 * as a gross bug that should be fixed right away.
+	 */
+	if (!r->access) {
+		bad_trap(vcpu, params, r, "register access");
+		return;
+	}
+
+	/* Skip instruction if instructed so */
+	if (likely(r->access(vcpu, params, r)))
+		kvm_incr_pc(vcpu);
+}
+
+void reset_vm_ftr_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *reg)
+{
+	u32 id = reg_to_encoding(reg);
+	struct kvm *kvm = vcpu->kvm;
+
+	if (test_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags))
+		return;
+
+	kvm_set_vm_id_reg(kvm, id, reg->reset(vcpu, reg));
+}
+
+void reset_vcpu_ftr_id_reg(struct kvm_vcpu *vcpu,
+			   const struct sys_reg_desc *reg)
+{
+	if (kvm_vcpu_initialized(vcpu))
+		return;
+
+	reg->reset(vcpu, reg);
+}
+
+static int demux_c15_get(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
+{
+	u32 val;
+	u32 __user *uval = uaddr;
+
+	/* Fail if we have unknown bits set. */
+	if (id & ~(KVM_REG_ARCH_MASK|KVM_REG_SIZE_MASK|KVM_REG_ARM_COPROC_MASK
+		   | ((1 << KVM_REG_ARM_COPROC_SHIFT)-1)))
+		return -ENOENT;
+
+	switch (id & KVM_REG_ARM_DEMUX_ID_MASK) {
+	case KVM_REG_ARM_DEMUX_ID_CCSIDR:
+		if (KVM_REG_SIZE(id) != 4)
+			return -ENOENT;
+		val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
+			>> KVM_REG_ARM_DEMUX_VAL_SHIFT;
+		if (val >= CSSELR_MAX)
+			return -ENOENT;
+
+		return put_user(get_ccsidr(vcpu, val), uval);
+	default:
+		return -ENOENT;
+	}
+}
+
+int __kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
+			      size_t n_descs,
+			      const struct sys_reg_desc sys_reg_descs[n_descs])
+{
+	void __user *uaddr = (void __user *)(unsigned long)reg->addr;
+
+	if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
+		return demux_c15_get(vcpu, reg->id, uaddr);
+
+	return kvm_sys_reg_get_user(vcpu, reg, sys_reg_descs, n_descs);
+}
+
+static int demux_c15_set(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
+{
+	u32 val, newval;
+	u32 __user *uval = uaddr;
+
+	/* Fail if we have unknown bits set. */
+	if (id & ~(KVM_REG_ARCH_MASK|KVM_REG_SIZE_MASK|KVM_REG_ARM_COPROC_MASK
+		   | ((1 << KVM_REG_ARM_COPROC_SHIFT)-1)))
+		return -ENOENT;
+
+	switch (id & KVM_REG_ARM_DEMUX_ID_MASK) {
+	case KVM_REG_ARM_DEMUX_ID_CCSIDR:
+		if (KVM_REG_SIZE(id) != 4)
+			return -ENOENT;
+		val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
+			>> KVM_REG_ARM_DEMUX_VAL_SHIFT;
+		if (val >= CSSELR_MAX)
+			return -ENOENT;
+
+		if (get_user(newval, uval))
+			return -EFAULT;
+
+		return set_ccsidr(vcpu, val, newval);
+	default:
+		return -ENOENT;
+	}
+}
+
+
+int __kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
+			      size_t n_descs,
+			      const struct sys_reg_desc sys_reg_descs[n_descs])
+
+{
+	void __user *uaddr = (void __user *)(unsigned long)reg->addr;
+
+	if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
+		return demux_c15_set(vcpu, reg->id, uaddr);
+
+	return kvm_sys_reg_set_user(vcpu, reg, sys_reg_descs, n_descs);
+}
+
+static unsigned int num_demux_regs(void)
+{
+	return CSSELR_MAX;
+}
+
+static int write_demux_regids(u64 __user *uindices)
+{
+	u64 val = KVM_REG_ARM64 | KVM_REG_SIZE_U32 | KVM_REG_ARM_DEMUX;
+	unsigned int i;
+
+	val |= KVM_REG_ARM_DEMUX_ID_CCSIDR;
+	for (i = 0; i < CSSELR_MAX; i++) {
+		if (put_user(val | i, uindices))
+			return -EFAULT;
+		uindices++;
+	}
+	return 0;
+}
+
+static u64 sys_reg_to_index(const struct sys_reg_desc *reg)
+{
+	return (KVM_REG_ARM64 | KVM_REG_SIZE_U64 |
+		KVM_REG_ARM64_SYSREG |
+		(reg->Op0 << KVM_REG_ARM64_SYSREG_OP0_SHIFT) |
+		(reg->Op1 << KVM_REG_ARM64_SYSREG_OP1_SHIFT) |
+		(reg->CRn << KVM_REG_ARM64_SYSREG_CRN_SHIFT) |
+		(reg->CRm << KVM_REG_ARM64_SYSREG_CRM_SHIFT) |
+		(reg->Op2 << KVM_REG_ARM64_SYSREG_OP2_SHIFT));
+}
+
+static bool copy_reg_to_user(const struct sys_reg_desc *reg, u64 __user **uind)
+{
+	u64 idx;
+
+	if (!*uind)
+		return true;
+
+	switch (reg_to_encoding(reg)) {
+	case SYS_CNTV_CVAL_EL0:
+		idx = KVM_REG_ARM_TIMER_CVAL;
+		break;
+	case SYS_CNTVCT_EL0:
+		idx = KVM_REG_ARM_TIMER_CNT;
+		break;
+	default:
+		idx = sys_reg_to_index(reg);
+	}
+
+	if (put_user(idx, *uind))
+		return false;
+
+	(*uind)++;
+	return true;
+}
+
+static int walk_one_sys_reg(const struct kvm_vcpu *vcpu,
+			    const struct sys_reg_desc *rd,
+			    u64 __user **uind,
+			    unsigned int *total)
+{
+	/*
+	 * Ignore registers we trap but don't save,
+	 * and for which no custom user accessor is provided.
+	 */
+	if (!(rd->reg || rd->get_user))
+		return 0;
+
+	if (sysreg_hidden(vcpu, rd))
+		return 0;
+
+	if (!copy_reg_to_user(rd, uind))
+		return -EFAULT;
+
+	(*total)++;
+	return 0;
+}
+
+/* Assumed ordered tables, see kvm_sys_reg_table_init. */
+static int walk_sys_regs(struct kvm_vcpu *vcpu,
+		  const struct sys_reg_desc *start,
+		  const struct sys_reg_desc *end2,
+		  u64 __user *uind)
+{
+	const struct sys_reg_desc *i2;
+	unsigned int total = 0;
+	int err;
+
+	i2 = start;
+
+	while (i2 != end2) {
+		err = walk_one_sys_reg(vcpu, i2++, &uind, &total);
+		if (err)
+			return err;
+	}
+	return total;
+}
+
+unsigned long __kvm_arm_num_sys_reg_descs(struct kvm_vcpu *vcpu, size_t n_descs,
+					  const struct sys_reg_desc sys_reg_descs[n_descs])
+{
+	return num_demux_regs()
+		+ walk_sys_regs(vcpu, sys_reg_descs, sys_reg_descs + n_descs, (u64 __user *)NULL);
+}
+
+int __kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices,
+				   size_t n_descs,
+				   const struct sys_reg_desc sys_reg_descs[n_descs])
+{
+	int err;
+
+	err = walk_sys_regs(vcpu, sys_reg_descs, sys_reg_descs + n_descs, uindices);
+	if (err < 0)
+		return err;
+	uindices += err;
+
+	return write_demux_regids(uindices);
+}
+
+bool check_sysreg_table(const struct sys_reg_desc *table, unsigned int n,
+			       bool reset_check)
+{
+	unsigned int i;
+
+	for (i = 0; i < n; i++) {
+		if (reset_check && table[i].reg && !table[i].reset) {
+			kvm_err("sys_reg table %pS entry %d (%s) lacks reset\n",
+				&table[i], i, table[i].name);
+			return false;
+		}
+
+		if (i && cmp_sys_reg(&table[i-1], &table[i]) >= 0) {
+			kvm_err("sys_reg table %pS entry %d (%s -> %s) out of order\n",
+				&table[i], i, table[i - 1].name, table[i].name);
+			return false;
+		}
+	}
+
+	return true;
+}
+
+/******************************************************************************
+ * Userspace API
+ *****************************************************************************/
+
+static bool index_to_params(u64 id, struct sys_reg_params *params)
+{
+	switch (id & KVM_REG_SIZE_MASK) {
+	case KVM_REG_SIZE_U64:
+		/* Any unused index bits means it's not valid. */
+		if (id & ~(KVM_REG_ARCH_MASK | KVM_REG_SIZE_MASK
+			      | KVM_REG_ARM_COPROC_MASK
+			      | KVM_REG_ARM64_SYSREG_OP0_MASK
+			      | KVM_REG_ARM64_SYSREG_OP1_MASK
+			      | KVM_REG_ARM64_SYSREG_CRN_MASK
+			      | KVM_REG_ARM64_SYSREG_CRM_MASK
+			      | KVM_REG_ARM64_SYSREG_OP2_MASK))
+			return false;
+		params->Op0 = ((id & KVM_REG_ARM64_SYSREG_OP0_MASK)
+			       >> KVM_REG_ARM64_SYSREG_OP0_SHIFT);
+		params->Op1 = ((id & KVM_REG_ARM64_SYSREG_OP1_MASK)
+			       >> KVM_REG_ARM64_SYSREG_OP1_SHIFT);
+		params->CRn = ((id & KVM_REG_ARM64_SYSREG_CRN_MASK)
+			       >> KVM_REG_ARM64_SYSREG_CRN_SHIFT);
+		params->CRm = ((id & KVM_REG_ARM64_SYSREG_CRM_MASK)
+			       >> KVM_REG_ARM64_SYSREG_CRM_SHIFT);
+		params->Op2 = ((id & KVM_REG_ARM64_SYSREG_OP2_MASK)
+			       >> KVM_REG_ARM64_SYSREG_OP2_SHIFT);
+		return true;
+	default:
+		return false;
+	}
+}
+
+const struct sys_reg_desc *get_reg_by_id(u64 id,
+					 const struct sys_reg_desc table[],
+					 unsigned int num)
+{
+	struct sys_reg_params params;
+
+	if (!index_to_params(id, &params))
+		return NULL;
+
+	return find_reg(&params, table, num);
+}
+
+/* Decode an index value, and find the sys_reg_desc entry. */
+static const struct sys_reg_desc *
+id_to_sys_reg_desc(struct kvm_vcpu *vcpu, u64 id,
+		   const struct sys_reg_desc table[], unsigned int num)
+
+{
+	const struct sys_reg_desc *r;
+
+	/* We only do sys_reg for now. */
+	if ((id & KVM_REG_ARM_COPROC_MASK) != KVM_REG_ARM64_SYSREG)
+		return NULL;
+
+	r = get_reg_by_id(id, table, num);
+
+	/* Not saved in the sys_reg array and not otherwise accessible? */
+	if (r && (!(r->reg || r->get_user) || sysreg_hidden(vcpu, r)))
+		r = NULL;
+
+	return r;
+}
+
+static u64 kvm_one_reg_to_id(const struct kvm_one_reg *reg)
+{
+	switch (reg->id) {
+	case KVM_REG_ARM_TIMER_CVAL:
+		return TO_ARM64_SYS_REG(CNTV_CVAL_EL0);
+	case KVM_REG_ARM_TIMER_CNT:
+		return TO_ARM64_SYS_REG(CNTVCT_EL0);
+	default:
+		return reg->id;
+	}
+}
+
+int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
+			 const struct sys_reg_desc table[], unsigned int num)
+{
+	u64 __user *uaddr = (u64 __user *)(unsigned long)reg->addr;
+	const struct sys_reg_desc *r;
+	u64 id = kvm_one_reg_to_id(reg);
+	u64 val;
+	int ret;
+
+	r = id_to_sys_reg_desc(vcpu, id, table, num);
+	if (!r || sysreg_hidden(vcpu, r))
+		return -ENOENT;
+
+	if (r->get_user) {
+		ret = (r->get_user)(vcpu, r, &val);
+	} else {
+		val = __vcpu_sys_reg(vcpu, r->reg);
+		ret = 0;
+	}
+
+	if (!ret)
+		ret = put_user(val, uaddr);
+
+	return ret;
+}
+
+int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg,
+			 const struct sys_reg_desc table[], unsigned int num)
+{
+	u64 __user *uaddr = (u64 __user *)(unsigned long)reg->addr;
+	const struct sys_reg_desc *r;
+	u64 id = kvm_one_reg_to_id(reg);
+	u64 val;
+	int ret;
+
+	if (get_user(val, uaddr))
+		return -EFAULT;
+
+	r = id_to_sys_reg_desc(vcpu, id, table, num);
+	if (!r || sysreg_hidden(vcpu, r))
+		return -ENOENT;
+
+	if (sysreg_user_write_ignore(vcpu, r))
+		return 0;
+
+	if (r->set_user) {
+		ret = (r->set_user)(vcpu, r, val);
+	} else {
+		__vcpu_assign_sys_reg(vcpu, r->reg, val);
+		ret = 0;
+	}
+
+	return ret;
+}
diff --git a/virt/kvm/arm64/trace.h b/virt/kvm/arm64/trace.h
index 0814000b7749..d791ee4ea2bf 100644
--- a/virt/kvm/arm64/trace.h
+++ b/virt/kvm/arm64/trace.h
@@ -36,6 +36,40 @@ TRACE_EVENT(kvm_mmio_nisv,
 		  __entry->far, __entry->vcpu_pc)
 );
 
+TRACE_EVENT(kvm_sys_access,
+	TP_PROTO(unsigned long vcpu_pc, struct sys_reg_params *params, const struct sys_reg_desc *reg),
+	TP_ARGS(vcpu_pc, params, reg),
+
+	TP_STRUCT__entry(
+		__field(unsigned long,			vcpu_pc)
+		__field(bool,				is_write)
+		__field(const char *,			name)
+		__field(u8,				Op0)
+		__field(u8,				Op1)
+		__field(u8,				CRn)
+		__field(u8,				CRm)
+		__field(u8,				Op2)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu_pc = vcpu_pc;
+		__entry->is_write = params->is_write;
+		__entry->name = reg->name;
+		__entry->Op0 = reg->Op0;
+		__entry->Op0 = reg->Op0;
+		__entry->Op1 = reg->Op1;
+		__entry->CRn = reg->CRn;
+		__entry->CRm = reg->CRm;
+		__entry->Op2 = reg->Op2;
+	),
+
+	TP_printk("PC: %lx %s (%d,%d,%d,%d,%d) %s",
+		  __entry->vcpu_pc, __entry->name ?: "UNKN",
+		  __entry->Op0, __entry->Op1, __entry->CRn,
+		  __entry->CRm, __entry->Op2,
+		  str_write_read(__entry->is_write))
+);
+
 #endif /* __KVM_ARM64_TRACE_H__ */
 
 /* This part must be outside protection */
-- 
2.53.0

[PATCH v1 22/26] KVM: arm64: Refactor core reg handling

[Steffen Eiden]

Refactor sysreg core handling. Before this all core regs are identified
with a memory address and reading/writing happened through accessing the
data at this address. However, for arm64 on s390 not all core registers
have a dedicated memory address.

Refactor such that the address function does not return an address but
actually does the read/write request. ELR_EL1 and SPSR_EL1 now use
vcpu_read_sys_reg/vcpu_write_sys_reg accessor functions, allowing s390
to provide custom implementations.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 virt/kvm/arm64/guest.c | 100 ++++++++++++++++++++++++++---------------
 1 file changed, 64 insertions(+), 36 deletions(-)

diff --git a/virt/kvm/arm64/guest.c b/virt/kvm/arm64/guest.c
index 35ba03033b4c..11509382d594 100644
--- a/virt/kvm/arm64/guest.c
+++ b/virt/kvm/arm64/guest.c
@@ -65,69 +65,96 @@ static int core_reg_size_from_offset(const struct kvm_vcpu *vcpu, u64 off)
 	return size;
 }
 
-static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
+static int core_reg_rw(struct kvm_vcpu *vcpu, u64 reg_id, void *valp, bool read)
 {
-	u64 off = core_reg_offset_from_id(reg->id);
+	u64 off = core_reg_offset_from_id(reg_id);
 	int size = core_reg_size_from_offset(vcpu, off);
+	void *addr;
 
-	if (size < 0)
-		return NULL;
+	if (size < 0 || (KVM_REG_SIZE(reg_id) != size))
+		return -EINVAL;
 
-	if (KVM_REG_SIZE(reg->id) != size)
-		return NULL;
+	switch (off) {
+	case KVM_REG_ARM_CORE_REG(elr_el1):
+		if (read)
+			*(u64 *)valp = vcpu_read_sys_reg(vcpu, ELR_EL1);
+		else
+			vcpu_write_sys_reg(vcpu, *(u64 *)valp, ELR_EL1);
+		return 0;
+
+	case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_EL1]):
+		if (read)
+			*(u64 *)valp = vcpu_read_sys_reg(vcpu, SPSR_EL1);
+		else
+			vcpu_write_sys_reg(vcpu, *(u64 *)valp, SPSR_EL1);
+		return 0;
+	}
 
 	switch (off) {
 	case KVM_REG_ARM_CORE_REG(regs.regs[0]) ...
 	     KVM_REG_ARM_CORE_REG(regs.regs[30]):
 		off -= KVM_REG_ARM_CORE_REG(regs.regs[0]);
 		off /= 2;
-		return &vcpu_gp_regs(vcpu)[off];
+		addr = &vcpu_gp_regs(vcpu)[off];
+		break;
 
 	case KVM_REG_ARM_CORE_REG(regs.sp):
-		return vcpu_sp_el0(vcpu);
+		addr = vcpu_sp_el0(vcpu);
+		break;
 
 	case KVM_REG_ARM_CORE_REG(regs.pc):
-		return vcpu_pc(vcpu);
+		addr = vcpu_pc(vcpu);
+		break;
 
 	case KVM_REG_ARM_CORE_REG(regs.pstate):
-		return vcpu_cpsr(vcpu);
+		addr = vcpu_cpsr(vcpu);
+		break;
 
 	case KVM_REG_ARM_CORE_REG(sp_el1):
-		return kvm_vcpu_get_sp_el1(vcpu);
-
-	case KVM_REG_ARM_CORE_REG(elr_el1):
-		return __ctxt_sys_reg(&vcpu->arch.ctxt, ELR_EL1);
-
-	case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_EL1]):
-		return __ctxt_sys_reg(&vcpu->arch.ctxt, SPSR_EL1);
+		addr = kvm_vcpu_get_sp_el1(vcpu);
+		break;
 
 	case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_ABT]):
-		return &vcpu->arch.ctxt.spsr_abt;
+		addr = &vcpu->arch.ctxt.spsr_abt;
+		break;
 
 	case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_UND]):
-		return &vcpu->arch.ctxt.spsr_und;
+		addr = &vcpu->arch.ctxt.spsr_und;
+		break;
 
 	case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_IRQ]):
-		return &vcpu->arch.ctxt.spsr_irq;
+		addr = &vcpu->arch.ctxt.spsr_irq;
+		break;
 
 	case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_FIQ]):
-		return &vcpu->arch.ctxt.spsr_fiq;
+		addr = &vcpu->arch.ctxt.spsr_fiq;
+		break;
 
 	case KVM_REG_ARM_CORE_REG(fp_regs.vregs[0]) ...
 	     KVM_REG_ARM_CORE_REG(fp_regs.vregs[31]):
 		off -= KVM_REG_ARM_CORE_REG(fp_regs.vregs[0]);
 		off /= 4;
-		return kvm_vcpu_get_vreg(vcpu, off);
+		addr = kvm_vcpu_get_vreg(vcpu, off);
+		break;
 
 	case KVM_REG_ARM_CORE_REG(fp_regs.fpsr):
-		return  kvm_vcpu_get_fpsr(vcpu);
+		addr = kvm_vcpu_get_fpsr(vcpu);
+		break;
 
 	case KVM_REG_ARM_CORE_REG(fp_regs.fpcr):
-		return  kvm_vcpu_get_fpcr(vcpu);
+		addr = kvm_vcpu_get_fpcr(vcpu);
+		break;
 
 	default:
-		return NULL;
+		return -EINVAL;
 	}
+
+	if (read)
+		memcpy(valp, addr, size);
+	else
+		memcpy(addr, valp, size);
+
+	return 0;
 }
 
 int get_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
@@ -140,7 +167,9 @@ int get_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	 */
 	__u32 __user *uaddr = (__u32 __user *)(unsigned long)reg->addr;
 	int nr_regs = sizeof(struct kvm_regs) / sizeof(__u32);
-	void *addr;
+	__uint128_t tmp;
+	void *valp = &tmp;
+	int ret;
 	u32 off;
 
 	/* Our ID is an index into the kvm_regs struct. */
@@ -149,11 +178,12 @@ int get_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	    (off + (KVM_REG_SIZE(reg->id) / sizeof(__u32))) >= nr_regs)
 		return -ENOENT;
 
-	addr = core_reg_addr(vcpu, reg);
-	if (!addr)
-		return -EINVAL;
 
-	if (copy_to_user(uaddr, addr, KVM_REG_SIZE(reg->id)))
+	ret = core_reg_rw(vcpu, reg->id, valp, true);
+	if (ret)
+		return ret;
+
+	if (copy_to_user(uaddr, valp, KVM_REG_SIZE(reg->id)))
 		return -EFAULT;
 
 	return 0;
@@ -164,7 +194,7 @@ int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	__u32 __user *uaddr = (__u32 __user *)(unsigned long)reg->addr;
 	int nr_regs = sizeof(struct kvm_regs) / sizeof(__u32);
 	__uint128_t tmp;
-	void *valp = &tmp, *addr;
+	void *valp = &tmp;
 	u64 off;
 	int err = 0;
 
@@ -174,10 +204,6 @@ int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	    (off + (KVM_REG_SIZE(reg->id) / sizeof(__u32))) >= nr_regs)
 		return -ENOENT;
 
-	addr = core_reg_addr(vcpu, reg);
-	if (!addr)
-		return -EINVAL;
-
 	if (KVM_REG_SIZE(reg->id) > sizeof(tmp))
 		return -EINVAL;
 
@@ -220,7 +246,9 @@ int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 		}
 	}
 
-	memcpy(addr, valp, KVM_REG_SIZE(reg->id));
+	err = core_reg_rw(vcpu, reg->id, valp, false);
+	if (err)
+		goto out;
 
 	if (*vcpu_cpsr(vcpu) & PSR_MODE32_BIT) {
 		int i, nr_reg;
-- 
2.53.0

[PATCH v1 23/26] KVM: s390: arm64: Implement feature sanitisation

[Steffen Eiden]

Implement feature detection and sanitization for arm64 KVM guests on
s390.

Query hardware capabilities using QAAF instruction and sanitize arm64 ID
registers. Provide accessor functions for queried capabilities and
integrate feature detection into module initialization sequence.

Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/kvm_feature.h          | 111 ++++++++++++
 arch/s390/include/asm/kvm_host_arm64.h       |   2 +-
 arch/s390/include/asm/kvm_host_arm64_types.h |   3 +-
 arch/s390/include/asm/kvm_nested.h           |   5 +
 arch/s390/kvm/arm64/Makefile                 |   1 +
 arch/s390/kvm/arm64/arm.c                    |   6 +
 arch/s390/kvm/arm64/feature.c                | 170 +++++++++++++++++++
 7 files changed, 296 insertions(+), 2 deletions(-)
 create mode 100644 arch/s390/include/asm/kvm_feature.h
 create mode 100644 arch/s390/kvm/arm64/feature.c

diff --git a/arch/s390/include/asm/kvm_feature.h b/arch/s390/include/asm/kvm_feature.h
new file mode 100644
index 000000000000..dda99c3b09af
--- /dev/null
+++ b/arch/s390/include/asm/kvm_feature.h
@@ -0,0 +1,111 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __S390_KVM_ARM64_FEATURE_H__
+#define __S390_KVM_ARM64_FEATURE_H__
+
+#include <linux/init.h>
+#include <linux/types.h>
+#include <asm/kvm_host_arm64_types.h>
+
+#include <kvm/arm64/kvm_feature.h>
+
+int __init kvm_arm_host_detect_features(void);
+
+/**
+ * kvm_sae_supported_sd_formats() - Retrieve supported SAE SD formats
+ *
+ * Return: Bitmap of supported SAE state description formats.
+ */
+static inline u32 kvm_sae_supported_sd_formats(void)
+{
+	extern struct qaaf_qmc_block __qaaf_qmp;
+
+	return __qaaf_qmp.ssdf;
+}
+
+/**
+ * kvm_sae_supported_sa_formats() - Retrieve supported SAE SA formats
+ *
+ * Return: Bitmap of supported SAE save area formats.
+ */
+static inline u32 kvm_sae_supported_sa_formats(void)
+{
+	extern struct qaaf_qmc_block __qaaf_qmp;
+
+	return __qaaf_qmp.ssaf;
+}
+
+/**
+ * kvm_sae_max_vcpus() - Retrieve maximum supported vcpus
+ *
+ * Return: Max number of vCPUs supported by the SAE instruction/the machine
+ *	as indicated by QAAF, not necessarily that supported by the host software.
+ */
+static inline u16 kvm_sae_max_vcpus(void)
+{
+	extern struct qaaf_qmc_block __qaaf_qmp;
+
+	/* QAAF QMP reports the max id not the max num */
+	return __qaaf_qmp.maxncpu + 1;
+}
+
+/**
+ * kvm_sae_irptc() - Retrieve supported SAE IRPTCs
+ *
+ * Return: Bitmap of supported SAE IRPTCs.
+ */
+static inline u64 kvm_sae_irptc(void)
+{
+	extern struct qaaf_qmc_block __qaaf_qmp;
+
+	return __qaaf_qmp.regs[QAAF_IRPTC];
+}
+
+/** 
+ * kvm_arm_host_sys_reg_by_name() - Get the value of a sys register
+ *
+ * @id - ARM (ID-)register name
+ *
+ * Get the value of a sys register describing the host configuration.
+ * The returned value indicates support by the machine (HW/FW)
+ * after some sanitisation. Equivalent to arm64's read_sanitised_ftr_reg,
+ * See `enum qaaf_regs` for supported registers. Use the ARM register name
+ * without the QAAF_REG_ prefix as defined in the qaaf_regs enum.
+ *
+ * Example:
+ *	kvm_arm_host_sys_reg_by_name(ID_AA64PFR0_EL1)
+ *
+ * Return: Sanitised HW value of the specified register
+ */
+#define kvm_arm_host_sys_reg_by_name(id) kvm_arm_host_sys_reg_by_id(SYS_##id)
+
+/** 
+ * kvm_arm_host_sys_reg_by_id() - Get the value of a sys register
+ *
+ * @id - ARM sysreg id
+ *
+ * Get the value of a sys register describing the host configuration.
+ * The returned value indicates support by the machine (HW/FW)
+ * after some sanitisation. Equivalent to arm64's read_sanitised_ftr_reg,
+ * See `enum qaaf_regs` for supported registers.
+ *
+ * Example:
+ *	kvm_arm_host_sys_reg_by_id(SYS_ID_AA64PFR0_EL1)
+ *
+ * Return: Sanitised HW value of the specified register
+ */
+u64 kvm_arm_host_sys_reg_by_id(u32 id);
+
+#define kvm_arm_host_has_feat(id, fld, limit)				\
+	id_has_feat(kvm_arm_host_sys_reg_by_name(id), id, fld, limit)
+
+#define kvm_has_fpmr(k)					\
+	(kvm_has_feat((k), ID_AA64PFR2_EL1, FPMR, IMP))
+
+#define kvm_has_s1poe(k)				\
+	(kvm_has_feat((k), ID_AA64MMFR3_EL1, S1POE, IMP))
+
+#define kvm_vcpu_has_pmu(_v) false
+#define kvm_has_mte(_k) false
+
+#endif /* __S390_KVM_ARM64_FEATURE_H__*/
diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index fcbe510cb868..f62daff3303b 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -6,6 +6,7 @@
 
 #include <asm/kvm_host_arm64_types.h>
 #include <asm/debug.h>
+#include <asm/kvm_feature.h>
 
 #define vcpu_gp_regs(v)		((v)->arch.sae_block.gpr)
 
@@ -123,7 +124,6 @@ struct kvm_vcpu_stat {
 #define _vcpu_test_and_clear_flag(v, flagset, ...)	\
 	__vcpu_test_and_clear_flag(&(v)->arch.flagset, __VA_ARGS__)
 
-#define kvm_has_mte(_kvm) false
 #define vcpu_has_sve(_vcpu) false
 #define vcpu_has_ptrauth(_vcpu) false
 
diff --git a/arch/s390/include/asm/kvm_host_arm64_types.h b/arch/s390/include/asm/kvm_host_arm64_types.h
index 16f7018a1714..45fa360601fb 100644
--- a/arch/s390/include/asm/kvm_host_arm64_types.h
+++ b/arch/s390/include/asm/kvm_host_arm64_types.h
@@ -206,7 +206,8 @@ enum {
 	/* 0x3f-0x41 reserved */
 	QAAF_REG_CNTFRQ_EL0		= 0x42,
 	QAAF_REG_CTR_EL0		= 0x43,
-	/* 0x44-0x49 reserved */
+	QAAF_REG_AIDR_EL1		= 0x44,
+	/* 0x43-0x49 reserved */
 	QAAF_IRPTC			= 0x4a,
 	/* 0x4b reserved */
 	QAAF_REG_ICH_VTR_EL2		= 0x4c,
diff --git a/arch/s390/include/asm/kvm_nested.h b/arch/s390/include/asm/kvm_nested.h
index 7158932e718b..f075df33277f 100644
--- a/arch/s390/include/asm/kvm_nested.h
+++ b/arch/s390/include/asm/kvm_nested.h
@@ -10,4 +10,9 @@ static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu)
 	return false;
 }
 
+static inline u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val)
+{
+	BUILD_BUG();
+}
+
 #endif /* ASM_KVM_NESTED_H */
diff --git a/arch/s390/kvm/arm64/Makefile b/arch/s390/kvm/arm64/Makefile
index d0aac34b8d2f..28deeb90efa9 100644
--- a/arch/s390/kvm/arm64/Makefile
+++ b/arch/s390/kvm/arm64/Makefile
@@ -9,6 +9,7 @@ ccflags-y += -I $(src) -I$(srctree)/arch/s390/kvm/gmap -DKVM_S390_ARM64
 
 kvm-arm64-obj := \
 	arm.o \
+	feature.o \
 	guest.o \
 	handle_exit.o \
 	inject_fault.o \
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index 48418c46e451..dc0e070f8a62 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -701,6 +701,8 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg)
 
 static int __init kvm_s390_arm64_init(void)
 {
+	int err = 0;
+
 	if (!sclp.has_aef) {
 		pr_info("SAE is not available\n");
 		return -ENXIO;
@@ -711,6 +713,10 @@ static int __init kvm_s390_arm64_init(void)
 		return -ENXIO;
 	}
 
+	err = kvm_arm_host_detect_features();
+	if (err)
+		return err;
+
 	return kvm_init_with_dev(sizeof(struct kvm_vcpu), 0, THIS_MODULE,
 				 KVM_DEV_NAME, MISC_DYNAMIC_MINOR);
 }
diff --git a/arch/s390/kvm/arm64/feature.c b/arch/s390/kvm/arm64/feature.c
new file mode 100644
index 000000000000..9a34c0866a2b
--- /dev/null
+++ b/arch/s390/kvm/arm64/feature.c
@@ -0,0 +1,170 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/kvm_host.h>
+#include <asm/kvm_host_arm64_types.h>
+#include <asm/kvm_feature.h>
+
+struct qaaf_qmc_block __qaaf_qmc;
+
+#define MASK_RESERVED(_qaafp, _id)				\
+	({							\
+		u64 *_reg = &((_qaafp)->regs[QAAF_REG_##_id]);	\
+		*_reg = (*_reg & ~_id##_RES0) | _id##_RES1;	\
+	})
+
+#define QAAF_FIELD_MODIFY(_idp, _id_field, _val) \
+	FIELD_MODIFY(_id_field##_MASK, _idp, _id_field##_##_val)
+
+#define MODIFY(_qaafp, _id, _field, _val) \
+	QAAF_FIELD_MODIFY(&((_qaafp)->regs[QAAF_REG_##_id]), _id##_##_field, _val)
+
+int __init kvm_arm_host_detect_features(void)
+{
+	qaaf_qmc(&__qaaf_qmc);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64ISAR0_EL1);
+
+	MODIFY(&__qaaf_qmc, ID_AA64ISAR1_EL1, LS64, NI);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64ISAR1_EL1);
+
+	MODIFY(&__qaaf_qmc, ID_AA64ISAR2_EL1, SYSINSTR_128, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64ISAR2_EL1, SYSREG_128, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64ISAR2_EL1, PAC_frac, NI);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64ISAR2_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64ISAR3_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64MMFR0_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64MMFR1_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64MMFR2_EL1);
+
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR3_EL1, Spec_FPACC, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR3_EL1, ADERR, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR3_EL1, SDERR, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR3_EL1, ANERR, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR3_EL1, SNERR, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR3_EL1, MEC, NI);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64MMFR3_EL1);
+
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR4_EL1, SRMASK, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR4_EL1, E3DSE, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR4_EL1, RMEGDI, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR4_EL1, FGWTE3, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64MMFR4_EL1, ASID2, NI);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64MMFR4_EL1);
+
+	MODIFY(&__qaaf_qmc, ID_AA64PFR0_EL1, SEL2, NI);
+	MODIFY(&__qaaf_qmc, ID_AA64PFR0_EL1, SVE, NI);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64PFR0_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64PFR1_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64PFR2_EL1);
+
+	MODIFY(&__qaaf_qmc, ID_AA64DFR0_EL1, PMUVer, NI);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64DFR0_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64DFR1_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64DFR2_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64AFR0_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64AFR1_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64ZFR0_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64SMFR0_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_AA64FPFR0_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_PFR0_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_PFR1_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_PFR2_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_DFR0_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_DFR1_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_MMFR0_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_MMFR1_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_MMFR2_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_MMFR3_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_MMFR4_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_MMFR5_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, ID_ISAR0_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_ISAR1_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_ISAR2_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_ISAR3_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_ISAR4_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_ISAR5_EL1);
+	MASK_RESERVED(&__qaaf_qmc, ID_ISAR6_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, MVFR0_EL1);
+	MASK_RESERVED(&__qaaf_qmc, MVFR1_EL1);
+	MASK_RESERVED(&__qaaf_qmc, MVFR2_EL1);
+
+	MASK_RESERVED(&__qaaf_qmc, CTR_EL0);
+
+	MASK_RESERVED(&__qaaf_qmc, ICH_VTR_EL2);
+
+	return 0;
+}
+
+#define _qaaf_reg_case(id) case SYS_##id: return __qaaf_qmc.regs[QAAF_REG_##id]
+
+u64 kvm_arm_host_sys_reg_by_id(u32 id)
+{
+	switch (id) {
+	_qaaf_reg_case(MIDR_EL1);
+	_qaaf_reg_case(MPIDR_EL1);
+	_qaaf_reg_case(REVIDR_EL1);
+	_qaaf_reg_case(ID_PFR0_EL1);
+	_qaaf_reg_case(ID_PFR1_EL1);
+	_qaaf_reg_case(ID_DFR0_EL1);
+	_qaaf_reg_case(ID_AFR0_EL1);
+	_qaaf_reg_case(ID_MMFR0_EL1);
+	_qaaf_reg_case(ID_MMFR1_EL1);
+	_qaaf_reg_case(ID_MMFR2_EL1);
+	_qaaf_reg_case(ID_MMFR3_EL1);
+	_qaaf_reg_case(ID_ISAR0_EL1);
+	_qaaf_reg_case(ID_ISAR1_EL1);
+	_qaaf_reg_case(ID_ISAR2_EL1);
+	_qaaf_reg_case(ID_ISAR3_EL1);
+	_qaaf_reg_case(ID_ISAR4_EL1);
+	_qaaf_reg_case(ID_ISAR5_EL1);
+	_qaaf_reg_case(ID_MMFR4_EL1);
+	_qaaf_reg_case(ID_ISAR6_EL1);
+	_qaaf_reg_case(MVFR0_EL1);
+	_qaaf_reg_case(MVFR1_EL1);
+	_qaaf_reg_case(MVFR2_EL1);
+	_qaaf_reg_case(ID_PFR2_EL1);
+	_qaaf_reg_case(ID_DFR1_EL1);
+	_qaaf_reg_case(ID_MMFR5_EL1);
+	_qaaf_reg_case(ID_AA64PFR0_EL1);
+	_qaaf_reg_case(ID_AA64PFR1_EL1);
+	_qaaf_reg_case(ID_AA64PFR2_EL1);
+	_qaaf_reg_case(ID_AA64ZFR0_EL1);
+	_qaaf_reg_case(ID_AA64SMFR0_EL1);
+	_qaaf_reg_case(ID_AA64FPFR0_EL1);
+	_qaaf_reg_case(ID_AA64DFR0_EL1);
+	_qaaf_reg_case(ID_AA64DFR1_EL1);
+	_qaaf_reg_case(ID_AA64DFR2_EL1);
+	_qaaf_reg_case(ID_AA64AFR0_EL1);
+	_qaaf_reg_case(ID_AA64AFR1_EL1);
+	_qaaf_reg_case(ID_AA64ISAR0_EL1);
+	_qaaf_reg_case(ID_AA64ISAR1_EL1);
+	_qaaf_reg_case(ID_AA64ISAR2_EL1);
+	_qaaf_reg_case(ID_AA64ISAR3_EL1);
+	_qaaf_reg_case(ID_AA64MMFR0_EL1);
+	_qaaf_reg_case(ID_AA64MMFR1_EL1);
+	_qaaf_reg_case(ID_AA64MMFR2_EL1);
+	_qaaf_reg_case(ID_AA64MMFR3_EL1);
+	_qaaf_reg_case(ID_AA64MMFR4_EL1);
+	_qaaf_reg_case(CNTFRQ_EL0);
+	_qaaf_reg_case(CTR_EL0);
+	_qaaf_reg_case(AIDR_EL1);
+	_qaaf_reg_case(ICH_VTR_EL2);
+	_qaaf_reg_case(PMMIR_EL1);
+	_qaaf_reg_case(PMCR_EL0);
+	_qaaf_reg_case(PMCEID0_EL0);
+	_qaaf_reg_case(PMCEID1_EL0);
+	default:
+		WARN(true, "Unknown system register 0x%x\n", id);
+		return 0xbad1234bad;
+	}
+}
-- 
2.53.0

[PATCH v1 24/26] KVM: s390: arm64: Implement sysreg handling

[Steffen Eiden]

Add arm64 system register support for the s390 KVM backend, including
register enumeration, userspace get/set handling, reset/finalization
hooks, and vCPU run integration for trapped sysreg accesses.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Co-developed-by: Nico Boehr <nrb@linux.ibm.com>
Signed-off-by: Nico Boehr <nrb@linux.ibm.com>
Co-developed-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/arm64/kvm/Makefile                |   1 -
 arch/s390/include/asm/kvm_emulate.h    |  34 ++
 arch/s390/include/asm/kvm_host_arm64.h |   2 +-
 arch/s390/kvm/arm64/Makefile           |   1 +
 arch/s390/kvm/arm64/arm.c              |  23 +-
 arch/s390/kvm/arm64/guest.c            |  20 +-
 arch/s390/kvm/arm64/handle_exit.c      |   1 +
 arch/s390/kvm/arm64/reset.c            |   5 +
 arch/s390/kvm/arm64/sys_regs.c         | 769 +++++++++++++++++++++++++
 arch/s390/kvm/arm64/trace.h            |  33 ++
 virt/kvm/arm64/Makefile.kvm            |   1 +
 11 files changed, 875 insertions(+), 15 deletions(-)
 create mode 100644 arch/s390/kvm/arm64/sys_regs.c
 create mode 100644 arch/s390/kvm/arm64/trace.h

diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index d55c7245aad9..5b4a8d002fc9 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -19,7 +19,6 @@ kvm-y += arm.o mmu.o psci.o hypercalls.o pvtime.o \
 	 guest.o debug.o reset.o sys_regs.o stacktrace.o \
 	 vgic-sys-reg-v3.o fpsimd.o pkvm.o \
 	 arch_timer.o trng.o vmid.o emulate-nested.o nested.o at.o \
-	 $(KVM_ARM64)/sys_regs.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
diff --git a/arch/s390/include/asm/kvm_emulate.h b/arch/s390/include/asm/kvm_emulate.h
index 93b9e777b51e..5549d44fda18 100644
--- a/arch/s390/include/asm/kvm_emulate.h
+++ b/arch/s390/include/asm/kvm_emulate.h
@@ -12,6 +12,7 @@
 
 #include <kvm/arm64/kvm_arm.h>
 #include <kvm/arm64/kvm_emulate.h>
+#include <clocksource/arm_arch_timer.h>
 
 static __always_inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)
 {
@@ -84,6 +85,39 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu)
 			      HCR_TID4 | HCR_TID5 | HCR_TIDCP;
 }
 
+static inline void vcpu_reset_hcrx(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.hcrx_elz = 0;
+	if (kvm_has_feat(vcpu->kvm, ID_AA64ISAR2_EL1, MOPS, IMP))
+		vcpu->arch.hcrx_elz |= HCRX_EL2_MSCEn ;
+}
+
+static inline void vcpu_reset_cptr(struct kvm_vcpu *vcpu)
+{
+	u64 cptr;
+
+	/* we unconditionally have E2H, so disable traps to EL2 for FP and SVE, if enabled */
+	cptr = CPACR_EL1_FPEN;
+	if (vcpu_has_sve(vcpu))
+		cptr |= CPACR_EL1_ZEN;
+	vcpu_write_host_sys_reg(vcpu, cptr, SYS_CPTR_EL2);
+}
+
+static inline void vcpu_reset_cnthctl(struct kvm_vcpu *vcpu)
+{
+	u64 cnthctl;
+
+	/* we unconditionally have E2H, so disable traps to EL2 for physical timer registers */
+	cnthctl = (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN) << 10;
+	vcpu_write_host_sys_reg(vcpu, cnthctl, SYS_CNTHCTL_EL2);
+}
+
+static inline void vcpu_reset_icc(struct kvm_vcpu *vcpu)
+{
+	/* ensure ICC_SRE_EL2.Enable = 1 so ICC_SRE_EL1 is handled in hardware */
+	vcpu_write_host_sys_reg(vcpu, 0xf, SYS_ICC_SRE_EL2);
+}
+
 static inline unsigned long vcpu_get_vsesr(struct kvm_vcpu *vcpu)
 {
 	WARN(true, "not implemented, just feat RAS");
diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index f62daff3303b..cc457d69621c 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -370,7 +370,7 @@ int __init kvm_sys_reg_table_init(void);
 
 static inline u64 kvm_sanitised_host_ftr_reg(u32 id)
 {
-	return 0xbad1234bad;
+	return kvm_arm_host_sys_reg_by_id(id);
 }
 
 #endif /* ASM_KVM_HOST_ARM64_H */
diff --git a/arch/s390/kvm/arm64/Makefile b/arch/s390/kvm/arm64/Makefile
index 28deeb90efa9..bec24b60a071 100644
--- a/arch/s390/kvm/arm64/Makefile
+++ b/arch/s390/kvm/arm64/Makefile
@@ -15,6 +15,7 @@ kvm-arm64-obj := \
 	inject_fault.o \
 	mmu.o \
 	reset.o \
+	sys_regs.o \
 
 kvm-arm64-obj += $(patsubst %.o,%-arm64.o,$(shared-arm64-obj))
 kvm-arm64-obj += $(patsubst %.o,%-arm64.o,$(kvm-y))
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index dc0e070f8a62..88b66552cff0 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -14,6 +14,7 @@
 
 #include <kvm/arm64/handle_exit.h>
 #include <kvm/arm64/kvm_emulate.h>
+#include <kvm/arm64/sys_regs.h>
 
 #include <gmap.h>
 
@@ -227,6 +228,7 @@ bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu)
 int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
+	int ret;
 
 	if (!kvm_vcpu_initialized(vcpu))
 		return -ENOEXEC;
@@ -237,6 +239,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
 	if (likely(READ_ONCE(vcpu->pid)))
 		return 0;
 
+	ret = kvm_finalize_sys_regs(vcpu);
+	if (ret)
+		return ret;
 
 	mutex_lock(&kvm->arch.config_lock);
 	set_bit(KVM_ARCH_FLAG_HAS_RAN_ONCE, &kvm->arch.flags);
@@ -363,6 +368,9 @@ static void arm_vcpu_run(struct kvm_vcpu *vcpu)
 	guest_enter_irqoff();
 	local_irq_enable();
 
+	vcpu_write_host_sys_reg(vcpu, vcpu->arch.hcr_elz, SYS_HCR_EL2);
+	vcpu_write_host_sys_reg(vcpu, vcpu->arch.hcrx_elz, SYS_HCRX_EL2);
+	vcpu_write_host_sys_reg(vcpu, vcpu->arch.mpidr, SYS_VMPIDR_EL2); //TODO implement mpidr plumbing
 	sae_block->icptr = 0;
 
 	sae64a(sae_block);
@@ -690,15 +698,6 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
 	return -ENOIOCTLCMD;
 }
 
-void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, enum vcpu_sysreg reg)
-{
-}
-
-u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg)
-{
-	return 0xbad12324bad;
-}
-
 static int __init kvm_s390_arm64_init(void)
 {
 	int err = 0;
@@ -717,6 +716,12 @@ static int __init kvm_s390_arm64_init(void)
 	if (err)
 		return err;
 
+	err = kvm_sys_reg_table_init();
+	if (err) {
+		kvm_info("Error initializing system register tables");
+		return err;
+	}
+
 	return kvm_init_with_dev(sizeof(struct kvm_vcpu), 0, THIS_MODULE,
 				 KVM_DEV_NAME, MISC_DYNAMIC_MINOR);
 }
diff --git a/arch/s390/kvm/arm64/guest.c b/arch/s390/kvm/arm64/guest.c
index 893d48037292..3287059f89c7 100644
--- a/arch/s390/kvm/arm64/guest.c
+++ b/arch/s390/kvm/arm64/guest.c
@@ -2,6 +2,8 @@
 #include <linux/kvm_host.h>
 #include <linux/kvm.h>
 
+#include <kvm/arm64/sys_regs.h>
+
 #include "guest.h"
 
 const struct kvm_stats_desc kvm_vm_stats_desc[] = {
@@ -42,12 +44,22 @@ const struct kvm_stats_header kvm_vcpu_stats_header = {
 
 int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
 {
-	return copy_core_reg_indices(vcpu, uindices);
+	int ret;
+
+	ret = copy_core_reg_indices(vcpu, uindices);
+	if (ret < 0)
+		return ret;
+	uindices += ret;
+
+	return kvm_arm_copy_sys_reg_indices(vcpu, uindices);
 }
 
 unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu)
 {
-	return num_core_regs(vcpu);
+	unsigned long num = num_core_regs(vcpu);
+
+	num += kvm_arm_num_sys_reg_descs(vcpu);
+	return num;
 }
 
 int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
@@ -60,7 +72,7 @@ int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	case KVM_REG_ARM_CORE:
 		return get_core_reg(vcpu, reg);
 	default:
-		return -EINVAL;
+		return kvm_arm_sys_reg_get_reg(vcpu, reg);
 	}
 }
 
@@ -74,7 +86,7 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 	case KVM_REG_ARM_CORE:
 		return set_core_reg(vcpu, reg);
 	default:
-		return -EINVAL;
+		return kvm_arm_sys_reg_set_reg(vcpu, reg);
 	}
 }
 
diff --git a/arch/s390/kvm/arm64/handle_exit.c b/arch/s390/kvm/arm64/handle_exit.c
index bccd644fb07c..53aa7f6c4745 100644
--- a/arch/s390/kvm/arm64/handle_exit.c
+++ b/arch/s390/kvm/arm64/handle_exit.c
@@ -47,6 +47,7 @@ static int handle_hvc(struct kvm_vcpu *vcpu)
 exit_handle_fn arm_exit_handlers[] = {
 	[0 ... ESR_ELx_EC_MAX]	= kvm_handle_unknown_ec,
 	[ESR_ELx_EC_HVC64]	= handle_hvc,
+	[ESR_ELx_EC_SYS64]	= kvm_handle_sys_reg,
 	[ESR_ELx_EC_IABT_LOW]	= kvm_handle_guest_abort,
 	[ESR_ELx_EC_DABT_LOW]	= kvm_handle_guest_abort,
 };
diff --git a/arch/s390/kvm/arm64/reset.c b/arch/s390/kvm/arm64/reset.c
index b38b36c72c72..19404bfb85b6 100644
--- a/arch/s390/kvm/arm64/reset.c
+++ b/arch/s390/kvm/arm64/reset.c
@@ -33,9 +33,14 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 		vcpu_put(vcpu);
 
 	kvm_reset_vcpu_core_regs(vcpu);
+	kvm_reset_sys_regs(vcpu);
 
 	/* Reset special registers */
 	vcpu_reset_hcr(vcpu);
+	vcpu_reset_hcrx(vcpu);
+	vcpu_reset_cptr(vcpu);
+	vcpu_reset_cnthctl(vcpu);
+	vcpu_reset_icc(vcpu);
 
 	if (reset_state.reset) {
 		*vcpu_pc(vcpu) = reset_state.pc;
diff --git a/arch/s390/kvm/arm64/sys_regs.c b/arch/s390/kvm/arm64/sys_regs.c
new file mode 100644
index 000000000000..0bcfdade8509
--- /dev/null
+++ b/arch/s390/kvm/arm64/sys_regs.c
@@ -0,0 +1,769 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/cpumask.h>
+#include <linux/cacheinfo.h>
+#include <linux/kvm_host.h>
+#include <linux/log2.h>
+#include <asm/sysreg-defs.h>
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_feature.h>
+#include <asm/kvm_nested.h>
+
+#include <kvm/arm64/sys_regs.h>
+
+#define CREATE_TRACE_POINTS
+#include "trace.h"
+
+#define SR_INVALID 0x8badf00d8badf00d
+
+enum sr_loc_attr {
+	SR_LOC_SAVE_AREA,	/* Register on SA, loaded (on cpu) */
+	SR_LOC_SPECIAL,		/* Register in state description or Special care register*/
+	SR_LOC_INVALID,		/* Register is unknown */
+};
+
+struct sr_loc {
+	enum sr_loc_attr loc;
+};
+
+#define SREG_RANGE(NAME) __##NAME##_BEGIN__ + 1 ... __##NAME##_END__ - 1
+static __always_inline void locate_register(const struct kvm_vcpu *vcpu,
+					    enum vcpu_sysreg reg,
+					    struct sr_loc *loc)
+{
+	switch (reg) {
+	case SREG_RANGE(STATE_DESC):
+	case SREG_RANGE(SPECIAL):
+		loc->loc = SR_LOC_SPECIAL;
+		break;
+	case SREG_RANGE(SAVE_AREA):
+		loc->loc = SR_LOC_SAVE_AREA;
+		break;
+	default:
+		WARN(true, "%s wants to read invalid register %x", __func__, reg);
+		loc->loc = SR_LOC_INVALID;
+	}
+}
+
+static __always_inline u64 read_special_sr(const struct kvm_vcpu *vcpu,
+					   enum vcpu_sysreg reg)
+{
+	switch (reg) {
+	case CLIDR_EL1:		return vcpu->arch.sys_reg_clidr_el1;
+	case CSSELR_EL1:	return vcpu->arch.sys_reg_csselr_el1;
+	case MPIDR_EL1:		return vcpu->arch.mpidr;
+	case CNTP_CTL_EL0:	return vcpu->arch.sae_block.cntp_ctl;
+	case CNTV_CTL_EL0:	return vcpu->arch.sae_block.cntv_ctl;
+	case CONTEXTIDR_EL1:	return vcpu->arch.sae_block.contextidr_el1;
+	case SP_EL1:		return vcpu->arch.sae_block.sp_el1;
+	default:
+		WARN(true, "%s wants to read non-special register %x", __func__, reg);
+		return SR_INVALID;
+	}
+}
+
+static __always_inline u64 read_sr_from_vcpu(const struct kvm_vcpu *vcpu,
+					     enum vcpu_sysreg reg)
+{
+	switch (reg) {
+	case ACTLR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_ACTLR_EL1);
+	case AFSR0_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_AFSR0_EL1);
+	case AFSR1_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_AFSR1_EL1);
+	case CNTFRQ_EL0:	return _vcpu_read_sys_reg(vcpu, SYS_CNTFRQ_EL0);
+	case CNTP_CVAL_EL0:	return _vcpu_read_sys_reg(vcpu, SYS_CNTP_CVAL_EL0);
+	case CNTV_CVAL_EL0:	return _vcpu_read_sys_reg(vcpu, SYS_CNTV_CVAL_EL0);
+	case DISR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_DISR_EL1);
+	case MIDR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_MIDR_EL1);
+	case OSLSR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_OSLSR_EL1);
+	case PAR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_PAR_EL1);
+	case SCTLR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_SCTLR_EL1);
+	case CPACR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_CPACR_EL1);
+	case VBAR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_VBAR_EL1);
+	case SPSR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_SPSR_EL1);
+	case ELR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_ELR_EL1);
+	case ESR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_ESR_EL1);
+	case TCR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_TCR_EL1);
+	case MAIR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_MAIR_EL1);
+	case TTBR0_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_TTBR0_EL1);
+	case TTBR1_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_TTBR1_EL1);
+	case FAR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_FAR_EL1);
+	case TPIDR_EL0:		return _vcpu_read_sys_reg(vcpu, SYS_TPIDR_EL0);
+	case TPIDR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_TPIDR_EL1);
+	case TPIDRRO_EL0:	return _vcpu_read_sys_reg(vcpu, SYS_TPIDRRO_EL0);
+	case CNTKCTL_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_CNTKCTL_EL1);
+	case ZCR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_ZCR_EL1);
+	case SCXTNUM_EL0:	return _vcpu_read_sys_reg(vcpu, SYS_SCXTNUM_EL0);
+	case SCXTNUM_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_SCXTNUM_EL1);
+	case APIBKEYLO_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APIBKEYLO_EL1);
+	case APIBKEYHI_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APIBKEYHI_EL1);
+	case APIAKEYLO_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APIAKEYLO_EL1);
+	case APIAKEYHI_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APIAKEYHI_EL1);
+	case APGAKEYLO_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APGAKEYLO_EL1);
+	case APGAKEYHI_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APGAKEYHI_EL1);
+	case APDBKEYLO_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APDBKEYLO_EL1);
+	case APDBKEYHI_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APDBKEYHI_EL1);
+	case APDAKEYLO_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APDAKEYLO_EL1);
+	case APDAKEYHI_EL1:	return _vcpu_read_sys_reg(vcpu, SYS_APDAKEYHI_EL1);
+	case MDSCR_EL1:		return _vcpu_read_sys_reg(vcpu, SYS_MDSCR_EL1);
+	default:
+		WARN(true, "%s: failed to resolve %x", __func__,  reg);
+		return SR_INVALID;
+	}
+}
+
+u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg)
+{
+	struct sr_loc loc = {};
+
+	locate_register(vcpu, reg, &loc);
+
+	if (unlikely(loc.loc == SR_LOC_INVALID)) {
+		WARN(true, "%s: failed to locate %u", __func__,  reg);
+		return SR_INVALID;
+	}
+
+	if (loc.loc == SR_LOC_SPECIAL)
+		return read_special_sr(vcpu, reg);
+
+	return read_sr_from_vcpu(vcpu, reg);
+}
+
+static __always_inline void write_special_sr(struct kvm_vcpu *vcpu, u64 val,
+					     enum vcpu_sysreg reg)
+{
+	switch (reg) {
+	case CLIDR_EL1:		vcpu->arch.sys_reg_clidr_el1 = val;		break;
+	case CSSELR_EL1:	vcpu->arch.sys_reg_csselr_el1 = val;		break;
+	case MPIDR_EL1:		vcpu->arch.mpidr = val;				break;
+	case CNTP_CTL_EL0: 	vcpu->arch.sae_block.cntp_ctl = val;		break;
+	case CNTV_CTL_EL0:	vcpu->arch.sae_block.cntv_ctl = val;		break;
+	case CONTEXTIDR_EL1:	vcpu->arch.sae_block.contextidr_el1 = val;	break;
+	case SP_EL1:		vcpu->arch.sae_block.sp_el1 = val;		break;
+	default:
+		WARN(true, "%s wants to write non-special register %x", __func__, reg);
+	}
+}
+
+static __always_inline void write_sr_to_vcpu(struct kvm_vcpu *vcpu, u64 val,
+					     enum vcpu_sysreg reg)
+{
+	switch (reg) {
+	case ACTLR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_ACTLR_EL1);		break;
+	case AFSR0_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_AFSR0_EL1);		break;
+	case AFSR1_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_AFSR1_EL1);		break;
+	case CNTFRQ_EL0:	_vcpu_write_sys_reg(vcpu, val, SYS_CNTFRQ_EL0);		break;
+	case CNTP_CVAL_EL0:	_vcpu_write_sys_reg(vcpu, val, SYS_CNTP_CVAL_EL0);	break;
+	case CNTV_CVAL_EL0:	_vcpu_write_sys_reg(vcpu, val, SYS_CNTV_CVAL_EL0);	break;
+	case DISR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_DISR_EL1);		break;
+	case MIDR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_MIDR_EL1);		break;
+	case PAR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_PAR_EL1);		break;
+	case OSLAR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_OSLAR_EL1);		break;
+	case SCTLR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_SCTLR_EL1);		break;
+	case CPACR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_CPACR_EL1);		break;
+	case VBAR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_VBAR_EL1);		break;
+	case SPSR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_SPSR_EL1);		break;
+	case ELR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_ELR_EL1);		break;
+	case ESR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_ESR_EL1);		break;
+	case TCR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_TCR_EL1);		break;
+	case MAIR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_MAIR_EL1);		break;
+	case TTBR0_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_TTBR0_EL1);		break;
+	case TTBR1_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_TTBR1_EL1);		break;
+	case FAR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_FAR_EL1);		break;
+	case TPIDR_EL0:		_vcpu_write_sys_reg(vcpu, val, SYS_TPIDR_EL0);		break;
+	case TPIDR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_TPIDR_EL1);		break;
+	case TPIDRRO_EL0:	_vcpu_write_sys_reg(vcpu, val, SYS_TPIDRRO_EL0);	break;
+	case CNTKCTL_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_CNTKCTL_EL1);	break;
+	case ZCR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_ZCR_EL1);		break;
+	case SCXTNUM_EL0:	_vcpu_write_sys_reg(vcpu, val, SYS_SCXTNUM_EL0);	break;
+	case SCXTNUM_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_SCXTNUM_EL1);	break;
+	case APIBKEYLO_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APIBKEYLO_EL1);	break;
+	case APIBKEYHI_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APIBKEYHI_EL1);	break;
+	case APIAKEYLO_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APIAKEYLO_EL1);	break;
+	case APIAKEYHI_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APIAKEYHI_EL1);	break;
+	case APGAKEYLO_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APGAKEYLO_EL1);	break;
+	case APGAKEYHI_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APGAKEYHI_EL1);	break;
+	case APDBKEYLO_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APDBKEYLO_EL1);	break;
+	case APDBKEYHI_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APDBKEYHI_EL1);	break;
+	case APDAKEYLO_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APDAKEYLO_EL1);	break;
+	case APDAKEYHI_EL1:	_vcpu_write_sys_reg(vcpu, val, SYS_APDAKEYHI_EL1);	break;
+	case MDSCR_EL1:		_vcpu_write_sys_reg(vcpu, val, SYS_MDSCR_EL1);		break;
+	default:
+		WARN(true, "%s: failed to resolve %x", __func__,  reg);
+	}
+}
+
+void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, enum vcpu_sysreg reg)
+{
+	struct sr_loc loc = {};
+
+	locate_register(vcpu, reg, &loc);
+
+	if (unlikely(loc.loc == SR_LOC_INVALID)) {
+		WARN(true, "%s: failed to locate %u", __func__,  reg);
+		return;
+	}
+
+	if (loc.loc == SR_LOC_SPECIAL) {
+		write_special_sr(vcpu, val, reg);
+		return;
+	}
+
+	write_sr_to_vcpu(vcpu, val, reg);
+}
+
+u64 vcpu_read_host_sys_reg(const struct kvm_vcpu *vcpu, int reg)
+{
+	switch (reg) {
+	case SYS_ICH_LR0_EL2:	return vcpu->arch.sae_block.ic_regs.ich_lrn_el2[0];
+	case SYS_ICH_LR1_EL2:	return vcpu->arch.sae_block.ic_regs.ich_lrn_el2[1];
+	case SYS_ICH_LR2_EL2:	return vcpu->arch.sae_block.ic_regs.ich_lrn_el2[2];
+	case SYS_ICH_LR3_EL2:	return vcpu->arch.sae_block.ic_regs.ich_lrn_el2[3];
+	case SYS_ICH_HCR_EL2:	return vcpu->arch.sae_block.ic_regs.ich_hcr_el2;
+	case SYS_ICH_AP0R0_EL2:	return vcpu->arch.sae_block.ic_regs.ich_ap0r0_el2;
+	case SYS_ICH_AP1R0_EL2:	return vcpu->arch.sae_block.ic_regs.ich_ap1r0_el2;
+	case SYS_ICH_VMCR_EL2:	return vcpu->arch.sae_block.ic_regs.ich_vmcr_el2;
+	case SYS_VSESR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_VSESR_EL2);
+	case SYS_HCR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HCR_EL2);
+	case SYS_CPTR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_CPTR_EL2);
+	case SYS_MDCR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_MDCR_EL2);
+	case SYS_HCRX_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HCRX_EL2);
+	case SYS_HFGRTR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HFGRTR_EL2);
+	case SYS_HFGWTR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HFGWTR_EL2);
+	case SYS_HDFGWTR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HDFGWTR_EL2);
+	case SYS_HDFGRTR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HDFGRTR_EL2);
+	case SYS_HFGITR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_HFGITR_EL2);
+	case SYS_CNTHCTL_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_CNTHCTL_EL2);
+	case SYS_ICC_SRE_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_ICC_SRE_EL2);
+	case SYS_VMPIDR_EL2:	return _vcpu_read_sys_reg(vcpu, SYS_VMPIDR_EL2);
+	default:
+		WARN(true, "%s wants to read non-host register %x", __func__, reg);
+		return SR_INVALID;
+	}
+}
+
+void vcpu_write_host_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
+{
+	switch (reg) {
+	case SYS_ICH_LR0_EL2:	vcpu->arch.sae_block.ic_regs.ich_lrn_el2[0] = val;	break;
+	case SYS_ICH_LR1_EL2:	vcpu->arch.sae_block.ic_regs.ich_lrn_el2[1] = val;	break;
+	case SYS_ICH_LR2_EL2:	vcpu->arch.sae_block.ic_regs.ich_lrn_el2[2] = val;	break;
+	case SYS_ICH_LR3_EL2:	vcpu->arch.sae_block.ic_regs.ich_lrn_el2[3] = val;	break;
+	case SYS_ICH_HCR_EL2:	vcpu->arch.sae_block.ic_regs.ich_hcr_el2 = val;		break;
+	case SYS_ICH_AP0R0_EL2:	vcpu->arch.sae_block.ic_regs.ich_ap0r0_el2 = val;	break;
+	case SYS_ICH_AP1R0_EL2:	vcpu->arch.sae_block.ic_regs.ich_ap1r0_el2 = val;	break;
+	case SYS_ICH_VMCR_EL2:	vcpu->arch.sae_block.ic_regs.ich_vmcr_el2 = val;	break;
+	case SYS_VSESR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_VSESR_EL2);		break;
+	case SYS_HCR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HCR_EL2);		break;
+	case SYS_CPTR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_CPTR_EL2);		break;
+	case SYS_MDCR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_MDCR_EL2);		break;
+	case SYS_HCRX_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HCRX_EL2);		break;
+	case SYS_HFGRTR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HFGRTR_EL2);		break;
+	case SYS_HFGWTR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HFGWTR_EL2);		break;
+	case SYS_HDFGWTR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HDFGWTR_EL2);	break;
+	case SYS_HDFGRTR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HDFGRTR_EL2);	break;
+	case SYS_HFGITR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_HFGITR_EL2);		break;
+	case SYS_CNTHCTL_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_CNTHCTL_EL2);	break;
+	case SYS_ICC_SRE_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_ICC_SRE_EL2);	break;
+	case SYS_VMPIDR_EL2:	_vcpu_write_sys_reg(vcpu, val, SYS_VMPIDR_EL2);		break;
+	default:
+		WARN(true, "%s wants to write non-host register %x", __func__, reg);
+	}
+}
+
+static int set_id_aa64dfr0_el1(struct kvm_vcpu *vcpu,
+			       const struct sys_reg_desc *rd,
+			       u64 val)
+{
+	u8 debugver = SYS_FIELD_GET(ID_AA64DFR0_EL1, DebugVer, val);
+	u8 pmuver = SYS_FIELD_GET(ID_AA64DFR0_EL1, PMUVer, val);
+
+	if (pmuver == ID_AA64DFR0_EL1_PMUVer_IMP_DEF)
+		return -EINVAL;
+
+	/*
+	 * ID_AA64DFR0_EL1.DebugVer is one of those awkward fields with a
+	 * nonzero minimum safe value.
+	 */
+	if (debugver < ID_AA64DFR0_EL1_DebugVer_IMP)
+		return -EINVAL;
+
+	return set_id_reg(vcpu, rd, val);
+}
+
+static int set_id_aa64pfr0_el1(struct kvm_vcpu *vcpu,
+			       const struct sys_reg_desc *rd, u64 user_val)
+{
+	/* Fail the guest's request to disable the AA64 ISA at EL{0,1,2} */
+	if (!FIELD_GET(ID_AA64PFR0_EL1_EL0, user_val) ||
+	    !FIELD_GET(ID_AA64PFR0_EL1_EL1, user_val) ||
+	    (vcpu_has_nv(vcpu) && !FIELD_GET(ID_AA64PFR0_EL1_EL2, user_val)))
+		return -EINVAL;
+
+	return set_id_reg(vcpu, rd, user_val);
+}
+
+int arm64_check_features(struct kvm_vcpu *vcpu,
+			 const struct sys_reg_desc *rd,
+			 u64 val)
+{
+	u32 id = reg_to_encoding(rd);
+	u64 host_val;
+	u64 old_val;
+
+	/*
+	 * This is a very minimal implementation to sanitise only the craziest
+	 * shenanigans we could get from userspace. The final goal is to share
+	 * the sanitisation with arm as well but this needs access to arm
+	 * (core) kernel code for which there is no suitable solution yet. TODO
+	 */
+
+	/*
+	 * If the register is RAZ we know the only safe value is 0.
+	 */
+	if (sysreg_visible_as_raz(vcpu, rd))
+		return val ? -E2BIG : 0;
+
+	host_val = kvm_arm_host_sys_reg_by_id(id);
+
+	switch (id) {
+	case SYS_ID_AA64MMFR0_EL1:
+		/* Forbid PRANGE values we do not know about */
+		if (SYS_FIELD_GET(ID_AA64MMFR0_EL1, PARANGE, host_val) >
+		    ID_AA64MMFR0_EL1_PARANGE_56)
+			return -E2BIG;
+		break;
+	case SYS_CTR_EL0:
+		/* forbid upgrading from VIPT to PIPT */
+		old_val = read_id_reg(vcpu, rd);
+		if (SYS_FIELD_GET(CTR_EL0, L1Ip, old_val) == CTR_EL0_L1Ip_VIPT &&
+		    SYS_FIELD_GET(CTR_EL0, L1Ip, val) == CTR_EL0_L1Ip_PIPT)
+			return -E2BIG;
+		break;
+	}
+	return 0;
+}
+
+static u64 __ro_after_init boot_cpu_midr_val;
+static u64 __ro_after_init boot_cpu_revidr_val;
+static u64 __ro_after_init boot_cpu_aidr_val;
+
+static void init_imp_id_regs(void)
+{
+	boot_cpu_midr_val = kvm_arm_host_sys_reg_by_name(MIDR_EL1);
+	boot_cpu_revidr_val = kvm_arm_host_sys_reg_by_name(REVIDR_EL1);
+	boot_cpu_aidr_val = 0xC0FFEE; // TODO qaaf support needed.
+}
+
+u64 reset_imp_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	switch (reg_to_encoding(r)) {
+	case SYS_MIDR_EL1:
+		return boot_cpu_midr_val;
+	case SYS_REVIDR_EL1:
+		return boot_cpu_revidr_val;
+	case SYS_AIDR_EL1:
+		return boot_cpu_aidr_val;
+	default:
+		KVM_BUG_ON(1, vcpu->kvm);
+		return 0;
+	}
+}
+
+bool access_imp_id_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+		       const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		return write_to_read_only(vcpu, p, r);
+
+	/*
+	 * Return the VM-scoped implementation ID register values if userspace
+	 * has made them writable.
+	 */
+	if (test_bit(KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, &vcpu->kvm->arch.flags))
+		return access_id_reg(vcpu, p, r);
+
+	switch (reg_to_encoding(r)) {
+	case SYS_REVIDR_EL1:
+		p->regval = boot_cpu_revidr_val; //no old api to conform to?
+		break;
+	case SYS_AIDR_EL1:
+		p->regval = boot_cpu_aidr_val;
+		break;
+	default:
+		WARN_ON_ONCE(1);
+	}
+
+	return true;
+}
+
+static int get_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 *val)
+{
+	*val = vcpu->arch.mpidr;
+	return 0;
+}
+
+static int set_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, u64 val)
+{
+	vcpu->arch.mpidr = val;
+	return 0;
+}
+
+static u64 reset_midr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	_vcpu_write_sys_reg(vcpu, boot_cpu_midr_val, SYS_MIDR_EL1);
+	return boot_cpu_midr_val;
+}
+
+static u64 reset_oslsr_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	u64 oslsr = r->val;
+
+	set_oslsr_el1(vcpu, r, oslsr);
+	return oslsr;
+}
+
+static int arch_timer_set_user(struct kvm_vcpu *vcpu,
+			       const struct sys_reg_desc *rd,
+			       u64 val)
+{
+	switch (reg_to_encoding(rd)) {
+	case SYS_CNTVCT_EL0:
+		vcpu->arch.sae_block.gpto = ptff_qagto(val);
+		return 0;
+	case SYS_CNTPCT_EL0:
+		return 0;
+	}
+
+	__vcpu_assign_sys_reg(vcpu, rd->reg, val);
+	return 0;
+}
+
+static int arch_timer_get_user(struct kvm_vcpu *vcpu,
+			       const struct sys_reg_desc *rd,
+			       u64 *val)
+{
+	switch (reg_to_encoding(rd)) {
+	case SYS_CNTVCT_EL0:
+		*val = ptff_qagpt(vcpu->arch.sae_block.gpto);
+		break;
+	case SYS_CNTPCT_EL0:
+		*val = ptff_qagpt(vcpu->arch.sae_block.gpto);
+		break;
+	default:
+		*val = __vcpu_sys_reg(vcpu, rd->reg);
+	}
+
+	return 0;
+}
+
+#define TIMER_REG(name) { \
+	SYS_DESC(SYS_##name),			\
+	.reset = reset_val,			\
+	.reg = name,				\
+	.get_user = arch_timer_get_user,	\
+	.set_user = arch_timer_set_user,	\
+}
+
+/*
+ * Architected system registers.
+ * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
+ */
+static const struct sys_reg_desc sys_reg_descs[] = {
+	/* Op0 = 2 */
+	{ SYS_DESC(SYS_OSLAR_EL1), trap_oslar_el1 },
+	{ SYS_DESC(SYS_OSLSR_EL1), trap_oslsr_el1, reset_oslsr_el1, OSLSR_EL1,
+	  OSLSR_EL1_OSLM_IMPLEMENTED, NULL, set_oslsr_el1 },
+
+	/* Op0 = 3 */
+	/* Op1 = 0 */
+	/* CRn = 0 */
+	/* CRm = 0 */
+
+	//should not trap, right?
+	{ SYS_DESC(SYS_MIDR_EL1), NULL, reset_midr, 0, GENMASK_ULL(31, 0),
+	  get_id_reg, set_imp_id_reg },
+	{ SYS_DESC(SYS_MPIDR_EL1), NULL, reset_mpidr, MPIDR_EL1, 0, get_mpidr,
+	  set_mpidr },
+	IMPLEMENTATION_ID(REVIDR_EL1, GENMASK_ULL(63, 0)),
+
+	/*
+	 * ID regs: all ID_SANITISED() entries here must have corresponding
+	 * entries in arm64_ftr_regs[].
+	 */
+
+	/* AArch64 mappings of the AArch32 ID registers */
+	/* CRm=1 */
+	ID_HIDDEN(ID_PFR0_EL1),
+	ID_HIDDEN(ID_PFR1_EL1),
+	ID_HIDDEN(ID_DFR0_EL1),
+	ID_HIDDEN(ID_AFR0_EL1),
+	ID_HIDDEN(ID_MMFR0_EL1),
+	ID_HIDDEN(ID_MMFR1_EL1),
+	ID_HIDDEN(ID_MMFR2_EL1),
+	ID_HIDDEN(ID_MMFR3_EL1),
+
+	/* CRm=2 */
+	ID_HIDDEN(ID_ISAR0_EL1),
+	ID_HIDDEN(ID_ISAR1_EL1),
+	ID_HIDDEN(ID_ISAR2_EL1),
+	ID_HIDDEN(ID_ISAR3_EL1),
+	ID_HIDDEN(ID_ISAR4_EL1),
+	ID_HIDDEN(ID_ISAR5_EL1),
+	ID_HIDDEN(ID_MMFR4_EL1),
+	ID_HIDDEN(ID_ISAR6_EL1),
+
+	/* CRm=3 */
+	ID_HIDDEN(MVFR0_EL1),
+	ID_HIDDEN(MVFR1_EL1),
+	ID_HIDDEN(MVFR2_EL1),
+	ID_UNALLOCATED(3, 3),
+	ID_HIDDEN(ID_PFR2_EL1),
+	ID_HIDDEN(ID_DFR1_EL1),
+	ID_HIDDEN(ID_MMFR5_EL1),
+	ID_UNALLOCATED(3, 7),
+
+	/* AArch64 ID registers */
+	/* CRm=4 */
+	ID_FILTERED(ID_AA64PFR0_EL1, id_aa64pfr0_el1,
+		    ~(ID_AA64PFR0_EL1_AMU | ID_AA64PFR0_EL1_MPAM |
+		      ID_AA64PFR0_EL1_SVE | ID_AA64PFR0_EL1_RAS |
+		      ID_AA64PFR0_EL1_GIC | ID_AA64PFR0_EL1_AdvSIMD |
+		      ID_AA64PFR0_EL1_FP)),
+	ID_WRITABLE(ID_AA64PFR1_EL1,
+		    ~(ID_AA64PFR1_EL1_PFAR | ID_AA64PFR1_EL1_DF2 |
+		      ID_AA64PFR1_EL1_MTEX | ID_AA64PFR1_EL1_THE |
+		      ID_AA64PFR1_EL1_GCS | ID_AA64PFR1_EL1_MTE_frac |
+		      ID_AA64PFR1_EL1_NMI | ID_AA64PFR1_EL1_RNDR_trap |
+		      ID_AA64PFR1_EL1_SME | ID_AA64PFR1_EL1_RES0 |
+		      ID_AA64PFR1_EL1_MPAM_frac | ID_AA64PFR1_EL1_RAS_frac |
+		      ID_AA64PFR1_EL1_MTE)),
+	ID_WRITABLE(ID_AA64PFR2_EL1, ID_AA64PFR2_EL1_FPMR),
+	ID_UNALLOCATED(4, 3),
+	ID_WRITABLE(ID_AA64ZFR0_EL1, ~ID_AA64ZFR0_EL1_RES0),
+	ID_HIDDEN(ID_AA64SMFR0_EL1),
+	ID_UNALLOCATED(4, 6),
+	ID_WRITABLE(ID_AA64FPFR0_EL1, ~ID_AA64FPFR0_EL1_RES0),
+
+	/* CRm=5 */
+	/*
+	 * Prior to FEAT_Debugv8.9, the architecture defines context-aware
+	 * breakpoints (CTX_CMPs) as the highest numbered breakpoints (BRPs).
+	 * KVM does not trap + emulate the breakpoint registers, and as such
+	 * cannot support a layout that misaligns with the underlying hardware.
+	 * While it may be possible to describe a subset that aligns with
+	 * hardware, just prevent changes to BRPs and CTX_CMPs altogether for
+	 * simplicity.
+	 *
+	 * See DDI0487K.a, section D2.8.3 Breakpoint types and linking
+	 * of breakpoints for more details.
+	 */
+	ID_FILTERED(ID_AA64DFR0_EL1, id_aa64dfr0_el1,
+		    ID_AA64DFR0_EL1_DoubleLock_MASK |
+			    ID_AA64DFR0_EL1_WRPs_MASK |
+			    ID_AA64DFR0_EL1_PMUVer_MASK |
+			    ID_AA64DFR0_EL1_DebugVer_MASK),
+	ID_SANITISED(ID_AA64DFR1_EL1),
+	ID_UNALLOCATED(5, 2),
+	ID_UNALLOCATED(5, 3),
+	ID_HIDDEN(ID_AA64AFR0_EL1),
+	ID_HIDDEN(ID_AA64AFR1_EL1),
+	ID_UNALLOCATED(5, 6),
+	ID_UNALLOCATED(5, 7),
+
+	/* CRm=6 */
+	ID_WRITABLE(ID_AA64ISAR0_EL1, ~ID_AA64ISAR0_EL1_RES0),
+	ID_WRITABLE(ID_AA64ISAR1_EL1,
+		    ~(ID_AA64ISAR1_EL1_GPI | ID_AA64ISAR1_EL1_GPA |
+		      ID_AA64ISAR1_EL1_API | ID_AA64ISAR1_EL1_APA)),
+	ID_WRITABLE(ID_AA64ISAR2_EL1,
+		    ~(ID_AA64ISAR2_EL1_RES0 | ID_AA64ISAR2_EL1_APA3 |
+		      ID_AA64ISAR2_EL1_GPA3)),
+	ID_WRITABLE(ID_AA64ISAR3_EL1,
+		    (ID_AA64ISAR3_EL1_FPRCVT | ID_AA64ISAR3_EL1_FAMINMAX)),
+	ID_UNALLOCATED(6, 4),
+	ID_UNALLOCATED(6, 5),
+	ID_UNALLOCATED(6, 6),
+	ID_UNALLOCATED(6, 7),
+
+	/* CRm=7 */
+	ID_FILTERED(ID_AA64MMFR0_EL1, id_aa64mmfr0_el1,
+		    ~(ID_AA64MMFR0_EL1_RES0 | ID_AA64MMFR0_EL1_ASIDBITS)),
+	ID_WRITABLE(ID_AA64MMFR1_EL1,
+		    ~(ID_AA64MMFR1_EL1_RES0 | ID_AA64MMFR1_EL1_HCX |
+		      ID_AA64MMFR1_EL1_TWED | ID_AA64MMFR1_EL1_XNX |
+		      ID_AA64MMFR1_EL1_VH | ID_AA64MMFR1_EL1_VMIDBits)),
+	ID_WRITABLE(ID_AA64MMFR2_EL1,
+		    ~(ID_AA64MMFR2_EL1_RES0 | ID_AA64MMFR2_EL1_EVT |
+		      ID_AA64MMFR2_EL1_FWB | ID_AA64MMFR2_EL1_IDS |
+		      ID_AA64MMFR2_EL1_NV | ID_AA64MMFR2_EL1_CCIDX)),
+	ID_WRITABLE(ID_AA64MMFR3_EL1,
+		    (ID_AA64MMFR3_EL1_TCRX | ID_AA64MMFR3_EL1_S1PIE |
+		     ID_AA64MMFR3_EL1_S1POE)),
+	ID_WRITABLE(ID_AA64MMFR4_EL1, ID_AA64MMFR4_EL1_NV_frac),
+	ID_UNALLOCATED(7, 5),
+	ID_UNALLOCATED(7, 6),
+	ID_UNALLOCATED(7, 7),
+
+	/* CRn = 1 */
+	{ SYS_DESC(SYS_SCTLR_EL1), NULL, reset_val, SCTLR_EL1, 0x00C50078 },
+	{ SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 },
+
+	/* CRn = 2 */
+	{ SYS_DESC(SYS_TTBR0_EL1), access_rw, reset_val, TTBR0_EL1, 0 },
+	{ SYS_DESC(SYS_TTBR1_EL1), access_rw, reset_val, TTBR1_EL1, 0 },
+	{ SYS_DESC(SYS_TCR_EL1), access_rw, reset_val, TCR_EL1, 0 },
+
+	{ SYS_DESC(SYS_PMMIR_EL1), trap_raz_wi },
+	/* CRn = 10 */
+	{ SYS_DESC(SYS_MAIR_EL1), NULL, reset_unknown, MAIR_EL1 },
+
+	/* CRn = 12 */
+	{ SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 },
+
+	/* CRn = 13 */
+	{ SYS_DESC(SYS_TPIDR_EL1), NULL, reset_unknown, TPIDR_EL1 },
+
+	/* Op1 = 1 */
+	/* CRn = 0 */
+	/* CRm = 0 */
+	{ SYS_DESC(SYS_CCSIDR_EL1), access_ccsidr },
+	{ SYS_DESC(SYS_CLIDR_EL1), access_clidr, reset_clidr, CLIDR_EL1,
+	  ~CLIDR_EL1_RES0, .set_user = set_clidr },
+	IMPLEMENTATION_ID(AIDR_EL1, GENMASK_ULL(63, 0)),
+	{ SYS_DESC(SYS_CSSELR_EL1), access_csselr, reset_unknown, CSSELR_EL1 },
+	ID_FILTERED(CTR_EL0, ctr_el0,
+		    CTR_EL0_DIC_MASK | CTR_EL0_IDC_MASK |
+			    CTR_EL0_DminLine_MASK | CTR_EL0_L1Ip_MASK |
+			    CTR_EL0_IminLine_MASK),
+
+	{ SYS_DESC(SYS_CNTFRQ_EL0), NULL, reset_val, CNTFRQ_EL0, 0x3B9ACA00 },
+	{ SYS_DESC(SYS_CNTVCT_EL0), .get_user = arch_timer_get_user,
+	  .set_user = arch_timer_set_user },
+	TIMER_REG(CNTP_CTL_EL0),
+	TIMER_REG(CNTP_CVAL_EL0),
+	TIMER_REG(CNTV_CTL_EL0),
+	TIMER_REG(CNTV_CVAL_EL0),
+
+};
+
+/*
+ * kvm_handle_sys_reg -- handles a system instruction or mrs/msr instruction
+ *			 trap on a guest execution
+ * @vcpu: The VCPU pointer
+ */
+int kvm_handle_sys_reg(struct kvm_vcpu *vcpu)
+{
+	const struct sys_reg_desc *desc = NULL;
+	struct sys_reg_params params;
+	unsigned long esr = kvm_vcpu_get_esr(vcpu);
+	int Rt = kvm_vcpu_sys_get_rt(vcpu);
+
+	trace_kvm_handle_sys_reg(esr);
+
+	params = esr_sys64_to_params(esr);
+	params.regval = vcpu_get_reg(vcpu, Rt);
+
+	/* System registers have Op0=={2,3}, as per DDI487 J.a C5.1.2 */
+	if (params.Op0 == 2 || params.Op0 == 3)
+		desc = find_reg(&params, sys_reg_descs, ARRAY_SIZE(sys_reg_descs));
+	else
+		WARN(true, "system instruction handling not supported");
+
+	if (!desc) {
+		if (!(params.Op0 == 3 && (params.CRn & 0b1011) == 0b1011))
+			print_sys_reg_msg(&params,
+					  "Unsupported guest access at: %lx\n",
+					  *vcpu_pc(vcpu));
+		kvm_inject_undefined(vcpu);
+		return 1;
+	} else {
+		perform_access(vcpu, &params, desc);
+	}
+
+	/* Read from system register? */
+	if (!params.is_write &&
+	    (params.Op0 == 2 || params.Op0 == 3))
+		vcpu_set_reg(vcpu, Rt, params.regval);
+
+	return 1;
+}
+
+void kvm_reset_sys_regs(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	unsigned long i;
+
+	for (i = 0; i < ARRAY_SIZE(sys_reg_descs); i++) {
+		const struct sys_reg_desc *r = &sys_reg_descs[i];
+
+		if (!r->reset)
+			continue;
+
+		if (is_vm_ftr_id_reg(reg_to_encoding(r))) {
+			reset_vm_ftr_id_reg(vcpu, r);
+			/*
+			 * Even VM only regs are stored in the save area of the vcpu
+			 * and need a meaningful initial value. Call reset because of this.
+			 */
+			r->reset(vcpu, r);
+		} else if (is_vcpu_ftr_id_reg(reg_to_encoding(r))) {
+			reset_vcpu_ftr_id_reg(vcpu, r);
+		} else {
+			r->reset(vcpu, r);
+		}
+	}
+	set_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags);
+}
+
+int kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu,
+			    const struct kvm_one_reg *reg)
+{
+	return __kvm_arm_sys_reg_get_reg(vcpu, reg, ARRAY_SIZE(sys_reg_descs),
+					 sys_reg_descs);
+}
+
+int __init kvm_sys_reg_table_init(void)
+{
+	init_imp_id_regs();
+
+	bool valid = check_sysreg_table(sys_reg_descs,
+					ARRAY_SIZE(sys_reg_descs), false);
+
+	if (!valid)
+		return -EINVAL;
+
+	return 0;
+}
+
+unsigned long kvm_arm_num_sys_reg_descs(struct kvm_vcpu *vcpu)
+{
+	return __kvm_arm_num_sys_reg_descs(vcpu, ARRAY_SIZE(sys_reg_descs),
+					   sys_reg_descs);
+}
+
+int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
+{
+	return __kvm_arm_copy_sys_reg_indices(vcpu, uindices, 
+					      ARRAY_SIZE(sys_reg_descs),
+					      sys_reg_descs);
+}
+
+int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu,
+			    const struct kvm_one_reg *reg)
+{
+	return __kvm_arm_sys_reg_set_reg(vcpu, reg, ARRAY_SIZE(sys_reg_descs),
+					 sys_reg_descs);
+}
+
+/*
+ * Perform last adjustments to the ID registers that are implied by the
+ * configuration outside of the ID regs themselves, as well as any
+ * initialisation that directly depend on these ID registers (such as
+ * RES0/RES1 behaviours). This is not the place to configure traps though.
+ *
+ * Because this can be called once per CPU, changes must be idempotent.
+ */
+int kvm_finalize_sys_regs(struct kvm_vcpu *vcpu)
+{
+	/* TODO GIC finalization */
+	return 0;
+}
diff --git a/arch/s390/kvm/arm64/trace.h b/arch/s390/kvm/arm64/trace.h
new file mode 100644
index 000000000000..015f380207ab
--- /dev/null
+++ b/arch/s390/kvm/arm64/trace.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#if !defined(_TRACE_ARM64_KVM_S390_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_ARM64_KVM_S390_H
+
+#include <linux/tracepoint.h>
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM kvm
+
+TRACE_EVENT(kvm_handle_sys_reg,
+	TP_PROTO(unsigned long esr),
+	TP_ARGS(esr),
+
+	TP_STRUCT__entry(
+		__field(unsigned long,	esr)
+	),
+
+	TP_fast_assign(
+		__entry->esr = esr;
+	),
+
+	TP_printk("ESR 0x%08lx", __entry->esr)
+);
+
+#endif /* _TRACE_ARM64_KVM_S390_H */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH ../../arch/s390/kvm/arm64
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE trace
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
diff --git a/virt/kvm/arm64/Makefile.kvm b/virt/kvm/arm64/Makefile.kvm
index c5e1db570a09..9a3df2446b8b 100644
--- a/virt/kvm/arm64/Makefile.kvm
+++ b/virt/kvm/arm64/Makefile.kvm
@@ -10,4 +10,5 @@ shared-arm64-obj := \
 	$(KVM_ARM64)/handle_exit.o \
 	$(KVM_ARM64)/mmio.o \
 	$(KVM_ARM64)/reset.o \
+	$(KVM_ARM64)/sys_regs.o \
 
-- 
2.53.0

[PATCH v1 25/26] KVM: s390: arm64: Implement exception injection

[Steffen Eiden]

Add exception injection support for s390 KVM arm64 architecture. This
implements the core exception entry mechanism and fault injection
functions needed for proper guest exception handling.

Co-developed-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host_arm64.h |   2 +
 arch/s390/kvm/arm64/Makefile           |   1 +
 arch/s390/kvm/arm64/arm.c              |  15 +---
 arch/s390/kvm/arm64/exception.c        | 105 +++++++++++++++++++++++++
 arch/s390/kvm/arm64/inject_fault.c     |  72 ++++++++++++++++-
 5 files changed, 182 insertions(+), 13 deletions(-)
 create mode 100644 arch/s390/kvm/arm64/exception.c

diff --git a/arch/s390/include/asm/kvm_host_arm64.h b/arch/s390/include/asm/kvm_host_arm64.h
index cc457d69621c..34d131b36e70 100644
--- a/arch/s390/include/asm/kvm_host_arm64.h
+++ b/arch/s390/include/asm/kvm_host_arm64.h
@@ -373,4 +373,6 @@ static inline u64 kvm_sanitised_host_ftr_reg(u32 id)
 	return kvm_arm_host_sys_reg_by_id(id);
 }
 
+void kvm_adjust_pc(struct kvm_vcpu *vcpu);
+
 #endif /* ASM_KVM_HOST_ARM64_H */
diff --git a/arch/s390/kvm/arm64/Makefile b/arch/s390/kvm/arm64/Makefile
index bec24b60a071..9d6b7d5bf588 100644
--- a/arch/s390/kvm/arm64/Makefile
+++ b/arch/s390/kvm/arm64/Makefile
@@ -9,6 +9,7 @@ ccflags-y += -I $(src) -I$(srctree)/arch/s390/kvm/gmap -DKVM_S390_ARM64
 
 kvm-arm64-obj := \
 	arm.o \
+	exception.o \
 	feature.o \
 	guest.o \
 	handle_exit.o \
diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index 88b66552cff0..c62e50002628 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -350,19 +350,11 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
 	return 0;
 }
 
-static void adjust_pc(struct kvm_vcpu *vcpu)
-{
-	if (vcpu_get_flag(vcpu, INCREMENT_PC)) {
-		kvm_skip_instr(vcpu);
-		vcpu_clear_flag(vcpu, INCREMENT_PC);
-	}
-}
-
 static void arm_vcpu_run(struct kvm_vcpu *vcpu)
 {
 	struct kvm_sae_block *sae_block = &vcpu->arch.sae_block;
 
-	adjust_pc(vcpu);
+	kvm_adjust_pc(vcpu);
 
 	local_irq_disable();
 	guest_enter_irqoff();
@@ -457,8 +449,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 
 	kvm_sigset_deactivate(vcpu);
 out:
-	if (unlikely(vcpu_get_flag(vcpu, INCREMENT_PC)))
-		adjust_pc(vcpu);
+	if (unlikely(vcpu_get_flag(vcpu, PENDING_EXCEPTION) ||
+		     vcpu_get_flag(vcpu, INCREMENT_PC)))
+		kvm_adjust_pc(vcpu);
 
 	save_vx_regs(vcpu->arch.ctxt.vregs);
 	kernel_fpu_end(&fpu_save, KERNEL_FPC | KERNEL_VXR);
diff --git a/arch/s390/kvm/arm64/exception.c b/arch/s390/kvm/arm64/exception.c
new file mode 100644
index 000000000000..871e4e2c710a
--- /dev/null
+++ b/arch/s390/kvm/arm64/exception.c
@@ -0,0 +1,105 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/kvm_host.h>
+#include <asm/kvm_emulate.h>
+#include <asm/sysreg-defs.h>
+
+#include <asm/pstate.h>
+
+/*
+ * This performs the exception entry at a given EL (@target_mode), stashing PC
+ * and PSTATE into ELR and SPSR respectively, and compute the new PC/PSTATE.
+ * The EL passed to this function *must* be a non-secure, privileged mode with
+ * bit 0 being set (PSTATE.SP == 1).
+ *
+ * When an exception is taken, most PSTATE fields are left unchanged in the
+ * handler. However, some are explicitly overridden (e.g. M[4:0]). Luckily all
+ * of the inherited bits have the same position in the AArch64/AArch32 SPSR_ELx
+ * layouts, so we don't need to shuffle these for exceptions from AArch32 EL0.
+ *
+ * For the SPSR_ELx layout for AArch64, see ARM DDI 0487E.a page C5-429.
+ * For the SPSR_ELx layout for AArch32, see ARM DDI 0487E.a page C5-426.
+ *
+ * Here we manipulate the fields in order of the AArch64 SPSR_ELx layout, from
+ * MSB to LSB.
+ */
+static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
+			      enum exception_type type)
+{
+	unsigned long sctlr, vbar, old_pstate, new, mode;
+	u64 exc_offset;
+
+	old_pstate = *vcpu_cpsr(vcpu);
+	mode = old_pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	if      (mode == target_mode)
+		exc_offset = CURRENT_EL_SP_ELx_VECTOR;
+	else if ((mode | PSR_MODE_THREAD_BIT) == target_mode)
+		exc_offset = CURRENT_EL_SP_EL0_VECTOR;
+	else if (!(mode & PSR_MODE32_BIT))
+		exc_offset = LOWER_EL_AArch64_VECTOR;
+	else
+		exc_offset = LOWER_EL_AArch32_VECTOR;
+
+	switch (target_mode) {
+	case PSR_MODE_EL1h:
+		vbar = vcpu_read_sys_reg(vcpu, VBAR_EL1);
+		sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+		vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
+		vcpu_write_sys_reg(vcpu, old_pstate, SPSR_EL1);
+		break;
+	default:
+		panic("Unknown PSR Mode: 0x%lx.", target_mode);
+	}
+
+	*vcpu_pc(vcpu) = vbar + exc_offset + type;
+
+	new = 0;
+
+	new |= (old_pstate & PSR_N_BIT);
+	new |= (old_pstate & PSR_Z_BIT);
+	new |= (old_pstate & PSR_C_BIT);
+	new |= (old_pstate & PSR_V_BIT);
+	new |= (old_pstate & PSR_DIT_BIT);
+	new |= (old_pstate & PSR_PAN_BIT);
+
+	if (!(sctlr & SCTLR_EL1_SPAN))
+		new |= PSR_PAN_BIT;
+
+	if (sctlr & SCTLR_ELx_DSSBS)
+		new |= PSR_SSBS_BIT;
+
+	new |= PSR_D_BIT;
+	new |= PSR_A_BIT;
+	new |= PSR_I_BIT;
+	new |= PSR_F_BIT;
+
+	new |= target_mode;
+
+	*vcpu_cpsr(vcpu) = new;
+}
+
+static void kvm_inject_exception(struct kvm_vcpu *vcpu)
+{
+	switch (vcpu_get_flag(vcpu, EXCEPT_MASK)) {
+	case unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC):
+		enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync);
+		break;
+	}
+}
+
+/*
+ * Adjust the guest PC (and potentially exception state) depending on
+ * flags provided by the emulation code.
+ */
+void kvm_adjust_pc(struct kvm_vcpu *vcpu)
+{
+	if (vcpu_get_flag(vcpu, PENDING_EXCEPTION)) {
+		kvm_inject_exception(vcpu);
+		vcpu_clear_flag(vcpu, PENDING_EXCEPTION);
+		vcpu_clear_flag(vcpu, EXCEPT_MASK);
+	} else if (vcpu_get_flag(vcpu, INCREMENT_PC)) {
+		kvm_skip_instr(vcpu);
+		vcpu_clear_flag(vcpu, INCREMENT_PC);
+	}
+}
diff --git a/arch/s390/kvm/arm64/inject_fault.c b/arch/s390/kvm/arm64/inject_fault.c
index 650c041efde2..0216ffbd62ab 100644
--- a/arch/s390/kvm/arm64/inject_fault.c
+++ b/arch/s390/kvm/arm64/inject_fault.c
@@ -2,6 +2,42 @@
 
 #include <asm/kvm_emulate.h>
 
+#define exception_esr_elx(__vcpu) ESR_EL1
+#define exception_far_elx(__vcpu) FAR_EL1
+
+static void pend_sync_exception(struct kvm_vcpu *vcpu)
+{
+	kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
+}
+
+static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)
+{
+	unsigned long cpsr = *vcpu_cpsr(vcpu);
+	bool is_aarch32 = vcpu_mode_is_32bit(vcpu);
+	u64 esr = 0;
+
+	if (kvm_vcpu_abt_iss1tw(vcpu))
+		esr |= ESR_ELx_FSC_SEA_TTW(kvm_vcpu_abt_gltl(vcpu));
+	else
+		esr |= ESR_ELx_FSC_EXTABT;
+
+	pend_sync_exception(vcpu);
+
+	if (kvm_vcpu_trap_il_is32bit(vcpu))
+		esr |= ESR_ELx_IL;
+
+	if (is_aarch32 || (cpsr & PSR_MODE_MASK) == PSR_MODE_EL0t)
+		esr |= (ESR_ELx_EC_IABT_LOW << ESR_ELx_EC_SHIFT);
+	else
+		esr |= (ESR_ELx_EC_IABT_CUR << ESR_ELx_EC_SHIFT);
+
+	if (!is_iabt)
+		esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT;
+
+	vcpu_write_sys_reg(vcpu, addr, ESR_EL1);
+	vcpu_write_sys_reg(vcpu, esr, FAR_EL1);
+}
+
 /**
  * kvm_inject_undefined - inject an undefined instruction into the guest
  * @vcpu: The vCPU in which to inject the exception
@@ -11,11 +47,43 @@
  */
 void kvm_inject_undefined(struct kvm_vcpu *vcpu)
 {
-	/* Stub until s390 supports arm64 sysregs TODO sysregs*/
+	u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);
+
+	pend_sync_exception(vcpu);
+
+	/*
+	 * Build an unknown exception, depending on the instruction
+	 * set.
+	 */
+	if (kvm_vcpu_trap_il_is32bit(vcpu))
+		esr |= ESR_ELx_IL;
+
+	vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu));
+}
+
+static void __kvm_inject_sea(struct kvm_vcpu *vcpu, bool iabt, u64 addr)
+{
+	inject_abt64(vcpu, iabt, addr);
 }
 
 int kvm_inject_sea(struct kvm_vcpu *vcpu, bool iabt, u64 addr)
 {
-	/* Stub until s390 supports arm64 sysregs TODO sysregs*/
+	lockdep_assert_held(&vcpu->mutex);
+
+	__kvm_inject_sea(vcpu, iabt, addr);
 	return 1;
 }
+
+void kvm_inject_size_fault(struct kvm_vcpu *vcpu)
+{
+	unsigned long addr, esr;
+
+	addr  = kvm_vcpu_get_fault_ipa(vcpu);
+	addr |= FAR_TO_FIPA_OFFSET(kvm_vcpu_get_hfar(vcpu));
+
+	__kvm_inject_sea(vcpu, kvm_vcpu_trap_is_iabt(vcpu), addr);
+
+	esr = vcpu_read_sys_reg(vcpu, exception_esr_elx(vcpu));
+	esr &= ~GENMASK_ULL(5, 0);
+	vcpu_write_sys_reg(vcpu, esr, exception_esr_elx(vcpu));
+}
-- 
2.53.0

[PATCH v1 26/26] KVM: s390: arm64: Finalize page fault handling

[Steffen Eiden]

Complete the page fault handling implementation by replacing temporary
error returns with proper ARM64 exception injection.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/kvm/arm64/mmu.c       | 62 ++++++++++++++++++++++-----------
 include/kvm/arm64/kvm_feature.h |  3 ++
 2 files changed, 44 insertions(+), 21 deletions(-)

diff --git a/arch/s390/kvm/arm64/mmu.c b/arch/s390/kvm/arm64/mmu.c
index 8759cbafbaff..4224503276fb 100644
--- a/arch/s390/kvm/arm64/mmu.c
+++ b/arch/s390/kvm/arm64/mmu.c
@@ -31,12 +31,14 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, gpa_t fault_ipa,
 	ret = kvm_s390_faultin_gfn(vcpu, NULL, &f);
 	if (ret <= 0)
 		return ret;
-	if (ret == PGM_ADDRESSING)
+	if (ret == PGM_ADDRESSING) {
 		/*
-		 * Without the relevant sysregs we cannot do anything for now.
-		 * Go back to userspace with an error. TODO sysreg handling
+		 * There is no page with the requested address. Inject size fault
+		 * which is the closest arm match to PGM-addressing
 		 */
-		return -ENOEXEC;
+		kvm_inject_size_fault(vcpu);
+		return 1;
+	}
 	KVM_BUG_ON(ret, vcpu->kvm);
 	return -EINVAL;
 }
@@ -69,6 +71,27 @@ static int kvm_handle_pic(struct kvm_vcpu *vcpu, bool *translation)
 	return 0;
 }
 
+static u32 kvm_get_pa_bits(struct kvm *kvm)
+{
+	u32 id_parange = get_idreg_field_enum(kvm, ID_AA64MMFR0_EL1, PARANGE);
+
+	switch (id_parange) {
+	case ID_AA64MMFR0_EL1_PARANGE_32: return 32;
+	case ID_AA64MMFR0_EL1_PARANGE_36: return 36;
+	case ID_AA64MMFR0_EL1_PARANGE_40: return 40;
+	case ID_AA64MMFR0_EL1_PARANGE_42: return 42;
+	case ID_AA64MMFR0_EL1_PARANGE_44: return 44;
+	case ID_AA64MMFR0_EL1_PARANGE_48: return 48;
+	case ID_AA64MMFR0_EL1_PARANGE_52: return 52;
+	case ID_AA64MMFR0_EL1_PARANGE_56: return 56;
+	}
+	/*
+	 * Future values must be higher than we know already.
+	 * See ARM DDI 0487C.a. Return a safe limit.
+	 */
+	return ID_AA64MMFR0_EL1_PARANGE_48;
+}
+
 int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 {
 	struct kvm_memory_slot *memslot;
@@ -98,16 +121,17 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 	}
 
 	if (translation) {
-		/*
-		 * For both cases:
-		 * Without the relevant sysregs we cannot do anything for now.
-		 * Go back to userspace with an error. TODO sysreg handling
-		 */
-		if (fault_ipa >= BIT_ULL(get_kvm_ipa_limit()))
-			return -ENOEXEC;
+		/* Beyond sanitised PA range (which is the IPA limit) */
+		if (fault_ipa >= BIT_ULL(kvm_get_pa_bits(vcpu->kvm))) {
+			kvm_inject_size_fault(vcpu);
+			return 1;
+		}
 
-		if (fault_ipa >= kvm_phys_size(vcpu->kvm))
-			return -ENOEXEC;
+		/* Falls between the IPA range and the PA range? */
+		if (fault_ipa >= kvm_phys_size(vcpu->kvm)) {
+			fault_ipa |= kvm_vcpu_get_hfar(vcpu) & GENMASK(11, 0);
+			return kvm_inject_sea(vcpu, is_iabt, fault_ipa);
+		}
 	}
 
 	idx = srcu_read_lock(&vcpu->kvm->srcu);
@@ -123,18 +147,14 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 		 * The guest has put either its instructions or its page-tables
 		 * somewhere it shouldn't have. Userspace won't be able to do
 		 * anything about this (there's no syndrome for a start).
-		 *
-		 * Without the relevant sysregs we cannot do anything for now.
-		 * Go back to userspace with an error. TODO sysreg handling
 		 */
-		if (is_iabt)
+		if (is_iabt) {
+			ret = kvm_inject_sea_iabt(vcpu, kvm_vcpu_get_hfar(vcpu));
 			goto out_unlock;
+		}
 
 		if (kvm_vcpu_abt_iss1tw(vcpu)) {
-			/*
-			 * Without the relevant sysregs we cannot do anything for now.
-			 * Go back to userspace with an error. TODO sysreg handling
-			 */
+			ret = kvm_inject_sea_dabt(vcpu, kvm_vcpu_get_hfar(vcpu));
 			goto out_unlock;
 		}
 
diff --git a/include/kvm/arm64/kvm_feature.h b/include/kvm/arm64/kvm_feature.h
index 945abbbf1aa8..193c9ca7045f 100644
--- a/include/kvm/arm64/kvm_feature.h
+++ b/include/kvm/arm64/kvm_feature.h
@@ -47,6 +47,9 @@
 
 #define kvm_has_feat_enum(kvm, ...) __kvm_has_feat_enum(kvm, __VA_ARGS__)
 
+#define get_idreg_field_enum(kvm, id, fld)				\
+	SYS_FIELD_GET(id, fld, kvm_read_vm_id_reg((kvm), SYS_##id))
+
 /* Check for a given level of PAuth support */
 #define kvm_has_pauth(k, l)						\
 	({								\
-- 
2.53.0

Re: [PATCH v1 00/26] KVM: arm64 on s390 System Register Handling

[Claudio Imbrenda]

On Fri, 29 May 2026 17:55:33 +0200
Steffen Eiden <seiden@linux.ibm.com> wrote:

> Add system register handling for KVM/arm64 on s390. Restructure and share
> KVM/arm64 code, introduce ARM guest management functions for s390 hosts,
> and implement host sysreg & exception handling.
> 
> Changes in detail:
> 
> arm64:
> 
> Refactor arm64 feature detection and ID register handling to make it
> generic and reusable across architectures.
> 
> Restructure ID register storage and core register handling. Refactor core
> registers to use functions instead of direct memory access.
> 
> Move arm64-specific definitions (CPU types, cache, KVM features, ID
> registers, system registers) to shared locations for reuse by other
> architectures.
> 
> s390:
> 
> Add s390 instruction support for ARM guest management: easr/sasr for
> system register access, QAAF for feature queries, and ptff extensions
> for guest time handling.
> 
> Implement complete sysreg handling for s390 including feature
> sanitisation, register enumeration and access, exception injection,
> and finalized page fault handling.
> 
> The series builds upon the foundation established in the first series and
> requires the first series v3[1] as base.

the series is now conveniently available on the  sae  branch on
https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git

> 
> 	Steffen
> 
> [1] https://lore.kernel.org/lkml/20260529155050.2902245-1-seiden@linux.ibm.com/
> 
> Andreas Grapentin (1):
>   KVM: arm64: Fix set_oslsr_el1 to write to OSLAR_EL1
> 
> Steffen Eiden (25):
>   KVM: arm64: Extract some feature related changes to kvm_feature.h
>   KVM: arm64: Remove __expand_field_sign_(un)signed
>   KVM: arm64: Generalize get_idreg_field_*()
>   KVM: arm64: Generalize kvm_cmp_feat_*()
>   KVM: arm64: Generalize kvm_has_feat_*
>   KVM: arm64: Remove get_idreg_field_*() and kvm_cmp_feat_*()
>   KVM: arm64: Remove kvm_has_feat_range
>   KVM: arm64: Split up feature sysreg sanitisation
>   KVM: arm64: Refactor idreg caching into dedicated structure
>   KVM: arm64: Move definitions from sys_regs.c to sys_regs.h
>   KVM: arm64: Add PVM_ prefix to avoid name collisions
>   s390: Introduce read/write ARM sysreg instructions
>   s390: Introduce Query Available Arm features
>   s390: Add functions to query arm guest time
>   KVM: s390: arm64: Add sysreg related functions and definitions
>   arm64: Extract cputype definitions.
>   arm64: Extract cache definitions
>   KVM: arm64: Share KVM feature detection macros
>   KVM: arm64: Share ID reg handling
>   KVM: arm64: Share sys-reg handling
>   KVM: arm64: Refactor core reg handling
>   KVM: s390: arm64: Implement feature sanitisation
>   KVM: s390: arm64: Implement sysreg handling
>   KVM: s390: arm64: Implement exception injection
>   KVM: s390: arm64: Finalize page fault handling
> 
>  arch/arm64/include/asm/cache.h                |   19 +-
>  arch/arm64/include/asm/cputype.h              |  246 +---
>  arch/arm64/include/asm/kvm_emulate.h          |    1 +
>  arch/arm64/include/asm/kvm_feature.h          |   27 +
>  arch/arm64/include/asm/kvm_host.h             |  137 +-
>  arch/arm64/include/asm/kvm_nested.h           |    1 +
>  arch/arm64/kvm/arm.c                          |    2 +-
>  arch/arm64/kvm/at.c                           |    1 +
>  arch/arm64/kvm/config.c                       |    3 +-
>  arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h    |    1 +
>  arch/arm64/kvm/hyp/nvhe/hyp-main.c            |    1 +
>  arch/arm64/kvm/hyp/nvhe/pkvm.c                |    7 +-
>  arch/arm64/kvm/hyp/nvhe/sys_regs.c            |   55 +-
>  arch/arm64/kvm/nested.c                       |    2 +-
>  arch/arm64/kvm/sys_regs.c                     | 1307 +----------------
>  arch/arm64/kvm/sys_regs.h                     |  276 ----
>  arch/arm64/kvm/trace_handle_exit.h            |   36 +-
>  arch/arm64/kvm/vgic-sys-reg-v3.c              |    2 +-
>  arch/arm64/kvm/vgic/vgic-init.c               |    1 +
>  arch/arm64/kvm/vgic/vgic.h                    |    1 +
>  arch/s390/include/asm/kvm_emulate.h           |   34 +
>  arch/s390/include/asm/kvm_feature.h           |  111 ++
>  arch/s390/include/asm/kvm_host_arm64.h        |  168 ++-
>  arch/s390/include/asm/kvm_host_arm64_types.h  |   97 ++
>  arch/s390/include/asm/kvm_nested.h            |    5 +
>  arch/s390/include/asm/sae-asm.h               |   48 +
>  arch/s390/include/asm/sae.h                   |   86 ++
>  arch/s390/include/asm/timex.h                 |   49 +
>  arch/s390/kernel/dis.c                        |    1 +
>  arch/s390/kernel/time.c                       |    1 +
>  arch/s390/kvm/arm64/Makefile                  |    3 +
>  arch/s390/kvm/arm64/arm.c                     |   44 +-
>  arch/s390/kvm/arm64/exception.c               |  105 ++
>  arch/s390/kvm/arm64/feature.c                 |  170 +++
>  arch/s390/kvm/arm64/guest.c                   |   20 +-
>  arch/s390/kvm/arm64/handle_exit.c             |    1 +
>  arch/s390/kvm/arm64/inject_fault.c            |   72 +-
>  arch/s390/kvm/arm64/mmu.c                     |   62 +-
>  arch/s390/kvm/arm64/reset.c                   |    5 +
>  arch/s390/kvm/arm64/sys_regs.c                |  769 ++++++++++
>  arch/s390/kvm/arm64/trace.h                   |   33 +
>  arch/s390/tools/opcodes.txt                   |    3 +
>  include/arch/arm64/asm/cache-defs.h           |   22 +
>  .../arch/arm64/asm/cputype-defs.h             |   92 +-
>  include/arch/arm64/asm/sysreg-defs.h          |    9 +
>  include/kvm/arm64/kvm_feature.h               |   68 +
>  include/kvm/arm64/kvm_host.h                  |   52 +
>  include/kvm/arm64/sys_regs.h                  |  548 +++++++
>  virt/kvm/arm64/Makefile.kvm                   |    1 +
>  virt/kvm/arm64/guest.c                        |  100 +-
>  virt/kvm/arm64/mmio.c                         |    1 +
>  virt/kvm/arm64/sys_regs.c                     | 1039 +++++++++++++
>  virt/kvm/arm64/trace.h                        |   34 +
>  53 files changed, 3838 insertions(+), 2141 deletions(-)
>  create mode 100644 arch/arm64/include/asm/kvm_feature.h
>  delete mode 100644 arch/arm64/kvm/sys_regs.h
>  create mode 100644 arch/s390/include/asm/kvm_feature.h
>  create mode 100644 arch/s390/include/asm/sae-asm.h
>  create mode 100644 arch/s390/kvm/arm64/exception.c
>  create mode 100644 arch/s390/kvm/arm64/feature.c
>  create mode 100644 arch/s390/kvm/arm64/sys_regs.c
>  create mode 100644 arch/s390/kvm/arm64/trace.h
>  create mode 100644 include/arch/arm64/asm/cache-defs.h
>  copy arch/arm64/include/asm/cputype.h => include/arch/arm64/asm/cputype-defs.h (85%)
>  create mode 100644 include/kvm/arm64/kvm_feature.h
>  create mode 100644 include/kvm/arm64/sys_regs.h
>  create mode 100644 virt/kvm/arm64/sys_regs.c
> 
> 
> base-commit: 4095afb932d1a98a6fcb3f4f490964949d0de338

Re: [PATCH v1 10/26] KVM: arm64: Fix set_oslsr_el1 to write to OSLAR_EL1

[Oliver Upton]

Hi,

On Fri, May 29, 2026 at 05:55:43PM +0200, Steffen Eiden wrote:
> From: Andreas Grapentin <gra@linux.ibm.com>
> 
> The set_oslsr_el1() function was incorrectly writing directly to the
> OSLSR_EL1 register, which is architecturally a read-only status register
> that reflects the state of the OS Lock.
> 
> Fix this by extracting the OSLK bit from the user-provided value and
> writing it to OSLAR_EL1 (OS Lock Access Register) instead, which is the
> proper control register for managing the OS Lock state. OSLSR_EL1 will
> then reflect this state when read.
> 
> This ensures the implementation follows the ARM architecture
> specification where OSLAR_EL1 controls the lock and OSLSR_EL1 provides
> status information.
> 
> Signed-off-by: Andreas Grapentin <gra@linux.ibm.com>
> Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>

The current behavior of KVM is correct. KVM treats OSLSR_EL1 as the
stateful representation of the OS lock and is RO from the guest POV.

We keep the UAPI straightforward by making this register RW from
userspace, such that the VMM can directly write back the value returned
from KVM_GET_ONE_REG.

Do you have another reason for using OSLAR_EL1 as the canonical
representation?

Thanks,
Oliver

Re: [PATCH v1 12/26] KVM: arm64: Add PVM_ prefix to avoid name collisions

[Oliver Upton]

On Fri, May 29, 2026 at 05:55:45PM +0200, Steffen Eiden wrote:
> Rename ID_UNALLOCATED to PVM_ID_UNALLOCATED and read_id_reg to
> pvm_read_id_reg to prevent future name collisions with other subsystems.
> While at it, fix whitespace issues in the macro invocations
> 
> Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>

No issues with the rename but do you even need the nVHE object at all
for s390?

Thanks,
Oliver

> ---
>  arch/arm64/kvm/hyp/nvhe/sys_regs.c | 48 +++++++++++++++---------------
>  1 file changed, 24 insertions(+), 24 deletions(-)
> 
> diff --git a/arch/arm64/kvm/hyp/nvhe/sys_regs.c b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
> index e8d773d38905..08b14053568b 100644
> --- a/arch/arm64/kvm/hyp/nvhe/sys_regs.c
> +++ b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
> @@ -282,8 +282,8 @@ static void inject_undef64(struct kvm_vcpu *vcpu)
>  	inject_sync64(vcpu, (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT));
>  }
>  
> -static u64 read_id_reg(const struct kvm_vcpu *vcpu,
> -		       struct sys_reg_desc const *r)
> +static u64 pvm_read_id_reg(const struct kvm_vcpu *vcpu,
> +			   struct sys_reg_desc const *r)
>  {
>  	struct kvm *kvm = vcpu->kvm;
>  	u32 reg = reg_to_encoding(r);
> @@ -341,7 +341,7 @@ static bool pvm_access_id_aarch64(struct kvm_vcpu *vcpu,
>  		return false;
>  	}
>  
> -	p->regval = read_id_reg(vcpu, r);
> +	p->regval = pvm_read_id_reg(vcpu, r);
>  	return true;
>  }
>  
> @@ -379,7 +379,7 @@ static bool pvm_idst_access(struct kvm_vcpu *vcpu,
>   * register with encoding Op0=3, Op1=0, CRn=0, CRm=crm, Op2=op2
>   * (1 <= crm < 8, 0 <= Op2 < 8).
>   */
> -#define ID_UNALLOCATED(crm, op2) {			\
> +#define PVM_ID_UNALLOCATED(crm, op2) {			\
>  	Op0(3), Op1(0), CRn(0), CRm(crm), Op2(op2),	\
>  	.access = pvm_access_id_aarch64,		\
>  }
> @@ -438,46 +438,46 @@ static const struct sys_reg_desc pvm_sys_reg_descs[] = {
>  	AARCH32(SYS_MVFR0_EL1),
>  	AARCH32(SYS_MVFR1_EL1),
>  	AARCH32(SYS_MVFR2_EL1),
> -	ID_UNALLOCATED(3,3),
> +	PVM_ID_UNALLOCATED(3, 3),
>  	AARCH32(SYS_ID_PFR2_EL1),
>  	AARCH32(SYS_ID_DFR1_EL1),
>  	AARCH32(SYS_ID_MMFR5_EL1),
> -	ID_UNALLOCATED(3,7),
> +	PVM_ID_UNALLOCATED(3, 7),
>  
>  	/* AArch64 ID registers */
>  	/* CRm=4 */
>  	AARCH64(SYS_ID_AA64PFR0_EL1),
>  	AARCH64(SYS_ID_AA64PFR1_EL1),
>  	AARCH64(SYS_ID_AA64PFR2_EL1),
> -	ID_UNALLOCATED(4,3),
> +	PVM_ID_UNALLOCATED(4, 3),
>  	AARCH64(SYS_ID_AA64ZFR0_EL1),
> -	ID_UNALLOCATED(4,5),
> -	ID_UNALLOCATED(4,6),
> -	ID_UNALLOCATED(4,7),
> +	PVM_ID_UNALLOCATED(4, 5),
> +	PVM_ID_UNALLOCATED(4, 6),
> +	PVM_ID_UNALLOCATED(4, 7),
>  	AARCH64(SYS_ID_AA64DFR0_EL1),
>  	AARCH64(SYS_ID_AA64DFR1_EL1),
> -	ID_UNALLOCATED(5,2),
> -	ID_UNALLOCATED(5,3),
> +	PVM_ID_UNALLOCATED(5, 2),
> +	PVM_ID_UNALLOCATED(5, 3),
>  	AARCH64(SYS_ID_AA64AFR0_EL1),
>  	AARCH64(SYS_ID_AA64AFR1_EL1),
> -	ID_UNALLOCATED(5,6),
> -	ID_UNALLOCATED(5,7),
> +	PVM_ID_UNALLOCATED(5, 6),
> +	PVM_ID_UNALLOCATED(5, 7),
>  	AARCH64(SYS_ID_AA64ISAR0_EL1),
>  	AARCH64(SYS_ID_AA64ISAR1_EL1),
>  	AARCH64(SYS_ID_AA64ISAR2_EL1),
> -	ID_UNALLOCATED(6,3),
> -	ID_UNALLOCATED(6,4),
> -	ID_UNALLOCATED(6,5),
> -	ID_UNALLOCATED(6,6),
> -	ID_UNALLOCATED(6,7),
> +	PVM_ID_UNALLOCATED(6, 3),
> +	PVM_ID_UNALLOCATED(6, 4),
> +	PVM_ID_UNALLOCATED(6, 5),
> +	PVM_ID_UNALLOCATED(6, 6),
> +	PVM_ID_UNALLOCATED(6, 7),
>  	AARCH64(SYS_ID_AA64MMFR0_EL1),
>  	AARCH64(SYS_ID_AA64MMFR1_EL1),
>  	AARCH64(SYS_ID_AA64MMFR2_EL1),
> -	ID_UNALLOCATED(7,3),
> -	ID_UNALLOCATED(7,4),
> -	ID_UNALLOCATED(7,5),
> -	ID_UNALLOCATED(7,6),
> -	ID_UNALLOCATED(7,7),
> +	PVM_ID_UNALLOCATED(7, 3),
> +	PVM_ID_UNALLOCATED(7, 4),
> +	PVM_ID_UNALLOCATED(7, 5),
> +	PVM_ID_UNALLOCATED(7, 6),
> +	PVM_ID_UNALLOCATED(7, 7),
>  
>  	/* Scalable Vector Registers are restricted. */
>  
> -- 
> 2.53.0
> 

Re: [PATCH v1 15/26] s390: Add functions to query arm guest time

[Oliver Upton]

On Fri, May 29, 2026 at 05:55:48PM +0200, Steffen Eiden wrote:
> Add functions to convert between ARM guest time (LSB0) and s390 host
> time (MSB0) using new ptff function codes.
> 
> Co-developed-by: Nico Boehr <nrb@linux.ibm.com>
> Signed-off-by: Nico Boehr <nrb@linux.ibm.com>
> Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
> ---
>  arch/s390/include/asm/timex.h | 49 +++++++++++++++++++++++++++++++++++
>  arch/s390/kernel/time.c       |  1 +
>  arch/s390/kvm/arm64/arm.c     |  9 ++++++-
>  3 files changed, 58 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/s390/include/asm/timex.h b/arch/s390/include/asm/timex.h
> index 49447b40f038..9ec22a28bbda 100644
> --- a/arch/s390/include/asm/timex.h
> +++ b/arch/s390/include/asm/timex.h
> @@ -99,6 +99,8 @@ extern unsigned char ptff_function_mask[16];
>  #define PTFF_QSI	0x02	/* query steering information */
>  #define PTFF_QPT	0x03	/* query physical clock */
>  #define PTFF_QUI	0x04	/* query UTC information */
> +#define PTFF_QAGTO	0x10	/* query arm guest time offset */
> +#define PTFF_QAGPT	0x11	/* query arm guest physical time offset */

Are these analogous to CNTVOFF_EL2 and CNTPOFF_EL2?

Thanks,
Oliver 

>  #define PTFF_ATO	0x40	/* adjust tod offset */
>  #define PTFF_STO	0x41	/* set tod offset */
>  #define PTFF_SFS	0x42	/* set fine steering rate */
> @@ -136,6 +138,17 @@ struct ptff_qui {
>  	unsigned int pad_0x5c[41];
>  } __packed;
>  
> +/*
> + * Query Arm Guest Time
> + * used for:
> + *	- Query Arm Guest Time Offset
> + *	- Query Arm Guest Physical Time
> + */
> +struct ptff_qagt {
> +	u64 in;
> +	u64 out;
> +};
> +
>  /*
>   * ptff - Perform timing facility function
>   * @ptff_block: Pointer to ptff parameter block
> @@ -286,4 +299,40 @@ static inline int tod_after_eq(unsigned long a, unsigned long b)
>  	return a >= b;
>  }
>  
> +/*
> + * ptff_qagto() -  Query Arm Guest Time Offset
> + *
> + * @physical_time: Arm guest physical time in MSb 0
> + *
> + * Converts Arm guest physical time in MSb 0 bit ordering
> + * into the Arm guest offset in LSb 0 bit ordering.
> + *
> + * Return: Arm guest time offset in LSb 0
> + */
> +static inline u64 ptff_qagto(u64 physical_time)
> +{
> +	struct ptff_qagt qagto = { .in = physical_time };
> +
> +	ptff(&qagto, sizeof(qagto), PTFF_QAGTO);
> +	return qagto.out;
> +}
> +
> +/*
> + * ptff_qagpt() - Query Arm Guest Physical Time
> + *
> + * @guest_time_offset: Arm guest time offset in MSb 0
> + *
> + * Converts Arm guest offset in MSb 0 bit ordering
> + * into the Arm guest physical time in LSb 0 bit ordering.
> + *
> + * Return: Arm guest physical time in LSb 0
> + * */
> +static inline u64 ptff_qagpt(u64 guest_time_offset)
> +{
> +	struct ptff_qagt qagpt = { .in = guest_time_offset };
> +
> +	ptff(&qagpt, sizeof(qagpt), PTFF_QAGPT);
> +	return qagpt.out;
> +}
> +
>  #endif
> diff --git a/arch/s390/kernel/time.c b/arch/s390/kernel/time.c
> index bd0df61d1907..2b989bebd220 100644
> --- a/arch/s390/kernel/time.c
> +++ b/arch/s390/kernel/time.c
> @@ -65,6 +65,7 @@ ATOMIC_NOTIFIER_HEAD(s390_epoch_delta_notifier);
>  EXPORT_SYMBOL(s390_epoch_delta_notifier);
>  
>  unsigned char ptff_function_mask[16];
> +EXPORT_SYMBOL(ptff_function_mask);
>  
>  static unsigned long lpar_offset;
>  static unsigned long initial_leap_seconds;
> diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
> index bf0866659421..636bbeda98a8 100644
> --- a/arch/s390/kvm/arm64/arm.c
> +++ b/arch/s390/kvm/arm64/arm.c
> @@ -692,8 +692,15 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
>  
>  static int __init kvm_s390_arm64_init(void)
>  {
> -	if (!sclp.has_aef)
> +	if (!sclp.has_aef) {
> +		pr_info("SAE is not available\n");
>  		return -ENXIO;
> +	}
> +
> +	if (!(ptff_query(PTFF_QAGTO) && ptff_query(PTFF_QAGPT))) {
> +		pr_info("PTFF for arm on s390 is not available\n");
> +		return -ENXIO;
> +	}
>  
>  	return kvm_init_with_dev(sizeof(struct kvm_vcpu), 0, THIS_MODULE,
>  				 KVM_DEV_NAME, MISC_DYNAMIC_MINOR);
> -- 
> 2.53.0
> 

Re: [PATCH v1 09/26] KVM: arm64: Refactor idreg caching into dedicated structure

[Oliver Upton]

On Fri, May 29, 2026 at 05:55:42PM +0200, Steffen Eiden wrote:
> +struct kvm_vm_id_regs {
> +	/*
> +	 * Emulated CPU ID registers per VM
> +	 * (Op0, Op1, CRn, CRm, Op2) of the ID registers to be saved in it
> +	 * is (3, 0, 0, crm, op2), where 1<=crm<8, 0<=op2<8.
> +	 *
> +	 * These emulated idregs are VM-wide, but accessed from the context of a vCPU.
> +	 * Atomic access to multiple idregs are guarded by kvm_arch.config_lock.
> +	 */
> +#define IDREG_IDX(id)		(((sys_reg_CRm(id) - 1) << 3) | sys_reg_Op2(id))
> +#define KVM_ARM_ID_REG_NUM	(IDREG_IDX(sys_reg(3, 0, 0, 7, 7)) + 1)
> +	u64 normal[KVM_ARM_ID_REG_NUM];

nit: ftr_reg would be slightly more clear.

Thanks,
Oliver

> +	u64 midr_el1;
> +	u64 revidr_el1;
> +	u64 aidr_el1;
> +	u64 ctr_el0;
> +};
> +
> +static inline u64 *__vm_id_reg(struct kvm_vm_id_regs *id_regs, u32 reg)
>  {
>  	switch (reg) {
>  	case sys_reg(3, 0, 0, 1, 0) ... sys_reg(3, 0, 0, 7, 7):
> -		return &ka->id_regs[IDREG_IDX(reg)];
> +		return &id_regs->normal[IDREG_IDX(reg)];
>  	case SYS_CTR_EL0:
> -		return &ka->ctr_el0;
> +		return &id_regs->ctr_el0;
>  	case SYS_MIDR_EL1:
> -		return &ka->midr_el1;
> +		return &id_regs->midr_el1;
>  	case SYS_REVIDR_EL1:
> -		return &ka->revidr_el1;
> +		return &id_regs->revidr_el1;
>  	case SYS_AIDR_EL1:
> -		return &ka->aidr_el1;
> +		return &id_regs->aidr_el1;
>  	default:
>  		WARN_ON_ONCE(1);
>  		return NULL;
> @@ -1419,7 +1423,7 @@ static inline u64 *__vm_id_reg(struct kvm_arch *ka, u32 reg)
>  }
>  
>  #define kvm_read_vm_id_reg(kvm, reg)					\
> -	({ u64 __val = *__vm_id_reg(&(kvm)->arch, reg); __val; })
> +	({ u64 __val = *__vm_id_reg(&(kvm)->arch.id_regs, reg); __val; })
>  
>  void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val);
>  
> diff --git a/arch/arm64/kvm/config.c b/arch/arm64/kvm/config.c
> index 014fe04daabf..58a439c3ab9c 100644
> --- a/arch/arm64/kvm/config.c
> +++ b/arch/arm64/kvm/config.c
> @@ -1398,7 +1398,7 @@ void __init check_feature_map(void)
>  
>  static bool idreg_feat_match(struct kvm *kvm, const struct reg_bits_to_feat_map *map)
>  {
> -	u64 regval = kvm->arch.id_regs[map->regidx];
> +	u64 regval = kvm->arch.id_regs.normal[map->regidx];
>  	u64 regfld = (regval >> map->shift) & GENMASK(map->width - 1, 0);
>  
>  	if (map->sign) {
> diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c
> index eb1c10120f9f..94620f142f42 100644
> --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c
> +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c
> @@ -343,7 +343,7 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
>  	DECLARE_BITMAP(allowed_features, KVM_VCPU_MAX_FEATURES);
>  
>  	/* CTR_EL0 is always under host control, even for protected VMs. */
> -	hyp_vm->kvm.arch.ctr_el0 = host_kvm->arch.ctr_el0;
> +	hyp_vm->kvm.arch.id_regs.ctr_el0 = host_kvm->arch.id_regs.ctr_el0;
>  
>  	/* Preserve the vgic model so that GICv3 emulation works */
>  	hyp_vm->kvm.arch.vgic.vgic_model = host_kvm->arch.vgic.vgic_model;
> @@ -358,7 +358,7 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc
>  			    KVM_VCPU_MAX_FEATURES);
>  
>  		if (test_bit(KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS, &host_arch_flags))
> -			hyp_vm->kvm.arch.midr_el1 = host_kvm->arch.midr_el1;
> +			hyp_vm->kvm.arch.id_regs.midr_el1 = host_kvm->arch.id_regs.midr_el1;
>  
>  		return;
>  	}
> @@ -493,7 +493,8 @@ static int vm_copy_id_regs(struct pkvm_hyp_vcpu *hyp_vcpu)
>  	if (test_and_set_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags))
>  		return 0;
>  
> -	memcpy(kvm->arch.id_regs, host_kvm->arch.id_regs, sizeof(kvm->arch.id_regs));
> +	memcpy(kvm->arch.id_regs.normal, host_kvm->arch.id_regs.normal,
> +	       sizeof(kvm->arch.id_regs.normal));
>  
>  	return 0;
>  }
> diff --git a/arch/arm64/kvm/hyp/nvhe/sys_regs.c b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
> index b5a0de84ce01..e8d773d38905 100644
> --- a/arch/arm64/kvm/hyp/nvhe/sys_regs.c
> +++ b/arch/arm64/kvm/hyp/nvhe/sys_regs.c
> @@ -292,7 +292,7 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu,
>  		return 0;
>  
>  	if (reg >= sys_reg(3, 0, 0, 1, 0) && reg <= sys_reg(3, 0, 0, 7, 7))
> -		return kvm->arch.id_regs[IDREG_IDX(reg)];
> +		return kvm->arch.id_regs.normal[IDREG_IDX(reg)];
>  
>  	return 0;
>  }
> @@ -543,7 +543,7 @@ void kvm_init_pvm_id_regs(struct kvm_vcpu *vcpu)
>  	 * for protected VMs.
>  	 */
>  	for (r = sys_reg(3, 0, 0, 4, 0); r <= sys_reg(3, 0, 0, 7, 7); r += sys_reg(0, 0, 0, 0, 1))
> -		ka->id_regs[IDREG_IDX(r)] = pvm_calc_id_reg(vcpu, r);
> +		ka->id_regs.normal[IDREG_IDX(r)] = pvm_calc_id_reg(vcpu, r);
>  
>  	set_bit(KVM_ARCH_FLAG_ID_REGS_INITIALIZED, &kvm->arch.flags);
>  }
> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> index b9aa892616ab..195ecdac7bd6 100644
> --- a/arch/arm64/kvm/sys_regs.c
> +++ b/arch/arm64/kvm/sys_regs.c
> @@ -2477,7 +2477,7 @@ static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
>  
>  void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val)
>  {
> -	u64 *p = __vm_id_reg(&kvm->arch, reg);
> +	u64 *p = __vm_id_reg(&kvm->arch.id_regs, reg);
>  
>  	lockdep_assert_held(&kvm->arch.config_lock);
>  
> -- 
> 2.53.0
> 

Re: [PATCH v1 10/26] KVM: arm64: Fix set_oslsr_el1 to write to OSLAR_EL1

[Andreas Grapentin]

On Jun 01 26, Oliver Upton wrote:
> The current behavior of KVM is correct. KVM treats OSLSR_EL1 as the
> stateful representation of the OS lock and is RO from the guest POV.
> 
> We keep the UAPI straightforward by making this register RW from
> userspace, such that the VMM can directly write back the value returned
> from KVM_GET_ONE_REG.
> 
> Do you have another reason for using OSLAR_EL1 as the canonical
> representation?

Ah, thanks for pointing that out!

this change must have bled into the shared KVM code by mistake, it was
supposed to live in the s390 specific code because it deals with an s390
specific quirk. It was not intended for this change to have any impact
on the native arm64 KVM implementation.

on s390, we need to map the userspace writes of OSLSR_EL1 to OSLAR_EL1,
because OSLSR_EL1 is write-only for the facilities that are available to
us to write system registers. But that detail should be confined to an
s390-specific implementation of set_oslsr_el1.

We'll amend this for v2.

Best,
Andreas

Re: [PATCH v1 15/26] s390: Add functions to query arm guest time

[Andreas Grapentin]

On Jun 01 26, Oliver Upton wrote:
> > Add functions to convert between ARM guest time (LSB0) and s390 host
> > time (MSB0) using new ptff function codes.
> > 
> > Co-developed-by: Nico Boehr <nrb@linux.ibm.com>
> > Signed-off-by: Nico Boehr <nrb@linux.ibm.com>
> > Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
> > ---
> >  arch/s390/include/asm/timex.h | 49 +++++++++++++++++++++++++++++++++++
> >  arch/s390/kernel/time.c       |  1 +
> >  arch/s390/kvm/arm64/arm.c     |  9 ++++++-
> >  3 files changed, 58 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/s390/include/asm/timex.h b/arch/s390/include/asm/timex.h
> > index 49447b40f038..9ec22a28bbda 100644
> > --- a/arch/s390/include/asm/timex.h
> > +++ b/arch/s390/include/asm/timex.h
> > @@ -99,6 +99,8 @@ extern unsigned char ptff_function_mask[16];
> >  #define PTFF_QSI	0x02	/* query steering information */
> >  #define PTFF_QPT	0x03	/* query physical clock */
> >  #define PTFF_QUI	0x04	/* query UTC information */
> > +#define PTFF_QAGTO	0x10	/* query arm guest time offset */
> > +#define PTFF_QAGPT	0x11	/* query arm guest physical time offset */
> 
> Are these analogous to CNTVOFF_EL2 and CNTPOFF_EL2?

They are somewhat related -- we don't have the EL2 counter offset
registers, but we still need to virtualize the guest counters, so
instead we have to maintain the guest counter offset separately.

However, to determine the correct guest counter offset, we are using
these new PTFF function codes to translate between the s390 and arm64
architected counter domains.

Best,
Andreas