[RFC PATCH v7 00/28] KVM: arm64: Add Statistical Profiling Extension (SPE) support

[RFC PATCH v7 00/28] KVM: arm64: Add Statistical Profiling Extension (SPE) support

[Alexandru Elisei]

For a description of the problem, and why using gup doesn't work, see the
previous iteration of the series, version v6 [1].

This series is based on v7.3-rc1.

I've completely changed the approach yet again, but I've kept Reviewed-by tags
for the some of the prerequisite patches. The highlights are:

* The entire memory is effectively pinned at stage 2 before the first VCPU is
run by having userspace use guest_memfd to back VM memory and then doing
KVM_PRE_FAULT_MEMORY on that memory.

This change alone simplifies the code immensly.

* The maximum buffer size does not need to be virtualized since the entire VM
memory is pinned, which means that PMBIDR_EL1 doesn't need to be trapped.
SPE features cannot be completely hidden from the guest, as some of them (most
of them?) add new fields to the records. The end result: PMBIDR_EL1 and
PMSIDR_EL1 don't need to be trapped, so the dependency on FEAT_FGT is dropped.

* It works with the SPE host driver built as a module; KVM requires that the
module is loaded for SPE to be virtualized.

The end result is that KVM support is mostly reduced to the userspace ABI to
configure the feature and context switching the SPE state.


Prerequisites
=============

There are several prerequisites:

* The series depends on KVM_PRE_FAULT_MEMORY being implemented for arm64.
There are two different implementations of that on the mailing list: [2], [3]. I
have chosen to use the series from Jack Thomson [2] (rebased on top of v7.3-rc1
+ the SPE patches) for the simple reason that it doesn't change the way KVM
handles stage 2 faults, and can be cherry-picked *after* the KVM SPE patches, to
allow sashiko to review the SPE patches.

* A patch to explicitely ignore MMU notifiers for guest_memfd-only memslots. The
patch is on Sean's radar to be merged once he finishes fixing a bug in a related
area [4].

I believe that it is possible to have memory unmapped from stage 2 (it involves
automatic NUMA balancing and userspace explicitly setting a mempolicy), but
haven't tested that; the patch codifies that guest_memfd memory should never be
unmapped without userspace requesting it (userspace can unmap the memory from
stage 2 by closing the guest_memfd file descriptor or punching a hole in the
descriptor).

* Patches to add dirty page logging support to guest_memfd-only memslots
(rebased on top of v7.3-rc1 + the SPE patches), as it's impossible today to
change the flags for a guest_memfd-backed memslot. These patches were sent to
the mailing list [5], and as far as I can tell the conclusion was that allowing
dirty page logging is a goal to be pursued for guest_memfd memslots. If we do
decide that dirty page logging for a VM with SPE is useful, I'm planning to
revisit dirty page logging for guest_memfd.

A branch with the prerequisites **after** the SPE series can be found at [6].
The prequisites come after the SPE patches so I can send this series with an
upstream base commit, which allows sashiko to review them.

kvmtool patches are at [7], and kvm-unit-tests are at [8].


How it works
============

1. Userspace sets the KVM_ARM_VCPU_SPE VCPU feature for all VCPUs, configures
the SPE interrupt number (required), assigns a SPE PMU instance to the VM
(required) and then calls KVM_ARM_VCPU_SPE_INIT to finalise configuration. All
of this is done via KVM_SET_DEVICE_ATTR on each VCPU.

Once the VCPU feature is set, KVM_ARM_VCPU_SPE_INIT must be executed before a
VCPU can be run.

2. Userspace uses guest_memfd-backed memslots to represent guest memory, and
pre-maps that memory at stage 2 using KVM_PRE_FAULT_MEMORY.

The order between steps 1 and 2 does not matter.

3. Userspace runs the VCPU(s).

For a VM with SPE, KVM will not check if memslots are backed by guest_memfd,
because KVM cannot know which memslots the guest will use as RAM and which have
a different use, even if they are not read-only. For example, pvtime.

To create a VM with SPE enabled with kvmtool:

# modprobe arm_spe_pmu # If not already loaded or built-in
$ lkvm run --kernel <image> --spe --guest-memfd --name spe-test --params "kpti=off <other_params>"

To test dirty page logging with kvmtool, use the debug interface:

# Create a VM with the name spe-test, as above, then:
$ lkvm debug --name spe-test --enable-dirty-track # to enable dirty page logging
or
$ lkvm debug --name spe-test --disable-dirty-track # to disable dirty page logging

To run the tests from kvm-unit-tests:

$ configure --arch=arm64 --cross-prefix=<cross_compile> --target=kvmtool
$ make clean && make
$ KVMTOOL=<path/to/kvmtool/binary> ./run_tests.sh -g spe


Migration
=========

This is tricky (read: hacky). Relying on KVM fixing up stage 2 faults triggered
SPE attempting to write to memory that is read-only doesn't really work, because
a partial record might be written to the buffer (PMBSR_EL1.DL set). To recover
from this, KVM, or better yet, the VMM, would need to learn to parse the buffer
and rewind PMBPTR_EL1 to the first by after the last complete record.

So what I did instead was to introduce a new hardware exit reason, and have
VCPUs which have the buffer enabled exit to userspace when dirty page logging is
enabled. A VCPU also exits to userspace if the buffer was disabled when dirty
page logging was enabled, and the buffer later becomes enabled, before dirty page
logging is disabled.

When dirty page logging is disabled, the stage 2 permissions are relaxed to
allow writes. The gory details are in patch #27 ("KVM: arm64: Handle dirty page
logging when SPE feature is set").

[1] https://lore.kernel.org/kvmarm/20251114160717.163230-1-alexandru.elisei@arm.com/
[2] https://lore.kernel.org/kvmarm/20260612162354.73378-3-jackabt.amazon@gmail.com/
[3] https://lore.kernel.org/kvmarm/20260825-kvm-arm-prefault-v1-0-befe8947702e@kernel.org/
[4] https://lore.kernel.org/kvmarm/amPCdAswlk24Xilf@google.com/
[5] https://lore.kernel.org/kvmarm/20260702142912.6395-1-alexandru.elisei@arm.com/
[6] https://gitlab.arm.com/linux-arm/linux-ae/-/tree/kvm-spe-v7
[7] https://gitlab.arm.com/linux-arm/kvmtool-ae/-/tree/kvm-spe-v7
[8] https://gitlab.arm.com/linux-arm/kvm-unit-tests-ae/-/tree/kvm-spe-v7

Alexandru Elisei (26):
  arm64/sysreg: Add the nVM field to PMBLIMITR_EL1
  arm64/sysreg: Define MDCR_EL2.E2PB values
  KVM: arm64: Add CONFIG_KVM_ARM_SPE Kconfig option
  perf: arm_spe_pmu: Move struct arm_spe_pmu to a separate header file
  perf: arm_spe_pmu: Add PMBIDR_EL1 and PMSIDR_EL1 to struct arm_spe_pmu
  KVM: arm64: Add KVM_CAP_ARM_SPE capability
  KVM: arm64: Add KVM_ARM_VCPU_SPE VCPU feature
  HACK! KVM: arm64: Disable SPE virtualization if protected KVM is
    enabled
  HACK! KVM: arm64: Enable SPE virtualization only in VHE mode
  HACK! KVM: arm64: Disable SPE virtualization if nested virt is enabled
  KVM: arm64: Add SPE VCPU device attribute to set the SPE device
  KVM: arm64: Add SPE VCPU device attribute to initialize SPE
  KVM: arm64: Use PMSVer from the assigned SPE instance
  KVM: arm64: Add SPE system registers to VCPU context
  KVM: arm64: Apply a RES0 mask to PMBLIMITR_EL1 writes
  KVM: arm64: config: Use functions from spe.c to test
    FEAT_SPE_{FnE,FDS}
  KVM: arm64: VHE: Context switch SPE state
  KVM: arm64: Allow guest SPE physical timestamps only if kernel allows
    it
  KVM: arm64: Handle SPE maintenance interrupts
  arm64: errata: Disable SPE in KVM
  KVM: arm64: Add kvm-arm.ignore_spe_errata kernel parameter
  arm64: errata: Don't enable guest buffer if misprogrammed
  KVM: arm64: at: Use callback for reading descriptor
  KVM: arm64: Map memory on a SPE stage 2 fault
  KVM: arm64: Handle dirty page logging when SPE feature is set
  KVM: arm64: Allow the creation of a SPE enabled VM

Sudeep Holla (2):
  KVM: arm64: Add a new VCPU device control group for SPE
  KVM: arm64: Add SPE VCPU device attribute to set the interrupt number

 .../admin-guide/kernel-parameters.txt         |   3 +
 Documentation/arch/arm64/silicon-errata.rst   |  37 +
 Documentation/virt/kvm/api.rst                |  19 +-
 Documentation/virt/kvm/devices/vcpu.rst       |  84 +++
 arch/arm64/Kconfig                            |  55 ++
 arch/arm64/include/asm/kvm_asm.h              |   1 +
 arch/arm64/include/asm/kvm_host.h             |  48 +-
 arch/arm64/include/asm/kvm_hyp.h              |  26 +-
 arch/arm64/include/asm/kvm_mmu.h              |   1 +
 arch/arm64/include/asm/kvm_nested.h           |   7 +
 arch/arm64/include/asm/kvm_pgtable.h          |  19 +
 arch/arm64/include/asm/kvm_pkvm.h             |   1 +
 arch/arm64/include/asm/kvm_spe.h              | 150 ++++
 arch/arm64/include/uapi/asm/kvm.h             |   6 +
 arch/arm64/kernel/cpu_errata.c                |  51 ++
 arch/arm64/kvm/Kconfig                        |  10 +
 arch/arm64/kvm/Makefile                       |   1 +
 arch/arm64/kvm/arm.c                          |  98 ++-
 arch/arm64/kvm/at.c                           |  22 +-
 arch/arm64/kvm/config.c                       |  30 +-
 arch/arm64/kvm/guest.c                        |  12 +
 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h |   2 +-
 arch/arm64/kvm/hyp/nvhe/hyp-main.c            |  34 +-
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         |   7 +-
 arch/arm64/kvm/hyp/pgtable.c                  |   8 +
 arch/arm64/kvm/hyp/vhe/Makefile               |   1 +
 arch/arm64/kvm/hyp/vhe/debug-sr.c             |  18 +
 arch/arm64/kvm/hyp/vhe/spe-sr.c               | 212 ++++++
 arch/arm64/kvm/hyp/vhe/switch.c               |   9 +
 arch/arm64/kvm/mmu.c                          |  72 ++
 arch/arm64/kvm/nested.c                       |   7 +-
 arch/arm64/kvm/pkvm.c                         |  27 +
 arch/arm64/kvm/pmu-emul.c                     |  11 +-
 arch/arm64/kvm/spe.c                          | 647 ++++++++++++++++++
 arch/arm64/kvm/sys_regs.c                     | 103 ++-
 arch/arm64/tools/cpucaps                      |   2 +
 arch/arm64/tools/sysreg                       |  11 +-
 drivers/perf/arm_spe_pmu.c                    |  46 +-
 include/linux/perf/arm_spe_pmu.h              |  66 ++
 include/uapi/linux/kvm.h                      |   1 +
 40 files changed, 1849 insertions(+), 116 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_spe.h
 create mode 100644 arch/arm64/kvm/hyp/vhe/spe-sr.c
 create mode 100644 arch/arm64/kvm/spe.c
 create mode 100644 include/linux/perf/arm_spe_pmu.h


base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
-- 
2.43.0

[RFC PATCH v7 01/28] arm64/sysreg: Add the nVM field to PMBLIMITR_EL1

[Alexandru Elisei]

Define PMBLIMITR_EL1.nVM, to be used by KVM.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/tools/sysreg | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg
index 94bf065c8ac7..a11a968eb707 100644
--- a/arch/arm64/tools/sysreg
+++ b/arch/arm64/tools/sysreg
@@ -3106,7 +3106,9 @@ EndSysreg
 
 Sysreg	PMBLIMITR_EL1	3	0	9	10	0
 Field	63:12	LIMIT
-Res0	11:6
+Res0	11:8
+Field	7	nVM
+Res0	6
 Field	5	PMFZ
 Res0	4:3
 Enum	2:1	FM
-- 
2.43.0

[RFC PATCH v7 02/28] arm64/sysreg: Define MDCR_EL2.E2PB values

[Alexandru Elisei]

KVM will make use of the different values for MDCR_EL2.E2PB, document them.

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/tools/sysreg | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg
index a11a968eb707..4485736a4f49 100644
--- a/arch/arm64/tools/sysreg
+++ b/arch/arm64/tools/sysreg
@@ -4045,7 +4045,12 @@ Field	17	HPMD
 Res0	16
 Field	15	EnSPM
 Field	14	TPMS
-Field	13:12	E2PB
+UnsignedEnum	13:12 E2PB
+	0b00	EL2
+	0b01	RESERVED
+	0b10	EL1_TRAP
+	0b11	EL1
+EndEnum
 Field	11	TDRA
 Field	10	TDOSA
 Field	9	TDA
-- 
2.43.0

[RFC PATCH v7 03/28] KVM: arm64: Add CONFIG_KVM_ARM_SPE Kconfig option

[Alexandru Elisei]

Add a new configuration option that will be used for KVM SPE emulation.
CONFIG_KVM_ARM_SPE depends on the SPE driver being loaded because:

1. The SPE driver maintains a cpumask of physical CPUs that support SPE,
   and that will be used by KVM to emulate SPE on heterogeneous systems.

2. KVM will rely on the SPE driver enabling the SPE interrupt at the GIC
   level.

The option will become user selectable after SPE support in KVM is
implemented.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/kvm/Kconfig | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 449154f9a485..fece4c02b97c 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -103,5 +103,14 @@ config PKVM_STACKTRACE
 	  If unsure, or not using protected nVHE (pKVM), say N.
 
 endif # NVHE_EL2_DEBUG
+
+config KVM_ARM_SPE
+	bool
+	depends on KVM && ARM_SPE_PMU
+	default n
+	help
+	  Adds support for Statistical Profiling Extension (SPE) in virtual
+	  machines.
+
 endif # KVM
 endif # VIRTUALIZATION
-- 
2.43.0

[RFC PATCH v7 04/28] perf: arm_spe_pmu: Move struct arm_spe_pmu to a separate header file

[Alexandru Elisei]

KVM will soon want to make use of struct arm_spe_pmu, move it to a separate
header where it will be easily accessible.

Cc: Will Deacon <will@kernel.org>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 drivers/perf/arm_spe_pmu.c       | 33 +--------------------
 include/linux/perf/arm_spe_pmu.h | 50 ++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 32 deletions(-)
 create mode 100644 include/linux/perf/arm_spe_pmu.h

diff --git a/drivers/perf/arm_spe_pmu.c b/drivers/perf/arm_spe_pmu.c
index b70f3fedf028..79c571a649d5 100644
--- a/drivers/perf/arm_spe_pmu.c
+++ b/drivers/perf/arm_spe_pmu.c
@@ -28,6 +28,7 @@
 #include <linux/of.h>
 #include <linux/perf_event.h>
 #include <linux/perf/arm_pmu.h>
+#include <linux/perf/arm_spe_pmu.h>
 #include <linux/platform_device.h>
 #include <linux/printk.h>
 #include <linux/slab.h>
@@ -67,38 +68,6 @@ struct arm_spe_pmu_buf {
 	void					*base;
 };
 
-struct arm_spe_pmu {
-	struct pmu				pmu;
-	struct platform_device			*pdev;
-	cpumask_t				supported_cpus;
-	struct hlist_node			hotplug_node;
-
-	int					irq; /* PPI */
-	u16					pmsver;
-	u16					min_period;
-	u16					counter_sz;
-
-#define SPE_PMU_FEAT_FILT_EVT			(1UL << 0)
-#define SPE_PMU_FEAT_FILT_TYP			(1UL << 1)
-#define SPE_PMU_FEAT_FILT_LAT			(1UL << 2)
-#define SPE_PMU_FEAT_ARCH_INST			(1UL << 3)
-#define SPE_PMU_FEAT_LDS			(1UL << 4)
-#define SPE_PMU_FEAT_ERND			(1UL << 5)
-#define SPE_PMU_FEAT_INV_FILT_EVT		(1UL << 6)
-#define SPE_PMU_FEAT_DISCARD			(1UL << 7)
-#define SPE_PMU_FEAT_EFT			(1UL << 8)
-#define SPE_PMU_FEAT_FDS			(1UL << 9)
-#define SPE_PMU_FEAT_DEV_PROBED			(1UL << 63)
-	u64					features;
-
-	u64					pmsevfr_res0;
-	u16					max_record_sz;
-	u16					align;
-	struct perf_output_handle __percpu	*handle;
-};
-
-#define to_spe_pmu(p) (container_of(p, struct arm_spe_pmu, pmu))
-
 /* Convert a free-running index from perf into an SPE buffer offset */
 #define PERF_IDX2OFF(idx, buf) \
 	((idx) % ((unsigned long)(buf)->nr_pages << PAGE_SHIFT))
diff --git a/include/linux/perf/arm_spe_pmu.h b/include/linux/perf/arm_spe_pmu.h
new file mode 100644
index 000000000000..af86fd184050
--- /dev/null
+++ b/include/linux/perf/arm_spe_pmu.h
@@ -0,0 +1,50 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Split from driver/perf/arm_spe_pmu.c
+ *
+ * Copyright (C) 2016 ARM Limited
+ */
+
+#ifndef __PERF_ARM_SPE_PMU_H__
+#define __PERF_ARM_SPE_PMU_H__
+
+#include <linux/cpumask.h>
+#include <linux/container_of.h>
+#include <linux/list.h>
+#include <linux/perf_event.h>
+#include <linux/platform_device.h>
+#include <linux/types.h>
+
+struct arm_spe_pmu {
+	struct pmu				pmu;
+	struct platform_device			*pdev;
+	cpumask_t				supported_cpus;
+	struct hlist_node			hotplug_node;
+
+	int					irq; /* PPI */
+	u16					pmsver;
+	u16					min_period;
+	u16					counter_sz;
+
+#define SPE_PMU_FEAT_FILT_EVT			(1UL << 0)
+#define SPE_PMU_FEAT_FILT_TYP			(1UL << 1)
+#define SPE_PMU_FEAT_FILT_LAT			(1UL << 2)
+#define SPE_PMU_FEAT_ARCH_INST			(1UL << 3)
+#define SPE_PMU_FEAT_LDS			(1UL << 4)
+#define SPE_PMU_FEAT_ERND			(1UL << 5)
+#define SPE_PMU_FEAT_INV_FILT_EVT		(1UL << 6)
+#define SPE_PMU_FEAT_DISCARD			(1UL << 7)
+#define SPE_PMU_FEAT_EFT			(1UL << 8)
+#define SPE_PMU_FEAT_FDS			(1UL << 9)
+#define SPE_PMU_FEAT_DEV_PROBED			(1UL << 63)
+	u64					features;
+
+	u64					pmsevfr_res0;
+	u16					max_record_sz;
+	u16					align;
+	struct perf_output_handle __percpu	*handle;
+};
+
+#define to_spe_pmu(p) (container_of(p, struct arm_spe_pmu, pmu))
+
+#endif /* __PERF_ARM_SPE_PMU_H__ */
-- 
2.43.0

[RFC PATCH v7 05/28] perf: arm_spe_pmu: Add PMBIDR_EL1 and PMSIDR_EL1 to struct arm_spe_pmu

[Alexandru Elisei]

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 drivers/perf/arm_spe_pmu.c       | 3 +++
 include/linux/perf/arm_spe_pmu.h | 2 ++
 2 files changed, 5 insertions(+)

diff --git a/drivers/perf/arm_spe_pmu.c b/drivers/perf/arm_spe_pmu.c
index 79c571a649d5..42ba0924bd02 100644
--- a/drivers/perf/arm_spe_pmu.c
+++ b/drivers/perf/arm_spe_pmu.c
@@ -1095,6 +1095,7 @@ static void __arm_spe_pmu_dev_probe(void *info)
 			"profiling buffer owned by higher exception level\n");
 		return;
 	}
+	spe_pmu->pmbidr_el1 = reg;
 
 	/* Minimum alignment. If it's out-of-range, then fail the probe */
 	fld = FIELD_GET(PMBIDR_EL1_ALIGN, reg);
@@ -1107,6 +1108,8 @@ static void __arm_spe_pmu_dev_probe(void *info)
 
 	/* It's now safe to read PMSIDR and figure out what we've got */
 	reg = read_sysreg_s(SYS_PMSIDR_EL1);
+	spe_pmu->pmsidr_el1 = reg;
+
 	if (FIELD_GET(PMSIDR_EL1_FE, reg))
 		spe_pmu->features |= SPE_PMU_FEAT_FILT_EVT;
 
diff --git a/include/linux/perf/arm_spe_pmu.h b/include/linux/perf/arm_spe_pmu.h
index af86fd184050..97b14e236fcd 100644
--- a/include/linux/perf/arm_spe_pmu.h
+++ b/include/linux/perf/arm_spe_pmu.h
@@ -21,6 +21,8 @@ struct arm_spe_pmu {
 	cpumask_t				supported_cpus;
 	struct hlist_node			hotplug_node;
 
+	u64					pmbidr_el1;
+	u64					pmsidr_el1;
 	int					irq; /* PPI */
 	u16					pmsver;
 	u16					min_period;
-- 
2.43.0

[RFC PATCH v7 06/28] KVM: arm64: Add KVM_CAP_ARM_SPE capability

[Alexandru Elisei]

Add the SPE capability that will be used by userspace to test if KVM
supports SPE virtualization.

The SPE driver supports heterogenous systems. Keep track of all the
available Statistical Profiling Units (SPUs) in the system, because KVM
will require the user to associate a VM with exactly one SPU.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/virt/kvm/api.rst   | 10 ++++++
 arch/arm64/include/asm/kvm_spe.h | 18 ++++++++++
 arch/arm64/kvm/Makefile          |  1 +
 arch/arm64/kvm/arm.c             |  4 +++
 arch/arm64/kvm/spe.c             | 56 ++++++++++++++++++++++++++++++++
 drivers/perf/arm_spe_pmu.c       |  4 +++
 include/linux/perf/arm_spe_pmu.h | 12 +++++++
 include/uapi/linux/kvm.h         |  1 +
 8 files changed, 106 insertions(+)
 create mode 100644 arch/arm64/include/asm/kvm_spe.h
 create mode 100644 arch/arm64/kvm/spe.c

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index e0430cc750c9..1ed5c389c7a4 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -9600,6 +9600,16 @@ take care to differentiate between these cases.
 The presence of this capability indicates that the nested KVM guest can
 start in ESA mode.
 
+8.48 KVM_CAP_ARM_SPE
+--------------------
+
+:Capability: KVM_CAP_ARM_SPE
+:Architectures: arm64
+:Type: vm
+
+This capability indicates that Statistical Profiling Extension (SPE)
+virtualization is available in KVM.
+
 9. Known KVM API problems
 =========================
 
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
new file mode 100644
index 000000000000..f05e6d3f5705
--- /dev/null
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2021 - ARM Ltd
+ */
+
+#ifndef __ARM64_KVM_SPE_H__
+#define __ARM64_KVM_SPE_H__
+
+#ifdef CONFIG_KVM_ARM_SPE
+bool kvm_supports_spe(void);
+#else
+static __always_inline bool kvm_supports_spe(void)
+{
+	return false;
+}
+#endif /* CONFIG_KVM_ARM_SPE */
+
+#endif /* __ARM64_KVM_SPE_H__ */
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 59612d2f277c..9ee271f713cd 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -29,6 +29,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
 kvm-$(CONFIG_HW_PERF_EVENTS)  += pmu-emul.o pmu.o
 kvm-$(CONFIG_ARM64_PTR_AUTH)  += pauth.o
 kvm-$(CONFIG_PTDUMP_STAGE2_DEBUGFS) += ptdump.o
+kvm-$(CONFIG_KVM_ARM_SPE)     += spe.o
 
 kvm-$(CONFIG_NVHE_EL2_TRACING) += hyp_trace.o
 
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 8b080804bc90..eca1184ef9dd 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -42,6 +42,7 @@
 #include <asm/kvm_nested.h>
 #include <asm/kvm_pkvm.h>
 #include <asm/kvm_ptrauth.h>
+#include <asm/kvm_spe.h>
 #include <asm/sections.h>
 #include <asm/stacktrace/nvhe.h>
 
@@ -468,6 +469,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ARM_PMU_V3_STRICT:
 		r = kvm_supports_guest_pmuv3();
 		break;
+	case KVM_CAP_ARM_SPE:
+		r = kvm_supports_spe();
+		break;
 	case KVM_CAP_ARM_INJECT_SERROR_ESR:
 		r = cpus_have_final_cap(ARM64_HAS_RAS_EXTN);
 		break;
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
new file mode 100644
index 000000000000..646806a60d66
--- /dev/null
+++ b/arch/arm64/kvm/spe.c
@@ -0,0 +1,56 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2021 - ARM Ltd
+ */
+
+#include <linux/cpumask.h>
+#include <linux/kvm_host.h>
+#include <linux/perf/arm_spe_pmu.h>
+
+#include <asm/kvm_spe.h>
+#include <asm/sysreg.h>
+
+static LIST_HEAD(spe_pmus);
+static DEFINE_MUTEX(spe_pmus_lock);
+
+struct spe_pmu_entry {
+	struct list_head link;
+	struct arm_spe_pmu *spe_pmu;
+};
+
+void kvm_spe_add_instance(struct arm_spe_pmu *spe_pmu)
+{
+	struct spe_pmu_entry *entry;
+
+	guard(mutex)(&spe_pmus_lock);
+
+	entry = kmalloc(sizeof(*entry), GFP_KERNEL);
+	if (!entry)
+		return;
+
+	entry->spe_pmu = spe_pmu;
+	list_add_tail(&entry->link, &spe_pmus);
+}
+EXPORT_SYMBOL_FOR_MODULES(kvm_spe_add_instance, "arm_spe_pmu");
+
+void kvm_spe_remove_instance(struct arm_spe_pmu *spe_pmu)
+{
+	struct spe_pmu_entry *entry, *tmp;
+
+	guard(mutex)(&spe_pmus_lock);
+
+	list_for_each_entry_safe(entry, tmp, &spe_pmus, link) {
+		if (entry->spe_pmu == spe_pmu) {
+			list_del(&entry->link);
+			kfree(entry);
+			break;
+		}
+	}
+}
+EXPORT_SYMBOL_FOR_MODULES(kvm_spe_remove_instance, "arm_spe_pmu");
+
+bool kvm_supports_spe(void)
+{
+	guard(mutex)(&spe_pmus_lock);
+	return !list_empty(&spe_pmus);
+}
diff --git a/drivers/perf/arm_spe_pmu.c b/drivers/perf/arm_spe_pmu.c
index 42ba0924bd02..f24b122a087c 100644
--- a/drivers/perf/arm_spe_pmu.c
+++ b/drivers/perf/arm_spe_pmu.c
@@ -1370,6 +1370,8 @@ static int arm_spe_pmu_device_probe(struct platform_device *pdev)
 	if (ret)
 		goto out_teardown_dev;
 
+	kvm_spe_add_instance(spe_pmu);
+
 	return 0;
 
 out_teardown_dev:
@@ -1383,6 +1385,8 @@ static void arm_spe_pmu_device_remove(struct platform_device *pdev)
 {
 	struct arm_spe_pmu *spe_pmu = platform_get_drvdata(pdev);
 
+	kvm_spe_remove_instance(spe_pmu);
+
 	arm_spe_pmu_perf_destroy(spe_pmu);
 	arm_spe_pmu_dev_teardown(spe_pmu);
 	free_percpu(spe_pmu->handle);
diff --git a/include/linux/perf/arm_spe_pmu.h b/include/linux/perf/arm_spe_pmu.h
index 97b14e236fcd..bdb703558a87 100644
--- a/include/linux/perf/arm_spe_pmu.h
+++ b/include/linux/perf/arm_spe_pmu.h
@@ -49,4 +49,16 @@ struct arm_spe_pmu {
 
 #define to_spe_pmu(p) (container_of(p, struct arm_spe_pmu, pmu))
 
+#ifdef CONFIG_KVM_ARM_SPE
+void kvm_spe_add_instance(struct arm_spe_pmu *spe_pmu);
+void kvm_spe_remove_instance(struct arm_spe_pmu *spe_pmu);
+#else
+static inline void kvm_spe_add_instance(struct arm_spe_pmu *spe_pmu)
+{
+}
+static inline void kvm_spe_remove_instance(struct arm_spe_pmu *spe_pmu)
+{
+}
+#endif /* CONFIG_KVM_ARM_SPE */
+
 #endif /* __PERF_ARM_SPE_PMU_H__ */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index ac2d77d14963..1565a609890b 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -999,6 +999,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_HPAGE_2G 249
 #define KVM_CAP_PPC_COMPAT_CAPS 250
 #define KVM_CAP_ARM_PMU_V3_STRICT 251
+#define KVM_CAP_ARM_SPE 252
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
-- 
2.43.0

[RFC PATCH v7 07/28] KVM: arm64: Add KVM_ARM_VCPU_SPE VCPU feature

[Alexandru Elisei]

Add a new VCPU feature that enables SPE virtualization when set by
userspace. KVM_VCPU_MAX_FEATURE will be bumped and the feature will
become selectable by userspace once SPE support in KVM is implemented.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/virt/kvm/api.rst    | 2 ++
 arch/arm64/include/asm/kvm_spe.h  | 5 +++++
 arch/arm64/include/uapi/asm/kvm.h | 1 +
 arch/arm64/kvm/arm.c              | 7 +++++++
 4 files changed, 15 insertions(+)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 1ed5c389c7a4..654736fac78c 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -3591,6 +3591,8 @@ Possible features:
 	  Depends on KVM_CAP_ARM_EL2_E2H0.
 	  KVM_ARM_VCPU_HAS_EL2 must also be set.
 
+        - KVM_ARM_VCPU_SPE: Enable SPE for the CPU. Depends on KVM_CAP_ARM_SPE.
+
 4.83 KVM_ARM_PREFERRED_TARGET
 -----------------------------
 
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index f05e6d3f5705..e452da82666d 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -8,11 +8,16 @@
 
 #ifdef CONFIG_KVM_ARM_SPE
 bool kvm_supports_spe(void);
+
+#define vcpu_has_spe(vcpu)					\
+	(vcpu_has_feature(vcpu, KVM_ARM_VCPU_SPE))
 #else
 static __always_inline bool kvm_supports_spe(void)
 {
 	return false;
 }
+
+#define vcpu_has_spe(vcpu)	false
 #endif /* CONFIG_KVM_ARM_SPE */
 
 #endif /* __ARM64_KVM_SPE_H__ */
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 019e5e3d892e..f7f751c00e6f 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -107,6 +107,7 @@ struct kvm_regs {
 #define KVM_ARM_VCPU_HAS_EL2		7 /* Support nested virtualization */
 #define KVM_ARM_VCPU_HAS_EL2_E2H0	8 /* Limit NV support to E2H RES0 */
 #define KVM_ARM_VCPU_PMU_V3_STRICT	9 /* No default PMU creation */
+#define KVM_ARM_VCPU_SPE		10 /* Support SPE in guest */
 
 struct kvm_vcpu_init {
 	__u32 target;
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index eca1184ef9dd..b9cee4cf293d 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1604,6 +1604,9 @@ static unsigned long system_supported_vcpu_features(void)
 	if (!cpus_have_final_cap(ARM64_HAS_NESTED_VIRT))
 		clear_bit(KVM_ARM_VCPU_HAS_EL2, &features);
 
+	if (!kvm_supports_spe())
+		clear_bit(KVM_ARM_VCPU_SPE, &features);
+
 	return features;
 }
 
@@ -1648,6 +1651,10 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu,
 	if (test_bit(KVM_ARM_VCPU_HAS_EL2, &features))
 		return -EINVAL;
 
+	/* SPE is incompatible with AArch32 */
+	if (test_bit(KVM_ARM_VCPU_SPE, &features))
+		return -EINVAL;
+
 	return 0;
 }
 
-- 
2.43.0

[RFC PATCH v7 08/28] HACK! KVM: arm64: Disable SPE virtualization if protected KVM is enabled

[Alexandru Elisei]

For RFC only.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/kvm/spe.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 646806a60d66..97160d1fae81 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -22,6 +22,10 @@ void kvm_spe_add_instance(struct arm_spe_pmu *spe_pmu)
 {
 	struct spe_pmu_entry *entry;
 
+	/* TODO: pKVM support */
+	if (is_protected_kvm_enabled())
+		return;
+
 	guard(mutex)(&spe_pmus_lock);
 
 	entry = kmalloc(sizeof(*entry), GFP_KERNEL);
-- 
2.43.0

[RFC PATCH v7 09/28] HACK! KVM: arm64: Enable SPE virtualization only in VHE mode

[Alexandru Elisei]

For RFC only.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/kvm/spe.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 97160d1fae81..00908aa2576b 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -22,8 +22,8 @@ void kvm_spe_add_instance(struct arm_spe_pmu *spe_pmu)
 {
 	struct spe_pmu_entry *entry;
 
-	/* TODO: pKVM support */
-	if (is_protected_kvm_enabled())
+	/* TODO: pKVM and nVHE support */
+	if (is_protected_kvm_enabled() || !has_vhe())
 		return;
 
 	guard(mutex)(&spe_pmus_lock);
-- 
2.43.0

[RFC PATCH v7 10/28] HACK! KVM: arm64: Disable SPE virtualization if nested virt is enabled

[Alexandru Elisei]

For RFC only.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/kvm/arm.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index b9cee4cf293d..763b102dc51f 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1640,6 +1640,11 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu,
 	    !test_bit(KVM_ARM_VCPU_PMU_V3, &features))
 		return -EINVAL;
 
+	/* TODO: NV support */
+	if (test_bit(KVM_ARM_VCPU_SPE, &features) &&
+	    test_bit(KVM_ARM_VCPU_HAS_EL2, &features))
+		return -EINVAL;
+
 	if (!test_bit(KVM_ARM_VCPU_EL1_32BIT, &features))
 		return 0;
 
-- 
2.43.0

[RFC PATCH v7 11/28] KVM: arm64: Add a new VCPU device control group for SPE

[Alexandru Elisei]

From: Sudeep Holla <sudeep.holla@arm.com>

Add a new VCPU device control group to control various aspects of KVM's SPE
emulation. Functionality will be added in later patches.

[ Alexandru E: Major rework ]

Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/virt/kvm/devices/vcpu.rst |  5 +++++
 arch/arm64/include/asm/kvm_spe.h        | 17 +++++++++++++++++
 arch/arm64/include/uapi/asm/kvm.h       |  1 +
 arch/arm64/kvm/guest.c                  | 10 ++++++++++
 arch/arm64/kvm/spe.c                    | 15 +++++++++++++++
 5 files changed, 48 insertions(+)

diff --git a/Documentation/virt/kvm/devices/vcpu.rst b/Documentation/virt/kvm/devices/vcpu.rst
index deb5c51bc00c..e0a7fa35dfd2 100644
--- a/Documentation/virt/kvm/devices/vcpu.rst
+++ b/Documentation/virt/kvm/devices/vcpu.rst
@@ -302,3 +302,8 @@ From the destination VMM process:
 
 7. Write the KVM_VCPU_TSC_OFFSET attribute for every vCPU with the
    respective value derived in the previous step.
+
+5. GROUP: KVM_ARM_VCPU_SPE_CTRL
+===============================
+
+:Architectures: ARM64
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index e452da82666d..1327064d0984 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -11,6 +11,10 @@ bool kvm_supports_spe(void);
 
 #define vcpu_has_spe(vcpu)					\
 	(vcpu_has_feature(vcpu, KVM_ARM_VCPU_SPE))
+
+int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
+int kvm_spe_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
+int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 #else
 static __always_inline bool kvm_supports_spe(void)
 {
@@ -18,6 +22,19 @@ static __always_inline bool kvm_supports_spe(void)
 }
 
 #define vcpu_has_spe(vcpu)	false
+
+static inline int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
+static inline int kvm_spe_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
+static inline int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
 #endif /* CONFIG_KVM_ARM_SPE */
 
 #endif /* __ARM64_KVM_SPE_H__ */
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index f7f751c00e6f..e8a68e34be63 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -446,6 +446,7 @@ enum {
 #define   KVM_ARM_VCPU_TIMER_IRQ_HPTIMER	3
 #define KVM_ARM_VCPU_PVTIME_CTRL	2
 #define   KVM_ARM_VCPU_PVTIME_IPA	0
+#define KVM_ARM_VCPU_SPE_CTRL		3
 
 /* KVM_IRQ_LINE irq field index values */
 #define KVM_ARM_IRQ_VCPU2_SHIFT		28
diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c
index b01d6622b872..56c5f937fcdb 100644
--- a/arch/arm64/kvm/guest.c
+++ b/arch/arm64/kvm/guest.c
@@ -25,6 +25,7 @@
 #include <asm/kvm.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_nested.h>
+#include <asm/kvm_spe.h>
 #include <asm/sigcontext.h>
 
 #include "trace.h"
@@ -918,6 +919,9 @@ int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu,
 	case KVM_ARM_VCPU_PVTIME_CTRL:
 		ret = kvm_arm_pvtime_set_attr(vcpu, attr);
 		break;
+	case KVM_ARM_VCPU_SPE_CTRL:
+		ret = kvm_spe_set_attr(vcpu, attr);
+		break;
 	default:
 		ret = -ENXIO;
 		break;
@@ -941,6 +945,9 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,
 	case KVM_ARM_VCPU_PVTIME_CTRL:
 		ret = kvm_arm_pvtime_get_attr(vcpu, attr);
 		break;
+	case KVM_ARM_VCPU_SPE_CTRL:
+		ret = kvm_spe_get_attr(vcpu, attr);
+		break;
 	default:
 		ret = -ENXIO;
 		break;
@@ -964,6 +971,9 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,
 	case KVM_ARM_VCPU_PVTIME_CTRL:
 		ret = kvm_arm_pvtime_has_attr(vcpu, attr);
 		break;
+	case KVM_ARM_VCPU_SPE_CTRL:
+		ret = kvm_spe_has_attr(vcpu, attr);
+		break;
 	default:
 		ret = -ENXIO;
 		break;
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 00908aa2576b..fe45ad4c703a 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -58,3 +58,18 @@ bool kvm_supports_spe(void)
 	guard(mutex)(&spe_pmus_lock);
 	return !list_empty(&spe_pmus);
 }
+
+int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
+
+int kvm_spe_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
+
+int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
-- 
2.43.0

[RFC PATCH v7 12/28] KVM: arm64: Add SPE VCPU device attribute to set the interrupt number

[Alexandru Elisei]

From: Sudeep Holla <sudeep.holla@arm.com>

Add KVM_ARM_VCPU_SPE_CTRL(KVM_ARM_VCPU_SPE_IRQ) to allow the user to set
the interrupt number for the buffer management interrupt.

[ Alexandru E: Split from "KVM: arm64: Add a new VCPU device control group
               for SPE" ]

Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/virt/kvm/devices/vcpu.rst | 22 +++++++
 arch/arm64/include/asm/kvm_host.h       |  2 +
 arch/arm64/include/asm/kvm_spe.h        | 11 ++++
 arch/arm64/include/uapi/asm/kvm.h       |  1 +
 arch/arm64/kvm/guest.c                  |  2 +
 arch/arm64/kvm/spe.c                    | 82 +++++++++++++++++++++++++
 6 files changed, 120 insertions(+)

diff --git a/Documentation/virt/kvm/devices/vcpu.rst b/Documentation/virt/kvm/devices/vcpu.rst
index e0a7fa35dfd2..d30b2f7fec9a 100644
--- a/Documentation/virt/kvm/devices/vcpu.rst
+++ b/Documentation/virt/kvm/devices/vcpu.rst
@@ -307,3 +307,25 @@ From the destination VMM process:
 ===============================
 
 :Architectures: ARM64
+
+5.1 ATTRIBUTE: KVM_ARM_VCPU_SPE_IRQ
+-----------------------------------
+
+:Parameters: in kvm_device_attr.addr the address for the Profiling Buffer
+             management interrupt number as a pointer to an int
+
+Returns:
+
+	 =======  ==========================================================
+	 -EFAULT  Error accessing the buffer management interrupt number
+	 -EINVAL  Invalid interrupt number or not using an in-kernel irqchip
+	 -ENODEV  KVM_ARM_VCPU_HAS_SPE VCPU feature not set
+	 -ENXIO   SPE not supported or not properly configured
+	 =======  ==========================================================
+
+Required.
+
+Specifies the Profiling Buffer management interrupt number. The interrupt number
+must be a PPI and the interrupt number must be the same for each VCPU. Arm
+recommends 21 as the interrupt number. SPE virtualization requires an in-kernel
+vGIC implementation.
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 27fe0cd5b2d7..e7332bd82bf3 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -27,6 +27,7 @@
 #include <asm/fpsimd.h>
 #include <asm/kvm.h>
 #include <asm/kvm_asm.h>
+#include <asm/kvm_spe.h>
 #include <asm/vncr_mapping.h>
 
 #define __KVM_HAVE_ARCH_INTC_INITIALIZED
@@ -918,6 +919,7 @@ struct kvm_vcpu_arch {
 	struct vgic_cpu vgic_cpu;
 	struct arch_timer_cpu timer_cpu;
 	struct kvm_pmu pmu;
+	struct kvm_vcpu_spe vcpu_spe;
 
 	/* vcpu power state */
 	struct kvm_mp_state mp_state;
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index 1327064d0984..8ded2d821bff 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -6,7 +6,15 @@
 #ifndef __ARM64_KVM_SPE_H__
 #define __ARM64_KVM_SPE_H__
 
+struct kvm;
+struct kvm_vcpu;
+struct kvm_device_attr;
+
 #ifdef CONFIG_KVM_ARM_SPE
+struct kvm_vcpu_spe {
+	int irq_num;		/* Buffer management interrupt number */
+};
+
 bool kvm_supports_spe(void);
 
 #define vcpu_has_spe(vcpu)					\
@@ -16,6 +24,9 @@ int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 int kvm_spe_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 #else
+struct kvm_vcpu_spe {
+};
+
 static __always_inline bool kvm_supports_spe(void)
 {
 	return false;
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index e8a68e34be63..b9e5019181eb 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -447,6 +447,7 @@ enum {
 #define KVM_ARM_VCPU_PVTIME_CTRL	2
 #define   KVM_ARM_VCPU_PVTIME_IPA	0
 #define KVM_ARM_VCPU_SPE_CTRL		3
+#define   KVM_ARM_VCPU_SPE_IRQ		0
 
 /* KVM_IRQ_LINE irq field index values */
 #define KVM_ARM_IRQ_VCPU2_SHIFT		28
diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c
index 56c5f937fcdb..f953b235b458 100644
--- a/arch/arm64/kvm/guest.c
+++ b/arch/arm64/kvm/guest.c
@@ -920,7 +920,9 @@ int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu,
 		ret = kvm_arm_pvtime_set_attr(vcpu, attr);
 		break;
 	case KVM_ARM_VCPU_SPE_CTRL:
+		mutex_lock(&vcpu->kvm->arch.config_lock);
 		ret = kvm_spe_set_attr(vcpu, attr);
+		mutex_unlock(&vcpu->kvm->arch.config_lock);
 		break;
 	default:
 		ret = -ENXIO;
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index fe45ad4c703a..dd5f80318ba4 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -59,17 +59,99 @@ bool kvm_supports_spe(void)
 	return !list_empty(&spe_pmus);
 }
 
+static bool kvm_spe_irq_is_valid(struct kvm *kvm, int irq)
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vcpu_spe *vcpu_spe;
+	unsigned long i;
+
+	kvm_for_each_vcpu(i, vcpu, kvm) {
+		vcpu_spe = &vcpu->arch.vcpu_spe;
+
+		if (!vcpu_spe->irq_num)
+			continue;
+
+		if (vcpu_spe->irq_num != irq)
+			return false;
+	}
+
+	return true;
+}
+
 int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 {
+	struct kvm_vcpu_spe *vcpu_spe = &vcpu->arch.vcpu_spe;
+	struct kvm *kvm = vcpu->kvm;
+
+	lockdep_assert_held(&kvm->arch.config_lock);
+
+	if (!vcpu_has_spe(vcpu))
+		return -ENODEV;
+
+	switch (attr->attr) {
+	case KVM_ARM_VCPU_SPE_IRQ: {
+		int __user *uaddr = (int __user *)(long)attr->addr;
+		int irq;
+
+		if (!irqchip_in_kernel(kvm))
+			return -EINVAL;
+
+		if (get_user(irq, uaddr))
+			return -EFAULT;
+
+		if (!irq_is_ppi(kvm, irq))
+			return -EINVAL;
+
+		if (!kvm_spe_irq_is_valid(kvm, irq))
+			return -EINVAL;
+
+		vcpu_spe->irq_num = irq;
+		return 0;
+	}
+	}
+
 	return -ENXIO;
 }
 
 int kvm_spe_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 {
+	struct kvm_vcpu_spe *vcpu_spe = &vcpu->arch.vcpu_spe;
+	struct kvm *kvm = vcpu->kvm;
+
+	if (!vcpu_has_spe(vcpu))
+		return -ENODEV;
+
+	switch (attr->attr) {
+	case KVM_ARM_VCPU_SPE_IRQ: {
+		int __user *uaddr = (int __user *)(long)attr->addr;
+		int irq;
+
+		if (!irqchip_in_kernel(kvm))
+			return -EINVAL;
+
+		if (!vcpu_spe->irq_num)
+			return -ENXIO;
+
+		irq = vcpu_spe->irq_num;
+		if (put_user(irq, uaddr))
+			return -EFAULT;
+
+		return 0;
+	}
+	}
+
 	return -ENXIO;
 }
 
 int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 {
+	if (!vcpu_has_spe(vcpu))
+		return -ENODEV;
+
+	switch(attr->attr) {
+	case KVM_ARM_VCPU_SPE_IRQ:
+		return 0;
+	}
+
 	return -ENXIO;
 }
-- 
2.43.0

[RFC PATCH v7 13/28] KVM: arm64: Add SPE VCPU device attribute to set the SPE device

[Alexandru Elisei]

To support systems where there is more than one SPE instance, or where not
all the PEs have SPE, add KVM_ARM_VCPU_SPE_CTRL(KVM_ARM_VCPU_SPE_ID) for
userspace to set the SPE instance it wants the virtual machine to use.
Similar to the PMU, it is entirely up to userspace to make sure the VCPUs
are run only on the physical CPUs which share this SPE instance.

If the ioctl is called for multiple VCPUs, userspace must use the same
SPE identifier for each of the VCPUs.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/virt/kvm/devices/vcpu.rst | 32 +++++++++
 arch/arm64/include/asm/kvm_host.h       |  4 ++
 arch/arm64/include/asm/kvm_spe.h        | 19 ++++++
 arch/arm64/include/uapi/asm/kvm.h       |  1 +
 arch/arm64/kvm/arm.c                    | 43 ++++++++++++
 arch/arm64/kvm/pmu-emul.c               | 11 ++-
 arch/arm64/kvm/spe.c                    | 90 +++++++++++++++++++++++++
 7 files changed, 198 insertions(+), 2 deletions(-)

diff --git a/Documentation/virt/kvm/devices/vcpu.rst b/Documentation/virt/kvm/devices/vcpu.rst
index d30b2f7fec9a..d60dd1daa67f 100644
--- a/Documentation/virt/kvm/devices/vcpu.rst
+++ b/Documentation/virt/kvm/devices/vcpu.rst
@@ -329,3 +329,35 @@ Specifies the Profiling Buffer management interrupt number. The interrupt number
 must be a PPI and the interrupt number must be the same for each VCPU. Arm
 recommends 21 as the interrupt number. SPE virtualization requires an in-kernel
 vGIC implementation.
+
+5.2 ATTRIBUTE: KVM_ARM_VCPU_SPE_ID
+------------------------------------------
+
+:Parameters: in kvm_device_attr.addr the address to an int representing the SPE
+             identifier.
+
+:Returns:
+
+	 =======  =========================================================
+	 -EBUSY   Virtual machine has already run
+	 -EFAULT  Error accessing the SPE identifier
+	 -EINVAL  A different SPE instance already assigned, or no physical
+                  CPUs found
+	 -ENXIO   SPE not supported or not properly configured, or the SPE
+                  module not loaded
+	 -ENODEV  KVM_ARM_VCPU_HAS_SPE VCPU feature not set
+	 =======  =========================================================
+
+Required.
+
+Request that the VCPU uses the specified SPE instance. The SPE identifier can be
+read from the 'type' file for the desired SPE instance under /sys/devices (or,
+equivalent, /sys/bus/event_source). Must be set for at least one VCPU, in which
+case all the other VCPUs will use the same SPE instance. Once a SPE instance has
+been set, attempting to set a different one will result in an error.
+
+Similar to KVM_ARM_VCPU_PMU_V3_CTRL(KVM_ARM_VCPU_PMU_SET_PMU), userspace is
+responsible for making sure that the VCPU is run only on physical CPUs which
+share the specified SPE instance. If the VCPU has a PMU assigned, and there are
+no physical CPUs which share the same SPE and PMU instance, an error is
+returned.
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index e7332bd82bf3..10fa223bdbda 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -382,6 +382,7 @@ struct kvm_arch {
 	 */
 	unsigned long *pmu_filter;
 	struct arm_pmu *arm_pmu;
+	struct kvm_spe kvm_spe;
 
 	cpumask_var_t supported_cpus;
 
@@ -1499,6 +1500,9 @@ void kvm_vcpu_put_vhe(struct kvm_vcpu *vcpu);
 int __init kvm_set_ipa_limit(void);
 u32 kvm_get_pa_bits(struct kvm *kvm);
 
+int kvm_update_supported_cpus(struct kvm *kvm, cpumask_t *pmu_cpus,
+			      cpumask_t *spe_cpus);
+
 #define __KVM_HAVE_ARCH_VM_ALLOC
 struct kvm *kvm_arch_alloc_vm(void);
 
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index 8ded2d821bff..b8afb419c851 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -11,6 +11,11 @@ struct kvm_vcpu;
 struct kvm_device_attr;
 
 #ifdef CONFIG_KVM_ARM_SPE
+struct arm_spe_pmu;
+struct kvm_spe {
+	struct arm_spe_pmu *spe_pmu;
+};
+
 struct kvm_vcpu_spe {
 	int irq_num;		/* Buffer management interrupt number */
 };
@@ -20,10 +25,17 @@ bool kvm_supports_spe(void);
 #define vcpu_has_spe(vcpu)					\
 	(vcpu_has_feature(vcpu, KVM_ARM_VCPU_SPE))
 
+void kvm_spe_destroy_vm(struct kvm *kvm);
+
 int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 int kvm_spe_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
+
+bool kvm_spe_pmu_set(struct kvm *kvm);
 #else
+struct kvm_spe {
+};
+
 struct kvm_vcpu_spe {
 };
 
@@ -34,6 +46,9 @@ static __always_inline bool kvm_supports_spe(void)
 
 #define vcpu_has_spe(vcpu)	false
 
+static inline void kvm_spe_destroy_vm(struct kvm *kvm)
+{
+}
 static inline int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 {
 	return -ENXIO;
@@ -46,6 +61,10 @@ static inline int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr
 {
 	return -ENXIO;
 }
+static inline bool kvm_spe_pmu_set(struct kvm *kvm)
+{
+	return false;
+}
 #endif /* CONFIG_KVM_ARM_SPE */
 
 #endif /* __ARM64_KVM_SPE_H__ */
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index b9e5019181eb..8ffa3e89d38c 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -448,6 +448,7 @@ enum {
 #define   KVM_ARM_VCPU_PVTIME_IPA	0
 #define KVM_ARM_VCPU_SPE_CTRL		3
 #define   KVM_ARM_VCPU_SPE_IRQ		0
+#define   KVM_ARM_VCPU_SPE_ID		1
 
 /* KVM_IRQ_LINE irq field index values */
 #define KVM_ARM_IRQ_VCPU2_SHIFT		28
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 763b102dc51f..132df44c3088 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -336,6 +336,8 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 	kfree(kvm->arch.sysreg_masks);
 	kvm_destroy_vcpus(kvm);
 
+	kvm_spe_destroy_vm(kvm);
+
 	kvm_unshare_hyp(kvm, kvm + 1);
 
 	kvm_arm_teardown_hypercalls(kvm);
@@ -612,6 +614,47 @@ void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu)
 
 }
 
+int kvm_update_supported_cpus(struct kvm *kvm, cpumask_t *pmu_cpus,
+			      cpumask_t *spe_cpus)
+{
+	cpumask_t *supported_cpus;
+	cpumask_t *cpus;
+	int ret;
+
+	if (!zalloc_cpumask_var(&supported_cpus, GFP_KERNEL_ACCOUNT))
+		return -ENOMEM;
+
+	cpumask_copy(supported_cpus, kvm->arch.supported_cpus);
+
+	lockdep_assert_held(&kvm->arch.config_lock);
+
+	ret = -EINVAL;
+	/* Cannot update for SPE and PMU at the same time. */
+	if (WARN_ON_ONCE(pmu_cpus && spe_cpus))
+		goto out;
+
+	if (pmu_cpus)
+		cpus = pmu_cpus;
+	else
+		cpus = spe_cpus;
+
+	if ((spe_cpus && kvm->arch.arm_pmu) ||
+	    (pmu_cpus && kvm_spe_pmu_set(kvm)))
+		cpumask_and(supported_cpus, cpus, supported_cpus);
+	else
+		cpumask_copy(supported_cpus, cpus);
+
+	if (cpumask_empty(supported_cpus))
+		goto out;
+
+	cpumask_copy(kvm->arch.supported_cpus, supported_cpus);
+	ret = 0;
+out:
+	free_cpumask_var(supported_cpus);
+
+	return ret;
+}
+
 static void vcpu_set_pauth_traps(struct kvm_vcpu *vcpu)
 {
 	if (vcpu_has_ptrauth(vcpu) && !is_protected_kvm_enabled()) {
diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c
index 5b1af7e2176f..59930e1bf357 100644
--- a/arch/arm64/kvm/pmu-emul.c
+++ b/arch/arm64/kvm/pmu-emul.c
@@ -1085,6 +1085,11 @@ int kvm_arm_set_default_pmu(struct kvm *kvm)
 	return 0;
 }
 
+static int kvm_pmu_update_supported_cpus(struct kvm *kvm, cpumask_t *pmu_cpus)
+{
+	return kvm_update_supported_cpus(kvm, pmu_cpus, NULL);
+}
+
 static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -1104,8 +1109,11 @@ static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id)
 				break;
 			}
 
+			ret = kvm_pmu_update_supported_cpus(kvm, &arm_pmu->supported_cpus);
+			if (ret)
+				break;
 			kvm_arm_set_pmu(kvm, arm_pmu);
-			cpumask_copy(kvm->arch.supported_cpus, &arm_pmu->supported_cpus);
+
 
 			/*
 			 * Since a specific PMU is explicitly selected,
@@ -1117,7 +1125,6 @@ static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id)
 				kvm->arch.pmmir_slots =
 					FIELD_GET(ARMV8_PMU_SLOTS,
 						  arm_pmu->reg_pmmir);
-			ret = 0;
 			break;
 		}
 	}
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index dd5f80318ba4..87fcdbbcbd67 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -59,6 +59,71 @@ bool kvm_supports_spe(void)
 	return !list_empty(&spe_pmus);
 }
 
+void kvm_spe_destroy_vm(struct kvm *kvm)
+{
+	struct arm_spe_pmu *spe_pmu;
+
+	guard(mutex)(&spe_pmus_lock);
+
+	spe_pmu = kvm->arch.kvm_spe.spe_pmu;
+	if (!spe_pmu)
+		return;
+
+	module_put(spe_pmu->pmu.module);
+}
+
+bool kvm_spe_pmu_set(struct kvm *kvm)
+{
+	return !!kvm->arch.kvm_spe.spe_pmu;
+}
+
+static int kvm_spe_update_supported_cpus(struct kvm *kvm, cpumask_t *spe_cpus)
+{
+	return kvm_update_supported_cpus(kvm, NULL, spe_cpus);
+}
+
+static int kvm_spe_set_spe_id(struct kvm_vcpu *vcpu, int spe_id)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_spe *kvm_spe = &kvm->arch.kvm_spe;
+	struct arm_spe_pmu *existing_spe_pmu, *spe_pmu = NULL;
+	struct spe_pmu_entry *entry;
+
+	if (kvm_vm_has_ran_once(kvm))
+		return -EBUSY;
+
+	guard(mutex)(&spe_pmus_lock);
+
+	list_for_each_entry(entry, &spe_pmus, link) {
+		if (entry->spe_pmu->pmu.type == spe_id) {
+			spe_pmu = entry->spe_pmu;
+			break;
+		}
+	}
+
+	if (!spe_pmu)
+		return -ENXIO;
+
+	existing_spe_pmu = kvm_spe->spe_pmu;
+	if (existing_spe_pmu) {
+		if (existing_spe_pmu != spe_pmu)
+			return -EINVAL;
+		return 0;
+	}
+
+	if (!try_module_get(spe_pmu->pmu.module))
+		return -ENXIO;
+
+	if (kvm_spe_update_supported_cpus(kvm, &spe_pmu->supported_cpus)) {
+		module_put(spe_pmu->pmu.module);
+		return -EINVAL;
+	}
+
+	kvm_spe->spe_pmu = spe_pmu;
+
+	return 0;
+}
+
 static bool kvm_spe_irq_is_valid(struct kvm *kvm, int irq)
 {
 	struct kvm_vcpu *vcpu;
@@ -108,6 +173,15 @@ int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 		vcpu_spe->irq_num = irq;
 		return 0;
 	}
+	case KVM_ARM_VCPU_SPE_ID: {
+		int __user *uaddr = (int __user *)(long)attr->addr;
+		int spe_id;
+
+		if (get_user(spe_id, uaddr))
+			return -EFAULT;
+
+		return kvm_spe_set_spe_id(vcpu, spe_id);
+	}
 	}
 
 	return -ENXIO;
@@ -117,6 +191,7 @@ int kvm_spe_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 {
 	struct kvm_vcpu_spe *vcpu_spe = &vcpu->arch.vcpu_spe;
 	struct kvm *kvm = vcpu->kvm;
+	struct kvm_spe *kvm_spe = &kvm->arch.kvm_spe;
 
 	if (!vcpu_has_spe(vcpu))
 		return -ENODEV;
@@ -138,6 +213,20 @@ int kvm_spe_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 
 		return 0;
 	}
+	case KVM_ARM_VCPU_SPE_ID: {
+		struct arm_spe_pmu *spe_pmu = kvm_spe->spe_pmu;
+		int __user *uaddr = (int __user *)(long)attr->addr;
+		int spe_id;
+
+		if (!spe_pmu)
+			return -ENXIO;
+
+		spe_id = spe_pmu->pmu.type;
+		if (put_user(spe_id, uaddr))
+			return -EFAULT;
+
+		return 0;
+	}
 	}
 
 	return -ENXIO;
@@ -150,6 +239,7 @@ int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 
 	switch(attr->attr) {
 	case KVM_ARM_VCPU_SPE_IRQ:
+	case KVM_ARM_VCPU_SPE_ID:
 		return 0;
 	}
 
-- 
2.43.0

[RFC PATCH v7 14/28] KVM: arm64: Add SPE VCPU device attribute to initialize SPE

[Alexandru Elisei]

Add KVM_ARM_VCPU_SPE_CTRL(KVM_ARM_VCPU_SPE_INIT) VCPU ioctl to initialize
SPE. Initialization must be done exactly once for each VCPU.

[ Alexandru E: Split from "KVM: arm64: Add a new VCPU device control group
	   for SPE" ]

Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/virt/kvm/devices/vcpu.rst | 28 ++++++++++++++++++++---
 arch/arm64/include/asm/kvm_spe.h        |  6 +++++
 arch/arm64/include/uapi/asm/kvm.h       |  1 +
 arch/arm64/kvm/arm.c                    |  6 +++++
 arch/arm64/kvm/spe.c                    | 30 +++++++++++++++++++++++++
 5 files changed, 68 insertions(+), 3 deletions(-)

diff --git a/Documentation/virt/kvm/devices/vcpu.rst b/Documentation/virt/kvm/devices/vcpu.rst
index d60dd1daa67f..e2eac48b9d28 100644
--- a/Documentation/virt/kvm/devices/vcpu.rst
+++ b/Documentation/virt/kvm/devices/vcpu.rst
@@ -317,6 +317,7 @@ From the destination VMM process:
 Returns:
 
 	 =======  ==========================================================
+	 -EBUSY   SPE already initialized
 	 -EFAULT  Error accessing the buffer management interrupt number
 	 -EINVAL  Invalid interrupt number or not using an in-kernel irqchip
 	 -ENODEV  KVM_ARM_VCPU_HAS_SPE VCPU feature not set
@@ -338,15 +339,15 @@ vGIC implementation.
 
 :Returns:
 
-	 =======  =========================================================
-	 -EBUSY   Virtual machine has already run
+	 =======  ===========================================================
+	 -EBUSY   Virtual machine has already run, or SPE already initialized
 	 -EFAULT  Error accessing the SPE identifier
 	 -EINVAL  A different SPE instance already assigned, or no physical
                   CPUs found
 	 -ENXIO   SPE not supported or not properly configured, or the SPE
                   module not loaded
 	 -ENODEV  KVM_ARM_VCPU_HAS_SPE VCPU feature not set
-	 =======  =========================================================
+	 =======  ===========================================================
 
 Required.
 
@@ -361,3 +362,24 @@ responsible for making sure that the VCPU is run only on physical CPUs which
 share the specified SPE instance. If the VCPU has a PMU assigned, and there are
 no physical CPUs which share the same SPE and PMU instance, an error is
 returned.
+
+5.3 ATTRIBUTE: KVM_ARM_VCPU_SPE_INIT
+-----------------------------------
+
+:Parameters: no additional parameter in kvm_device_attr.addr
+
+Returns:
+
+	 =======  ============================================
+	 -EBUSY   SPE already initialized for this VCPU
+	 -ENXIO   SPE not supported or not properly configured
+	 =======  ============================================
+
+Required.
+
+Request initialization of the Statistical Profiling Extension for this VCPU.
+Must be done last, after SPE has been fully configured for the VCPU, and after
+the in-kernel irqchip has been initialized.
+
+KVM will refuse to run the VCPU and KVM_RUN will return an error if the SPE
+feature has been set for the VCPU, but SPE hasn't been initialized.
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index b8afb419c851..a6999709b298 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -18,6 +18,7 @@ struct kvm_spe {
 
 struct kvm_vcpu_spe {
 	int irq_num;		/* Buffer management interrupt number */
+	bool initialized;	/* SPE initialized for the VCPU */
 };
 
 bool kvm_supports_spe(void);
@@ -25,6 +26,7 @@ bool kvm_supports_spe(void);
 #define vcpu_has_spe(vcpu)					\
 	(vcpu_has_feature(vcpu, KVM_ARM_VCPU_SPE))
 
+int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu);
 void kvm_spe_destroy_vm(struct kvm *kvm);
 
 int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
@@ -46,6 +48,10 @@ static __always_inline bool kvm_supports_spe(void)
 
 #define vcpu_has_spe(vcpu)	false
 
+static inline int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu)
+{
+	return 0;
+}
 static inline void kvm_spe_destroy_vm(struct kvm *kvm)
 {
 }
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 8ffa3e89d38c..c695aae484d3 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -449,6 +449,7 @@ enum {
 #define KVM_ARM_VCPU_SPE_CTRL		3
 #define   KVM_ARM_VCPU_SPE_IRQ		0
 #define   KVM_ARM_VCPU_SPE_ID		1
+#define   KVM_ARM_VCPU_SPE_INIT		2
 
 /* KVM_IRQ_LINE irq field index values */
 #define KVM_ARM_IRQ_VCPU2_SHIFT		28
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 132df44c3088..c9a026c61ce7 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1026,6 +1026,12 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
 			return ret;
 	}
 
+	if (vcpu_has_spe(vcpu)) {
+		ret = kvm_spe_vcpu_first_run_init(vcpu);
+		if (ret)
+			return ret;
+	}
+
 	ret = vgic_v5_finalize_ppi_state(kvm);
 	if (ret)
 		return ret;
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 87fcdbbcbd67..fa87d224c8af 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -59,6 +59,16 @@ bool kvm_supports_spe(void)
 	return !list_empty(&spe_pmus);
 }
 
+int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_spe *vcpu_spe = &vcpu->arch.vcpu_spe;
+
+	if (!vcpu_spe->initialized)
+		return -EINVAL;
+
+	return 0;
+}
+
 void kvm_spe_destroy_vm(struct kvm *kvm)
 {
 	struct arm_spe_pmu *spe_pmu;
@@ -147,12 +157,16 @@ int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 {
 	struct kvm_vcpu_spe *vcpu_spe = &vcpu->arch.vcpu_spe;
 	struct kvm *kvm = vcpu->kvm;
+	struct kvm_spe *kvm_spe = &kvm->arch.kvm_spe;
 
 	lockdep_assert_held(&kvm->arch.config_lock);
 
 	if (!vcpu_has_spe(vcpu))
 		return -ENODEV;
 
+	if (vcpu_spe->initialized)
+		return -EBUSY;
+
 	switch (attr->attr) {
 	case KVM_ARM_VCPU_SPE_IRQ: {
 		int __user *uaddr = (int __user *)(long)attr->addr;
@@ -182,6 +196,21 @@ int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 
 		return kvm_spe_set_spe_id(vcpu, spe_id);
 	}
+	case KVM_ARM_VCPU_SPE_INIT:
+		if (!vcpu_spe->irq_num)
+			return -ENXIO;
+
+		if (!kvm_spe->spe_pmu)
+			return -ENXIO;
+
+		if (!vgic_initialized(kvm))
+			return -ENXIO;
+
+		if (kvm_vgic_set_owner(vcpu, vcpu_spe->irq_num, vcpu_spe))
+			return -ENXIO;
+
+		vcpu_spe->initialized = true;
+		return 0;
 	}
 
 	return -ENXIO;
@@ -240,6 +269,7 @@ int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 	switch(attr->attr) {
 	case KVM_ARM_VCPU_SPE_IRQ:
 	case KVM_ARM_VCPU_SPE_ID:
+	case KVM_ARM_VCPU_SPE_INIT:
 		return 0;
 	}
 
-- 
2.43.0

[RFC PATCH v7 15/28] KVM: arm64: Use PMSVer from the assigned SPE instance

[Alexandru Elisei]

KVM allows SPE to be virtualized on heterogeneous system, where hardware
can implement multiple SPE instances for different CPUs, or where SPE
might even be missing on some of the physical CPUs.

KVM resets the VCPU ID registers following a KVM_ARM_VCPU_INIT ioctl.
Using the value for ID_AA64DFR0_EL1.PMSVer on the physical CPU where the
ioctl is executing is unreliable, since userspace might want to later
assign a different SPE instance, with a different PMSVer value. Worse yet,
it is be possible that the VCPU reset is being performed on a physical CPU
without SPE. Be consistent and use PMSVer for the SPE instance assigned to
the VM to initialize ID_AA64DFR0_EL1.PMSVer.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/include/asm/kvm_spe.h | 10 ++++++++++
 arch/arm64/kvm/spe.c             | 24 ++++++++++++++++++++++++
 arch/arm64/kvm/sys_regs.c        |  8 +++++++-
 3 files changed, 41 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index a6999709b298..4bae3e6b05c8 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -29,11 +29,14 @@ bool kvm_supports_spe(void);
 int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu);
 void kvm_spe_destroy_vm(struct kvm *kvm);
 
+void kvm_spe_finalize_idregs(struct kvm *kvm);
+
 int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 int kvm_spe_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 
 bool kvm_spe_pmu_set(struct kvm *kvm);
+u8 kvm_spe_get_pmsver(struct kvm *kvm);
 #else
 struct kvm_spe {
 };
@@ -55,6 +58,9 @@ static inline int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu)
 static inline void kvm_spe_destroy_vm(struct kvm *kvm)
 {
 }
+static inline void kvm_spe_finalize_idregs(struct kvm *kvm)
+{
+}
 static inline int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 {
 	return -ENXIO;
@@ -71,6 +77,10 @@ static inline bool kvm_spe_pmu_set(struct kvm *kvm)
 {
 	return false;
 }
+static inline u8 kvm_spe_get_pmsver(struct kvm *kvm)
+{
+	return 0;
+}
 #endif /* CONFIG_KVM_ARM_SPE */
 
 #endif /* __ARM64_KVM_SPE_H__ */
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index fa87d224c8af..4426d614a9e0 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -87,6 +87,30 @@ bool kvm_spe_pmu_set(struct kvm *kvm)
 	return !!kvm->arch.kvm_spe.spe_pmu;
 }
 
+void kvm_spe_finalize_idregs(struct kvm *kvm)
+{
+	struct arm_spe_pmu *spe_pmu = kvm->arch.kvm_spe.spe_pmu;
+	u64 reg;
+
+	if (!spe_pmu)
+		return;
+
+	reg = kvm_read_vm_id_reg(kvm, SYS_ID_AA64DFR0_EL1);
+	reg &= ~ID_AA64DFR0_EL1_PMSVer_MASK;
+	reg |= SYS_FIELD_PREP(ID_AA64DFR0_EL1, PMSVer, spe_pmu->pmsver);
+	kvm_set_vm_id_reg(kvm, SYS_ID_AA64DFR0_EL1, reg);
+}
+
+u8 kvm_spe_get_pmsver(struct kvm *kvm)
+{
+	struct arm_spe_pmu *spe_pmu = kvm->arch.kvm_spe.spe_pmu;
+
+	if (!spe_pmu)
+		return 0;
+
+	return spe_pmu->pmsver;
+}
+
 static int kvm_spe_update_supported_cpus(struct kvm *kvm, cpumask_t *spe_cpus)
 {
 	return kvm_update_supported_cpus(kvm, NULL, spe_cpus);
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 44aae52c473d..a6bb6884b965 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -2220,8 +2220,11 @@ static u64 sanitise_id_aa64dfr0_el1(const struct kvm_vcpu *vcpu, u64 val)
 		val |= SYS_FIELD_PREP(ID_AA64DFR0_EL1, PMUVer,
 				      kvm_arm_pmu_get_pmuver_limit());
 
-	/* Hide SPE from guests */
+	/* Only initialize the SPE version if the vCPU has the feature. */
 	val &= ~ID_AA64DFR0_EL1_PMSVer_MASK;
+	if (vcpu_has_spe(vcpu))
+		val |= SYS_FIELD_PREP(ID_AA64DFR0_EL1, PMSVer,
+				      kvm_spe_get_pmsver(vcpu->kvm));
 
 	/* Hide BRBE from guests */
 	val &= ~ID_AA64DFR0_EL1_BRBE_MASK;
@@ -5928,6 +5931,9 @@ int kvm_finalize_sys_regs(struct kvm_vcpu *vcpu)
 		kvm_vgic_finalize_idregs(kvm);
 	}
 
+	if (vcpu_has_spe(vcpu))
+		kvm_spe_finalize_idregs(kvm);
+
 	return 0;
 }
 
-- 
2.43.0

[RFC PATCH v7 16/28] KVM: arm64: Add SPE system registers to VCPU context

[Alexandru Elisei]

Add the SPE registers to the VCPU context. Guest writes to the service bit,
which change the state of the SPE maintenance interrupt, will be handled in
a subsequent patch.

Only the buffer registers are trapped, because KVM will need to keep track
of the buffer enabled state for dirty page logging, errata handling and to
sanitise writes to PMBLIMITR_EL1 based on the features exposed to the VM.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/include/asm/kvm_host.h | 13 +++++++
 arch/arm64/include/asm/kvm_spe.h  | 11 ++++++
 arch/arm64/kvm/spe.c              | 29 ++++++++++++++++
 arch/arm64/kvm/sys_regs.c         | 56 ++++++++++++++++++++++++-------
 4 files changed, 96 insertions(+), 13 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 10fa223bdbda..b92b19229c36 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -483,6 +483,19 @@ enum vcpu_sysreg {
 	PMOVSSET_EL0,	/* Overflow Flag Status Set Register */
 	PMUSERENR_EL0,	/* User Enable Register */
 
+	/* SPE registers */
+	PMSCR_EL1,
+	PMSNEVFR_EL1,
+	PMSICR_EL1,
+	PMSIRR_EL1,
+	PMSFCR_EL1,
+	PMSEVFR_EL1,
+	PMSLATFR_EL1,
+	PMBLIMITR_EL1,
+	PMBPTR_EL1,
+	PMBSR_EL1,
+	PMSDSFR_EL1,
+
 	/* Pointer Authentication Registers in a strict increasing order. */
 	APIAKEYLO_EL1,
 	APIAKEYHI_EL1,
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index 4bae3e6b05c8..56dc9e660bef 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -37,6 +37,9 @@ int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 
 bool kvm_spe_pmu_set(struct kvm *kvm);
 u8 kvm_spe_get_pmsver(struct kvm *kvm);
+
+bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val);
+u64 kvm_spe_read_sysreg(struct kvm_vcpu *vcpu, int reg);
 #else
 struct kvm_spe {
 };
@@ -81,6 +84,14 @@ static inline u8 kvm_spe_get_pmsver(struct kvm *kvm)
 {
 	return 0;
 }
+static inline bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val)
+{
+	return true;
+}
+static inline u64 kvm_spe_read_sysreg(struct kvm_vcpu *vcpu, int reg)
+{
+	return 0;
+}
 #endif /* CONFIG_KVM_ARM_SPE */
 
 #endif /* __ARM64_KVM_SPE_H__ */
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 4426d614a9e0..6a00a44c73ed 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -7,6 +7,7 @@
 #include <linux/kvm_host.h>
 #include <linux/perf/arm_spe_pmu.h>
 
+#include <asm/kvm_emulate.h>
 #include <asm/kvm_spe.h>
 #include <asm/sysreg.h>
 
@@ -82,6 +83,34 @@ void kvm_spe_destroy_vm(struct kvm *kvm)
 	module_put(spe_pmu->pmu.module);
 }
 
+bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val)
+{
+	switch (reg) {
+	case PMBLIMITR_EL1:
+	case PMBSR_EL1:
+	case PMBPTR_EL1:
+		__vcpu_assign_sys_reg(vcpu, reg, val);
+		break;
+	default:
+		WARN_ON_ONCE("unexpected trap");
+	}
+
+	return true;
+}
+
+u64 kvm_spe_read_sysreg(struct kvm_vcpu *vcpu, int reg)
+{
+	switch (reg) {
+	case PMBLIMITR_EL1:
+	case PMBSR_EL1:
+	case PMBPTR_EL1:
+		return __vcpu_sys_reg(vcpu, reg);
+	default:
+		WARN_ON_ONCE("unexpected trap");
+		return 0;
+	}
+}
+
 bool kvm_spe_pmu_set(struct kvm *kvm)
 {
 	return !!kvm->arch.kvm_spe.spe_pmu;
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index a6bb6884b965..89b01b6ac783 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -1554,6 +1554,28 @@ static int set_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
 	return 0;
 }
 
+static unsigned int spe_visibility(const struct kvm_vcpu *vcpu,
+				   const struct sys_reg_desc *r)
+{
+	if (vcpu_has_spe(vcpu))
+		return 0;
+
+	return REG_HIDDEN;
+}
+
+static bool access_spe_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+			   const struct sys_reg_desc *r)
+{
+	u64 val = p->regval;
+	int reg = r->reg;
+
+	if (p->is_write)
+		return kvm_spe_write_sysreg(vcpu, reg, val);
+
+	p->regval = kvm_spe_read_sysreg(vcpu, reg);
+	return true;
+}
+
 /* Silly macro to expand the DBG{BCR,BVR,WVR,WCR}n_EL1 registers in one go */
 #define DBG_BCR_BVR_WCR_WVR_EL1(n)					\
 	{ SYS_DESC(SYS_DBGBVRn_EL1(n)),					\
@@ -1586,6 +1608,14 @@ static int set_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
 	  .reset = reset_pmevtyper,					\
 	  .access = access_pmu_evtyper, .reg = (PMEVTYPER0_EL0 + n), }
 
+#define SPE_SYS_REG(name)						\
+	SYS_DESC(SYS_##name), .reg = name, .access = access_spe_reg,	\
+	.reset = reset_val, .val = 0, .visibility = spe_visibility
+
+#define SPE_UNTRAPPED_REG(name)						\
+	SYS_DESC(SYS_##name), .reg = name, .access = undef_access,	\
+	.reset = reset_val, .val = 0, .visibility = spe_visibility
+
 /* Macro to expand the AMU counter and type registers*/
 #define AMU_AMEVCNTR0_EL0(n) { SYS_DESC(SYS_AMEVCNTR0_EL0(n)), undef_access }
 #define AMU_AMEVTYPER0_EL0(n) { SYS_DESC(SYS_AMEVTYPER0_EL0(n)), undef_access }
@@ -3525,19 +3555,19 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_FAR_EL1), access_vm_reg, reset_unknown, FAR_EL1 },
 	{ SYS_DESC(SYS_PAR_EL1), NULL, reset_unknown, PAR_EL1 },
 
-	{ SYS_DESC(SYS_PMSCR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMSNEVFR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMSICR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMSIRR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMSFCR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMSEVFR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMSLATFR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMSIDR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMBLIMITR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMBPTR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMBSR_EL1), undef_access },
-	{ SYS_DESC(SYS_PMSDSFR_EL1), undef_access },
-	/* PMBIDR_EL1 is not trapped */
+	{ SPE_UNTRAPPED_REG(PMSCR_EL1) },
+	{ SPE_UNTRAPPED_REG(PMSNEVFR_EL1) },
+	{ SPE_UNTRAPPED_REG(PMSICR_EL1) },
+	{ SPE_UNTRAPPED_REG(PMSIRR_EL1) },
+	{ SPE_UNTRAPPED_REG(PMSFCR_EL1) },
+	{ SPE_UNTRAPPED_REG(PMSEVFR_EL1) },
+	{ SPE_UNTRAPPED_REG(PMSLATFR_EL1) },
+	{ SYS_DESC(SYS_PMSIDR_EL1), .access = undef_access },
+	{ SPE_SYS_REG(PMBLIMITR_EL1) },
+	{ SPE_SYS_REG(PMBPTR_EL1) },
+	{ SPE_SYS_REG(PMBSR_EL1) },
+	{ SPE_UNTRAPPED_REG(PMSDSFR_EL1) },
+	/* PMBIDR_EL1 is not trapped and read-only */
 
 	{ PMU_SYS_REG(PMINTENSET_EL1),
 	  .access = access_pminten, .reg = PMINTENSET_EL1,
-- 
2.43.0

[RFC PATCH v7 17/28] KVM: arm64: Apply a RES0 mask to PMBLIMITR_EL1 writes

[Alexandru Elisei]

PMBLIMITR_EL1 is special, because it has feature enable fields. If those
features are hidden from a guest, ignore them when the guest or the VMM
attempt to set them.

The least significant bits of the buffer limit address are RES0 based on
the smallest granule implemented. Make sure they are treated as zero even
if the smallest granule exposed to the VM is larger than what the hardware
implements.

Note that it is up to the user to write to the SPE buffer registers only
after the VM features have been configured in the VM id registers.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/virt/kvm/devices/vcpu.rst |  3 ++
 arch/arm64/include/asm/kvm_spe.h        |  7 +++++
 arch/arm64/kvm/spe.c                    | 40 +++++++++++++++++++++++--
 arch/arm64/kvm/sys_regs.c               | 38 +++++++++++++++++++++--
 4 files changed, 83 insertions(+), 5 deletions(-)

diff --git a/Documentation/virt/kvm/devices/vcpu.rst b/Documentation/virt/kvm/devices/vcpu.rst
index e2eac48b9d28..eb352760869c 100644
--- a/Documentation/virt/kvm/devices/vcpu.rst
+++ b/Documentation/virt/kvm/devices/vcpu.rst
@@ -383,3 +383,6 @@ the in-kernel irqchip has been initialized.
 
 KVM will refuse to run the VCPU and KVM_RUN will return an error if the SPE
 feature has been set for the VCPU, but SPE hasn't been initialized.
+
+Attempting to access any of the SPE registers until SPE has been initialized on
+the VCPU will fail with -EBUSY.
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index 56dc9e660bef..3d8e9720daca 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -14,6 +14,7 @@ struct kvm_device_attr;
 struct arm_spe_pmu;
 struct kvm_spe {
 	struct arm_spe_pmu *spe_pmu;
+	u64 pmblimitr_el1_res0_mask;
 };
 
 struct kvm_vcpu_spe {
@@ -26,6 +27,8 @@ bool kvm_supports_spe(void);
 #define vcpu_has_spe(vcpu)					\
 	(vcpu_has_feature(vcpu, KVM_ARM_VCPU_SPE))
 
+bool kvm_vcpu_spe_initialized(struct kvm_vcpu *vcpu);
+
 int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu);
 void kvm_spe_destroy_vm(struct kvm *kvm);
 
@@ -54,6 +57,10 @@ static __always_inline bool kvm_supports_spe(void)
 
 #define vcpu_has_spe(vcpu)	false
 
+static inline bool kvm_vcpu_spe_initialized(struct kvm_vcpu *vcpu)
+{
+	return false;
+}
 static inline int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu)
 {
 	return 0;
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 6a00a44c73ed..8c173a1f48bd 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -60,13 +60,43 @@ bool kvm_supports_spe(void)
 	return !list_empty(&spe_pmus);
 }
 
-int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu)
+bool kvm_vcpu_spe_initialized(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_spe *vcpu_spe = &vcpu->arch.vcpu_spe;
 
-	if (!vcpu_spe->initialized)
+	return vcpu_spe->initialized;
+}
+
+static void kvm_spe_compute_pmblimitr_el1_res0_mask(struct kvm *kvm)
+{
+	u64 res0_mask = GENMASK_ULL(11, 8) | BIT(6) | GENMASK_ULL(4, 3);
+
+	if (!kvm_has_feat(kvm, ID_AA64DFR2_EL1, SPE_nVM, IMP))
+		res0_mask |= PMBLIMITR_EL1_nVM_MASK;
+
+	if (!kvm_has_feat_enum(kvm, ID_AA64MMFR0_EL1, TGRAN4, NI))
+		goto set_mask;
+
+	if (kvm_has_feat(kvm, ID_AA64MMFR0_EL1, TGRAN16, IMP))
+		res0_mask |= GENMASK_ULL(13, 12);
+	else
+		res0_mask |= GENMASK_ULL(15, 12);
+
+set_mask:
+	WRITE_ONCE(kvm->arch.kvm_spe.pmblimitr_el1_res0_mask, res0_mask);
+}
+
+int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_spe *kvm_spe = &kvm->arch.kvm_spe;
+
+	if (!kvm_vcpu_spe_initialized(vcpu))
 		return -EINVAL;
 
+	if (!kvm_spe->pmblimitr_el1_res0_mask)
+		kvm_spe_compute_pmblimitr_el1_res0_mask(kvm);
+
 	return 0;
 }
 
@@ -85,8 +115,12 @@ void kvm_spe_destroy_vm(struct kvm *kvm)
 
 bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val)
 {
+	struct kvm_spe *kvm_spe = &vcpu->kvm->arch.kvm_spe;
+
 	switch (reg) {
 	case PMBLIMITR_EL1:
+		val &= ~kvm_spe->pmblimitr_el1_res0_mask;
+		fallthrough;
 	case PMBSR_EL1:
 	case PMBPTR_EL1:
 		__vcpu_assign_sys_reg(vcpu, reg, val);
@@ -217,7 +251,7 @@ int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 	if (!vcpu_has_spe(vcpu))
 		return -ENODEV;
 
-	if (vcpu_spe->initialized)
+	if (kvm_vcpu_spe_initialized(vcpu))
 		return -EBUSY;
 
 	switch (attr->attr) {
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 89b01b6ac783..4c4bf88c9aa5 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -1563,6 +1563,38 @@ static unsigned int spe_visibility(const struct kvm_vcpu *vcpu,
 	return REG_HIDDEN;
 }
 
+static int set_user_spe_sysreg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
+			       u64 val)
+{
+	int reg = r->reg;
+
+	if (!kvm_vcpu_spe_initialized(vcpu))
+		return -EBUSY;
+
+	if (reg == PMBLIMITR_EL1 || reg == PMBPTR_EL1 || reg == PMBSR_EL1)
+		kvm_spe_write_sysreg(vcpu, reg, val);
+	else
+		__vcpu_assign_sys_reg(vcpu, reg, val);
+
+	return 0;
+}
+
+static int get_user_spe_sysreg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
+			       u64 *val)
+{
+	int reg = r->reg;
+
+	if (!kvm_vcpu_spe_initialized(vcpu))
+		return -EBUSY;
+
+	if (reg == PMBLIMITR_EL1 || reg == PMBPTR_EL1 || reg == PMBSR_EL1)
+		*val = kvm_spe_read_sysreg(vcpu, reg);
+	else
+		*val = __vcpu_sys_reg(vcpu, reg);
+
+	return 0;
+}
+
 static bool access_spe_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
 			   const struct sys_reg_desc *r)
 {
@@ -1610,11 +1642,13 @@ static bool access_spe_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
 
 #define SPE_SYS_REG(name)						\
 	SYS_DESC(SYS_##name), .reg = name, .access = access_spe_reg,	\
-	.reset = reset_val, .val = 0, .visibility = spe_visibility
+	.reset = reset_val, .val = 0, .visibility = spe_visibility,	\
+	.set_user = set_user_spe_sysreg, .get_user = get_user_spe_sysreg
 
 #define SPE_UNTRAPPED_REG(name)						\
 	SYS_DESC(SYS_##name), .reg = name, .access = undef_access,	\
-	.reset = reset_val, .val = 0, .visibility = spe_visibility
+	.reset = reset_val, .val = 0, .visibility = spe_visibility,	\
+	.set_user = set_user_spe_sysreg, .get_user = get_user_spe_sysreg
 
 /* Macro to expand the AMU counter and type registers*/
 #define AMU_AMEVCNTR0_EL0(n) { SYS_DESC(SYS_AMEVCNTR0_EL0(n)), undef_access }
-- 
2.43.0

[RFC PATCH v7 18/28] KVM: arm64: config: Use functions from spe.c to test FEAT_SPE_{FnE,FDS}

[Alexandru Elisei]

KVM's FGU mechanism will trap the registers introduced by FEAT_SPE_FnE and
FEAT_SPE_FDS if the feature is not present for the VM. Move the functions
that check for the presence of these features out of config.c and into
spe.c, since that's where the bulk of SPE virtualization lies.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/include/asm/kvm_spe.h | 11 +++++++++++
 arch/arm64/kvm/config.c          | 30 +++++-------------------------
 arch/arm64/kvm/spe.c             | 22 ++++++++++++++++++++++
 3 files changed, 38 insertions(+), 25 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index 3d8e9720daca..618051dcf59f 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -43,6 +43,9 @@ u8 kvm_spe_get_pmsver(struct kvm *kvm);
 
 bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val);
 u64 kvm_spe_read_sysreg(struct kvm_vcpu *vcpu, int reg);
+
+bool kvm_spe_has_feat_spe_fne(struct kvm *kvm);
+bool kvm_spe_has_feat_spe_fds(struct kvm *kvm);
 #else
 struct kvm_spe {
 };
@@ -99,6 +102,14 @@ static inline u64 kvm_spe_read_sysreg(struct kvm_vcpu *vcpu, int reg)
 {
 	return 0;
 }
+static inline bool kvm_spe_has_feat_spe_fne(struct kvm *kvm)
+{
+	return false;
+}
+static inline bool kvm_spe_has_feat_spe_fds(struct kvm *kvm)
+{
+	return false;
+}
 #endif /* CONFIG_KVM_ARM_SPE */
 
 #endif /* __ARM64_KVM_SPE_H__ */
diff --git a/arch/arm64/kvm/config.c b/arch/arm64/kvm/config.c
index 1053676551af..78e414ed736d 100644
--- a/arch/arm64/kvm/config.c
+++ b/arch/arm64/kvm/config.c
@@ -7,6 +7,7 @@
 #include <linux/kvm_host.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_nested.h>
+#include <asm/kvm_spe.h>
 #include <asm/sysreg.h>
 
 /*
@@ -159,7 +160,6 @@ struct reg_feat_map_desc {
 #define FEAT_GICv3		ID_AA64PFR0_EL1, GIC, IMP
 #define FEAT_LOR		ID_AA64MMFR1_EL1, LO, IMP
 #define FEAT_SPEv1p2		ID_AA64DFR0_EL1, PMSVer, V1P2
-#define FEAT_SPEv1p4		ID_AA64DFR0_EL1, PMSVer, V1P4
 #define FEAT_SPEv1p5		ID_AA64DFR0_EL1, PMSVer, V1P5
 #define FEAT_ATS1A		ID_AA64ISAR2_EL1, ATS1A, IMP
 #define FEAT_SPECRES2		ID_AA64ISAR1_EL1, SPECRES, COSP_RCTX
@@ -292,26 +292,6 @@ static bool feat_sme_smps(struct kvm *kvm)
 		(read_sysreg_s(SYS_SMIDR_EL1) & SMIDR_EL1_SMPS));
 }
 
-static bool feat_spe_fds(struct kvm *kvm)
-{
-	/*
-	 * Revisit this if KVM ever supports SPE -- this really should
-	 * look at the guest's view of PMSIDR_EL1.
-	 */
-	return (kvm_has_feat(kvm, FEAT_SPEv1p4) &&
-		(read_sysreg_s(SYS_PMSIDR_EL1) & PMSIDR_EL1_FDS));
-}
-
-static bool feat_spe_fne(struct kvm *kvm)
-{
-	/*
-	 * Revisit this if KVM ever supports SPE -- this really should
-	 * look at the guest's view of PMSIDR_EL1.
-	 */
-	return (kvm_has_feat(kvm, FEAT_SPEv1p2) &&
-		(read_sysreg_s(SYS_PMSIDR_EL1) & PMSIDR_EL1_FnE));
-}
-
 static bool feat_trbe_mpam(struct kvm *kvm)
 {
 	/*
@@ -547,7 +527,7 @@ static const struct reg_bits_to_feat_map hdfgrtr_feat_map[] = {
 		   HDFGRTR_EL2_PMBPTR_EL1	|
 		   HDFGRTR_EL2_PMBLIMITR_EL1,
 		   FEAT_SPE),
-	NEEDS_FEAT(HDFGRTR_EL2_nPMSNEVFR_EL1, feat_spe_fne),
+	NEEDS_FEAT(HDFGRTR_EL2_nPMSNEVFR_EL1, kvm_spe_has_feat_spe_fne),
 	NEEDS_FEAT(HDFGRTR_EL2_nBRBDATA		|
 		   HDFGRTR_EL2_nBRBCTL		|
 		   HDFGRTR_EL2_nBRBIDR,
@@ -615,7 +595,7 @@ static const struct reg_bits_to_feat_map hdfgwtr_feat_map[] = {
 		   HDFGWTR_EL2_PMBPTR_EL1	|
 		   HDFGWTR_EL2_PMBLIMITR_EL1,
 		   FEAT_SPE),
-	NEEDS_FEAT(HDFGWTR_EL2_nPMSNEVFR_EL1, feat_spe_fne),
+	NEEDS_FEAT(HDFGWTR_EL2_nPMSNEVFR_EL1, kvm_spe_has_feat_spe_fne),
 	NEEDS_FEAT(HDFGWTR_EL2_nBRBDATA		|
 		   HDFGWTR_EL2_nBRBCTL,
 		   FEAT_BRBE),
@@ -851,7 +831,7 @@ static const struct reg_bits_to_feat_map hdfgrtr2_feat_map[] = {
 		   HDFGRTR2_EL2_nPMSSDATA,
 		   FEAT_PMUv3_SS),
 	NEEDS_FEAT(HDFGRTR2_EL2_nPMIAR_EL1, FEAT_SEBEP),
-	NEEDS_FEAT(HDFGRTR2_EL2_nPMSDSFR_EL1, feat_spe_fds),
+	NEEDS_FEAT(HDFGRTR2_EL2_nPMSDSFR_EL1, kvm_spe_has_feat_spe_fds),
 	NEEDS_FEAT(HDFGRTR2_EL2_nPMBMAR_EL1, FEAT_SPE_nVM),
 	NEEDS_FEAT(HDFGRTR2_EL2_nSPMACCESSR_EL1	|
 		   HDFGRTR2_EL2_nSPMCNTEN	|
@@ -884,7 +864,7 @@ static const struct reg_bits_to_feat_map hdfgwtr2_feat_map[] = {
 		   feat_pmuv3p9),
 	NEEDS_FEAT(HDFGWTR2_EL2_nPMSSCR_EL1, FEAT_PMUv3_SS),
 	NEEDS_FEAT(HDFGWTR2_EL2_nPMIAR_EL1, FEAT_SEBEP),
-	NEEDS_FEAT(HDFGWTR2_EL2_nPMSDSFR_EL1, feat_spe_fds),
+	NEEDS_FEAT(HDFGWTR2_EL2_nPMSDSFR_EL1, kvm_spe_has_feat_spe_fds),
 	NEEDS_FEAT(HDFGWTR2_EL2_nPMBMAR_EL1, FEAT_SPE_nVM),
 	NEEDS_FEAT(HDFGWTR2_EL2_nSPMACCESSR_EL1	|
 		   HDFGWTR2_EL2_nSPMCNTEN	|
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 8c173a1f48bd..3b285b45332b 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -113,6 +113,28 @@ void kvm_spe_destroy_vm(struct kvm *kvm)
 	module_put(spe_pmu->pmu.module);
 }
 
+bool kvm_spe_has_feat_spe_fds(struct kvm *kvm)
+{
+	struct arm_spe_pmu *spe_pmu = kvm->arch.kvm_spe.spe_pmu;
+
+	if (!spe_pmu)
+		return false;
+
+	return kvm_has_feat(kvm, ID_AA64DFR0_EL1, PMSVer, V1P4) &&
+	       FIELD_GET(PMSIDR_EL1_FDS, spe_pmu->pmsidr_el1);
+}
+
+bool kvm_spe_has_feat_spe_fne(struct kvm *kvm)
+{
+	struct arm_spe_pmu *spe_pmu = kvm->arch.kvm_spe.spe_pmu;
+
+	if (!spe_pmu)
+		return false;
+
+	return kvm_has_feat(kvm, ID_AA64DFR0_EL1, PMSVer, V1P2) &&
+	       FIELD_GET(PMSIDR_EL1_FnE, spe_pmu->pmsidr_el1);
+}
+
 bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val)
 {
 	struct kvm_spe *kvm_spe = &vcpu->kvm->arch.kvm_spe;
-- 
2.43.0

[RFC PATCH v7 19/28] KVM: arm64: VHE: Context switch SPE state

[Alexandru Elisei]

Save and restore the SPE register state when a VCPU is run. The SPE
resources are shared between the guest and the host (the resources are not
partitioned) so KVM has to save the host state, change the owning
translation regime, and then restore the guest state before entering a
guest. The sequence is performed in reverse when exiting a guest.

Somewhat unexpectedly, the owning regime is not modified in
kvm_arm_setup_mdcr_el2(), where MDCR_EL2 is written in the VHE case. That's
because after kvm_arm_setup_mdcr_el2() there is a window where the VCPU
runs with interrupts enabled, and during this window perf on the host might
install a new event on the physical CPU via an IPI. If the owning regime
were modified in kvm_arm_setup_mdcr_el2(), the buffer would use the guest
EL1 translation tables to write to memory, leading at best to buffer
management events due to faults, at worst to random memory corruption.

Note that now the host now stops profiling KVM in
__kvm_spe_save_host_state_vhe(), as opposed to when the guest starts
executing with PMSCR_EL1.{E1SPE,E0SPE} = {0,0} following the ERET to EL1.

Buffer management event interrupts will be handled later.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/include/asm/kvm_host.h |   1 +
 arch/arm64/include/asm/kvm_hyp.h  |  26 +++-
 arch/arm64/include/asm/kvm_spe.h  |   4 +
 arch/arm64/kvm/arm.c              |   4 +-
 arch/arm64/kvm/hyp/vhe/Makefile   |   1 +
 arch/arm64/kvm/hyp/vhe/debug-sr.c |  18 +++
 arch/arm64/kvm/hyp/vhe/spe-sr.c   | 209 ++++++++++++++++++++++++++++++
 arch/arm64/kvm/hyp/vhe/switch.c   |   9 ++
 arch/arm64/kvm/spe.c              |   9 ++
 9 files changed, 279 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm64/kvm/hyp/vhe/spe-sr.c

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index b92b19229c36..81abfb705aeb 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -796,6 +796,7 @@ struct kvm_host_data {
 		struct kvm_guest_debug_arch regs;
 		/* Statistical profiling extension */
 		u64 pmscr_el1;
+		u64 pmscr_el2;
 		u64 pmblimitr_el1;
 		/* Self-hosted trace */
 		u64 trfcr_el1;
diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h
index 4974492744cc..a98c0d445260 100644
--- a/arch/arm64/include/asm/kvm_hyp.h
+++ b/arch/arm64/include/asm/kvm_hyp.h
@@ -119,7 +119,31 @@ void __debug_switch_to_host(struct kvm_vcpu *vcpu);
 #ifdef __KVM_NVHE_HYPERVISOR__
 void __debug_save_host_buffers_nvhe(struct kvm_vcpu *vcpu);
 void __debug_restore_host_buffers_nvhe(struct kvm_vcpu *vcpu);
-#endif
+#else
+#ifdef CONFIG_KVM_ARM_SPE
+void __kvm_spe_save_host_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt);
+void __kvm_spe_restore_guest_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *guest_ctxt);
+void __kvm_spe_save_guest_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *guest_ctxt);
+void __kvm_spe_restore_host_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt);
+#else
+static inline void __kvm_spe_save_host_state_vhe(struct kvm_vcpu *vcpu,
+						 struct kvm_cpu_context *host_ctxt)
+{
+}
+static inline void __kvm_spe_restore_guest_state_vhe(struct kvm_vcpu *vcpu,
+						     struct kvm_cpu_context *guest_ctxt)
+{
+}
+static inline void __kvm_spe_save_guest_state_vhe(struct kvm_vcpu *vcpu,
+						  struct kvm_cpu_context *guest_ctxt)
+{
+}
+static inline void __kvm_spe_restore_host_state_vhe(struct kvm_vcpu *vcpu,
+						    struct kvm_cpu_context *host_ctxt)
+{
+}
+#endif /* CONFIG_KVM_ARM_SPE */
+#endif /* __KVM_NVHE_HYPERVISOR__ */
 
 u64 __guest_enter(struct kvm_vcpu *vcpu);
 
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index 618051dcf59f..ad0ad6ace93c 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -30,6 +30,7 @@ bool kvm_supports_spe(void);
 bool kvm_vcpu_spe_initialized(struct kvm_vcpu *vcpu);
 
 int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu);
+void kvm_vcpu_put_spe_vhe(struct kvm_vcpu *vcpu);
 void kvm_spe_destroy_vm(struct kvm *kvm);
 
 void kvm_spe_finalize_idregs(struct kvm *kvm);
@@ -68,6 +69,9 @@ static inline int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu)
 {
 	return 0;
 }
+static inline void kvm_vcpu_put_spe_vhe(struct kvm_vcpu *vcpu)
+{
+}
 static inline void kvm_spe_destroy_vm(struct kvm *kvm)
 {
 }
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index c9a026c61ce7..5992efc4f94b 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -804,8 +804,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 
 	kvm_vcpu_put_debug(vcpu);
 	kvm_arch_vcpu_put_fp(vcpu);
-	if (has_vhe())
+	if (has_vhe()) {
+		kvm_vcpu_put_spe_vhe(vcpu);
 		kvm_vcpu_put_vhe(vcpu);
+	}
 	kvm_timer_vcpu_put(vcpu);
 	kvm_vgic_put(vcpu);
 	kvm_vcpu_pmu_restore_host(vcpu);
diff --git a/arch/arm64/kvm/hyp/vhe/Makefile b/arch/arm64/kvm/hyp/vhe/Makefile
index d6b3475145c0..e757d7a24212 100644
--- a/arch/arm64/kvm/hyp/vhe/Makefile
+++ b/arch/arm64/kvm/hyp/vhe/Makefile
@@ -11,3 +11,4 @@ CFLAGS_switch.o += -Wno-override-init
 obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o
 obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
 	 ../hyp-entry.o ../exception.o ../vgic-v5-sr.o
+obj-$(CONFIG_KVM_ARM_SPE) += spe-sr.o
diff --git a/arch/arm64/kvm/hyp/vhe/debug-sr.c b/arch/arm64/kvm/hyp/vhe/debug-sr.c
index 0100339b09e0..7b03f2611259 100644
--- a/arch/arm64/kvm/hyp/vhe/debug-sr.c
+++ b/arch/arm64/kvm/hyp/vhe/debug-sr.c
@@ -13,9 +13,27 @@
 void __debug_switch_to_guest(struct kvm_vcpu *vcpu)
 {
 	__debug_switch_to_guest_common(vcpu);
+	if (vcpu_has_spe(vcpu)) {
+		u64 mdcr_el2 = vcpu->arch.mdcr_el2;
+
+		mdcr_el2 &= ~MDCR_EL2_E2PB;
+		/* Set buffer owner to EL1 and trap the buffer registers. */
+		mdcr_el2 |= FIELD_PREP(MDCR_EL2_E2PB, MDCR_EL2_E2PB_EL1_TRAP);
+
+		/* Do not trap sampling control registers. */
+		mdcr_el2 &= ~MDCR_EL2_TPMS;
+		write_sysreg(mdcr_el2, mdcr_el2);
+		/* Synchronise MDCR_EL2 and HCR_EL2 writes. */
+		isb();
+	}
 }
 
 void __debug_switch_to_host(struct kvm_vcpu *vcpu)
 {
+	if (vcpu_has_spe(vcpu)) {
+		write_sysreg(*host_data_ptr(host_debug_state.mdcr_el2), mdcr_el2);
+		/* Synchronise MDCR_EL2 and HCR_EL2 writes. */
+		isb();
+	}
 	__debug_switch_to_host_common(vcpu);
 }
diff --git a/arch/arm64/kvm/hyp/vhe/spe-sr.c b/arch/arm64/kvm/hyp/vhe/spe-sr.c
new file mode 100644
index 000000000000..978620ad99fa
--- /dev/null
+++ b/arch/arm64/kvm/hyp/vhe/spe-sr.c
@@ -0,0 +1,209 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2021 - ARM Ltd
+ */
+
+#include <linux/kvm_host.h>
+
+#include <asm/kvm_hyp.h>
+#include <asm/kprobes.h>
+#include <asm/kvm_spe.h>
+
+static void __kvm_spe_save_sampling_regs(struct kvm *kvm, struct kvm_cpu_context *ctxt)
+{
+	/*
+	 * This is dependent on the VM features, and not the hardware features,
+	 * even for the host, because (a) guest accesses to PMSNEVFR_EL1 and
+	 * PMSDSFR_EL1 result in an undefined exception in the guest and (b)
+	 * they don't affect the guest unless the guest explicitly enables them,
+	 * and a well-behaved guest shouldn't enable them when the corresponding
+	 * feature is not advertised.
+	 */
+	if (kvm_spe_has_feat_spe_fne(kvm))
+		ctxt_sys_reg(ctxt, PMSNEVFR_EL1) = read_sysreg_s(SYS_PMSNEVFR_EL1);
+	ctxt_sys_reg(ctxt, PMSICR_EL1) = read_sysreg_s(SYS_PMSICR_EL1);
+	ctxt_sys_reg(ctxt, PMSIRR_EL1) = read_sysreg_s(SYS_PMSIRR_EL1);
+	ctxt_sys_reg(ctxt, PMSFCR_EL1) = read_sysreg_s(SYS_PMSFCR_EL1);
+	ctxt_sys_reg(ctxt, PMSEVFR_EL1) = read_sysreg_s(SYS_PMSEVFR_EL1);
+	ctxt_sys_reg(ctxt, PMSLATFR_EL1) = read_sysreg_s(SYS_PMSLATFR_EL1);
+	if (kvm_spe_has_feat_spe_fds(kvm))
+		ctxt_sys_reg(ctxt, PMSDSFR_EL1) = read_sysreg_s(SYS_PMSDSFR_EL1);
+}
+
+static void __kvm_spe_restore_sampling_regs(struct kvm *kvm, struct kvm_cpu_context *ctxt)
+{
+	if (kvm_spe_has_feat_spe_fne(kvm))
+		write_sysreg_s(ctxt_sys_reg(ctxt, PMSNEVFR_EL1), SYS_PMSNEVFR_EL1);
+	write_sysreg_s(ctxt_sys_reg(ctxt, PMSICR_EL1), SYS_PMSICR_EL1);
+	write_sysreg_s(ctxt_sys_reg(ctxt, PMSIRR_EL1), SYS_PMSIRR_EL1);
+	write_sysreg_s(ctxt_sys_reg(ctxt, PMSFCR_EL1), SYS_PMSFCR_EL1);
+	write_sysreg_s(ctxt_sys_reg(ctxt, PMSEVFR_EL1), SYS_PMSEVFR_EL1);
+	write_sysreg_s(ctxt_sys_reg(ctxt, PMSLATFR_EL1), SYS_PMSLATFR_EL1);
+	if (kvm_spe_has_feat_spe_fds(kvm))
+		write_sysreg_s(ctxt_sys_reg(ctxt, PMSDSFR_EL1), SYS_PMSDSFR_EL1);
+}
+
+/*
+ * Before
+ *  - PMSCR_EL2.E2SPE = 0 or 1
+ *  - PMBLIMITR_EL1.E = 0 or 1
+ *  - PMBSR_EL1.S = 0 or 1
+ *
+ * After:
+ *  - PMSCR_EL2.E2SPE = 0
+ *  - PMBLIMITR_EL1.E = 0
+ *  - PMBSR_EL1.S = 0
+ */
+void __kvm_spe_save_host_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt)
+{
+	u64 pmblimitr, pmscr_el2, pmbsr;
+
+	pmscr_el2 = read_sysreg_el2(SYS_PMSCR);
+	if (FIELD_GET(PMSCR_EL1_E1SPE, pmscr_el2)) {
+		write_sysreg_el2(0, SYS_PMSCR);
+		isb();
+	}
+
+	pmblimitr = read_sysreg_s(SYS_PMBLIMITR_EL1);
+	if (FIELD_GET(PMBLIMITR_EL1_E, pmblimitr)) {
+		psb_csync();
+		dsb(nsh);
+		write_sysreg_s(0, SYS_PMBLIMITR_EL1);
+		/* Update PMBPTR_EL1 and PMBSR_EL1. */
+		isb();
+	}
+
+	pmbsr = read_sysreg_s(SYS_PMBSR_EL1);
+	if (FIELD_GET(PMBSR_EL1_S, pmbsr)) {
+		write_sysreg_s(0, SYS_PMBSR_EL1);
+		isb();
+	}
+
+	__kvm_spe_save_sampling_regs(vcpu->kvm, host_ctxt);
+
+	ctxt_sys_reg(host_ctxt, PMBLIMITR_EL1) = pmblimitr;
+	ctxt_sys_reg(host_ctxt, PMBPTR_EL1) = read_sysreg_s(SYS_PMBPTR_EL1);
+	ctxt_sys_reg(host_ctxt, PMBSR_EL1) = pmbsr;
+
+	*host_data_ptr(host_debug_state.pmscr_el2) = pmscr_el2;
+}
+NOKPROBE_SYMBOL(__kvm_spe_save_host_state_vhe);
+
+static bool __kvm_spe_profiling_buffer_enabled(u64 pmblimitr, u64 pmbsr)
+{
+	return FIELD_GET(PMBLIMITR_EL1_E, pmblimitr) && !FIELD_GET(PMBSR_EL1_S, pmbsr);
+}
+
+/*
+ * Before:
+ *  - PMSCR_EL2.E2PE = 0
+ *  - PMBLIMITR_EL1.E = 0
+ *  - PMBSR_EL1.S = 0
+ *
+ * After:
+ *  - PMSCR_EL2.E2SPE = 0
+ *  - PMBLIMITR_EL1.E = 0 or 1
+ *  - PMBSR_EL1.S = 0
+ */
+void __kvm_spe_restore_guest_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *guest_ctxt)
+{
+	u64 pmblimitr, pmbsr;
+
+	pmbsr = ctxt_sys_reg(guest_ctxt, PMBSR_EL1);
+	pmblimitr = ctxt_sys_reg(guest_ctxt, PMBLIMITR_EL1);
+	if (__kvm_spe_profiling_buffer_enabled(pmblimitr, pmbsr)) {
+		write_sysreg_s(ctxt_sys_reg(guest_ctxt, PMBPTR_EL1), SYS_PMBPTR_EL1);
+		isb();
+		write_sysreg_s(pmblimitr, SYS_PMBLIMITR_EL1);
+		isb();
+		/* A buffer management event preserves fields in PMBSR_EL1 */
+		write_sysreg_s(pmbsr, SYS_PMBSR_EL1);
+	}
+
+	__kvm_spe_restore_sampling_regs(vcpu->kvm, guest_ctxt);
+
+	write_sysreg_el1(ctxt_sys_reg(guest_ctxt, PMSCR_EL1), SYS_PMSCR);
+	write_sysreg_el2(0, SYS_PMSCR);
+}
+NOKPROBE_SYMBOL(__kvm_spe_restore_guest_state_vhe);
+
+/*
+ * Before:
+ *  - PMSCR_EL2.E2SPE = 0
+ *  - PMBLIMITR_EL1.E = 0 or 1
+ *  - PMBSR_EL1.S = 0 or 1
+ *
+ * After:
+ *  - PMSCR_EL2.E2SPE = 0
+ *  - PMBLIMITR_EL1.E = 0
+ *  - PMBSR_EL1.S = 0
+ */
+void __kvm_spe_save_guest_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *guest_ctxt)
+{
+	u64 pmblimitr, pmbsr;
+
+	pmblimitr = ctxt_sys_reg(guest_ctxt, PMBLIMITR_EL1);
+	pmbsr = ctxt_sys_reg(guest_ctxt, PMBSR_EL1);
+
+	/*
+	 * Update PMBPTR_EL1 and PMBSR_EL1 only if profiling was enabled
+	 * when the guest state was resumed.
+	 */
+	if (__kvm_spe_profiling_buffer_enabled(pmblimitr, pmbsr)) {
+		psb_csync();
+		dsb(nsh);
+		write_sysreg_s(0, SYS_PMBLIMITR_EL1);
+		/* Advance PMBPTR_EL1. */
+		isb();
+
+		/* Hardware updates to PMBSR_EL1 are not handled, yet. */
+		ctxt_sys_reg(guest_ctxt, PMBPTR_EL1) = read_sysreg_s(SYS_PMBPTR_EL1);
+	}
+
+	__kvm_spe_save_sampling_regs(vcpu->kvm, guest_ctxt);
+
+	ctxt_sys_reg(guest_ctxt, PMSCR_EL1) = read_sysreg_el1(SYS_PMSCR);
+}
+NOKPROBE_SYMBOL(__kvm_spe_save_guest_state_vhe);
+
+/*
+ * Before:
+ *  - PMSCR_EL2.E2SPE = 0
+ *  - PMBLIMITR_EL1.E = 0
+ *  - PMBSR_EL1.S = 0
+ *
+ * After:
+ *  - PMSCR_EL2.E2SPE = 0 or 1
+ *  - PMBLIMITR_EL1.E = 0 or 1
+ *  - PMBSR_EL1.S = 0 or 1
+ */
+void __kvm_spe_restore_host_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt)
+{
+	u64 pmbsr, pmblimitr;
+
+	/* Synchronise MDCR_EL2.{E2PB,TPMS} write. */
+	isb();
+
+	pmbsr = ctxt_sys_reg(host_ctxt, PMBSR_EL1);
+	pmblimitr = ctxt_sys_reg(host_ctxt, PMBLIMITR_EL1);
+
+	write_sysreg_s(pmbsr, SYS_PMBSR_EL1);
+	write_sysreg_s(ctxt_sys_reg(host_ctxt, PMBPTR_EL1), SYS_PMBPTR_EL1);
+	/*
+	 * If the enable bit is set:
+	 *  - If PMBSR_EL1.S = 0, synchronise the write to PMBPTR_EL1.
+	 *  - If PMBSR_EL1.S = 1, synchronise the write to PMBSR_EL1, which
+	 *  disables the buffer.
+	 */
+	if (FIELD_GET(PMBLIMITR_EL1_E, pmblimitr))
+		isb();
+
+	write_sysreg_s(pmblimitr, SYS_PMBLIMITR_EL1);
+	if (__kvm_spe_profiling_buffer_enabled(pmblimitr, pmbsr))
+		isb();
+
+	__kvm_spe_restore_sampling_regs(vcpu->kvm, host_ctxt);
+
+	write_sysreg_el2(*host_data_ptr(host_debug_state.pmscr_el2), SYS_PMSCR);
+}
+NOKPROBE_SYMBOL(__kvm_spe_restore_host_state_vhe);
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index 7875911c0506..a6817a10ae3b 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -613,6 +613,8 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
 	fpsimd_lazy_switch_to_guest(vcpu);
 
 	sysreg_save_host_state_vhe(host_ctxt);
+	if (vcpu_has_spe(vcpu))
+		__kvm_spe_save_host_state_vhe(vcpu, host_ctxt);
 
 	/*
 	 * Note that ARM erratum 1165522 requires us to configure both stage 1
@@ -625,7 +627,10 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
 	__kvm_adjust_pc(vcpu);
 
 	sysreg_restore_guest_state_vhe(guest_ctxt);
+
 	__debug_switch_to_guest(vcpu);
+	if (vcpu_has_spe(vcpu))
+		__kvm_spe_restore_guest_state_vhe(vcpu, guest_ctxt);
 
 	do {
 		/* Jump in the fire! */
@@ -635,12 +640,16 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
 	} while (fixup_guest_exit(vcpu, &exit_code));
 
 	sysreg_save_guest_state_vhe(guest_ctxt);
+	if (vcpu_has_spe(vcpu))
+		__kvm_spe_save_guest_state_vhe(vcpu, guest_ctxt);
 
 	__deactivate_traps(vcpu);
 
 	sysreg_restore_host_state_vhe(host_ctxt);
 
 	__debug_switch_to_host(vcpu);
+	if (vcpu_has_spe(vcpu))
+		__kvm_spe_restore_host_state_vhe(vcpu, host_ctxt);
 
 	/*
 	 * Ensure that all system register writes above have taken effect
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 3b285b45332b..af1e19cb8e9a 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -100,6 +100,15 @@ int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu)
 	return 0;
 }
 
+void kvm_vcpu_put_spe_vhe(struct kvm_vcpu *vcpu)
+{
+	if (!vcpu_has_spe(vcpu) || unlikely(vcpu_on_unsupported_cpu(vcpu)))
+		return;
+
+	/* See kvm_debug_init_vhe() */
+	write_sysreg_el1(0, SYS_PMSCR);
+}
+
 void kvm_spe_destroy_vm(struct kvm *kvm)
 {
 	struct arm_spe_pmu *spe_pmu;
-- 
2.43.0

[RFC PATCH v7 20/28] KVM: arm64: Allow guest SPE physical timestamps only if kernel allows it

[Alexandru Elisei]

The SPE driver allows userspace to use physical timestamps for records only
if the kernel allows it, do the same for a virtual machine.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/include/asm/kvm_spe.h | 1 +
 arch/arm64/kvm/hyp/vhe/spe-sr.c  | 5 +++--
 arch/arm64/kvm/spe.c             | 8 ++++++++
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index ad0ad6ace93c..8a55ea0de4e5 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -15,6 +15,7 @@ struct arm_spe_pmu;
 struct kvm_spe {
 	struct arm_spe_pmu *spe_pmu;
 	u64 pmblimitr_el1_res0_mask;
+	u64 guest_pmscr_el2;
 };
 
 struct kvm_vcpu_spe {
diff --git a/arch/arm64/kvm/hyp/vhe/spe-sr.c b/arch/arm64/kvm/hyp/vhe/spe-sr.c
index 978620ad99fa..3c32a1338cc8 100644
--- a/arch/arm64/kvm/hyp/vhe/spe-sr.c
+++ b/arch/arm64/kvm/hyp/vhe/spe-sr.c
@@ -107,6 +107,7 @@ static bool __kvm_spe_profiling_buffer_enabled(u64 pmblimitr, u64 pmbsr)
  */
 void __kvm_spe_restore_guest_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *guest_ctxt)
 {
+	struct kvm *kvm = vcpu->kvm;
 	u64 pmblimitr, pmbsr;
 
 	pmbsr = ctxt_sys_reg(guest_ctxt, PMBSR_EL1);
@@ -120,10 +121,10 @@ void __kvm_spe_restore_guest_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_con
 		write_sysreg_s(pmbsr, SYS_PMBSR_EL1);
 	}
 
-	__kvm_spe_restore_sampling_regs(vcpu->kvm, guest_ctxt);
+	__kvm_spe_restore_sampling_regs(kvm, guest_ctxt);
 
 	write_sysreg_el1(ctxt_sys_reg(guest_ctxt, PMSCR_EL1), SYS_PMSCR);
-	write_sysreg_el2(0, SYS_PMSCR);
+	write_sysreg_el2(kvm->arch.kvm_spe.guest_pmscr_el2, SYS_PMSCR);
 }
 NOKPROBE_SYMBOL(__kvm_spe_restore_guest_state_vhe);
 
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index af1e19cb8e9a..68b751bdb384 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -3,9 +3,11 @@
  * Copyright (C) 2021 - ARM Ltd
  */
 
+#include <linux/capability.h>
 #include <linux/cpumask.h>
 #include <linux/kvm_host.h>
 #include <linux/perf/arm_spe_pmu.h>
+#include <linux/perf_event.h>
 
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_spe.h>
@@ -97,6 +99,12 @@ int kvm_spe_vcpu_first_run_init(struct kvm_vcpu *vcpu)
 	if (!kvm_spe->pmblimitr_el1_res0_mask)
 		kvm_spe_compute_pmblimitr_el1_res0_mask(kvm);
 
+	if (kvm_vm_has_ran_once(kvm))
+		return 0;
+
+	if (perf_allow_kernel())
+		kvm_spe->guest_pmscr_el2 = PMSCR_EL2_PCT_PHYS;
+
 	return 0;
 }
 
-- 
2.43.0

[RFC PATCH v7 21/28] KVM: arm64: Handle SPE maintenance interrupts

[Alexandru Elisei]

Inject the maintenance interrupt when the guest writes 1 to PMBSR_EL1.S,
and deassert the interrupt when the service bit is cleared.

Re-inject all maintenance interrupts raised by the SPE hardware while
the guest was running.

Save the value of the hardware PMBSR_EL1 register in a separate
variable, instead of updating the VCPU sysreg directly. That's to allow
kvm_spe_sync_hwstate() to discern when the service bit was set by the
hardware, and not by the guest.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/include/asm/kvm_spe.h | 13 ++++++++++
 arch/arm64/kvm/arm.c             |  2 ++
 arch/arm64/kvm/hyp/vhe/spe-sr.c  |  4 +--
 arch/arm64/kvm/spe.c             | 43 ++++++++++++++++++++++++++++++++
 arch/arm64/kvm/sys_regs.c        |  7 +++++-
 5 files changed, 66 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index 8a55ea0de4e5..385b1157e578 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -19,8 +19,10 @@ struct kvm_spe {
 };
 
 struct kvm_vcpu_spe {
+	u64 hw_pmbsr_el1;	/* Updated on hardware management event */
 	int irq_num;		/* Buffer management interrupt number */
 	bool initialized;	/* SPE initialized for the VCPU */
+	bool irq_level;		/* Virtual buffer management interrupt level */
 };
 
 bool kvm_supports_spe(void);
@@ -43,11 +45,15 @@ int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 bool kvm_spe_pmu_set(struct kvm *kvm);
 u8 kvm_spe_get_pmsver(struct kvm *kvm);
 
+u64 kvm_spe_reset_sysreg(struct kvm_vcpu *vcpu, int reg);
+
 bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val);
 u64 kvm_spe_read_sysreg(struct kvm_vcpu *vcpu, int reg);
 
 bool kvm_spe_has_feat_spe_fne(struct kvm *kvm);
 bool kvm_spe_has_feat_spe_fds(struct kvm *kvm);
+
+void kvm_spe_sync_hwstate(struct kvm_vcpu *vcpu);
 #else
 struct kvm_spe {
 };
@@ -99,6 +105,10 @@ static inline u8 kvm_spe_get_pmsver(struct kvm *kvm)
 {
 	return 0;
 }
+static inline u64 kvm_spe_reset_sysreg(struct kvm_vcpu *vcpu, int reg)
+{
+	return 0;
+}
 static inline bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val)
 {
 	return true;
@@ -115,6 +125,9 @@ static inline bool kvm_spe_has_feat_spe_fds(struct kvm *kvm)
 {
 	return false;
 }
+static inline void kvm_spe_sync_hwstate(struct kvm_vcpu *vcpu)
+{
+}
 #endif /* CONFIG_KVM_ARM_SPE */
 
 #endif /* __ARM64_KVM_SPE_H__ */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 5992efc4f94b..689fa175f5f4 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1421,6 +1421,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		if (kvm_vcpu_has_pmu(vcpu))
 			kvm_pmu_sync_hwstate(vcpu);
 
+		kvm_spe_sync_hwstate(vcpu);
+
 		/*
 		 * Sync the vgic state before syncing the timer state because
 		 * the timer code needs to know if the virtual timer
diff --git a/arch/arm64/kvm/hyp/vhe/spe-sr.c b/arch/arm64/kvm/hyp/vhe/spe-sr.c
index 3c32a1338cc8..df35025db19a 100644
--- a/arch/arm64/kvm/hyp/vhe/spe-sr.c
+++ b/arch/arm64/kvm/hyp/vhe/spe-sr.c
@@ -154,10 +154,10 @@ void __kvm_spe_save_guest_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_contex
 		psb_csync();
 		dsb(nsh);
 		write_sysreg_s(0, SYS_PMBLIMITR_EL1);
-		/* Advance PMBPTR_EL1. */
+		/* Advance PMBPTR_EL1 and PMBSR_EL1. */
 		isb();
 
-		/* Hardware updates to PMBSR_EL1 are not handled, yet. */
+		vcpu->arch.vcpu_spe.hw_pmbsr_el1 = read_sysreg_s(SYS_PMBSR_EL1);
 		ctxt_sys_reg(guest_ctxt, PMBPTR_EL1) = read_sysreg_s(SYS_PMBPTR_EL1);
 	}
 
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 68b751bdb384..89ebeef1c223 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -21,6 +21,8 @@ struct spe_pmu_entry {
 	struct arm_spe_pmu *spe_pmu;
 };
 
+static void kvm_spe_update_irq_level(struct kvm_vcpu *vcpu, bool level);
+
 void kvm_spe_add_instance(struct arm_spe_pmu *spe_pmu)
 {
 	struct spe_pmu_entry *entry;
@@ -152,6 +154,14 @@ bool kvm_spe_has_feat_spe_fne(struct kvm *kvm)
 	       FIELD_GET(PMSIDR_EL1_FnE, spe_pmu->pmsidr_el1);
 }
 
+u64 kvm_spe_reset_sysreg(struct kvm_vcpu *vcpu, int reg)
+{
+	if (reg == PMBSR_EL1 && kvm_vcpu_spe_initialized(vcpu))
+		kvm_spe_update_irq_level(vcpu, false);
+
+	return 0;
+}
+
 bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val)
 {
 	struct kvm_spe *kvm_spe = &vcpu->kvm->arch.kvm_spe;
@@ -168,6 +178,11 @@ bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val)
 		WARN_ON_ONCE("unexpected trap");
 	}
 
+	if (reg == PMBSR_EL1) {
+		val = __vcpu_sys_reg(vcpu, PMBSR_EL1);
+		kvm_spe_update_irq_level(vcpu, FIELD_GET(PMBSR_EL1_S, val));
+	}
+
 	return true;
 }
 
@@ -184,6 +199,34 @@ u64 kvm_spe_read_sysreg(struct kvm_vcpu *vcpu, int reg)
 	}
 }
 
+static void kvm_spe_update_irq_level(struct kvm_vcpu *vcpu, bool level)
+{
+	struct kvm_vcpu_spe *vcpu_spe = &vcpu->arch.vcpu_spe;
+	int ret;
+
+	if (vcpu_spe->irq_level == level)
+		return;
+
+	ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu, vcpu_spe->irq_num, level, vcpu_spe);
+	if (!WARN_ON_ONCE(ret))
+		vcpu_spe->irq_level = level;
+}
+
+void kvm_spe_sync_hwstate(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_spe *vcpu_spe = &vcpu->arch.vcpu_spe;
+
+	if (!vcpu_has_spe(vcpu))
+		return;
+
+	if (!FIELD_GET(PMBSR_EL1_S, vcpu_spe->hw_pmbsr_el1))
+		return;
+
+	__vcpu_assign_sys_reg(vcpu, PMBSR_EL1, vcpu_spe->hw_pmbsr_el1);
+	vcpu_spe->hw_pmbsr_el1 = 0;
+	kvm_spe_update_irq_level(vcpu, true);
+}
+
 bool kvm_spe_pmu_set(struct kvm *kvm)
 {
 	return !!kvm->arch.kvm_spe.spe_pmu;
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 4c4bf88c9aa5..ebe1c933f8d5 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -1595,6 +1595,11 @@ static int get_user_spe_sysreg(struct kvm_vcpu *vcpu, const struct sys_reg_desc
 	return 0;
 }
 
+static u64 reset_spe_sysreg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	return kvm_spe_reset_sysreg(vcpu, r->reg);
+}
+
 static bool access_spe_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
 			   const struct sys_reg_desc *r)
 {
@@ -1642,7 +1647,7 @@ static bool access_spe_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
 
 #define SPE_SYS_REG(name)						\
 	SYS_DESC(SYS_##name), .reg = name, .access = access_spe_reg,	\
-	.reset = reset_val, .val = 0, .visibility = spe_visibility,	\
+	.reset = reset_spe_sysreg, .visibility = spe_visibility,	\
 	.set_user = set_user_spe_sysreg, .get_user = get_user_spe_sysreg
 
 #define SPE_UNTRAPPED_REG(name)						\
-- 
2.43.0

[RFC PATCH v7 22/28] arm64: errata: Disable SPE in KVM

[Alexandru Elisei]

When stage 1 is enabled and buffer virtual address VA is mapped as
read-only, and hardware dirty bit management is disabled at stage 1 and
stage 2, if stage 2 is enabled, SPE might write to address VA instead of
generating a fault.

This doesn't affect the SPE driver when running on baremetal, since the
memory is mapped at stage 1 with write permissions.

KVM does not use hardware dirty bit management and this erratum allows a
guest to write to read-only memory as long as that address is mapped at
stage 2.

Since KVM allows userspace to use SPE on heterogenous systems, where there
might be different SPE instances, tie the workaround to a particular SPE
instance and do not allow userspace to use that instance when it is
affected.

Several parts are affected:

* ARM Neoverse-N1 (MP050), SDEN v35, erratum 3023823
  https://support.arm.com/documentation/SDEN-885747/35-0
* ARM Neoverse-N2 (MP128), SDEN v21, erratum 3031178
  https://support.arm.com/documentation/SDEN-1982442/21-0
* ARM Neoverse-V1 (MP076), SDEN v23, erratum 3028884
  https://support.arm.com/documentation/SDEN-1401781/23-0
* ARM Neoverse-V2 (MP158), SDEN v13, erratum 3031173
  https://support.arm.com/documentation/SDEN-2332927/13-0
* ARM Cortex-A78 (MP102), SDEN v25, erratum 3031174
  https://support.arm.com/documentation/SDEN-1401784/25-0
* ARM Cortex-A78C:
    * MP138: SDEN v20, erratum 3031176
      https://support.arm.com/documentation/SDEN-1707916/20-0
    * MP154: SDEN v14, erratum 3031177
      https://support.arm.com/documentation/SDEN-2004089/14-0
* ARM Cortex-A78AE (MP105), SDEN v22, erratum 3031175
  https://support.arm.com/documentation/SDEN-1707912/22-0
* ARM Cortex-X1 (MP077), SDEN v25, erratum 3031174
  https://support.arm.com/documentation/SDEN-1401782/25-0
* ARM Cortex-X1C (MP136), SDEN v20, erratum 3031176
  https://support.arm.com/documentation/SDEN-1707914/20-0
* ARM Cortex-X3 (MP141), SDEN v18, erratum 3022726
  https://support.arm.com/documentation/SDEN-2055130/18-0
* ARM Cortex-X4 (MP161), SDEN v13, erratum 3022725
  https://support.arm.com/documentation/109148/13-0

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/arch/arm64/silicon-errata.rst | 23 ++++++++++++++++
 arch/arm64/Kconfig                          | 30 +++++++++++++++++++++
 arch/arm64/kernel/cpu_errata.c              | 25 +++++++++++++++++
 arch/arm64/kvm/Kconfig                      |  1 +
 arch/arm64/kvm/spe.c                        |  3 +++
 arch/arm64/tools/cpucaps                    |  1 +
 drivers/perf/arm_spe_pmu.c                  |  3 +++
 include/linux/perf/arm_spe_pmu.h            |  1 +
 8 files changed, 87 insertions(+)

diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst
index ac3248b9f2f3..a29f8fbc16a8 100644
--- a/Documentation/arch/arm64/silicon-errata.rst
+++ b/Documentation/arch/arm64/silicon-errata.rst
@@ -149,13 +149,20 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A78      | #4193791        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-A78      | #3031174        | ARM64_ERRATUM_3023823       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A78AE    | #4193793        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-A78AE    | #3031175        | ARM64_ERRATUM_3023823       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A78C     | #3324346,       | ARM64_ERRATUM_3194386       |
 |                |                 | #3324347        |                             |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A78C     | #4193794        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-A78C     | #3031176,       | ARM64_ERRATUM_3023823       |
+|                |                 | #3031177        |                             |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A710     | #2119858        | ARM64_ERRATUM_2119858       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A710     | #2054223        | ARM64_ERRATUM_2054223       |
@@ -180,10 +187,14 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X1       | #4193791        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-X1       | #3031174        | ARM64_ERRATUM_3023823       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X1C      | #3324346        | ARM64_ERRATUM_3194386       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X1C      | #4193792        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-X1C      | #3031176        | ARM64_ERRATUM_3023823       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X2       | #2119858        | ARM64_ERRATUM_2119858       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X2       | #2224489        | ARM64_ERRATUM_2224489       |
@@ -196,10 +207,14 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X3       | #4193786        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-X3       | #3022726        | ARM64_ERRATUM_3023823       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X4       | #3194386        | ARM64_ERRATUM_3194386       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X4       | #4118414        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-X4       | #3022725        | ARM64_ERRATUM_3023823       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X925     | #3324334        | ARM64_ERRATUM_3194386       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X925     | #4193781        | ARM64_ERRATUM_4118414       |
@@ -216,6 +231,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-N1     | #4193800        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Neoverse-N1     | #3023823        | ARM64_ERRATUM_3023823       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-N2     | #2139208        | ARM64_ERRATUM_2139208       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-N2     | #2067961        | ARM64_ERRATUM_2067961       |
@@ -226,6 +243,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-N2     | #4193789        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Neoverse-N2     | #3031178        | ARM64_ERRATUM_3023823       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-N3     | #3456111        | ARM64_ERRATUM_3194386       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-V1     | #1619801        | N/A                         |
@@ -234,10 +253,14 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-V1     | #4193790        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Neoverse-V1     | #3028884        | ARM64_ERRATUM_3023823       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-V2     | #3324336        | ARM64_ERRATUM_3194386       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-V2     | #4193787        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Neoverse-V2     | #3031173        | ARM64_ERRATUM_3023823       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-V3     | #3312417        | ARM64_ERRATUM_3194386       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-V3     | #4193784        | ARM64_ERRATUM_4118414       |
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b5a51b0ef944..ca6759d16d27 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1464,6 +1464,36 @@ config SOCIONEXT_SYNQUACER_PREITS
 
 	  If unsure, say Y.
 
+config ARM64_ERRATUM_3023823
+	bool "SPE might write to pages which lack write permission at Stage-1 or Stage-2"
+	depends on KVM_ARM_SPE
+	default y
+	help
+	  This option adds a workaround for the following errata:
+
+	  * ARM Neoverse-N1 erratum 3023823
+	  * ARM Neoverse-N2 erratum 3031178
+	  * ARM Neoverse-V1 erratum 3028884
+	  * ARM Neoverse-V2 erratum 3031173
+	  * ARM Cortex-A78 erratum 3031174
+	  * ARM Cortex-A78C errata 3031176 and 3031177
+	  * ARM Cortex-A78AE erratum 3031175
+	  * ARM Cortex-X1 erratum 3031174
+	  * ARM Cortex-X1C erratum 3031176
+	  * ARM Cortex-X3 erratum 3022726
+	  * ARM Cortex-X4 erratum 3022725
+
+	  When stage 1 is enabled, and stage 2 is either enabled or disabled,
+	  and buffer virtual address VA is mapped as read-only, and hardware
+	  dirty bit management is disabled at stage 1, and stage 2 if stage 2
+	  is enabled, SPE might write to address VA instead of generating a
+	  fault.
+
+	  The work around is to disable SPE for a virtual machine. The host
+	  driver is unaffected.
+
+	  If unsure, say Y.
+
 endmenu # "ARM errata workarounds via the alternatives framework"
 
 choice
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 5db8f0619e4b..ab0558fbd5b2 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -630,6 +630,24 @@ static void cpu_enable_sme_dvmsync(const struct arm64_cpu_capabilities *__unused
 }
 #endif
 
+#ifdef CONFIG_ARM64_ERRATUM_3023823
+static const struct midr_range erratum_3023823_list[] = {
+	MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N1),
+	MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N2),
+	MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V1),
+	MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V2),
+	MIDR_ALL_VERSIONS(MIDR_CORTEX_A78),
+	MIDR_ALL_VERSIONS(MIDR_CORTEX_A78C),
+	MIDR_ALL_VERSIONS(MIDR_CORTEX_A78AE),
+	MIDR_ALL_VERSIONS(MIDR_CORTEX_X1),
+	MIDR_ALL_VERSIONS(MIDR_CORTEX_X1C),
+	MIDR_ALL_VERSIONS(MIDR_CORTEX_X3),
+	/* Cortex-X4 r0p0 to r0p1 */
+	MIDR_REV_RANGE(MIDR_CORTEX_X4, 0, 0, 1),
+	{},
+};
+#endif
+
 #ifdef CONFIG_AMPERE_ERRATUM_AC03_CPU_38
 static const struct midr_range erratum_ac03_cpu_38_list[] = {
 	MIDR_ALL_VERSIONS(MIDR_AMPERE1),
@@ -987,6 +1005,13 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
 		MIDR_FIXED(MIDR_CPU_VAR_REV(1, 2), BIT(0)),
 	},
 #endif
+#ifdef CONFIG_ARM64_ERRATUM_3023823
+	{
+		.desc = "SPE might write to read-only pages",
+		.capability = ARM64_WORKAROUND_3023823,
+		ERRATA_MIDR_RANGE_LIST(erratum_3023823_list),
+	},
+#endif
 #ifdef CONFIG_ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD
 	{
 		.desc = "ARM errata 2966298, 3117295",
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index fece4c02b97c..2bc8e34c538d 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -104,6 +104,7 @@ config PKVM_STACKTRACE
 
 endif # NVHE_EL2_DEBUG
 
+
 config KVM_ARM_SPE
 	bool
 	depends on KVM && ARM_SPE_PMU
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 89ebeef1c223..ce68789d184b 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -290,6 +290,9 @@ static int kvm_spe_set_spe_id(struct kvm_vcpu *vcpu, int spe_id)
 		return 0;
 	}
 
+	if (spe_pmu->has_workaround_3023823)
+		return -EINVAL;
+
 	if (!try_module_get(spe_pmu->pmu.module))
 		return -ENXIO;
 
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 2775ba3359cf..5f997851b6d7 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -132,3 +132,4 @@ WORKAROUND_REPEAT_TLBI_SYNC
 WORKAROUND_SPECULATIVE_AT
 WORKAROUND_SPECULATIVE_SSBS
 WORKAROUND_SPECULATIVE_UNPRIV_LOAD
+WORKAROUND_3023823
diff --git a/drivers/perf/arm_spe_pmu.c b/drivers/perf/arm_spe_pmu.c
index f24b122a087c..928ea94f2ed6 100644
--- a/drivers/perf/arm_spe_pmu.c
+++ b/drivers/perf/arm_spe_pmu.c
@@ -1198,6 +1198,9 @@ static void __arm_spe_pmu_dev_probe(void *info)
 	write_sysreg_s(U64_MAX, SYS_PMSEVFR_EL1);
 	spe_pmu->pmsevfr_res0 = ~read_sysreg_s(SYS_PMSEVFR_EL1);
 
+	if (this_cpu_has_cap(ARM64_WORKAROUND_3023823))
+		spe_pmu->has_workaround_3023823 = true;
+
 	dev_info(dev,
 		 "probed SPEv1.%d for CPUs %*pbl [max_record_sz %u, align %u, features 0x%llx]\n",
 		 spe_pmu->pmsver - 1, cpumask_pr_args(&spe_pmu->supported_cpus),
diff --git a/include/linux/perf/arm_spe_pmu.h b/include/linux/perf/arm_spe_pmu.h
index bdb703558a87..edf58c51fb94 100644
--- a/include/linux/perf/arm_spe_pmu.h
+++ b/include/linux/perf/arm_spe_pmu.h
@@ -45,6 +45,7 @@ struct arm_spe_pmu {
 	u16					max_record_sz;
 	u16					align;
 	struct perf_output_handle __percpu	*handle;
+	bool					has_workaround_3023823;
 };
 
 #define to_spe_pmu(p) (container_of(p, struct arm_spe_pmu, pmu))
-- 
2.43.0

[RFC PATCH v7 23/28] KVM: arm64: Add kvm-arm.ignore_spe_errata kernel parameter

[Alexandru Elisei]

Add a new kernel command line parameter, kvm-arm.ignore_spe_errata with the
effect of ignoring any errata that might inhibit userspace from creating a
SPE enabled virtual machine.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/admin-guide/kernel-parameters.txt |  3 +++
 arch/arm64/kvm/spe.c                            | 10 +++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 68647ff4bdd2..2d91ce8e8c4b 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3299,6 +3299,9 @@ Kernel parameters
 
 			notrap: clear WFI instruction trap
 
+	kvm-arm.ignore_spe_errata=
+			[KVM,ARM] Ignore SPE errata and expose SPE to guests.
+
 	kvm_cma_resv_ratio=n [PPC,EARLY]
 			Reserves given percentage from system memory area for
 			contiguous memory allocation for KVM hash pagetable
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index ce68789d184b..3e82edc8be85 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -21,8 +21,16 @@ struct spe_pmu_entry {
 	struct arm_spe_pmu *spe_pmu;
 };
 
+static bool ignore_spe_errata;
+
 static void kvm_spe_update_irq_level(struct kvm_vcpu *vcpu, bool level);
 
+static int __init kvm_ignore_spe_errata(char *arg)
+{
+	return kstrtobool(arg, &ignore_spe_errata);
+}
+early_param("kvm-arm.ignore_spe_errata", kvm_ignore_spe_errata);
+
 void kvm_spe_add_instance(struct arm_spe_pmu *spe_pmu)
 {
 	struct spe_pmu_entry *entry;
@@ -290,7 +298,7 @@ static int kvm_spe_set_spe_id(struct kvm_vcpu *vcpu, int spe_id)
 		return 0;
 	}
 
-	if (spe_pmu->has_workaround_3023823)
+	if (spe_pmu->has_workaround_3023823 && !ignore_spe_errata)
 		return -EINVAL;
 
 	if (!try_module_get(spe_pmu->pmu.module))
-- 
2.43.0

[RFC PATCH v7 24/28] arm64: errata: Don't enable guest buffer if misprogrammed

[Alexandru Elisei]

When PMBPTR_EL1 is programmed to be equal to or greater than
PMBLIMITR_EL1.Limit, SPE is enabled and PMBSR_EL1.S is zero, the CPU might
deadlock.

This doesn't affect correctly written software, like the SPE driver, but a
buggy KVM guest might trigger the erratum. The workaround for KVM is not to
enable the hardware buffer, and, as a consequence, profiling. According
to ARM DDI0487M.c, rule R_PGGHS, this is one of the acceptable
behaviours when the buffer is misprogrammed.

Several parts are affected:

* ARM Neoverse-N1 (MP050), SDEN v35, erratum 1978083
  https://support.arm.com/documentation/SDEN-885747/35-0
* ARM Neoverse-V1 (MP076), SDEN v23, erratum 1978082
  https://support.arm.com/documentation/SDEN-1401781/23-0
* ARM Cortex-X1 (MP077), SDEN v25, erratum 2004055
  https://support.arm.com/documentation/SDEN-1401782/25-0
* ARM Cortex-X1C (MP136), SDEN v20, erratum 2004056
  https://support.arm.com/documentation/SDEN-1707914/20-0
* ARM Cortex-A78 (MP102), SDEN v25, erratum 2004055
  https://support.arm.com/documentation/SDEN-1401784/25-0
* ARM Cortex-A78AE (MP105), SDEN v22, erratum 2004059
  https://support.arm.com/documentation/SDEN-1707912/22-0
* ARM Cortex-A78C (MP138), SDEN v20, erratum 2004056
  https://support.arm.com/documentation/SDEN-1707916/20-0

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/arch/arm64/silicon-errata.rst | 14 +++++++++++
 arch/arm64/Kconfig                          | 25 ++++++++++++++++++++
 arch/arm64/include/asm/kvm_spe.h            |  1 +
 arch/arm64/kernel/cpu_errata.c              | 26 +++++++++++++++++++++
 arch/arm64/kvm/hyp/vhe/spe-sr.c             |  6 +++--
 arch/arm64/kvm/spe.c                        | 22 +++++++++++++++++
 arch/arm64/tools/cpucaps                    |  1 +
 drivers/perf/arm_spe_pmu.c                  |  3 +++
 include/linux/perf/arm_spe_pmu.h            |  1 +
 9 files changed, 97 insertions(+), 2 deletions(-)

diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst
index a29f8fbc16a8..a5477cae7d70 100644
--- a/Documentation/arch/arm64/silicon-errata.rst
+++ b/Documentation/arch/arm64/silicon-errata.rst
@@ -151,10 +151,14 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A78      | #3031174        | ARM64_ERRATUM_3023823       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-A78      | #2004055        | ARM64_ERRATUM_2004055       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A78AE    | #4193793        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A78AE    | #3031175        | ARM64_ERRATUM_3023823       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-A78AE    | #2004059        | ARM64_ERRATUM_2004055       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A78C     | #3324346,       | ARM64_ERRATUM_3194386       |
 |                |                 | #3324347        |                             |
 +----------------+-----------------+-----------------+-----------------------------+
@@ -163,6 +167,8 @@ stable kernels.
 | ARM            | Cortex-A78C     | #3031176,       | ARM64_ERRATUM_3023823       |
 |                |                 | #3031177        |                             |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-A78C     | #2004056        | ARM64_ERRATUM_2004055       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A710     | #2119858        | ARM64_ERRATUM_2119858       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A710     | #2054223        | ARM64_ERRATUM_2054223       |
@@ -189,12 +195,16 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X1       | #3031174        | ARM64_ERRATUM_3023823       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-X1       | #2004055        | ARM64_ERRATUM_2004055       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X1C      | #3324346        | ARM64_ERRATUM_3194386       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X1C      | #4193792        | ARM64_ERRATUM_4118414       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X1C      | #3031176        | ARM64_ERRATUM_3023823       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-X1C      | #2004056        | ARM64_ERRATUM_2004055       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X2       | #2119858        | ARM64_ERRATUM_2119858       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-X2       | #2224489        | ARM64_ERRATUM_2224489       |
@@ -233,6 +243,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-N1     | #3023823        | ARM64_ERRATUM_3023823       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Neoverse-N1     | #1978083        | ARM64_ERRATUM_2004055       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-N2     | #2139208        | ARM64_ERRATUM_2139208       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-N2     | #2067961        | ARM64_ERRATUM_2067961       |
@@ -255,6 +267,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-V1     | #3028884        | ARM64_ERRATUM_3023823       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Neoverse-V1     | #1978082        | ARM64_ERRATUM_2004055       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-V2     | #3324336        | ARM64_ERRATUM_3194386       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-V2     | #4193787        | ARM64_ERRATUM_4118414       |
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index ca6759d16d27..d9b27ee90ab5 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1494,6 +1494,31 @@ config ARM64_ERRATUM_3023823
 
 	  If unsure, say Y.
 
+config ARM64_ERRATUM_2004055
+	bool "Incorrect programming of PMBPTR_EL1 might result in a deadlock"
+	depends on KVM_ARM_SPE
+	default y
+	help
+	  This option adds a workaround for the following errata:
+
+	  * ARM Neoverse-N1 erratum 1978083
+	  * ARM Neoverse-V1 erratum 1978082
+	  * ARM Cortex-X1 erratum 2004055
+	  * ARM Cortex-X1C erratum 2004056
+	  * ARM Cortex-A78 erratum 2004055
+	  * ARM Cortex-A78AE erratum 2004059
+	  * ARM Cortex-A78C errata 2004056
+
+	  When PMBPTR_EL1 is programmed to be equal to or greater than
+	  PMBLIMITR_EL1.Limit, SPE is enabled and PMBSR_EL1.S is zero, the CPU
+	  might deadlock.
+
+	  The work around is for KVM to behave as if profiling is disabled when
+	  the buffer is enabled, PMSBSR_EL1.S is zero and
+	  PMBPTR_EL1.PTR >= PMBLIMITR_EL1.Limit.
+
+	  If unsure, say Y.
+
 endmenu # "ARM errata workarounds via the alternatives framework"
 
 choice
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index 385b1157e578..88f8b1bb6864 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -23,6 +23,7 @@ struct kvm_vcpu_spe {
 	int irq_num;		/* Buffer management interrupt number */
 	bool initialized;	/* SPE initialized for the VCPU */
 	bool irq_level;		/* Virtual buffer management interrupt level */
+	bool buffer_misprogrammed;
 };
 
 bool kvm_supports_spe(void);
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index ab0558fbd5b2..0b9620893763 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -648,6 +648,25 @@ static const struct midr_range erratum_3023823_list[] = {
 };
 #endif
 
+#ifdef CONFIG_ARM64_ERRATUM_2004055
+static const struct midr_range erratum_2004055_list[] = {
+	MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N1),
+	/* Neoverse-V1 r0p0 to r1p1 */
+	MIDR_RANGE(MIDR_NEOVERSE_V1, 0, 0, 1, 1),
+	/* Cortex-X1 r0p0 to r1p1 */
+	MIDR_RANGE(MIDR_CORTEX_X1, 0, 0, 1, 1),
+	/* Cortex-X1 r0p0 */
+	MIDR_REV(MIDR_CORTEX_X1C, 0, 0),
+	/* Cortex-A78 r0p0 to r1p1 */
+	MIDR_RANGE(MIDR_CORTEX_A78, 0, 0, 1, 1),
+	/* Cortex-A78 r0p0 to r1p1 */
+	MIDR_RANGE(MIDR_CORTEX_A78AE, 0, 0, 0, 1),
+	/* Cortex-A78C r0p0 */
+	MIDR_REV(MIDR_CORTEX_A78C, 0, 0),
+	{},
+};
+#endif
+
 #ifdef CONFIG_AMPERE_ERRATUM_AC03_CPU_38
 static const struct midr_range erratum_ac03_cpu_38_list[] = {
 	MIDR_ALL_VERSIONS(MIDR_AMPERE1),
@@ -1012,6 +1031,13 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
 		ERRATA_MIDR_RANGE_LIST(erratum_3023823_list),
 	},
 #endif
+#ifdef CONFIG_ARM64_ERRATUM_2004055
+	{
+		.desc = "Incorrect programming of PMBPTR_EL1 leads to deadlock",
+		.capability = ARM64_WORKAROUND_2004055,
+		ERRATA_MIDR_RANGE_LIST(erratum_2004055_list),
+	},
+#endif
 #ifdef CONFIG_ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD
 	{
 		.desc = "ARM errata 2966298, 3117295",
diff --git a/arch/arm64/kvm/hyp/vhe/spe-sr.c b/arch/arm64/kvm/hyp/vhe/spe-sr.c
index df35025db19a..ec991f4cc348 100644
--- a/arch/arm64/kvm/hyp/vhe/spe-sr.c
+++ b/arch/arm64/kvm/hyp/vhe/spe-sr.c
@@ -112,7 +112,8 @@ void __kvm_spe_restore_guest_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_con
 
 	pmbsr = ctxt_sys_reg(guest_ctxt, PMBSR_EL1);
 	pmblimitr = ctxt_sys_reg(guest_ctxt, PMBLIMITR_EL1);
-	if (__kvm_spe_profiling_buffer_enabled(pmblimitr, pmbsr)) {
+	if (__kvm_spe_profiling_buffer_enabled(pmblimitr, pmbsr) &&
+	    !vcpu->arch.vcpu_spe.buffer_misprogrammed) {
 		write_sysreg_s(ctxt_sys_reg(guest_ctxt, PMBPTR_EL1), SYS_PMBPTR_EL1);
 		isb();
 		write_sysreg_s(pmblimitr, SYS_PMBLIMITR_EL1);
@@ -150,7 +151,8 @@ void __kvm_spe_save_guest_state_vhe(struct kvm_vcpu *vcpu, struct kvm_cpu_contex
 	 * Update PMBPTR_EL1 and PMBSR_EL1 only if profiling was enabled
 	 * when the guest state was resumed.
 	 */
-	if (__kvm_spe_profiling_buffer_enabled(pmblimitr, pmbsr)) {
+	if (__kvm_spe_profiling_buffer_enabled(pmblimitr, pmbsr) &&
+	    !vcpu->arch.vcpu_spe.buffer_misprogrammed) {
 		psb_csync();
 		dsb(nsh);
 		write_sysreg_s(0, SYS_PMBLIMITR_EL1);
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index 3e82edc8be85..ef138562f939 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -170,6 +170,25 @@ u64 kvm_spe_reset_sysreg(struct kvm_vcpu *vcpu, int reg)
 	return 0;
 }
 
+static void kvm_spe_apply_workaround_2004055(struct kvm_vcpu *vcpu)
+{
+	u64 pmblimitr_el1 = __vcpu_sys_reg(vcpu, PMBLIMITR_EL1);
+	u64 pmbsr_el1 = __vcpu_sys_reg(vcpu, PMBSR_EL1);
+	u64 ptr, limit;
+
+	if (FIELD_GET(PMBSR_EL1_S, pmbsr_el1) ||
+	    !FIELD_GET(PMBLIMITR_EL1_E, pmblimitr_el1))
+		return;
+
+	ptr = FIELD_GET(PMBPTR_EL1_PTR, __vcpu_sys_reg(vcpu, PMBPTR_EL1));
+	limit = FIELD_GET(PMBLIMITR_EL1_LIMIT, pmblimitr_el1) << 12;
+
+	if (ptr >= limit)
+		vcpu->arch.vcpu_spe.buffer_misprogrammed = true;
+	else
+		vcpu->arch.vcpu_spe.buffer_misprogrammed = false;
+}
+
 bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val)
 {
 	struct kvm_spe *kvm_spe = &vcpu->kvm->arch.kvm_spe;
@@ -186,6 +205,9 @@ bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val)
 		WARN_ON_ONCE("unexpected trap");
 	}
 
+	if (unlikely(kvm_spe->spe_pmu->has_workaround_2004055))
+		kvm_spe_apply_workaround_2004055(vcpu);
+
 	if (reg == PMBSR_EL1) {
 		val = __vcpu_sys_reg(vcpu, PMBSR_EL1);
 		kvm_spe_update_irq_level(vcpu, FIELD_GET(PMBSR_EL1_S, val));
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 5f997851b6d7..e001a6ea5564 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -133,3 +133,4 @@ WORKAROUND_SPECULATIVE_AT
 WORKAROUND_SPECULATIVE_SSBS
 WORKAROUND_SPECULATIVE_UNPRIV_LOAD
 WORKAROUND_3023823
+WORKAROUND_2004055
diff --git a/drivers/perf/arm_spe_pmu.c b/drivers/perf/arm_spe_pmu.c
index 928ea94f2ed6..bca8cd3bca16 100644
--- a/drivers/perf/arm_spe_pmu.c
+++ b/drivers/perf/arm_spe_pmu.c
@@ -1201,6 +1201,9 @@ static void __arm_spe_pmu_dev_probe(void *info)
 	if (this_cpu_has_cap(ARM64_WORKAROUND_3023823))
 		spe_pmu->has_workaround_3023823 = true;
 
+	if (this_cpu_has_cap(ARM64_WORKAROUND_2004055))
+		spe_pmu->has_workaround_2004055 = true;
+
 	dev_info(dev,
 		 "probed SPEv1.%d for CPUs %*pbl [max_record_sz %u, align %u, features 0x%llx]\n",
 		 spe_pmu->pmsver - 1, cpumask_pr_args(&spe_pmu->supported_cpus),
diff --git a/include/linux/perf/arm_spe_pmu.h b/include/linux/perf/arm_spe_pmu.h
index edf58c51fb94..d5c510ebf785 100644
--- a/include/linux/perf/arm_spe_pmu.h
+++ b/include/linux/perf/arm_spe_pmu.h
@@ -46,6 +46,7 @@ struct arm_spe_pmu {
 	u16					align;
 	struct perf_output_handle __percpu	*handle;
 	bool					has_workaround_3023823;
+	bool					has_workaround_2004055;
 };
 
 #define to_spe_pmu(p) (container_of(p, struct arm_spe_pmu, pmu))
-- 
2.43.0

[RFC PATCH v7 25/28] KVM: arm64: at: Use callback for reading descriptor

[Alexandru Elisei]

Allow callers of __kvm_translate_va() to use a custom function for reading
the translation table descriptor from guest memory. This will be useful for
handling stage 2 faults reported by SPE, to map the stage 1 translation
tables at stage 2.

Suggested-by: James Morse <james.morse@arm.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/include/asm/kvm_nested.h |  7 +++++++
 arch/arm64/kvm/at.c                 | 22 ++++++++++++++--------
 arch/arm64/kvm/nested.c             |  7 ++++---
 3 files changed, 25 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h
index 1ed708335809..d812137dcd9e 100644
--- a/arch/arm64/include/asm/kvm_nested.h
+++ b/arch/arm64/include/asm/kvm_nested.h
@@ -350,8 +350,15 @@ struct s1_walk_filter {
 	void	*priv;
 };
 
+typedef int (*read_desc_fn)(struct kvm_vcpu *, u64, u64 *,
+			    struct s1_walk_info *);
+
+int kvm_vcpu_read_s1_desc(struct kvm_vcpu *vcpu, u64 pa, u64 *desc,
+			  struct s1_walk_info *wi);
+
 struct s1_walk_info {
 	struct s1_walk_filter	*filter;
+	read_desc_fn		read_s1_desc;
 	u64	     		baddr;
 	enum trans_regime	regime;
 	unsigned int		max_oa_bits;
diff --git a/arch/arm64/kvm/at.c b/arch/arm64/kvm/at.c
index 0926426b8798..6e77beced96c 100644
--- a/arch/arm64/kvm/at.c
+++ b/arch/arm64/kvm/at.c
@@ -224,6 +224,9 @@ static int setup_s1_walk(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 	unsigned int stride, x;
 	bool va55, tbi, lva, upper_range;
 
+	if (!wi->read_s1_desc)
+		return -EINVAL;
+
 	va55 = va & BIT(55);
 	upper_range = va55 && wi->regime != TR_EL2;
 
@@ -427,8 +430,8 @@ static int setup_s1_walk(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 	return -EFAULT;
 }
 
-static int kvm_read_s1_desc(struct kvm_vcpu *vcpu, u64 pa, u64 *desc,
-			    struct s1_walk_info *wi)
+int kvm_vcpu_read_s1_desc(struct kvm_vcpu *vcpu, u64 pa, u64 *desc,
+			  struct s1_walk_info *wi)
 {
 	u64 val;
 	int r;
@@ -513,7 +516,7 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 				return ret;
 		}
 
-		ret = kvm_read_s1_desc(vcpu, ipa, &desc, wi);
+		ret = wi->read_s1_desc(vcpu, ipa, &desc, wi);
 		if (ret) {
 			fail_s1_walk(wr, ESR_ELx_FSC_SEA_TTW(level), false);
 			return ret;
@@ -1324,6 +1327,7 @@ static int handle_at_slow(struct kvm_vcpu *vcpu, u32 op, u64 vaddr, u64 *par)
 	bool perm_fail = false;
 	int ret, idx;
 
+	wi.read_s1_desc = kvm_vcpu_read_s1_desc;
 	wi.regime = compute_translation_regime(vcpu, op);
 	wi.as_el0 = (op == OP_AT_S1E0R || op == OP_AT_S1E0W);
 	wi.pan = (op == OP_AT_S1E1RP || op == OP_AT_S1E1WP) &&
@@ -1657,9 +1661,10 @@ int __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
 }
 
 /*
- * Translate a VA for a given EL in a given translation regime, with
- * or without PAN. This requires wi->{regime, as_el0, pan} to be
- * set. The rest of the wi and wr should be 0-initialised.
+ * Translate a VA for a given EL in a given translation regime, with or without
+ * PAN. This requires wi->{regime, as_el0, pan, read_s1_desc} to be set.
+ * Setting wi->filter is optional. The rest of the wi and wr should be
+ * 0-initialised.
  */
 int __kvm_translate_va(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
 		       struct s1_walk_result *wr, u64 va)
@@ -1716,8 +1721,9 @@ int __kvm_find_s1_desc_level(struct kvm_vcpu *vcpu, u64 va, u64 ipa, int *level)
 			.fn	= match_s1_desc,
 			.priv	= &dm,
 		},
-		.as_el0	= false,
-		.pan	= false,
+		.read_s1_desc	= kvm_vcpu_read_s1_desc,
+		.as_el0		= false,
+		.pan		= false,
 	};
 	struct s1_walk_result wr = {};
 	int ret;
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 17123f0b6dab..e3e71628f8c5 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -1435,9 +1435,10 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
 			invalidate_vncr(vcpu->kvm, vt);
 
 		vt->wi = (struct s1_walk_info) {
-			.regime	= TR_EL20,
-			.as_el0	= false,
-			.pan	= false,
+			.read_s1_desc	= kvm_vcpu_read_s1_desc,
+			.regime		= TR_EL20,
+			.as_el0		= false,
+			.pan		= false,
 		};
 		vt->wr = (struct s1_walk_result){};
 	}
-- 
2.43.0

[RFC PATCH v7 26/28] KVM: arm64: Map memory on a SPE stage 2 fault

[Alexandru Elisei]

Make sure SPE in a guest can make progress even if something goes terribly
wrong in KVM by mapping the memory on stage 2 fault SPE buffer management
event.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/include/asm/kvm_host.h |  2 +
 arch/arm64/kvm/arm.c              |  4 +-
 arch/arm64/kvm/mmu.c              | 53 ++++++++++++++++++
 arch/arm64/kvm/spe.c              | 93 ++++++++++++++++++++++++++++++-
 4 files changed, 148 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 81abfb705aeb..31be131c8486 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -1464,6 +1464,8 @@ void kvm_debug_handle_oslar(struct kvm_vcpu *vcpu, u64 val);
 #define kvm_guest_owns_debug_regs(vcpu)		\
 	((vcpu)->arch.debug_owner == VCPU_DEBUG_GUEST_OWNED)
 
+int kvm_map_gpa(struct kvm_vcpu *vcpu, u64 gpa, u8 fsc, bool s1ptw);
+
 int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu,
 			       struct kvm_device_attr *attr);
 int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 689fa175f5f4..7c7d0616db53 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1421,8 +1421,6 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		if (kvm_vcpu_has_pmu(vcpu))
 			kvm_pmu_sync_hwstate(vcpu);
 
-		kvm_spe_sync_hwstate(vcpu);
-
 		/*
 		 * Sync the vgic state before syncing the timer state because
 		 * the timer code needs to know if the virtual timer
@@ -1472,6 +1470,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 
 		preempt_enable();
 
+		kvm_spe_sync_hwstate(vcpu);
+
 		/*
 		 * The ARMv8 architecture doesn't give the hypervisor
 		 * a mechanism to prevent a guest from dropping to AArch32 EL0
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 9ba86450fe4a..d7e7365c5c5a 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2796,3 +2796,56 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool was_enabled)
 
 	trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled);
 }
+
+int kvm_map_gpa(struct kvm_vcpu *vcpu, u64 gpa, u8 fsc, bool s1ptw)
+{
+	struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault;
+	struct kvm_vcpu_fault_info fault_backup = *fault_info;
+	struct kvm_s2_fault_desc s2fd = {
+		.vcpu		= vcpu,
+		.fault_ipa	= gpa,
+	};
+	struct kvm_memory_slot *memslot;
+	struct kvm *kvm = vcpu->kvm;
+	gfn_t gfn;
+	hva_t hva;
+	int ret;
+
+	lockdep_assert(srcu_read_lock_held(&kvm->srcu));
+
+	if (WARN_ON_ONCE(is_protected_kvm_enabled() || vcpu_has_nv(vcpu)))
+		return -EOPNOTSUPP;
+
+	if (esr_fsc_is_access_flag_fault(fsc)) {
+		handle_access_fault(vcpu, gpa);
+		return 0;
+	}
+
+	gfn = gpa_to_gfn(gpa);
+	memslot = gfn_to_memslot(kvm, gfn);
+	if (!memslot)
+		return -ENOENT;
+	hva = gfn_to_hva_memslot(memslot, gfn);
+	if (kvm_is_error_hva(hva))
+		return -EFAULT;
+
+	s2fd.memslot = memslot;
+	s2fd.hva = hva;
+
+	fault_info->esr_el2 = FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_DABT_LOW) |
+			      ESR_ELx_IL | FIELD_PREP(ESR_ELx_FSC, fsc);
+	if (s1ptw)
+		fault_info->esr_el2 |= ESR_ELx_S1PTW;
+	else
+		fault_info->esr_el2 |= ESR_ELx_WNR;
+	fault_info->hpfar_el2 = FIELD_PREP(HPFAR_EL2_FIPA, gpa >> 12);
+
+	if (kvm_slot_has_gmem(memslot))
+		ret = gmem_abort(&s2fd);
+	else
+		ret = user_mem_abort(&s2fd);
+
+	*fault_info = fault_backup;
+
+	return ret;
+}
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index ef138562f939..e5f3d0dea7c5 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -10,6 +10,7 @@
 #include <linux/perf_event.h>
 
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
 #include <asm/kvm_spe.h>
 #include <asm/sysreg.h>
 
@@ -242,17 +243,105 @@ static void kvm_spe_update_irq_level(struct kvm_vcpu *vcpu, bool level)
 		vcpu_spe->irq_level = level;
 }
 
+static bool kvm_spe_should_handle_fault(struct kvm_vcpu *vcpu)
+{
+	u64 pmbsr = vcpu->arch.vcpu_spe.hw_pmbsr_el1;
+	u64 pmbsr_ec = FIELD_GET(PMBSR_EL1_EC, pmbsr);
+
+	if (pmbsr_ec != PMBSR_EL1_EC_FAULT_S2)
+		return false;
+
+	if (FIELD_GET(PMBSR_EL1_EA, pmbsr))
+		return false;
+
+	if (esr_fsc_is_translation_fault(pmbsr) ||
+	    esr_fsc_is_permission_fault(pmbsr) ||
+	    esr_fsc_is_access_flag_fault(pmbsr))
+		return true;
+
+	return false;
+}
+
+static int kvm_spe_read_s1_desc(struct kvm_vcpu *vcpu, u64 gpa, u64 *desc,
+				struct s1_walk_info *wi)
+{
+	u64 pmbsr = vcpu->arch.vcpu_spe.hw_pmbsr_el1;
+	int ret;
+
+	ret = kvm_vcpu_read_s1_desc(vcpu, gpa, desc, wi);
+	if (ret)
+		return ret;
+
+	/* TODO: this should be a hardware capability check */
+	if (kvm_has_feat(vcpu->kvm, ID_AA64DFR2_EL1, SPE_EXC, IMP) &&
+	    !(FIELD_GET(ESR_ELx_S1PTW, pmbsr)))
+		return 0;
+
+	/*
+	 * No choice but to assume that the s2 fault happened on a s1 table
+	 * walk.
+	 */
+	return kvm_map_gpa(vcpu, gpa, FIELD_GET(ESR_ELx_FSC, pmbsr), true);
+}
+
+static int kvm_handle_spe_fault(struct kvm_vcpu *vcpu)
+{
+	struct s1_walk_info wi = {
+		.read_s1_desc	= kvm_spe_read_s1_desc,
+		.regime		= TR_EL10,
+		.as_el0		= false,
+		.pan		= false,
+	};
+	struct s1_walk_result wr = {};
+	u64 pmbptr = __vcpu_sys_reg(vcpu, PMBPTR_EL1);
+	u64 pmbsr = vcpu->arch.vcpu_spe.hw_pmbsr_el1;
+	int ret;
+
+	guard(srcu)(&vcpu->kvm->srcu);
+
+	ret = __kvm_translate_va(vcpu, &wi, &wr, pmbptr);
+	if (ret == -EAGAIN)
+		return 0;
+	if (ret)
+		return ret;
+
+	/* Stage 2 fault on a table walk was handled during the translation. */
+	/* TODO: this should be a hardware capability check */
+	if (kvm_has_feat(vcpu->kvm, ID_AA64DFR2_EL1, SPE_EXC, IMP) &&
+	    FIELD_GET(ESR_ELx_S1PTW, pmbsr))
+		return 0;
+
+	return kvm_map_gpa(vcpu, wr.pa, FIELD_GET(ESR_ELx_FSC, pmbsr), false);
+}
+
 void kvm_spe_sync_hwstate(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_spe *vcpu_spe = &vcpu->arch.vcpu_spe;
+	u64 hw_pmbsr_el1;
+	int ret;
 
 	if (!vcpu_has_spe(vcpu))
 		return;
 
-	if (!FIELD_GET(PMBSR_EL1_S, vcpu_spe->hw_pmbsr_el1))
+	hw_pmbsr_el1 = vcpu_spe->hw_pmbsr_el1;
+	if (!FIELD_GET(PMBSR_EL1_S, hw_pmbsr_el1))
 		return;
 
-	__vcpu_assign_sys_reg(vcpu, PMBSR_EL1, vcpu_spe->hw_pmbsr_el1);
+	if (kvm_spe_should_handle_fault(vcpu)) {
+		pr_debug("SPE stage 2 fault PMBSR_EL1=0x%llx PMBPTR_EL1=0x%llx",
+			 hw_pmbsr_el1, __vcpu_sys_reg(vcpu, PMBPTR_EL1));
+		ret = kvm_handle_spe_fault(vcpu);
+		if (ret)
+			goto reinject;
+		/* Let the guest know about the partial record. */
+		if (FIELD_GET(PMBSR_EL1_DL, hw_pmbsr_el1))
+			goto reinject;
+		vcpu_spe->hw_pmbsr_el1 = 0;
+		return;
+	}
+
+reinject:
+	__vcpu_assign_sys_reg(vcpu, PMBSR_EL1, hw_pmbsr_el1);
 	vcpu_spe->hw_pmbsr_el1 = 0;
 	kvm_spe_update_irq_level(vcpu, true);
 }
-- 
2.43.0

[RFC PATCH v7 27/28] KVM: arm64: Handle dirty page logging when SPE feature is set

[Alexandru Elisei]

Dirty page logging requires that memory is marked as read-only at stage 2,
to track when a guest writes to memory.

For SPE, this will cause a stage 2 fault when a record is written to
memory, and a profiling blackout window. Avoid it by refusing to run a VCPU
with the buffer enabled when at least one memslot is logging dirty pages.

Though not prohibited by KVM, to avoid the aforementioned blackout window,
when the VCPU exits with KVM_EXIT_FAIL_ENTRY and
hardware_entry_failure_reason set to KVM_EXIT_FAIL_ENTRY_SPE_DIRTY_LOGGING,
userspace should not run that VCPU until dirty page logging is disabled.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 Documentation/virt/kvm/api.rst                |  7 +-
 arch/arm64/include/asm/kvm_asm.h              |  1 +
 arch/arm64/include/asm/kvm_host.h             | 24 +++---
 arch/arm64/include/asm/kvm_mmu.h              |  1 +
 arch/arm64/include/asm/kvm_pgtable.h          | 19 +++++
 arch/arm64/include/asm/kvm_pkvm.h             |  1 +
 arch/arm64/include/asm/kvm_spe.h              | 16 ++++
 arch/arm64/include/uapi/asm/kvm.h             |  1 +
 arch/arm64/kvm/arm.c                          | 27 ++++++-
 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h |  2 +-
 arch/arm64/kvm/hyp/nvhe/hyp-main.c            | 34 +++++---
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         |  7 +-
 arch/arm64/kvm/hyp/pgtable.c                  |  8 ++
 arch/arm64/kvm/mmu.c                          | 19 +++++
 arch/arm64/kvm/pkvm.c                         | 27 +++++++
 arch/arm64/kvm/spe.c                          | 79 +++++++++++++++++++
 16 files changed, 246 insertions(+), 27 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 654736fac78c..00e85e79e145 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -1422,7 +1422,12 @@ pages in the host.
 The flags field supports two flags: KVM_MEM_LOG_DIRTY_PAGES and
 KVM_MEM_READONLY.  The former can be set to instruct KVM to keep track of
 writes to memory within the slot.  See KVM_GET_DIRTY_LOG ioctl to know how to
-use it.  The latter can be set, if KVM_CAP_READONLY_MEM capability allows it,
+use it. For arm64, setting and then clearing KVM_MEM_LOG_DIRTY_PAGES before
+creating a VCPU with the KVM_ARM_VCPU_SPE feature can lead to profiling
+blackout windows in the guest, caused by SPE encountering stage 2 faults when
+attempting to write to memory.
+
+KVM_MEM_READONLY can be set if KVM_CAP_READONLY_MEM capability allows it,
 to make a new slot read-only.  In this case, writes to this memory will be
 posted to userspace as KVM_EXIT_MMIO exits.
 
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index e5b92ac09e69..6251cbc6b105 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -100,6 +100,7 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___pkvm_host_unshare_guest,
 	__KVM_HOST_SMCCC_FUNC___pkvm_host_relax_perms_guest,
 	__KVM_HOST_SMCCC_FUNC___pkvm_host_wrprotect_guest,
+	__KVM_HOST_SMCCC_FUNC___pkvm_host_write_enable_guest,
 	__KVM_HOST_SMCCC_FUNC___pkvm_host_test_clear_young_guest,
 	__KVM_HOST_SMCCC_FUNC___pkvm_host_mkyoung_guest,
 	__KVM_HOST_SMCCC_FUNC___pkvm_reserve_vm,
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 31be131c8486..ff77f6ac3adf 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -45,17 +45,19 @@
 
 #define KVM_REQ_SLEEP \
 	KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
-#define KVM_REQ_IRQ_PENDING		KVM_ARCH_REQ(1)
-#define KVM_REQ_VCPU_RESET		KVM_ARCH_REQ(2)
-#define KVM_REQ_RECORD_STEAL		KVM_ARCH_REQ(3)
-#define KVM_REQ_RELOAD_GICv4		KVM_ARCH_REQ(4)
-#define KVM_REQ_RELOAD_PMU		KVM_ARCH_REQ(5)
-#define KVM_REQ_SUSPEND			KVM_ARCH_REQ(6)
-#define KVM_REQ_RESYNC_PMU_EL0		KVM_ARCH_REQ(7)
-#define KVM_REQ_NESTED_S2_UNMAP		KVM_ARCH_REQ(8)
-#define KVM_REQ_GUEST_HYP_IRQ_PENDING	KVM_ARCH_REQ(9)
-#define KVM_REQ_MAP_L1_VNCR_EL2		KVM_ARCH_REQ(10)
-#define KVM_REQ_VGIC_PROCESS_UPDATE	KVM_ARCH_REQ(11)
+#define KVM_REQ_IRQ_PENDING			KVM_ARCH_REQ(1)
+#define KVM_REQ_VCPU_RESET			KVM_ARCH_REQ(2)
+#define KVM_REQ_RECORD_STEAL			KVM_ARCH_REQ(3)
+#define KVM_REQ_RELOAD_GICv4			KVM_ARCH_REQ(4)
+#define KVM_REQ_RELOAD_PMU			KVM_ARCH_REQ(5)
+#define KVM_REQ_SUSPEND				KVM_ARCH_REQ(6)
+#define KVM_REQ_RESYNC_PMU_EL0			KVM_ARCH_REQ(7)
+#define KVM_REQ_NESTED_S2_UNMAP			KVM_ARCH_REQ(8)
+#define KVM_REQ_GUEST_HYP_IRQ_PENDING		KVM_ARCH_REQ(9)
+#define KVM_REQ_MAP_L1_VNCR_EL2			KVM_ARCH_REQ(10)
+#define KVM_REQ_VGIC_PROCESS_UPDATE		KVM_ARCH_REQ(11)
+#define KVM_REQ_SPE_HANDLE_DIRTY_LOGGING	KVM_ARCH_REQ(12)
+#define KVM_REQ_SPE_EXIT_DIRTY_LOGGING		KVM_ARCH_REQ(13)
 
 #define KVM_DIRTY_LOG_MANUAL_CAPS   (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
 				     KVM_DIRTY_LOG_INITIALLY_SET)
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 6eae7e7e2a68..5509e3e64abc 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -173,6 +173,7 @@ void kvm_stage2_unmap_range(struct kvm_s2_mmu *mmu, phys_addr_t start,
 			    u64 size, bool may_block);
 void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end);
 void kvm_stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end);
+void kvm_stage2_write_enable_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end);
 
 void stage2_unmap_vm(struct kvm *kvm);
 int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long type);
diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h
index 41a8687938eb..53c8dd62dcd5 100644
--- a/arch/arm64/include/asm/kvm_pgtable.h
+++ b/arch/arm64/include/asm/kvm_pgtable.h
@@ -735,6 +735,25 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size);
  */
 int kvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size);
 
+/**
+ * kvm_pgtable_stage2_write_enable() - Grant write permission to the stage-2
+ * 				       address range without TLB invalidation.
+ * @pgt:	Page-table structure initialised by kvm_pgtable_stage2_init*().
+ * @addr:	Intermediate physical address from which to write-enable.
+ * @size:	Size of the range.
+ *
+ * The offset of @addr within a page is ignored and @size is rounded-up to
+ * the next page boundary.
+ *
+ * Note that it is the caller's responsibility to invalidate the TLB after
+ * calling this function to ensure that the updated permissions are visible
+ * to the CPUs.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int kvm_pgtable_stage2_write_enable(struct kvm_pgtable *pgt, u64 addr,
+				    u64 size);
+
 /**
  * kvm_pgtable_stage2_mkyoung() - Set the access flag in a page-table entry.
  * @pgt:	Page-table structure initialised by kvm_pgtable_stage2_init*().
diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h
index beea00e693a0..a1aa03d13937 100644
--- a/arch/arm64/include/asm/kvm_pkvm.h
+++ b/arch/arm64/include/asm/kvm_pkvm.h
@@ -215,6 +215,7 @@ int pkvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phy
 			    enum kvm_pgtable_walk_flags flags);
 int pkvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size);
 int pkvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size);
+int pkvm_pgtable_stage2_write_enable(struct kvm_pgtable *pgt, u64 addr, u64 size);
 int pkvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size);
 bool pkvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, u64 size, bool mkold);
 int pkvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, enum kvm_pgtable_prot prot,
diff --git a/arch/arm64/include/asm/kvm_spe.h b/arch/arm64/include/asm/kvm_spe.h
index 88f8b1bb6864..d710f64e00be 100644
--- a/arch/arm64/include/asm/kvm_spe.h
+++ b/arch/arm64/include/asm/kvm_spe.h
@@ -9,6 +9,7 @@
 struct kvm;
 struct kvm_vcpu;
 struct kvm_device_attr;
+struct kvm_memory_slot;
 
 #ifdef CONFIG_KVM_ARM_SPE
 struct arm_spe_pmu;
@@ -39,6 +40,10 @@ void kvm_spe_destroy_vm(struct kvm *kvm);
 
 void kvm_spe_finalize_idregs(struct kvm *kvm);
 
+void kvm_spe_add_dirty_logging(struct kvm *kvm, const struct kvm_memory_slot *slot);
+void kvm_spe_remove_dirty_logging(struct kvm *kvm, const struct kvm_memory_slot *slot);
+void kvm_spe_handle_dirty_logging_request(struct kvm_vcpu *vcpu);
+
 int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 int kvm_spe_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
 int kvm_spe_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr);
@@ -86,6 +91,17 @@ static inline void kvm_spe_destroy_vm(struct kvm *kvm)
 static inline void kvm_spe_finalize_idregs(struct kvm *kvm)
 {
 }
+static inline void kvm_spe_add_dirty_logging(struct kvm *kvm,
+					     const struct kvm_memory_slot *slot)
+{
+}
+static inline void kvm_spe_remove_dirty_logging(struct kvm *kvm,
+						const struct kvm_memory_slot *slot)
+{
+}
+static inline void kvm_spe_handle_dirty_logging_request(struct kvm_vcpu *vcpu)
+{
+}
 static inline int kvm_spe_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
 {
 	return -ENXIO;
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index c695aae484d3..a86fa4f9d9e4 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -511,6 +511,7 @@ enum {
 
 /* run->fail_entry.hardware_entry_failure_reason codes. */
 #define KVM_EXIT_FAIL_ENTRY_CPU_UNSUPPORTED	(1ULL << 0)
+#define KVM_EXIT_FAIL_ENTRY_SPE_DIRTY_LOGGING	(1ULL << 1)
 
 enum kvm_smccc_filter_action {
 	KVM_SMCCC_FILTER_HANDLE = 0,
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 7c7d0616db53..73f321b3ed2d 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1172,6 +1172,15 @@ static int kvm_vcpu_suspend(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+static void kvm_vcpu_prepare_exit_fail_entry(struct kvm_vcpu *vcpu, u64 reason)
+{
+	struct kvm_run *run = vcpu->run;
+
+	run->exit_reason = KVM_EXIT_FAIL_ENTRY;
+	run->fail_entry.hardware_entry_failure_reason = reason;
+	run->fail_entry.cpu = smp_processor_id();
+}
+
 /**
  * check_vcpu_requests - check and handle pending vCPU requests
  * @vcpu:	the VCPU pointer
@@ -1223,6 +1232,20 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu)
 		if (kvm_check_request(KVM_REQ_SUSPEND, vcpu))
 			return kvm_vcpu_suspend(vcpu);
 
+		if (kvm_check_request(KVM_REQ_SPE_HANDLE_DIRTY_LOGGING, vcpu))
+			kvm_spe_handle_dirty_logging_request(vcpu);
+
+		/*
+		 * The request is consumed on purpose, to allow userspace to
+		 * resume the VCPU at the expense of profiling blackout windows
+		 * if it so wishes.
+		 */
+		if (kvm_check_request(KVM_REQ_SPE_EXIT_DIRTY_LOGGING, vcpu)) {
+			kvm_vcpu_prepare_exit_fail_entry(vcpu,
+					KVM_EXIT_FAIL_ENTRY_SPE_DIRTY_LOGGING);
+			return 0;
+		}
+
 		if (kvm_dirty_ring_check_request(vcpu))
 			return 0;
 
@@ -1285,9 +1308,7 @@ static bool kvm_vcpu_exit_request(struct kvm_vcpu *vcpu, int *ret)
 	}
 
 	if (unlikely(vcpu_on_unsupported_cpu(vcpu))) {
-		run->exit_reason = KVM_EXIT_FAIL_ENTRY;
-		run->fail_entry.hardware_entry_failure_reason = KVM_EXIT_FAIL_ENTRY_CPU_UNSUPPORTED;
-		run->fail_entry.cpu = smp_processor_id();
+		kvm_vcpu_prepare_exit_fail_entry(vcpu, KVM_EXIT_FAIL_ENTRY_CPU_UNSUPPORTED);
 		*ret = 0;
 		return true;
 	}
diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
index 29935c7da1de..dadb56d62820 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
@@ -47,7 +47,7 @@ int __pkvm_host_share_guest(u64 pfn, u64 gfn, u64 nr_pages, struct pkvm_hyp_vcpu
 			    enum kvm_pgtable_prot prot);
 int __pkvm_host_unshare_guest(u64 gfn, u64 nr_pages, struct pkvm_hyp_vm *hyp_vm);
 int __pkvm_host_relax_perms_guest(u64 gfn, struct pkvm_hyp_vcpu *vcpu, enum kvm_pgtable_prot prot);
-int __pkvm_host_wrprotect_guest(u64 gfn, u64 nr_pages, struct pkvm_hyp_vm *hyp_vm);
+int __pkvm_host_toggle_wp_guest(u64 gfn, u64 nr_pages, struct pkvm_hyp_vm *hyp_vm, bool wp);
 int __pkvm_host_test_clear_young_guest(u64 gfn, u64 nr_pages, bool mkold, struct pkvm_hyp_vm *vm);
 int __pkvm_host_mkyoung_guest(u64 gfn, struct pkvm_hyp_vcpu *vcpu);
 
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 9a3b92e626ad..34c442f1529b 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -487,22 +487,37 @@ static void handle___pkvm_host_relax_perms_guest(struct kvm_cpu_context *host_ct
 	cpu_reg(host_ctxt, 1) = ret;
 }
 
-static void handle___pkvm_host_wrprotect_guest(struct kvm_cpu_context *host_ctxt)
+static int host_toggle_wp_guest(pkvm_handle_t handle, u64 gfn, u64 nr_pages, bool wp)
 {
-	DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1);
-	DECLARE_REG(u64, gfn, host_ctxt, 2);
-	DECLARE_REG(u64, nr_pages, host_ctxt, 3);
 	struct pkvm_hyp_vm *hyp_vm;
-	int ret = -EINVAL;
+	int ret;
 
 	hyp_vm = get_np_pkvm_hyp_vm(handle);
 	if (!hyp_vm)
-		goto out;
+		return -EINVAL;
 
-	ret = __pkvm_host_wrprotect_guest(gfn, nr_pages, hyp_vm);
+	ret = __pkvm_host_toggle_wp_guest(gfn, nr_pages, hyp_vm, wp);
 	put_pkvm_hyp_vm(hyp_vm);
-out:
-	cpu_reg(host_ctxt, 1) = ret;
+
+	return ret;
+}
+
+static void handle___pkvm_host_wrprotect_guest(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1);
+	DECLARE_REG(u64, gfn, host_ctxt, 2);
+	DECLARE_REG(u64, nr_pages, host_ctxt, 3);
+
+	cpu_reg(host_ctxt, 1) = host_toggle_wp_guest(handle, gfn, nr_pages, true);
+}
+
+static void handle___pkvm_host_write_enable_guest(struct kvm_cpu_context *host_ctxt)
+{
+	DECLARE_REG(pkvm_handle_t, handle, host_ctxt, 1);
+	DECLARE_REG(u64, gfn, host_ctxt, 2);
+	DECLARE_REG(u64, nr_pages, host_ctxt, 3);
+
+	cpu_reg(host_ctxt, 1) = host_toggle_wp_guest(handle, gfn, nr_pages, false);
 }
 
 static void handle___pkvm_host_test_clear_young_guest(struct kvm_cpu_context *host_ctxt)
@@ -942,6 +957,7 @@ static const hcall_t host_hcall[] = {
 	HANDLE_FUNC(__pkvm_host_unshare_guest),
 	HANDLE_FUNC(__pkvm_host_relax_perms_guest),
 	HANDLE_FUNC(__pkvm_host_wrprotect_guest),
+	HANDLE_FUNC(__pkvm_host_write_enable_guest),
 	HANDLE_FUNC(__pkvm_host_test_clear_young_guest),
 	HANDLE_FUNC(__pkvm_host_mkyoung_guest),
 	HANDLE_FUNC(__pkvm_reserve_vm),
diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
index 39aa8911f62c..8e5405280c9d 100644
--- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c
+++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
@@ -1650,7 +1650,7 @@ int __pkvm_host_relax_perms_guest(u64 gfn, struct pkvm_hyp_vcpu *vcpu, enum kvm_
 	return ret;
 }
 
-int __pkvm_host_wrprotect_guest(u64 gfn, u64 nr_pages, struct pkvm_hyp_vm *vm)
+int __pkvm_host_toggle_wp_guest(u64 gfn, u64 nr_pages, struct pkvm_hyp_vm *vm, bool wp)
 {
 	u64 size, ipa = hyp_pfn_to_phys(gfn);
 	int ret;
@@ -1664,7 +1664,10 @@ int __pkvm_host_wrprotect_guest(u64 gfn, u64 nr_pages, struct pkvm_hyp_vm *vm)
 
 	assert_host_shared_guest(vm, ipa, size);
 	guest_lock_component(vm);
-	ret = kvm_pgtable_stage2_wrprotect(&vm->pgt, ipa, size);
+	if (wp)
+		ret = kvm_pgtable_stage2_wrprotect(&vm->pgt, ipa, size);
+	else
+		ret = kvm_pgtable_stage2_write_enable(&vm->pgt, ipa, size);
 	guest_unlock_component(vm);
 
 	return ret;
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index b74dd5ce1efd..07e43f607eae 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -1293,6 +1293,14 @@ int kvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size)
 					KVM_PGTABLE_WALK_IGNORE_EAGAIN);
 }
 
+int kvm_pgtable_stage2_write_enable(struct kvm_pgtable *pgt, u64 addr, u64 size)
+{
+	return stage2_update_leaf_attrs(pgt, addr, size,
+					KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W, 0,
+					NULL, NULL,
+					KVM_PGTABLE_WALK_IGNORE_EAGAIN);
+}
+
 void kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr,
 				enum kvm_pgtable_walk_flags flags)
 {
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index d7e7365c5c5a..e1dfc7d5db3e 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1240,6 +1240,17 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
 	return ret;
 }
 
+/**
+ * kvm_stage2_write_enable_range() - grant write permision for the memory range
+ * @mmu:        The KVM stage-2 MMU pointer
+ * @addr:	Start address of range
+ * @end:	End address of range
+ */
+void kvm_stage2_write_enable_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end)
+{
+	stage2_apply_range_resched(mmu, addr, end, KVM_PGT_FN(kvm_pgtable_stage2_write_enable));
+}
+
 /**
  * kvm_stage2_wp_range() - write protect stage2 memory region range
  * @mmu:        The KVM stage-2 MMU pointer
@@ -2578,6 +2589,14 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 				   enum kvm_mr_change change)
 {
 	bool log_dirty_pages = new && new->flags & KVM_MEM_LOG_DIRTY_PAGES;
+	bool had_dirty_logging = old && old->flags & KVM_MEM_LOG_DIRTY_PAGES;
+
+	if (change != KVM_MR_DELETE && had_dirty_logging != log_dirty_pages) {
+		if (log_dirty_pages)
+			kvm_spe_add_dirty_logging(kvm, new);
+		else
+			kvm_spe_remove_dirty_logging(kvm, new);
+	}
 
 	/*
 	 * At this point memslot has been committed and there is an
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index 8e4c6e4bec12..5ba2fa8010cc 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -488,6 +488,33 @@ int pkvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size)
 	return __pkvm_pgtable_stage2_unshare(pgt, addr, addr + size);
 }
 
+int pkvm_pgtable_stage2_write_enable(struct kvm_pgtable *pgt, u64 addr, u64 size)
+{
+	struct kvm *kvm = kvm_s2_mmu_to_kvm(pgt->mmu);
+	pkvm_handle_t handle = kvm->arch.pkvm.handle;
+	struct pkvm_mapping *mapping;
+	u64 gfn, end_gfn, nr_pages;
+	int ret = 0;
+
+	if (WARN_ON(kvm_vm_is_protected(kvm)))
+		return -EPERM;
+
+	end_gfn = (addr + size) >> PAGE_SHIFT;
+
+	lockdep_assert_held(&kvm->mmu_lock);
+	for_each_mapping_in_range_safe(pgt, addr, addr + size, mapping) {
+		gfn = max(addr >> PAGE_SHIFT, mapping->gfn);
+		nr_pages = min_t(u64, end_gfn - gfn, mapping->nr_pages);
+		ret = kvm_call_hyp_nvhe(__pkvm_host_write_enable_guest, handle, gfn,
+					nr_pages);
+		if (WARN_ON(ret))
+			break;
+	}
+
+	return ret;
+}
+
+
 int pkvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size)
 {
 	struct kvm *kvm = kvm_s2_mmu_to_kvm(pgt->mmu);
diff --git a/arch/arm64/kvm/spe.c b/arch/arm64/kvm/spe.c
index e5f3d0dea7c5..f3310465f40e 100644
--- a/arch/arm64/kvm/spe.c
+++ b/arch/arm64/kvm/spe.c
@@ -10,6 +10,7 @@
 #include <linux/perf_event.h>
 
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_mmu.h>
 #include <asm/kvm_nested.h>
 #include <asm/kvm_spe.h>
 #include <asm/sysreg.h>
@@ -141,6 +142,81 @@ void kvm_spe_destroy_vm(struct kvm *kvm)
 	module_put(spe_pmu->pmu.module);
 }
 
+void kvm_spe_add_dirty_logging(struct kvm *kvm, const struct kvm_memory_slot *slot)
+{
+	struct kvm_vcpu *vcpu;
+	unsigned long i;
+
+	lockdep_assert_held(&kvm->slots_lock);
+
+	/*
+	 * No point in handling a regular memslot, because they cannot guarantee
+	 * that memory remains mapped at stage 2.
+	 *
+	 * Memory described by a read-only memslot is always read-only, so SPE
+	 * cannot write to it anyway.
+	 */
+	if (!kvm_slot_has_gmem(slot) || (slot->flags & KVM_MEM_READONLY))
+		return;
+
+	kvm_for_each_vcpu(i, vcpu, kvm) {
+		if (!vcpu_has_spe(vcpu))
+			continue;
+		kvm_make_request(KVM_REQ_SPE_HANDLE_DIRTY_LOGGING, vcpu);
+		if (vcpu_has_run_once(vcpu))
+			__kvm_vcpu_kick(vcpu, true);
+	}
+}
+
+void kvm_spe_remove_dirty_logging(struct kvm *kvm, const struct kvm_memory_slot *slot)
+{
+	phys_addr_t start, end;
+	struct kvm_vcpu *vcpu;
+	unsigned long i;
+
+	lockdep_assert_held(&kvm->slots_lock);
+
+	if (!kvm_slot_has_gmem(slot) || (slot->flags & KVM_MEM_READONLY))
+		return;
+
+	kvm_for_each_vcpu(i, vcpu, kvm) {
+		/*
+		 * Write-enabling an entire memslot can be expensive, do it only
+		 * if the VM has SPE.
+		 */
+		if (vcpu_has_spe(vcpu)) {
+			start = slot->base_gfn << PAGE_SHIFT;
+			end = (slot->base_gfn + slot->npages) << PAGE_SHIFT;
+
+			write_lock(&kvm->mmu_lock);
+			kvm_stage2_write_enable_range(&kvm->arch.mmu, start, end);
+			write_unlock(&kvm->mmu_lock);
+			kvm_flush_remote_tlbs_memslot(kvm, slot);
+			return;
+		}
+	}
+}
+
+static bool kvm_spe_should_exit_dirty_logging(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	u64 pmblimitr, pmbsr;
+
+	if (!atomic_read(&kvm->nr_memslots_dirty_logging))
+		return false;
+
+	pmblimitr = __vcpu_sys_reg(vcpu, PMBLIMITR_EL1);
+	pmbsr = __vcpu_sys_reg(vcpu, PMBSR_EL1);
+
+	return FIELD_GET(PMBLIMITR_EL1_E, pmblimitr) && !FIELD_GET(PMBSR_EL1_S, pmbsr);
+}
+
+void kvm_spe_handle_dirty_logging_request(struct kvm_vcpu *vcpu)
+{
+	if (kvm_spe_should_exit_dirty_logging(vcpu))
+		kvm_make_request(KVM_REQ_SPE_EXIT_DIRTY_LOGGING, vcpu);
+}
+
 bool kvm_spe_has_feat_spe_fds(struct kvm *kvm)
 {
 	struct arm_spe_pmu *spe_pmu = kvm->arch.kvm_spe.spe_pmu;
@@ -214,6 +290,9 @@ bool kvm_spe_write_sysreg(struct kvm_vcpu *vcpu, int reg, u64 val)
 		kvm_spe_update_irq_level(vcpu, FIELD_GET(PMBSR_EL1_S, val));
 	}
 
+	if (kvm_spe_should_exit_dirty_logging(vcpu))
+		kvm_make_request(KVM_REQ_SPE_EXIT_DIRTY_LOGGING, vcpu);
+
 	return true;
 }
 
-- 
2.43.0

[RFC PATCH v7 28/28] KVM: arm64: Allow the creation of a SPE enabled VM

[Alexandru Elisei]

Everything is in place, allow userspace to enable SPE for a virtual
machine.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
---
 arch/arm64/include/asm/kvm_host.h | 2 +-
 arch/arm64/kvm/Kconfig            | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index ff77f6ac3adf..32b30ffdcf77 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -40,7 +40,7 @@
 
 #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS
 
-#define KVM_VCPU_MAX_FEATURES 10
+#define KVM_VCPU_MAX_FEATURES 11
 #define KVM_VCPU_VALID_FEATURES	(BIT(KVM_VCPU_MAX_FEATURES) - 1)
 
 #define KVM_REQ_SLEEP \
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 2bc8e34c538d..a285b18eb1cc 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -106,9 +106,9 @@ endif # NVHE_EL2_DEBUG
 
 
 config KVM_ARM_SPE
-	bool
+	bool "Support SPE in guest"
 	depends on KVM && ARM_SPE_PMU
-	default n
+	default y
 	help
 	  Adds support for Statistical Profiling Extension (SPE) in virtual
 	  machines.
-- 
2.43.0