[PATCH v3 0/7] Add support for FEAT_{LS64, LS64_V} and related tests

[PATCH v3 0/7] Add support for FEAT_{LS64, LS64_V} and related tests

[Yicong Yang]

From: Yicong Yang <yangyicong@hisilicon.com>

Armv8.7 introduces single-copy atomic 64-byte loads and stores
instructions and its variants named under FEAT_{LS64, LS64_V}.
Add support for Armv8.7 FEAT_{LS64, LS64_V}:
- Add identifying and enabling in the cpufeature list
- Expose the support of these features to userspace through HWCAP3
  and cpuinfo
- Add related hwcap test
- Handle the trap of unsupported memory (normal/uncacheable) access in a VM

A real scenario for this feature is that the userspace driver can make use of
this to implement direct WQE (workqueue entry) - a mechanism to fill WQE
directly into the hardware.

Picked Marc's 2 patches form [1] for handling the LS64 trap in a VM on emulated
MMIO and the introduce of KVM_EXIT_ARM_LDST64B.

[1] https://lore.kernel.org/linux-arm-kernel/20240815125959.2097734-1-maz@kernel.org/

Tested with hwcap test [*]:
[*] https://lore.kernel.org/linux-arm-kernel/20250331094320.35226-5-yangyicong@huawei.com/
On host:
root@localhost:/tmp# dmesg | grep "All CPU(s) started"
[    0.504846] CPU: All CPU(s) started at EL2
root@localhost:/tmp# ./hwcap
[...]
# LS64 present
ok 217 cpuinfo_match_LS64
ok 218 sigill_LS64
ok 219 # SKIP sigbus_LS64
# LS64_V present
ok 220 cpuinfo_match_LS64_V
ok 221 sigill_LS64_V
ok 222 # SKIP sigbus_LS64_V
# 115 skipped test(s) detected. Consider enabling relevant config options to improve coverage.
# Totals: pass:107 fail:0 xfail:0 xpass:0 skip:115 error:0

On guest:
root@localhost:/# dmesg | grep "All CPU(s) started"
[    0.451482] CPU: All CPU(s) started at EL1
root@localhost:/mnt# ./hwcap
[...]
# LS64 present
ok 217 cpuinfo_match_LS64
ok 218 sigill_LS64
ok 219 # SKIP sigbus_LS64
# LS64_V present
ok 220 cpuinfo_match_LS64_V
ok 221 sigill_LS64_V
ok 222 # SKIP sigbus_LS64_V
# 115 skipped test(s) detected. Consider enabling relevant config options to improve coverage.
# Totals: pass:107 fail:0 xfail:0 xpass:0 skip:115 error:0

Change since v2:
- Handle the LS64 fault to userspace and allow userspace to inject LS64 fault
- Reorder the patches to make KVM handling prior to feature support
Link: https://lore.kernel.org/linux-arm-kernel/20250331094320.35226-1-yangyicong@huawei.com/

Change since v1:
- Drop the support for LS64_ACCDATA
- handle the DABT of unsupported memory type after checking the memory attributes
Link: https://lore.kernel.org/linux-arm-kernel/20241202135504.14252-1-yangyicong@huawei.com/

Marc Zyngier (2):
  KVM: arm64: Add exit to userspace on {LD,ST}64B* outside of memslots
  KVM: arm64: Add documentation for KVM_EXIT_ARM_LDST64B

Yicong Yang (5):
  KVM: arm64: Handle DABT caused by LS64* instructions on unsupported
    memory
  KVM: arm/arm64: Allow user injection of unsupported exclusive/atomic
    DABT
  arm64: Provide basic EL2 setup for FEAT_{LS64, LS64_V} usage at EL0/1
  arm64: Add support for FEAT_{LS64, LS64_V}
  KVM: arm64: Enable FEAT_{LS64, LS64_V} in the supported guest

 Documentation/arch/arm64/booting.rst    | 12 ++++++
 Documentation/arch/arm64/elf_hwcaps.rst |  6 +++
 Documentation/virt/kvm/api.rst          | 43 +++++++++++++++++----
 arch/arm64/include/asm/el2_setup.h      | 12 +++++-
 arch/arm64/include/asm/esr.h            |  8 ++++
 arch/arm64/include/asm/hwcap.h          |  2 +
 arch/arm64/include/asm/kvm_emulate.h    |  7 ++++
 arch/arm64/include/uapi/asm/hwcap.h     |  2 +
 arch/arm64/include/uapi/asm/kvm.h       |  3 +-
 arch/arm64/kernel/cpufeature.c          | 51 +++++++++++++++++++++++++
 arch/arm64/kernel/cpuinfo.c             |  2 +
 arch/arm64/kvm/guest.c                  |  4 ++
 arch/arm64/kvm/inject_fault.c           | 29 ++++++++++++++
 arch/arm64/kvm/mmio.c                   | 27 ++++++++++++-
 arch/arm64/kvm/mmu.c                    | 21 +++++++++-
 arch/arm64/tools/cpucaps                |  2 +
 include/uapi/linux/kvm.h                |  3 +-
 17 files changed, 222 insertions(+), 12 deletions(-)

-- 
2.24.0

[PATCH v3 1/7] KVM: arm64: Add exit to userspace on {LD,ST}64B* outside of memslots

[Yicong Yang]

From: Marc Zyngier <maz@kernel.org>

The main use of {LD,ST}64B* is to talk to a device, which is hopefully
directly assigned to the guest and requires no additional handling.

However, this does not preclude a VMM from exposing a virtual device
to the guest, and to allow 64 byte accesses as part of the programming
interface. A direct consequence of this is that we need to be able
to forward such access to userspace.

Given that such a contraption is very unlikely to ever exist, we choose
to offer a limited service: userspace gets (as part of a new exit reason)
the ESR, the IPA, and that's it. It is fully expected to handle the full
semantics of the instructions, deal with ACCDATA, the return values and
increment PC. Much fun.

A canonical implementation can also simply inject an abort and be done
with it. Frankly, don't try to do anything else unless you have time
to waste.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 arch/arm64/kvm/mmio.c    | 27 ++++++++++++++++++++++++++-
 include/uapi/linux/kvm.h |  3 ++-
 2 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kvm/mmio.c b/arch/arm64/kvm/mmio.c
index ab365e839874..04520f2f6010 100644
--- a/arch/arm64/kvm/mmio.c
+++ b/arch/arm64/kvm/mmio.c
@@ -157,6 +157,9 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
 	bool is_write;
 	int len;
 	u8 data_buf[8];
+	u64 esr;
+
+	esr = kvm_vcpu_get_esr(vcpu);
 
 	/*
 	 * No valid syndrome? Ask userspace for help if it has
@@ -166,7 +169,7 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
 	 * though, so directly deliver an exception to the guest.
 	 */
 	if (!kvm_vcpu_dabt_isvalid(vcpu)) {
-		trace_kvm_mmio_nisv(*vcpu_pc(vcpu), kvm_vcpu_get_esr(vcpu),
+		trace_kvm_mmio_nisv(*vcpu_pc(vcpu), esr,
 				    kvm_vcpu_get_hfar(vcpu), fault_ipa);
 
 		if (vcpu_is_protected(vcpu)) {
@@ -185,6 +188,28 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
 		return -ENOSYS;
 	}
 
+	/*
+	 * When (DFSC == 0b00xxxx || DFSC == 0b10101x) && DFSC != 0b0000xx
+	 * ESR_EL2[12:11] describe the Load/Store Type. This allows us to
+	 * punt the LD64B/ST64B/ST64BV/ST64BV0 instructions to luserspace,
+	 * which will have to provide a full emulation of these 4
+	 * instructions.  No, we don't expect this do be fast.
+	 *
+	 * We rely on traps being set if the corresponding features are not
+	 * enabled, so if we get here, userspace has promised us to handle
+	 * it already.
+	 */
+	switch (kvm_vcpu_trap_get_fault(vcpu)) {
+	case 0b000100 ... 0b001111:
+	case 0b101010 ... 0b101011:
+		if (FIELD_GET(GENMASK(12, 11), esr)) {
+			run->exit_reason = KVM_EXIT_ARM_LDST64B;
+			run->arm_nisv.esr_iss = esr & ~(u64)ESR_ELx_FSC;
+			run->arm_nisv.fault_ipa = fault_ipa;
+			return 0;
+		}
+	}
+
 	/*
 	 * Prepare MMIO operation. First decode the syndrome data we get
 	 * from the CPU. Then try if some in-kernel emulation feels
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 37891580d05d..bfebaf25e188 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -179,6 +179,7 @@ struct kvm_xen_exit {
 #define KVM_EXIT_LOONGARCH_IOCSR  38
 #define KVM_EXIT_MEMORY_FAULT     39
 #define KVM_EXIT_TDX              40
+#define KVM_EXIT_ARM_LDST64B      41
 
 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
@@ -401,7 +402,7 @@ struct kvm_run {
 		} eoi;
 		/* KVM_EXIT_HYPERV */
 		struct kvm_hyperv_exit hyperv;
-		/* KVM_EXIT_ARM_NISV */
+		/* KVM_EXIT_ARM_NISV / KVM_EXIT_ARM_LDST64B */
 		struct {
 			__u64 esr_iss;
 			__u64 fault_ipa;
-- 
2.24.0

[PATCH v3 2/7] KVM: arm64: Add documentation for KVM_EXIT_ARM_LDST64B

[Yicong Yang]

From: Marc Zyngier <maz@kernel.org>

Add a bit of documentation for KVM_EXIT_ARM_LDST64B so that userspace
knows what to expect.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 Documentation/virt/kvm/api.rst | 43 ++++++++++++++++++++++++++++------
 1 file changed, 36 insertions(+), 7 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 9abf93ee5f65..54799aafe357 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -1300,12 +1300,13 @@ userspace, for example because of missing instruction syndrome decode
 information or because there is no device mapped at the accessed IPA, then
 userspace can ask the kernel to inject an external abort using the address
 from the exiting fault on the VCPU. It is a programming error to set
-ext_dabt_pending after an exit which was not either KVM_EXIT_MMIO or
-KVM_EXIT_ARM_NISV. This feature is only available if the system supports
-KVM_CAP_ARM_INJECT_EXT_DABT. This is a helper which provides commonality in
-how userspace reports accesses for the above cases to guests, across different
-userspace implementations. Nevertheless, userspace can still emulate all Arm
-exceptions by manipulating individual registers using the KVM_SET_ONE_REG API.
+ext_dabt_pending after an exit which was not either KVM_EXIT_MMIO,
+KVM_EXIT_ARM_NISV, or KVM_EXIT_ARM_LDST64B. This feature is only available if
+the system supports KVM_CAP_ARM_INJECT_EXT_DABT. This is a helper which
+provides commonality in how userspace reports accesses for the above cases to
+guests, across different userspace implementations. Nevertheless, userspace
+can still emulate all Arm exceptions by manipulating individual registers
+using the KVM_SET_ONE_REG API.
 
 See KVM_GET_VCPU_EVENTS for the data structure.
 
@@ -7002,12 +7003,14 @@ in send_page or recv a buffer to recv_page).
 
 ::
 
-		/* KVM_EXIT_ARM_NISV */
+		/* KVM_EXIT_ARM_NISV / KVM_EXIT_ARM_LDST64B */
 		struct {
 			__u64 esr_iss;
 			__u64 fault_ipa;
 		} arm_nisv;
 
+- KVM_EXIT_ARM_NISV:
+
 Used on arm64 systems. If a guest accesses memory not in a memslot,
 KVM will typically return to userspace and ask it to do MMIO emulation on its
 behalf. However, for certain classes of instructions, no instruction decode
@@ -7041,6 +7044,32 @@ Note that although KVM_CAP_ARM_NISV_TO_USER will be reported if
 queried outside of a protected VM context, the feature will not be
 exposed if queried on a protected VM file descriptor.
 
+- KVM_EXIT_ARM_LDST64B:
+
+Used on arm64 systems. When a guest using a LD64B, ST64B, ST64BV, ST64BV0,
+outside of a memslot, KVM will return to userspace with KVM_EXIT_ARM_LDST64B,
+exposing the relevant ESR_EL2 information and faulting IPA, similarly to
+KVM_EXIT_ARM_NISV.
+
+Userspace is supposed to fully emulate the instructions, which includes:
+
+	- fetch of the operands for a store, including ACCDATA_EL1 in the case
+	  of a ST64BV0 instruction
+	- deal with the endianness if the guest is big-endian
+	- emulate the access, including the delivery of an exception if the
+	  access didn't succeed
+	- provide a return value in the case of ST64BV/ST64BV0
+	- return the data in the case of a load
+	- increment PC if the instruction was successfully executed
+
+Note that there is no expectation of performance for this emulation, as it
+involves a large number of interaction with the guest state. It is, however,
+expected that the instruction's semantics are preserved, specially the
+single-copy atomicity property of the 64 byte access.
+
+This exit reason must be handled if userspace sets ID_AA64ISAR1_EL1.LS64 to a
+non-zero value, indicating that FEAT_LS64* is enabled.
+
 ::
 
 		/* KVM_EXIT_X86_RDMSR / KVM_EXIT_X86_WRMSR */
-- 
2.24.0

[PATCH v3 3/7] KVM: arm64: Handle DABT caused by LS64* instructions on unsupported memory

[Yicong Yang]

From: Yicong Yang <yangyicong@hisilicon.com>

If FEAT_LS64WB not supported, FEAT_LS64* instructions only support
to access Device/Uncacheable memory, otherwise a data abort for
unsupported Exclusive or atomic access (0x35) is generated per spec.
It's implementation defined whether the target exception level is
routed and is possible to implemented as route to EL2 on a VHE VM
according to DDI0487K.a Section C3.2.12.2 Single-copy atomic 64-byte
load/store.

If it's implemented as generate the DABT to the final enabled stage
(stage-2), since no valid ISV indicated in the ESR, it's better for
the userspace to decide how to handle it. Reuse the
NISV_IO_ABORT_TO_USER path with exit reason KVM_EXIT_ARM_LDST64B.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 arch/arm64/include/asm/esr.h |  8 ++++++++
 arch/arm64/kvm/mmu.c         | 21 ++++++++++++++++++++-
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h
index e1deed824464..63cd17f830da 100644
--- a/arch/arm64/include/asm/esr.h
+++ b/arch/arm64/include/asm/esr.h
@@ -124,6 +124,7 @@
 #define ESR_ELx_FSC_SEA_TTW(n)	(0x14 + (n))
 #define ESR_ELx_FSC_SECC	(0x18)
 #define ESR_ELx_FSC_SECC_TTW(n)	(0x1c + (n))
+#define ESR_ELx_FSC_EXCL_ATOMIC	(0x35)
 #define ESR_ELx_FSC_ADDRSZ	(0x00)
 
 /*
@@ -488,6 +489,13 @@ static inline bool esr_fsc_is_access_flag_fault(unsigned long esr)
 	       (esr == ESR_ELx_FSC_ACCESS_L(0));
 }
 
+static inline bool esr_fsc_is_excl_atomic_fault(unsigned long esr)
+{
+	esr = esr & ESR_ELx_FSC;
+
+	return esr == ESR_ELx_FSC_EXCL_ATOMIC;
+}
+
 static inline bool esr_fsc_is_addr_sz_fault(unsigned long esr)
 {
 	esr &= ESR_ELx_FSC;
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 2942ec92c5a4..5f05d1c4b5a2 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1665,6 +1665,24 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	if (exec_fault && device)
 		return -ENOEXEC;
 
+	/*
+	 * Target address is normal memory on the Host. We come here
+	 * because:
+	 * 1) Guest map it as device memory and perform LS64 operations
+	 * 2) VMM report it as device memory mistakenly
+	 * Hand it to the userspace.
+	 */
+	if (esr_fsc_is_excl_atomic_fault(kvm_vcpu_get_esr(vcpu))) {
+		struct kvm_run *run = vcpu->run;
+
+		run->exit_reason = KVM_EXIT_ARM_LDST64B;
+		run->arm_nisv.esr_iss = kvm_vcpu_dabt_iss_nisv_sanitized(vcpu);
+		run->arm_nisv.fault_ipa = fault_ipa |
+			(kvm_vcpu_get_hfar(vcpu) & (vma_pagesize - 1));
+
+		return -EAGAIN;
+	}
+
 	/*
 	 * Potentially reduce shadow S2 permissions to match the guest's own
 	 * S2. For exec faults, we'd only reach this point if the guest
@@ -1850,7 +1868,8 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 	/* Check the stage-2 fault is trans. fault or write fault */
 	if (!esr_fsc_is_translation_fault(esr) &&
 	    !esr_fsc_is_permission_fault(esr) &&
-	    !esr_fsc_is_access_flag_fault(esr)) {
+	    !esr_fsc_is_access_flag_fault(esr) &&
+	    !esr_fsc_is_excl_atomic_fault(esr)) {
 		kvm_err("Unsupported FSC: EC=%#x xFSC=%#lx ESR_EL2=%#lx\n",
 			kvm_vcpu_trap_get_class(vcpu),
 			(unsigned long)kvm_vcpu_trap_get_fault(vcpu),
-- 
2.24.0

Re: [PATCH v3 3/7] KVM: arm64: Handle DABT caused by LS64* instructions on unsupported memory

[Marc Zyngier]

On Thu, 26 Jun 2025 09:09:02 +0100,
Yicong Yang <yangyicong@huawei.com> wrote:
> 
> From: Yicong Yang <yangyicong@hisilicon.com>
> 
> If FEAT_LS64WB not supported, FEAT_LS64* instructions only support
> to access Device/Uncacheable memory, otherwise a data abort for
> unsupported Exclusive or atomic access (0x35) is generated per spec.
> It's implementation defined whether the target exception level is
> routed and is possible to implemented as route to EL2 on a VHE VM
> according to DDI0487K.a Section C3.2.12.2 Single-copy atomic 64-byte
> load/store.

Nit: in DDI0487L.b (the latest as I write), this is in C3.2.6.

> 
> If it's implemented as generate the DABT to the final enabled stage
> (stage-2), since no valid ISV indicated in the ESR, it's better for
> the userspace to decide how to handle it. Reuse the
> NISV_IO_ABORT_TO_USER path with exit reason KVM_EXIT_ARM_LDST64B.
> 
> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  arch/arm64/include/asm/esr.h |  8 ++++++++
>  arch/arm64/kvm/mmu.c         | 21 ++++++++++++++++++++-
>  2 files changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h
> index e1deed824464..63cd17f830da 100644
> --- a/arch/arm64/include/asm/esr.h
> +++ b/arch/arm64/include/asm/esr.h
> @@ -124,6 +124,7 @@
>  #define ESR_ELx_FSC_SEA_TTW(n)	(0x14 + (n))
>  #define ESR_ELx_FSC_SECC	(0x18)
>  #define ESR_ELx_FSC_SECC_TTW(n)	(0x1c + (n))
> +#define ESR_ELx_FSC_EXCL_ATOMIC	(0x35)
>  #define ESR_ELx_FSC_ADDRSZ	(0x00)
>  
>  /*
> @@ -488,6 +489,13 @@ static inline bool esr_fsc_is_access_flag_fault(unsigned long esr)
>  	       (esr == ESR_ELx_FSC_ACCESS_L(0));
>  }
>  
> +static inline bool esr_fsc_is_excl_atomic_fault(unsigned long esr)
> +{
> +	esr = esr & ESR_ELx_FSC;
> +
> +	return esr == ESR_ELx_FSC_EXCL_ATOMIC;
> +}
> +
>  static inline bool esr_fsc_is_addr_sz_fault(unsigned long esr)
>  {
>  	esr &= ESR_ELx_FSC;
> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index 2942ec92c5a4..5f05d1c4b5a2 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -1665,6 +1665,24 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
>  	if (exec_fault && device)
>  		return -ENOEXEC;
>  
> +	/*
> +	 * Target address is normal memory on the Host. We come here
> +	 * because:
> +	 * 1) Guest map it as device memory and perform LS64 operations
> +	 * 2) VMM report it as device memory mistakenly
> +	 * Hand it to the userspace.
> +	 */
> +	if (esr_fsc_is_excl_atomic_fault(kvm_vcpu_get_esr(vcpu))) {
> +		struct kvm_run *run = vcpu->run;
> +
> +		run->exit_reason = KVM_EXIT_ARM_LDST64B;
> +		run->arm_nisv.esr_iss = kvm_vcpu_dabt_iss_nisv_sanitized(vcpu);
> +		run->arm_nisv.fault_ipa = fault_ipa |
> +			(kvm_vcpu_get_hfar(vcpu) & (vma_pagesize - 1));
> +
> +		return -EAGAIN;
> +	}

I'm not sure that's the right thing to do.

If:

- the guest was told it doesn't have LS64WB,

- it was told that some range is memory,

- it uses that range as device,

- thanks to FWB the resulting memory type is "Normal-Cacheable"

- which results in an Unsupported Atomic exception

why would we involve the VMM at all? The VMM clearly said it didn't
want to be involved in this (we have a memslot).

I think we should simply inject the corresponding S1 fault back into
the guest.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v3 3/7] KVM: arm64: Handle DABT caused by LS64* instructions on unsupported memory

[Yicong Yang]

On 2025/6/26 16:51, Marc Zyngier wrote:
> On Thu, 26 Jun 2025 09:09:02 +0100,
> Yicong Yang <yangyicong@huawei.com> wrote:
>>
>> From: Yicong Yang <yangyicong@hisilicon.com>
>>
>> If FEAT_LS64WB not supported, FEAT_LS64* instructions only support
>> to access Device/Uncacheable memory, otherwise a data abort for
>> unsupported Exclusive or atomic access (0x35) is generated per spec.
>> It's implementation defined whether the target exception level is
>> routed and is possible to implemented as route to EL2 on a VHE VM
>> according to DDI0487K.a Section C3.2.12.2 Single-copy atomic 64-byte
>> load/store.
> 
> Nit: in DDI0487L.b (the latest as I write), this is in C3.2.6.
> 

will update the reference.

>>
>> If it's implemented as generate the DABT to the final enabled stage
>> (stage-2), since no valid ISV indicated in the ESR, it's better for
>> the userspace to decide how to handle it. Reuse the
>> NISV_IO_ABORT_TO_USER path with exit reason KVM_EXIT_ARM_LDST64B.
>>
>> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
>> ---
>>  arch/arm64/include/asm/esr.h |  8 ++++++++
>>  arch/arm64/kvm/mmu.c         | 21 ++++++++++++++++++++-
>>  2 files changed, 28 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h
>> index e1deed824464..63cd17f830da 100644
>> --- a/arch/arm64/include/asm/esr.h
>> +++ b/arch/arm64/include/asm/esr.h
>> @@ -124,6 +124,7 @@
>>  #define ESR_ELx_FSC_SEA_TTW(n)	(0x14 + (n))
>>  #define ESR_ELx_FSC_SECC	(0x18)
>>  #define ESR_ELx_FSC_SECC_TTW(n)	(0x1c + (n))
>> +#define ESR_ELx_FSC_EXCL_ATOMIC	(0x35)
>>  #define ESR_ELx_FSC_ADDRSZ	(0x00)
>>  
>>  /*
>> @@ -488,6 +489,13 @@ static inline bool esr_fsc_is_access_flag_fault(unsigned long esr)
>>  	       (esr == ESR_ELx_FSC_ACCESS_L(0));
>>  }
>>  
>> +static inline bool esr_fsc_is_excl_atomic_fault(unsigned long esr)
>> +{
>> +	esr = esr & ESR_ELx_FSC;
>> +
>> +	return esr == ESR_ELx_FSC_EXCL_ATOMIC;
>> +}
>> +
>>  static inline bool esr_fsc_is_addr_sz_fault(unsigned long esr)
>>  {
>>  	esr &= ESR_ELx_FSC;
>> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
>> index 2942ec92c5a4..5f05d1c4b5a2 100644
>> --- a/arch/arm64/kvm/mmu.c
>> +++ b/arch/arm64/kvm/mmu.c
>> @@ -1665,6 +1665,24 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
>>  	if (exec_fault && device)
>>  		return -ENOEXEC;
>>  
>> +	/*
>> +	 * Target address is normal memory on the Host. We come here
>> +	 * because:
>> +	 * 1) Guest map it as device memory and perform LS64 operations
>> +	 * 2) VMM report it as device memory mistakenly
>> +	 * Hand it to the userspace.
>> +	 */
>> +	if (esr_fsc_is_excl_atomic_fault(kvm_vcpu_get_esr(vcpu))) {
>> +		struct kvm_run *run = vcpu->run;
>> +
>> +		run->exit_reason = KVM_EXIT_ARM_LDST64B;
>> +		run->arm_nisv.esr_iss = kvm_vcpu_dabt_iss_nisv_sanitized(vcpu);
>> +		run->arm_nisv.fault_ipa = fault_ipa |
>> +			(kvm_vcpu_get_hfar(vcpu) & (vma_pagesize - 1));
>> +
>> +		return -EAGAIN;
>> +	}
> 
> I'm not sure that's the right thing to do.
> 
> If:
> 
> - the guest was told it doesn't have LS64WB,
> 
> - it was told that some range is memory,
> 
> - it uses that range as device,
> 
> - thanks to FWB the resulting memory type is "Normal-Cacheable"
> 
> - which results in an Unsupported Atomic exception
> 
> why would we involve the VMM at all? The VMM clearly said it didn't
> want to be involved in this (we have a memslot).
> 

ok I thought we should make VMM do the decision in all the cases(both
here and emulated MMIO) based on the last discussion[*], I may
misunderstand it. If this is the case...

> I think we should simply inject the corresponding S1 fault back into
> the guest.
> 

let's simply inject a corresponding DABT back here and only make the VMM
handle the emulated MMIO case. will update if no further comment.

thanks.

[*] https://lore.kernel.org/linux-arm-kernel/Z_NkHWStDJLo0cmY@linux.dev/

Re: [PATCH v3 3/7] KVM: arm64: Handle DABT caused by LS64* instructions on unsupported memory

[Marc Zyngier]

On Thu, 26 Jun 2025 12:39:41 +0100,
Yicong Yang <yangyicong@huawei.com> wrote:
> 
> On 2025/6/26 16:51, Marc Zyngier wrote:
> > On Thu, 26 Jun 2025 09:09:02 +0100,
> > Yicong Yang <yangyicong@huawei.com> wrote:

[...]

> >>
> >> +	/*
> >> +	 * Target address is normal memory on the Host. We come here
> >> +	 * because:
> >> +	 * 1) Guest map it as device memory and perform LS64 operations
> >> +	 * 2) VMM report it as device memory mistakenly
> >> +	 * Hand it to the userspace.
> >> +	 */
> >> +	if (esr_fsc_is_excl_atomic_fault(kvm_vcpu_get_esr(vcpu))) {
> >> +		struct kvm_run *run = vcpu->run;
> >> +
> >> +		run->exit_reason = KVM_EXIT_ARM_LDST64B;
> >> +		run->arm_nisv.esr_iss = kvm_vcpu_dabt_iss_nisv_sanitized(vcpu);
> >> +		run->arm_nisv.fault_ipa = fault_ipa |
> >> +			(kvm_vcpu_get_hfar(vcpu) & (vma_pagesize - 1));
> >> +
> >> +		return -EAGAIN;
> >> +	}
> > 
> > I'm not sure that's the right thing to do.
> > 
> > If:
> > 
> > - the guest was told it doesn't have LS64WB,
> > 
> > - it was told that some range is memory,
> > 
> > - it uses that range as device,
> > 
> > - thanks to FWB the resulting memory type is "Normal-Cacheable"
> > 
> > - which results in an Unsupported Atomic exception
> > 
> > why would we involve the VMM at all? The VMM clearly said it didn't
> > want to be involved in this (we have a memslot).
> > 
> 
> ok I thought we should make VMM do the decision in all the cases(both
> here and emulated MMIO) based on the last discussion[*], I may
> misunderstand it. If this is the case...
> 
> > I think we should simply inject the corresponding S1 fault back into
> > the guest.
> > 
> 
> let's simply inject a corresponding DABT back here and only make the VMM
> handle the emulated MMIO case. will update if no further comment.

A permission fault at S2 for a R/O memslot should definitely be
relayed to userspace. But the question is whether the HW would report
a permission fault or an unsupported atomic or exclusive fault (UAoEF
for short).

If the HW supports LS64WB, I'd fully expect to get a permission fault,
not an UAoEF, and we can perfectly report this to userspace with full
decode information (though this doesn't fit in the KVM_EXIT_MMIO
structure -- that's "only" an ABI problem).

If it doesn't, then we have a much bigger issue, and I don't think we
can realistically triage the exception in a meaningful way -- we just
can't know the reason why we failed, and we don't even know whether
this was a load or store.

Overall, I can see two options here:

- we limit the LS64 support to HW that supports LS64WB (too bad for
  the other implementations, which is 100% of them). We can always
  triage the exception correctly, and we're unlikely to ever take an
  UAoEF in this context.

- we define that R/O memslots do not support LS64 accesses at all,
  which is always a valid implementation -- the architecture makes no
  provision of which pieces of addressable memory supports an access
  type or another. With that, we can always inject the UAoEF back into
  the guest without any further triaging.

Oliver, what do you think?

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v3 3/7] KVM: arm64: Handle DABT caused by LS64* instructions on unsupported memory

[Yicong Yang]

On 2025/6/27 21:12, Marc Zyngier wrote:
> On Thu, 26 Jun 2025 12:39:41 +0100,
> Yicong Yang <yangyicong@huawei.com> wrote:
>>
>> On 2025/6/26 16:51, Marc Zyngier wrote:
>>> On Thu, 26 Jun 2025 09:09:02 +0100,
>>> Yicong Yang <yangyicong@huawei.com> wrote:
> 
> [...]
> 
>>>>
>>>> +	/*
>>>> +	 * Target address is normal memory on the Host. We come here
>>>> +	 * because:
>>>> +	 * 1) Guest map it as device memory and perform LS64 operations
>>>> +	 * 2) VMM report it as device memory mistakenly
>>>> +	 * Hand it to the userspace.
>>>> +	 */
>>>> +	if (esr_fsc_is_excl_atomic_fault(kvm_vcpu_get_esr(vcpu))) {
>>>> +		struct kvm_run *run = vcpu->run;
>>>> +
>>>> +		run->exit_reason = KVM_EXIT_ARM_LDST64B;
>>>> +		run->arm_nisv.esr_iss = kvm_vcpu_dabt_iss_nisv_sanitized(vcpu);
>>>> +		run->arm_nisv.fault_ipa = fault_ipa |
>>>> +			(kvm_vcpu_get_hfar(vcpu) & (vma_pagesize - 1));
>>>> +
>>>> +		return -EAGAIN;
>>>> +	}
>>>
>>> I'm not sure that's the right thing to do.
>>>
>>> If:
>>>
>>> - the guest was told it doesn't have LS64WB,
>>>
>>> - it was told that some range is memory,
>>>
>>> - it uses that range as device,
>>>
>>> - thanks to FWB the resulting memory type is "Normal-Cacheable"
>>>
>>> - which results in an Unsupported Atomic exception
>>>
>>> why would we involve the VMM at all? The VMM clearly said it didn't
>>> want to be involved in this (we have a memslot).
>>>
>>
>> ok I thought we should make VMM do the decision in all the cases(both
>> here and emulated MMIO) based on the last discussion[*], I may
>> misunderstand it. If this is the case...
>>
>>> I think we should simply inject the corresponding S1 fault back into
>>> the guest.
>>>
>>
>> let's simply inject a corresponding DABT back here and only make the VMM
>> handle the emulated MMIO case. will update if no further comment.
> 
> A permission fault at S2 for a R/O memslot should definitely be
> relayed to userspace. But the question is whether the HW would report
> a permission fault or an unsupported atomic or exclusive fault (UAoEF
> for short).
> 
> If the HW supports LS64WB, I'd fully expect to get a permission fault,
> not an UAoEF, and we can perfectly report this to userspace with full
> decode information (though this doesn't fit in the KVM_EXIT_MMIO
> structure -- that's "only" an ABI problem).
> 
> If it doesn't, then we have a much bigger issue, and I don't think we
> can realistically triage the exception in a meaningful way -- we just
> can't know the reason why we failed, and we don't even know whether
> this was a load or store.
> 
> Overall, I can see two options here:
> 
> - we limit the LS64 support to HW that supports LS64WB (too bad for
>   the other implementations, which is 100% of them). We can always
>   triage the exception correctly, and we're unlikely to ever take an
>   UAoEF in this context.
> 
> - we define that R/O memslots do not support LS64 accesses at all,
>   which is always a valid implementation -- the architecture makes no
>   provision of which pieces of addressable memory supports an access
>   type or another. With that, we can always inject the UAoEF back into
>   the guest without any further triaging.
> 

I'd vote for 2nd solution. this is only the exception case unless trigger it
purposely, the instructions are used for device memory in the normal
cases. It's unfair to forbid the use of it on LS64WB lacked platforms and we
do support LS64 but no LS64WB.

Injecting the UAoEF back will make the handling consistent with host.

thanks.

[PATCH v3 4/7] KVM: arm/arm64: Allow user injection of unsupported exclusive/atomic DABT

[Yicong Yang]

From: Yicong Yang <yangyicong@hisilicon.com>

The unsupported exclusive/atomic DABT exception is hand to the
userspace. Provide a way for the userspace to inject this DABT
to the guest if they want to imitate how this is handled on the
host.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 arch/arm64/include/asm/kvm_emulate.h |  1 +
 arch/arm64/include/uapi/asm/kvm.h    |  3 ++-
 arch/arm64/kvm/guest.c               |  4 ++++
 arch/arm64/kvm/inject_fault.c        | 29 ++++++++++++++++++++++++++++
 4 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 0720898f563e..df141ae77019 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -47,6 +47,7 @@ void kvm_skip_instr32(struct kvm_vcpu *vcpu);
 void kvm_inject_undefined(struct kvm_vcpu *vcpu);
 void kvm_inject_vabt(struct kvm_vcpu *vcpu);
 void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr);
+void kvm_inject_dabt_excl_atomic(struct kvm_vcpu *vcpu, unsigned long addr);
 void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr);
 void kvm_inject_size_fault(struct kvm_vcpu *vcpu);
 
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index ed5f3892674c..69985acda668 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -184,8 +184,9 @@ struct kvm_vcpu_events {
 		__u8 serror_pending;
 		__u8 serror_has_esr;
 		__u8 ext_dabt_pending;
+		__u8 ext_dabt_excl_atom_pending;
 		/* Align it to 8 bytes */
-		__u8 pad[5];
+		__u8 pad[4];
 		__u64 serror_esr;
 	} exception;
 	__u32 reserved[12];
diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c
index 2196979a24a3..47bc09ea50c3 100644
--- a/arch/arm64/kvm/guest.c
+++ b/arch/arm64/kvm/guest.c
@@ -839,6 +839,7 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
 	bool serror_pending = events->exception.serror_pending;
 	bool has_esr = events->exception.serror_has_esr;
 	bool ext_dabt_pending = events->exception.ext_dabt_pending;
+	bool ext_dabt_excl_atom_pending = events->exception.ext_dabt_excl_atom_pending;
 
 	if (serror_pending && has_esr) {
 		if (!cpus_have_final_cap(ARM64_HAS_RAS_EXTN))
@@ -855,6 +856,9 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
 	if (ext_dabt_pending)
 		kvm_inject_dabt(vcpu, kvm_vcpu_get_hfar(vcpu));
 
+	if (ext_dabt_excl_atom_pending)
+		kvm_inject_dabt_excl_atomic(vcpu, kvm_vcpu_get_hfar(vcpu));
+
 	return 0;
 }
 
diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c
index a640e839848e..d64650a1aefe 100644
--- a/arch/arm64/kvm/inject_fault.c
+++ b/arch/arm64/kvm/inject_fault.c
@@ -171,6 +171,35 @@ void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr)
 		inject_abt64(vcpu, false, addr);
 }
 
+/**
+ * kvm_inject_dabt_excl_atomic - inject a data abort for unsupported exclusive
+ *				 or atomic access
+ * @vcpu: The VCPU to receive the data abort
+ * @addr: The address to report in the DFAR
+ *
+ * It is assumed that this code is called from the VCPU thread and that the
+ * VCPU therefore is not currently executing guest code.
+ */
+void kvm_inject_dabt_excl_atomic(struct kvm_vcpu *vcpu, unsigned long addr)
+{
+	u64 esr = 0;
+
+	/* Reuse the general DABT injection routine and modify the DFSC */
+	kvm_inject_dabt(vcpu, addr);
+
+	if (match_target_el(vcpu, unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC))) {
+		esr = vcpu_read_sys_reg(vcpu, ESR_EL1);
+		esr &= ~ESR_ELx_FSC;
+		esr |= ESR_ELx_FSC_EXCL_ATOMIC;
+		vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
+	} else {
+		esr = vcpu_read_sys_reg(vcpu, ESR_EL2);
+		esr &= ~ESR_ELx_FSC;
+		esr |= ESR_ELx_FSC_EXCL_ATOMIC;
+		vcpu_write_sys_reg(vcpu, esr, ESR_EL2);
+	}
+}
+
 /**
  * kvm_inject_pabt - inject a prefetch abort into the guest
  * @vcpu: The VCPU to receive the prefetch abort
-- 
2.24.0

Re: [PATCH v3 4/7] KVM: arm/arm64: Allow user injection of unsupported exclusive/atomic DABT

[Yicong Yang]

On 2025/6/26 16:09, Yicong Yang wrote:
> From: Yicong Yang <yangyicong@hisilicon.com>
> 
> The unsupported exclusive/atomic DABT exception is hand to the
> userspace. Provide a way for the userspace to inject this DABT
> to the guest if they want to imitate how this is handled on the
> host.
> 

Tested LS64 fault in VM using kvmtool with below patch, debug information added.
The LS64 DABT injection works as expected.

# Perform LS64 on emulated MMIO
root@localhost:/mnt# lspci -tv
-[0000:00]-+-00.0  Device 1af4:1049
           \-01.0  Device 1af4:1041
root@localhost:/mnt# ./ls64.o -d 0000:00:00.0 -b 2
Start address of 0000:00:00.0 BAR2 is 0x0
mappded va is 0xffff82d20000 addr is 0x4120e8
  Info: esr_iss 93c09000 fault_ipa 50000000 // kvmtool debug information
  Info: correct mapping but emulated MMIO // kvmtool debug information
test FEAT_LS64
Bus error
# Perform LS64 on normal memory
root@localhost:/mnt# ./ls64.o -a
mappded va is 0xffffa5400000 addr is 0x4120e8
test FEAT_LS64
  Info: esr_iss 35 fault_ipa 83971000 // kvmtool debug information
  Info: Injecting DABT since incorrect Guest memory attribute // kvmtool debug information
Bus error

diff --git a/arm/aarch64/include/asm/kvm.h b/arm/aarch64/include/asm/kvm.h
index 66736ff..d3cd866 100644
--- a/arm/aarch64/include/asm/kvm.h
+++ b/arm/aarch64/include/asm/kvm.h
@@ -186,8 +186,9 @@ struct kvm_vcpu_events {
 		__u8 serror_pending;
 		__u8 serror_has_esr;
 		__u8 ext_dabt_pending;
+		__u8 ext_dabt_excl_atom_pending;
 		/* Align it to 8 bytes */
-		__u8 pad[5];
+		__u8 pad[4];
 		__u64 serror_esr;
 	} exception;
 	__u32 reserved[12];
diff --git a/include/kvm/kvm.h b/include/kvm/kvm.h
index eb23e2f..56b985d 100644
--- a/include/kvm/kvm.h
+++ b/include/kvm/kvm.h
@@ -129,6 +129,8 @@ bool kvm__emulate_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u8 *data, u32 len, u
 int kvm__destroy_mem(struct kvm *kvm, u64 guest_phys, u64 size, void *userspace_addr);
 int kvm__register_mem(struct kvm *kvm, u64 guest_phys, u64 size, void *userspace_addr,
 		      enum kvm_mem_type type);
+bool kvm__valid_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u32 len);
+
 static inline int kvm__register_ram(struct kvm *kvm, u64 guest_phys, u64 size,
 				    void *userspace_addr)
 {
diff --git a/include/linux/kvm.h b/include/linux/kvm.h
index 502ea63..fa01051 100644
--- a/include/linux/kvm.h
+++ b/include/linux/kvm.h
@@ -178,6 +178,7 @@ struct kvm_xen_exit {
 #define KVM_EXIT_NOTIFY           37
 #define KVM_EXIT_LOONGARCH_IOCSR  38
 #define KVM_EXIT_MEMORY_FAULT     39
+#define KVM_EXIT_ARM_LDST64B      41

 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
diff --git a/kvm-cpu.c b/kvm-cpu.c
index 7362f2e..f544cf4 100644
--- a/kvm-cpu.c
+++ b/kvm-cpu.c
@@ -238,6 +238,42 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
 				goto exit_kvm;
 			};
 			break;
+		case KVM_EXIT_ARM_LDST64B: {
+			struct kvm_run *kvm_run = cpu->kvm_run;
+			__u64 ipa = kvm_run->arm_nisv.fault_ipa;
+			int ret;
+
+			pr_info("esr_iss %llx fault_ipa %llx",
+				kvm_run->arm_nisv.esr_iss, ipa);
+
+			if (!kvm__valid_mmio(cpu, ipa, 64)) {
+				struct kvm_vcpu_events events = {
+					.exception.ext_dabt_excl_atom_pending = 1,
+				};
+
+				pr_info("Injecting DABT since incorrect Guest memory attribute");
+
+				ret = ioctl(cpu->vcpu_fd, KVM_SET_VCPU_EVENTS, &events);
+				if (ret) {
+					pr_err("err inject DABT");
+					goto panic_kvm;
+				}
+			} else {
+				struct kvm_vcpu_events events = {
+					.exception.ext_dabt_excl_atom_pending = 1,
+				};
+
+				pr_info("correct mapping but emulated MMIO");
+
+				ret = ioctl(cpu->vcpu_fd, KVM_SET_VCPU_EVENTS, &events);
+				if (ret) {
+					pr_err("err inject DABT");
+					goto panic_kvm;
+				}
+			}
+
+			break;
+		}
 		default: {
 			bool ret;

diff --git a/mmio.c b/mmio.c
index 231ce91..7071d3a 100644
--- a/mmio.c
+++ b/mmio.c
@@ -195,6 +195,11 @@ bool kvm__deregister_iotrap(struct kvm *kvm, u64 phys_addr, unsigned int flags)
 	return true;
 }

+bool kvm__valid_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u32 len)
+{
+	return !!mmio_get(&mmio_tree, phys_addr, len);
+}
+
 bool kvm__emulate_mmio(struct kvm_cpu *vcpu, u64 phys_addr, u8 *data,
 		       u32 len, u8 is_write)
 {


> Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
> ---
>  arch/arm64/include/asm/kvm_emulate.h |  1 +
>  arch/arm64/include/uapi/asm/kvm.h    |  3 ++-
>  arch/arm64/kvm/guest.c               |  4 ++++
>  arch/arm64/kvm/inject_fault.c        | 29 ++++++++++++++++++++++++++++
>  4 files changed, 36 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
> index 0720898f563e..df141ae77019 100644
> --- a/arch/arm64/include/asm/kvm_emulate.h
> +++ b/arch/arm64/include/asm/kvm_emulate.h
> @@ -47,6 +47,7 @@ void kvm_skip_instr32(struct kvm_vcpu *vcpu);
>  void kvm_inject_undefined(struct kvm_vcpu *vcpu);
>  void kvm_inject_vabt(struct kvm_vcpu *vcpu);
>  void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr);
> +void kvm_inject_dabt_excl_atomic(struct kvm_vcpu *vcpu, unsigned long addr);
>  void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr);
>  void kvm_inject_size_fault(struct kvm_vcpu *vcpu);
>  
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index ed5f3892674c..69985acda668 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -184,8 +184,9 @@ struct kvm_vcpu_events {
>  		__u8 serror_pending;
>  		__u8 serror_has_esr;
>  		__u8 ext_dabt_pending;
> +		__u8 ext_dabt_excl_atom_pending;
>  		/* Align it to 8 bytes */
> -		__u8 pad[5];
> +		__u8 pad[4];
>  		__u64 serror_esr;
>  	} exception;
>  	__u32 reserved[12];
> diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c
> index 2196979a24a3..47bc09ea50c3 100644
> --- a/arch/arm64/kvm/guest.c
> +++ b/arch/arm64/kvm/guest.c
> @@ -839,6 +839,7 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
>  	bool serror_pending = events->exception.serror_pending;
>  	bool has_esr = events->exception.serror_has_esr;
>  	bool ext_dabt_pending = events->exception.ext_dabt_pending;
> +	bool ext_dabt_excl_atom_pending = events->exception.ext_dabt_excl_atom_pending;
>  
>  	if (serror_pending && has_esr) {
>  		if (!cpus_have_final_cap(ARM64_HAS_RAS_EXTN))
> @@ -855,6 +856,9 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
>  	if (ext_dabt_pending)
>  		kvm_inject_dabt(vcpu, kvm_vcpu_get_hfar(vcpu));
>  
> +	if (ext_dabt_excl_atom_pending)
> +		kvm_inject_dabt_excl_atomic(vcpu, kvm_vcpu_get_hfar(vcpu));
> +
>  	return 0;
>  }
>  
> diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c
> index a640e839848e..d64650a1aefe 100644
> --- a/arch/arm64/kvm/inject_fault.c
> +++ b/arch/arm64/kvm/inject_fault.c
> @@ -171,6 +171,35 @@ void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr)
>  		inject_abt64(vcpu, false, addr);
>  }
>  
> +/**
> + * kvm_inject_dabt_excl_atomic - inject a data abort for unsupported exclusive
> + *				 or atomic access
> + * @vcpu: The VCPU to receive the data abort
> + * @addr: The address to report in the DFAR
> + *
> + * It is assumed that this code is called from the VCPU thread and that the
> + * VCPU therefore is not currently executing guest code.
> + */
> +void kvm_inject_dabt_excl_atomic(struct kvm_vcpu *vcpu, unsigned long addr)
> +{
> +	u64 esr = 0;
> +
> +	/* Reuse the general DABT injection routine and modify the DFSC */
> +	kvm_inject_dabt(vcpu, addr);
> +
> +	if (match_target_el(vcpu, unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC))) {
> +		esr = vcpu_read_sys_reg(vcpu, ESR_EL1);
> +		esr &= ~ESR_ELx_FSC;
> +		esr |= ESR_ELx_FSC_EXCL_ATOMIC;
> +		vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
> +	} else {
> +		esr = vcpu_read_sys_reg(vcpu, ESR_EL2);
> +		esr &= ~ESR_ELx_FSC;
> +		esr |= ESR_ELx_FSC_EXCL_ATOMIC;
> +		vcpu_write_sys_reg(vcpu, esr, ESR_EL2);
> +	}
> +}
> +
>  /**
>   * kvm_inject_pabt - inject a prefetch abort into the guest
>   * @vcpu: The VCPU to receive the prefetch abort
> 

[PATCH v3 5/7] arm64: Provide basic EL2 setup for FEAT_{LS64, LS64_V} usage at EL0/1

[Yicong Yang]

From: Yicong Yang <yangyicong@hisilicon.com>

Instructions introduced by FEAT_{LS64, LS64_V} is controlled by
HCRX_EL2.{EnALS, EnASR}. Configure all of these to allow usage
at EL0/1.

This doesn't mean these instructions are always available in
EL0/1 if provided. The hypervisor still have the control at
runtime.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
For note, Will raise the question HCRX setting here won't retain after a
world switch. This is fine now after commit ffea7c73d181 ("KVM: arm64: Properly save/restore HCRX_EL2")

 arch/arm64/include/asm/el2_setup.h | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
index ba5df0df02a4..9a0ab7dc9db1 100644
--- a/arch/arm64/include/asm/el2_setup.h
+++ b/arch/arm64/include/asm/el2_setup.h
@@ -57,9 +57,19 @@
         /* Enable GCS if supported */
 	mrs_s	x1, SYS_ID_AA64PFR1_EL1
 	ubfx	x1, x1, #ID_AA64PFR1_EL1_GCS_SHIFT, #4
-	cbz	x1, .Lset_hcrx_\@
+	cbz	x1, .Lskip_gcs_hcrx_\@
 	orr	x0, x0, #HCRX_EL2_GCSEn
 
+.Lskip_gcs_hcrx_\@:
+	/* Enable LS64, LS64_V if supported */
+	mrs_s	x1, SYS_ID_AA64ISAR1_EL1
+	ubfx	x1, x1, #ID_AA64ISAR1_EL1_LS64_SHIFT, #4
+	cbz	x1, .Lset_hcrx_\@
+	orr	x0, x0, #HCRX_EL2_EnALS
+	cmp	x1, #ID_AA64ISAR1_EL1_LS64_LS64_V
+	b.lt	.Lset_hcrx_\@
+	orr	x0, x0, #HCRX_EL2_EnASR
+
 .Lset_hcrx_\@:
 	msr_s	SYS_HCRX_EL2, x0
 .Lskip_hcrx_\@:
-- 
2.24.0

[PATCH v3 6/7] arm64: Add support for FEAT_{LS64, LS64_V}

[Yicong Yang]

From: Yicong Yang <yangyicong@hisilicon.com>

Armv8.7 introduces single-copy atomic 64-byte loads and stores
instructions and its variants named under FEAT_{LS64, LS64_V}.
These features are identified by ID_AA64ISAR1_EL1.LS64 and the
use of such instructions in userspace (EL0) can be trapped. In
order to support the use of corresponding instructions in userspace:
- Make ID_AA64ISAR1_EL1.LS64 visbile to userspace
- Add identifying and enabling in the cpufeature list
- Expose these support of these features to userspace through HWCAP3
  and cpuinfo

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 Documentation/arch/arm64/booting.rst    | 12 ++++++
 Documentation/arch/arm64/elf_hwcaps.rst |  6 +++
 arch/arm64/include/asm/hwcap.h          |  2 +
 arch/arm64/include/uapi/asm/hwcap.h     |  2 +
 arch/arm64/kernel/cpufeature.c          | 51 +++++++++++++++++++++++++
 arch/arm64/kernel/cpuinfo.c             |  2 +
 arch/arm64/tools/cpucaps                |  2 +
 7 files changed, 77 insertions(+)

diff --git a/Documentation/arch/arm64/booting.rst b/Documentation/arch/arm64/booting.rst
index ee9b790c0d72..837823d49212 100644
--- a/Documentation/arch/arm64/booting.rst
+++ b/Documentation/arch/arm64/booting.rst
@@ -483,6 +483,18 @@ Before jumping into the kernel, the following conditions must be met:
 
    - MDCR_EL3.TPM (bit 6) must be initialized to 0b0
 
+  For CPUs support for 64-byte loads and stores without status (FEAT_LS64):
+
+  - If the kernel is entered at EL1 and EL2 is present:
+
+    - HCRX_EL2.EnALS (bit 1) must be initialised to 0b1.
+
+  For CPUs support for 64-byte loads and stores with status (FEAT_LS64_V):
+
+  - If the kernel is entered at EL1 and EL2 is present:
+
+    - HCRX_EL2.EnASR (bit 2) must be initialised to 0b1.
+
 The requirements described above for CPU mode, caches, MMUs, architected
 timers, coherency and system registers apply to all CPUs.  All CPUs must
 enter the kernel in the same exception level.  Where the values documented
diff --git a/Documentation/arch/arm64/elf_hwcaps.rst b/Documentation/arch/arm64/elf_hwcaps.rst
index 69d7afe56853..9e6db258ff48 100644
--- a/Documentation/arch/arm64/elf_hwcaps.rst
+++ b/Documentation/arch/arm64/elf_hwcaps.rst
@@ -435,6 +435,12 @@ HWCAP2_SME_SF8DP4
 HWCAP2_POE
     Functionality implied by ID_AA64MMFR3_EL1.S1POE == 0b0001.
 
+HWCAP3_LS64
+    Functionality implied by ID_AA64ISAR1_EL1.LS64 == 0b0001.
+
+HWCAP3_LS64_V
+    Functionality implied by ID_AA64ISAR1_EL1.LS64 == 0b0010.
+
 4. Unused AT_HWCAP bits
 -----------------------
 
diff --git a/arch/arm64/include/asm/hwcap.h b/arch/arm64/include/asm/hwcap.h
index 1c3f9617d54f..f45ab66d3466 100644
--- a/arch/arm64/include/asm/hwcap.h
+++ b/arch/arm64/include/asm/hwcap.h
@@ -176,6 +176,8 @@
 #define KERNEL_HWCAP_POE		__khwcap2_feature(POE)
 
 #define __khwcap3_feature(x)		(const_ilog2(HWCAP3_ ## x) + 128)
+#define KERNEL_HWCAP_LS64              __khwcap3_feature(LS64)
+#define KERNEL_HWCAP_LS64_V            __khwcap3_feature(LS64_V)
 
 /*
  * This yields a mask that user programs can use to figure out what
diff --git a/arch/arm64/include/uapi/asm/hwcap.h b/arch/arm64/include/uapi/asm/hwcap.h
index 705a7afa8e58..88579dad778d 100644
--- a/arch/arm64/include/uapi/asm/hwcap.h
+++ b/arch/arm64/include/uapi/asm/hwcap.h
@@ -143,5 +143,7 @@
 /*
  * HWCAP3 flags - for AT_HWCAP3
  */
+#define HWCAP3_LS64		(1UL << 0)
+#define HWCAP3_LS64_V		(1UL << 1)
 
 #endif /* _UAPI__ASM_HWCAP_H */
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index b34044e20128..5e730a522c76 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -238,6 +238,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar0[] = {
 };
 
 static const struct arm64_ftr_bits ftr_id_aa64isar1[] = {
+	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_EL1_LS64_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_EL1_XS_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_EL1_I8MM_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_EL1_DGH_SHIFT, 4, 0),
@@ -2291,6 +2292,38 @@ static void cpu_enable_e0pd(struct arm64_cpu_capabilities const *cap)
 }
 #endif /* CONFIG_ARM64_E0PD */
 
+static bool has_ls64(const struct arm64_cpu_capabilities *entry, int __unused)
+{
+	u64 ls64;
+
+	ls64 = cpuid_feature_extract_field(__read_sysreg_by_encoding(entry->sys_reg),
+					   entry->field_pos, entry->sign);
+
+	if (ls64 == ID_AA64ISAR1_EL1_LS64_NI ||
+	    ls64 > ID_AA64ISAR1_EL1_LS64_LS64_ACCDATA)
+		return false;
+
+	if (entry->capability == ARM64_HAS_LS64 &&
+	    ls64 >= ID_AA64ISAR1_EL1_LS64_LS64)
+		return true;
+
+	if (entry->capability == ARM64_HAS_LS64_V &&
+	    ls64 >= ID_AA64ISAR1_EL1_LS64_LS64_V)
+		return true;
+
+	return false;
+}
+
+static void cpu_enable_ls64(struct arm64_cpu_capabilities const *cap)
+{
+	sysreg_clear_set(sctlr_el1, SCTLR_EL1_EnALS, SCTLR_EL1_EnALS);
+}
+
+static void cpu_enable_ls64_v(struct arm64_cpu_capabilities const *cap)
+{
+	sysreg_clear_set(sctlr_el1, SCTLR_EL1_EnASR, SCTLR_EL1_EnASR);
+}
+
 #ifdef CONFIG_ARM64_PSEUDO_NMI
 static bool can_use_gic_priorities(const struct arm64_cpu_capabilities *entry,
 				   int scope)
@@ -3061,6 +3094,22 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.matches = has_pmuv3,
 	},
 #endif
+	{
+		.desc = "LS64",
+		.capability = ARM64_HAS_LS64,
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.matches = has_ls64,
+		.cpu_enable = cpu_enable_ls64,
+		ARM64_CPUID_FIELDS(ID_AA64ISAR1_EL1, LS64, LS64)
+	},
+	{
+		.desc = "LS64_V",
+		.capability = ARM64_HAS_LS64_V,
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.matches = has_ls64,
+		.cpu_enable = cpu_enable_ls64_v,
+		ARM64_CPUID_FIELDS(ID_AA64ISAR1_EL1, LS64, LS64_V)
+	},
 	{},
 };
 
@@ -3173,6 +3222,8 @@ static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = {
 	HWCAP_CAP(ID_AA64ISAR1_EL1, BF16, EBF16, CAP_HWCAP, KERNEL_HWCAP_EBF16),
 	HWCAP_CAP(ID_AA64ISAR1_EL1, DGH, IMP, CAP_HWCAP, KERNEL_HWCAP_DGH),
 	HWCAP_CAP(ID_AA64ISAR1_EL1, I8MM, IMP, CAP_HWCAP, KERNEL_HWCAP_I8MM),
+	HWCAP_CAP(ID_AA64ISAR1_EL1, LS64, LS64, CAP_HWCAP, KERNEL_HWCAP_LS64),
+	HWCAP_CAP(ID_AA64ISAR1_EL1, LS64, LS64_V, CAP_HWCAP, KERNEL_HWCAP_LS64_V),
 	HWCAP_CAP(ID_AA64ISAR2_EL1, LUT, IMP, CAP_HWCAP, KERNEL_HWCAP_LUT),
 	HWCAP_CAP(ID_AA64ISAR3_EL1, FAMINMAX, IMP, CAP_HWCAP, KERNEL_HWCAP_FAMINMAX),
 	HWCAP_CAP(ID_AA64MMFR2_EL1, AT, IMP, CAP_HWCAP, KERNEL_HWCAP_USCAT),
diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c
index c1f2b6b04b41..3cca8d0a36f1 100644
--- a/arch/arm64/kernel/cpuinfo.c
+++ b/arch/arm64/kernel/cpuinfo.c
@@ -81,6 +81,8 @@ static const char *const hwcap_str[] = {
 	[KERNEL_HWCAP_PACA]		= "paca",
 	[KERNEL_HWCAP_PACG]		= "pacg",
 	[KERNEL_HWCAP_GCS]		= "gcs",
+	[KERNEL_HWCAP_LS64]		= "ls64",
+	[KERNEL_HWCAP_LS64_V]		= "ls64_v",
 	[KERNEL_HWCAP_DCPODP]		= "dcpodp",
 	[KERNEL_HWCAP_SVE2]		= "sve2",
 	[KERNEL_HWCAP_SVEAES]		= "sveaes",
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 10effd4cff6b..8930208a1b07 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -43,6 +43,8 @@ HAS_HCX
 HAS_LDAPR
 HAS_LPA2
 HAS_LSE_ATOMICS
+HAS_LS64
+HAS_LS64_V
 HAS_MOPS
 HAS_NESTED_VIRT
 HAS_PAN
-- 
2.24.0

[PATCH v3 7/7] KVM: arm64: Enable FEAT_{LS64, LS64_V} in the supported guest

[Yicong Yang]

From: Yicong Yang <yangyicong@hisilicon.com>

Using FEAT_{LS64, LS64_V} instructions in a guest is also controlled
by HCRX_EL2.{EnALS, EnASR}. Enable it if guest has related feature.

Signed-off-by: Yicong Yang <yangyicong@hisilicon.com>
---
 arch/arm64/include/asm/kvm_emulate.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index df141ae77019..3e000fff1c1d 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -628,6 +628,12 @@ static inline void vcpu_set_hcrx(struct kvm_vcpu *vcpu)
 
 		if (kvm_has_fpmr(kvm))
 			vcpu->arch.hcrx_el2 |= HCRX_EL2_EnFPM;
+
+		if (kvm_has_feat(kvm, ID_AA64ISAR1_EL1, LS64, LS64))
+			vcpu->arch.hcrx_el2 |= HCRX_EL2_EnALS;
+
+		if (kvm_has_feat(kvm, ID_AA64ISAR1_EL1, LS64, LS64_V))
+			vcpu->arch.hcrx_el2 |= HCRX_EL2_EnASR;
 	}
 }
 #endif /* __ARM64_KVM_EMULATE_H__ */
-- 
2.24.0