* [PATCH 00/13] Accelerate KVM RISC-V when running as a guest
@ 2024-07-19 16:09 Anup Patel
2024-07-19 16:09 ` [PATCH 01/13] RISC-V: KVM: Order the object files alphabetically Anup Patel
` (12 more replies)
0 siblings, 13 replies; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
The KVM RISC-V hypervisor might be running as a guest under some other
host hypervisor in which case the complete H-extension functionality will
be trap-n-emulated by the host hypervisor. In this case, the KVM RISC-V
performance can be accelerated using the SBI nested acceleration (NACL)
extension if the host hypervisor provides it.
These series extends KVM RISC-V to use SBI NACL extension whenever
underlying SBI implementation (aka host hypervisor) provides it.
These patches can also be found in the riscv_sbi_nested_v1 branch at:
https://github.com/avpatel/linux.git
To test these patches, run KVM RISC-V as Guest under latest Xvisor
found at: https://github.com/xvisor/xvisor.git
For the steps to test on Xvisor, refer the Xvisor documentation
<xvisor_source>/docs/riscv/riscv64-qemu.txt with two small changes:
1) In step#11, make sure compressed kvm.ko, guest kernel image, and
kvmtool are present in the rootfs.img
2) In step#14, make sure AIA is available to Xvisor by using
"virt,aia=aplic-imsic" as the QEMU machine name.
Anup Patel (13):
RISC-V: KVM: Order the object files alphabetically
RISC-V: KVM: Save/restore HSTATUS in C source
RISC-V: KVM: Save/restore SCOUNTEREN in C source
RISC-V: KVM: Break down the __kvm_riscv_switch_to() into macros
RISC-V: KVM: Replace aia_set_hvictl() with aia_hvictl_value()
RISC-V: KVM: Don't setup SGEI for zero guest external interrupts
RISC-V: Add defines for the SBI nested acceleration extension
RISC-V: KVM: Add common nested acceleration support
RISC-V: KVM: Use nacl_csr_xyz() for accessing H-extension CSRs
RISC-V: KVM: Use nacl_csr_xyz() for accessing AIA CSRs
RISC-V: KVM: Use SBI sync SRET call when available
RISC-V: KVM: Save trap CSRs in kvm_riscv_vcpu_enter_exit()
RISC-V: KVM: Use NACL HFENCEs for KVM request based HFENCEs
arch/riscv/include/asm/kvm_nacl.h | 237 ++++++++++++++++++++++++++++++
arch/riscv/include/asm/sbi.h | 120 +++++++++++++++
arch/riscv/kvm/Makefile | 27 ++--
arch/riscv/kvm/aia.c | 114 +++++++++-----
arch/riscv/kvm/main.c | 53 ++++++-
arch/riscv/kvm/mmu.c | 4 +-
arch/riscv/kvm/nacl.c | 152 +++++++++++++++++++
arch/riscv/kvm/tlb.c | 57 ++++---
arch/riscv/kvm/vcpu.c | 184 +++++++++++++++++------
arch/riscv/kvm/vcpu_switch.S | 137 ++++++++++-------
arch/riscv/kvm/vcpu_timer.c | 28 ++--
11 files changed, 935 insertions(+), 178 deletions(-)
create mode 100644 arch/riscv/include/asm/kvm_nacl.h
create mode 100644 arch/riscv/kvm/nacl.c
--
2.34.1
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH 01/13] RISC-V: KVM: Order the object files alphabetically
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-16 19:03 ` [01/13] " Atish Patra
2024-07-19 16:09 ` [PATCH 02/13] RISC-V: KVM: Save/restore HSTATUS in C source Anup Patel
` (11 subsequent siblings)
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
Order the object files alphabetically in the Makefile so that
it is very predictable inserting new object files in the future.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/kvm/Makefile | 26 ++++++++++++++------------
1 file changed, 14 insertions(+), 12 deletions(-)
diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile
index c2cacfbc06a0..c1eac0d093de 100644
--- a/arch/riscv/kvm/Makefile
+++ b/arch/riscv/kvm/Makefile
@@ -9,27 +9,29 @@ include $(srctree)/virt/kvm/Makefile.kvm
obj-$(CONFIG_KVM) += kvm.o
+# Ordered alphabetically
+kvm-y += aia.o
+kvm-y += aia_aplic.o
+kvm-y += aia_device.o
+kvm-y += aia_imsic.o
kvm-y += main.o
-kvm-y += vm.o
-kvm-y += vmid.o
-kvm-y += tlb.o
kvm-y += mmu.o
+kvm-y += tlb.o
kvm-y += vcpu.o
kvm-y += vcpu_exit.o
kvm-y += vcpu_fp.o
-kvm-y += vcpu_vector.o
kvm-y += vcpu_insn.o
kvm-y += vcpu_onereg.o
-kvm-y += vcpu_switch.o
+kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_pmu.o
kvm-y += vcpu_sbi.o
-kvm-$(CONFIG_RISCV_SBI_V01) += vcpu_sbi_v01.o
kvm-y += vcpu_sbi_base.o
-kvm-y += vcpu_sbi_replace.o
kvm-y += vcpu_sbi_hsm.o
+kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_sbi_pmu.o
+kvm-y += vcpu_sbi_replace.o
kvm-y += vcpu_sbi_sta.o
+kvm-$(CONFIG_RISCV_SBI_V01) += vcpu_sbi_v01.o
+kvm-y += vcpu_switch.o
kvm-y += vcpu_timer.o
-kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_pmu.o vcpu_sbi_pmu.o
-kvm-y += aia.o
-kvm-y += aia_device.o
-kvm-y += aia_aplic.o
-kvm-y += aia_imsic.o
+kvm-y += vcpu_vector.o
+kvm-y += vm.o
+kvm-y += vmid.o
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 02/13] RISC-V: KVM: Save/restore HSTATUS in C source
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
2024-07-19 16:09 ` [PATCH 01/13] RISC-V: KVM: Order the object files alphabetically Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-16 19:27 ` [02/13] " Atish Patra
2024-07-19 16:09 ` [PATCH 03/13] RISC-V: KVM: Save/restore SCOUNTEREN " Anup Patel
` (10 subsequent siblings)
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
We will be optimizing HSTATUS CSR access via shared memory setup
using the SBI nested acceleration extension. To facilitate this,
we first move HSTATUS save/restore in kvm_riscv_vcpu_enter_exit().
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/kvm/vcpu.c | 9 +++++++++
arch/riscv/kvm/vcpu_switch.S | 36 +++++++++++++-----------------------
2 files changed, 22 insertions(+), 23 deletions(-)
diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
index 449e5bb948c2..93b1ce043482 100644
--- a/arch/riscv/kvm/vcpu.c
+++ b/arch/riscv/kvm/vcpu.c
@@ -720,9 +720,18 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v
*/
static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
{
+ struct kvm_cpu_context *gcntx = &vcpu->arch.guest_context;
+ struct kvm_cpu_context *hcntx = &vcpu->arch.host_context;
+
kvm_riscv_vcpu_swap_in_guest_state(vcpu);
guest_state_enter_irqoff();
+
+ hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
+
__kvm_riscv_switch_to(&vcpu->arch);
+
+ gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
+
vcpu->arch.last_exit_cpu = vcpu->cpu;
guest_state_exit_irqoff();
kvm_riscv_vcpu_swap_in_host_state(vcpu);
diff --git a/arch/riscv/kvm/vcpu_switch.S b/arch/riscv/kvm/vcpu_switch.S
index 0c26189aa01c..f83643c4fdb9 100644
--- a/arch/riscv/kvm/vcpu_switch.S
+++ b/arch/riscv/kvm/vcpu_switch.S
@@ -43,35 +43,30 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
/* Load Guest CSR values */
REG_L t0, (KVM_ARCH_GUEST_SSTATUS)(a0)
- REG_L t1, (KVM_ARCH_GUEST_HSTATUS)(a0)
- REG_L t2, (KVM_ARCH_GUEST_SCOUNTEREN)(a0)
- la t4, .Lkvm_switch_return
- REG_L t5, (KVM_ARCH_GUEST_SEPC)(a0)
+ REG_L t1, (KVM_ARCH_GUEST_SCOUNTEREN)(a0)
+ la t3, .Lkvm_switch_return
+ REG_L t4, (KVM_ARCH_GUEST_SEPC)(a0)
/* Save Host and Restore Guest SSTATUS */
csrrw t0, CSR_SSTATUS, t0
- /* Save Host and Restore Guest HSTATUS */
- csrrw t1, CSR_HSTATUS, t1
-
/* Save Host and Restore Guest SCOUNTEREN */
- csrrw t2, CSR_SCOUNTEREN, t2
+ csrrw t1, CSR_SCOUNTEREN, t1
/* Save Host STVEC and change it to return path */
- csrrw t4, CSR_STVEC, t4
+ csrrw t3, CSR_STVEC, t3
/* Save Host SSCRATCH and change it to struct kvm_vcpu_arch pointer */
- csrrw t3, CSR_SSCRATCH, a0
+ csrrw t2, CSR_SSCRATCH, a0
/* Restore Guest SEPC */
- csrw CSR_SEPC, t5
+ csrw CSR_SEPC, t4
/* Store Host CSR values */
REG_S t0, (KVM_ARCH_HOST_SSTATUS)(a0)
- REG_S t1, (KVM_ARCH_HOST_HSTATUS)(a0)
- REG_S t2, (KVM_ARCH_HOST_SCOUNTEREN)(a0)
- REG_S t3, (KVM_ARCH_HOST_SSCRATCH)(a0)
- REG_S t4, (KVM_ARCH_HOST_STVEC)(a0)
+ REG_S t1, (KVM_ARCH_HOST_SCOUNTEREN)(a0)
+ REG_S t2, (KVM_ARCH_HOST_SSCRATCH)(a0)
+ REG_S t3, (KVM_ARCH_HOST_STVEC)(a0)
/* Restore Guest GPRs (except A0) */
REG_L ra, (KVM_ARCH_GUEST_RA)(a0)
@@ -153,8 +148,7 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
REG_L t1, (KVM_ARCH_HOST_STVEC)(a0)
REG_L t2, (KVM_ARCH_HOST_SSCRATCH)(a0)
REG_L t3, (KVM_ARCH_HOST_SCOUNTEREN)(a0)
- REG_L t4, (KVM_ARCH_HOST_HSTATUS)(a0)
- REG_L t5, (KVM_ARCH_HOST_SSTATUS)(a0)
+ REG_L t4, (KVM_ARCH_HOST_SSTATUS)(a0)
/* Save Guest SEPC */
csrr t0, CSR_SEPC
@@ -168,18 +162,14 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
/* Save Guest and Restore Host SCOUNTEREN */
csrrw t3, CSR_SCOUNTEREN, t3
- /* Save Guest and Restore Host HSTATUS */
- csrrw t4, CSR_HSTATUS, t4
-
/* Save Guest and Restore Host SSTATUS */
- csrrw t5, CSR_SSTATUS, t5
+ csrrw t4, CSR_SSTATUS, t4
/* Store Guest CSR values */
REG_S t0, (KVM_ARCH_GUEST_SEPC)(a0)
REG_S t2, (KVM_ARCH_GUEST_A0)(a0)
REG_S t3, (KVM_ARCH_GUEST_SCOUNTEREN)(a0)
- REG_S t4, (KVM_ARCH_GUEST_HSTATUS)(a0)
- REG_S t5, (KVM_ARCH_GUEST_SSTATUS)(a0)
+ REG_S t4, (KVM_ARCH_GUEST_SSTATUS)(a0)
/* Restore Host GPRs (except A0 and T0-T6) */
REG_L ra, (KVM_ARCH_HOST_RA)(a0)
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 03/13] RISC-V: KVM: Save/restore SCOUNTEREN in C source
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
2024-07-19 16:09 ` [PATCH 01/13] RISC-V: KVM: Order the object files alphabetically Anup Patel
2024-07-19 16:09 ` [PATCH 02/13] RISC-V: KVM: Save/restore HSTATUS in C source Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-16 21:36 ` Atish Patra
2024-07-19 16:09 ` [PATCH 04/13] RISC-V: KVM: Break down the __kvm_riscv_switch_to() into macros Anup Patel
` (9 subsequent siblings)
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
The SCOUNTEREN CSR need not be saved/restored in the low-level
__kvm_riscv_switch_to() function hence move the SCOUNTEREN CSR
save/restore to the kvm_riscv_vcpu_swap_in_guest_state() and
kvm_riscv_vcpu_swap_in_host_state() functions in C sources.
Also, re-arrange the CSR save/restore and related GPR usage in
the low-level __kvm_riscv_switch_to() low-level function.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/kvm/vcpu.c | 2 ++
arch/riscv/kvm/vcpu_switch.S | 52 +++++++++++++++---------------------
2 files changed, 23 insertions(+), 31 deletions(-)
diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
index 93b1ce043482..957e1a5e081b 100644
--- a/arch/riscv/kvm/vcpu.c
+++ b/arch/riscv/kvm/vcpu.c
@@ -691,6 +691,7 @@ static __always_inline void kvm_riscv_vcpu_swap_in_guest_state(struct kvm_vcpu *
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
+ vcpu->arch.host_scounteren = csr_swap(CSR_SCOUNTEREN, csr->scounteren);
vcpu->arch.host_senvcfg = csr_swap(CSR_SENVCFG, csr->senvcfg);
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN) &&
(cfg->hstateen0 & SMSTATEEN0_SSTATEEN0))
@@ -704,6 +705,7 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
+ csr->scounteren = csr_swap(CSR_SCOUNTEREN, vcpu->arch.host_scounteren);
csr->senvcfg = csr_swap(CSR_SENVCFG, vcpu->arch.host_senvcfg);
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN) &&
(cfg->hstateen0 & SMSTATEEN0_SSTATEEN0))
diff --git a/arch/riscv/kvm/vcpu_switch.S b/arch/riscv/kvm/vcpu_switch.S
index f83643c4fdb9..3f8cbc21a644 100644
--- a/arch/riscv/kvm/vcpu_switch.S
+++ b/arch/riscv/kvm/vcpu_switch.S
@@ -43,30 +43,25 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
/* Load Guest CSR values */
REG_L t0, (KVM_ARCH_GUEST_SSTATUS)(a0)
- REG_L t1, (KVM_ARCH_GUEST_SCOUNTEREN)(a0)
- la t3, .Lkvm_switch_return
- REG_L t4, (KVM_ARCH_GUEST_SEPC)(a0)
+ la t1, .Lkvm_switch_return
+ REG_L t2, (KVM_ARCH_GUEST_SEPC)(a0)
/* Save Host and Restore Guest SSTATUS */
csrrw t0, CSR_SSTATUS, t0
- /* Save Host and Restore Guest SCOUNTEREN */
- csrrw t1, CSR_SCOUNTEREN, t1
-
/* Save Host STVEC and change it to return path */
- csrrw t3, CSR_STVEC, t3
-
- /* Save Host SSCRATCH and change it to struct kvm_vcpu_arch pointer */
- csrrw t2, CSR_SSCRATCH, a0
+ csrrw t1, CSR_STVEC, t1
/* Restore Guest SEPC */
- csrw CSR_SEPC, t4
+ csrw CSR_SEPC, t2
+
+ /* Save Host SSCRATCH and change it to struct kvm_vcpu_arch pointer */
+ csrrw t3, CSR_SSCRATCH, a0
/* Store Host CSR values */
REG_S t0, (KVM_ARCH_HOST_SSTATUS)(a0)
- REG_S t1, (KVM_ARCH_HOST_SCOUNTEREN)(a0)
- REG_S t2, (KVM_ARCH_HOST_SSCRATCH)(a0)
- REG_S t3, (KVM_ARCH_HOST_STVEC)(a0)
+ REG_S t1, (KVM_ARCH_HOST_STVEC)(a0)
+ REG_S t3, (KVM_ARCH_HOST_SSCRATCH)(a0)
/* Restore Guest GPRs (except A0) */
REG_L ra, (KVM_ARCH_GUEST_RA)(a0)
@@ -145,31 +140,26 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
REG_S t6, (KVM_ARCH_GUEST_T6)(a0)
/* Load Host CSR values */
- REG_L t1, (KVM_ARCH_HOST_STVEC)(a0)
- REG_L t2, (KVM_ARCH_HOST_SSCRATCH)(a0)
- REG_L t3, (KVM_ARCH_HOST_SCOUNTEREN)(a0)
- REG_L t4, (KVM_ARCH_HOST_SSTATUS)(a0)
-
- /* Save Guest SEPC */
- csrr t0, CSR_SEPC
+ REG_L t0, (KVM_ARCH_HOST_STVEC)(a0)
+ REG_L t1, (KVM_ARCH_HOST_SSCRATCH)(a0)
+ REG_L t2, (KVM_ARCH_HOST_SSTATUS)(a0)
/* Save Guest A0 and Restore Host SSCRATCH */
- csrrw t2, CSR_SSCRATCH, t2
+ csrrw t1, CSR_SSCRATCH, t1
- /* Restore Host STVEC */
- csrw CSR_STVEC, t1
+ /* Save Guest SEPC */
+ csrr t3, CSR_SEPC
- /* Save Guest and Restore Host SCOUNTEREN */
- csrrw t3, CSR_SCOUNTEREN, t3
+ /* Restore Host STVEC */
+ csrw CSR_STVEC, t0
/* Save Guest and Restore Host SSTATUS */
- csrrw t4, CSR_SSTATUS, t4
+ csrrw t2, CSR_SSTATUS, t2
/* Store Guest CSR values */
- REG_S t0, (KVM_ARCH_GUEST_SEPC)(a0)
- REG_S t2, (KVM_ARCH_GUEST_A0)(a0)
- REG_S t3, (KVM_ARCH_GUEST_SCOUNTEREN)(a0)
- REG_S t4, (KVM_ARCH_GUEST_SSTATUS)(a0)
+ REG_S t1, (KVM_ARCH_GUEST_A0)(a0)
+ REG_S t2, (KVM_ARCH_GUEST_SSTATUS)(a0)
+ REG_S t3, (KVM_ARCH_GUEST_SEPC)(a0)
/* Restore Host GPRs (except A0 and T0-T6) */
REG_L ra, (KVM_ARCH_HOST_RA)(a0)
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 04/13] RISC-V: KVM: Break down the __kvm_riscv_switch_to() into macros
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
` (2 preceding siblings ...)
2024-07-19 16:09 ` [PATCH 03/13] RISC-V: KVM: Save/restore SCOUNTEREN " Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-16 21:37 ` Atish Patra
2024-07-19 16:09 ` [PATCH 05/13] RISC-V: KVM: Replace aia_set_hvictl() with aia_hvictl_value() Anup Patel
` (8 subsequent siblings)
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
Break down the __kvm_riscv_switch_to() function into macros so that
these macros can be later re-used by SBI NACL extension based low-level
switch function.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/kvm/vcpu_switch.S | 52 +++++++++++++++++++++++++++---------
1 file changed, 40 insertions(+), 12 deletions(-)
diff --git a/arch/riscv/kvm/vcpu_switch.S b/arch/riscv/kvm/vcpu_switch.S
index 3f8cbc21a644..9f13e5ce6a18 100644
--- a/arch/riscv/kvm/vcpu_switch.S
+++ b/arch/riscv/kvm/vcpu_switch.S
@@ -11,11 +11,7 @@
#include <asm/asm-offsets.h>
#include <asm/csr.h>
- .text
- .altmacro
- .option norelax
-
-SYM_FUNC_START(__kvm_riscv_switch_to)
+.macro SAVE_HOST_GPRS
/* Save Host GPRs (except A0 and T0-T6) */
REG_S ra, (KVM_ARCH_HOST_RA)(a0)
REG_S sp, (KVM_ARCH_HOST_SP)(a0)
@@ -40,10 +36,12 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
REG_S s9, (KVM_ARCH_HOST_S9)(a0)
REG_S s10, (KVM_ARCH_HOST_S10)(a0)
REG_S s11, (KVM_ARCH_HOST_S11)(a0)
+.endm
+.macro SAVE_HOST_AND_RESTORE_GUEST_CSRS __resume_addr
/* Load Guest CSR values */
REG_L t0, (KVM_ARCH_GUEST_SSTATUS)(a0)
- la t1, .Lkvm_switch_return
+ la t1, \__resume_addr
REG_L t2, (KVM_ARCH_GUEST_SEPC)(a0)
/* Save Host and Restore Guest SSTATUS */
@@ -62,7 +60,9 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
REG_S t0, (KVM_ARCH_HOST_SSTATUS)(a0)
REG_S t1, (KVM_ARCH_HOST_STVEC)(a0)
REG_S t3, (KVM_ARCH_HOST_SSCRATCH)(a0)
+.endm
+.macro RESTORE_GUEST_GPRS
/* Restore Guest GPRs (except A0) */
REG_L ra, (KVM_ARCH_GUEST_RA)(a0)
REG_L sp, (KVM_ARCH_GUEST_SP)(a0)
@@ -97,13 +97,9 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
/* Restore Guest A0 */
REG_L a0, (KVM_ARCH_GUEST_A0)(a0)
+.endm
- /* Resume Guest */
- sret
-
- /* Back to Host */
- .align 2
-.Lkvm_switch_return:
+.macro SAVE_GUEST_GPRS
/* Swap Guest A0 with SSCRATCH */
csrrw a0, CSR_SSCRATCH, a0
@@ -138,7 +134,9 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
REG_S t4, (KVM_ARCH_GUEST_T4)(a0)
REG_S t5, (KVM_ARCH_GUEST_T5)(a0)
REG_S t6, (KVM_ARCH_GUEST_T6)(a0)
+.endm
+.macro SAVE_GUEST_AND_RESTORE_HOST_CSRS
/* Load Host CSR values */
REG_L t0, (KVM_ARCH_HOST_STVEC)(a0)
REG_L t1, (KVM_ARCH_HOST_SSCRATCH)(a0)
@@ -160,7 +158,9 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
REG_S t1, (KVM_ARCH_GUEST_A0)(a0)
REG_S t2, (KVM_ARCH_GUEST_SSTATUS)(a0)
REG_S t3, (KVM_ARCH_GUEST_SEPC)(a0)
+.endm
+.macro RESTORE_HOST_GPRS
/* Restore Host GPRs (except A0 and T0-T6) */
REG_L ra, (KVM_ARCH_HOST_RA)(a0)
REG_L sp, (KVM_ARCH_HOST_SP)(a0)
@@ -185,6 +185,34 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
REG_L s9, (KVM_ARCH_HOST_S9)(a0)
REG_L s10, (KVM_ARCH_HOST_S10)(a0)
REG_L s11, (KVM_ARCH_HOST_S11)(a0)
+.endm
+
+ .text
+ .altmacro
+ .option norelax
+
+ /*
+ * Parameters:
+ * A0 <= Pointer to struct kvm_vcpu_arch
+ */
+SYM_FUNC_START(__kvm_riscv_switch_to)
+ SAVE_HOST_GPRS
+
+ SAVE_HOST_AND_RESTORE_GUEST_CSRS .Lkvm_switch_return
+
+ RESTORE_GUEST_GPRS
+
+ /* Resume Guest using SRET */
+ sret
+
+ /* Back to Host */
+ .align 2
+.Lkvm_switch_return:
+ SAVE_GUEST_GPRS
+
+ SAVE_GUEST_AND_RESTORE_HOST_CSRS
+
+ RESTORE_HOST_GPRS
/* Return to C code */
ret
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 05/13] RISC-V: KVM: Replace aia_set_hvictl() with aia_hvictl_value()
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
` (3 preceding siblings ...)
2024-07-19 16:09 ` [PATCH 04/13] RISC-V: KVM: Break down the __kvm_riscv_switch_to() into macros Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-16 21:39 ` Atish Patra
2024-07-19 16:09 ` [PATCH 06/13] RISC-V: KVM: Don't setup SGEI for zero guest external interrupts Anup Patel
` (7 subsequent siblings)
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
The aia_set_hvictl() internally writes the HVICTL CSR which makes
it difficult optimize the CSR write using SBI NACL extension for
kvm_riscv_vcpu_aia_update_hvip() function so replace aia_set_hvictl()
with new aia_hvictl_value() which only computes the HVICTL value.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/kvm/aia.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/arch/riscv/kvm/aia.c b/arch/riscv/kvm/aia.c
index 2967d305c442..17ae4a7c0e94 100644
--- a/arch/riscv/kvm/aia.c
+++ b/arch/riscv/kvm/aia.c
@@ -51,7 +51,7 @@ static int aia_find_hgei(struct kvm_vcpu *owner)
return hgei;
}
-static void aia_set_hvictl(bool ext_irq_pending)
+static inline unsigned long aia_hvictl_value(bool ext_irq_pending)
{
unsigned long hvictl;
@@ -62,7 +62,7 @@ static void aia_set_hvictl(bool ext_irq_pending)
hvictl = (IRQ_S_EXT << HVICTL_IID_SHIFT) & HVICTL_IID;
hvictl |= ext_irq_pending;
- csr_write(CSR_HVICTL, hvictl);
+ return hvictl;
}
#ifdef CONFIG_32BIT
@@ -130,7 +130,7 @@ void kvm_riscv_vcpu_aia_update_hvip(struct kvm_vcpu *vcpu)
#ifdef CONFIG_32BIT
csr_write(CSR_HVIPH, vcpu->arch.aia_context.guest_csr.hviph);
#endif
- aia_set_hvictl(!!(csr->hvip & BIT(IRQ_VS_EXT)));
+ csr_write(CSR_HVICTL, aia_hvictl_value(!!(csr->hvip & BIT(IRQ_VS_EXT))));
}
void kvm_riscv_vcpu_aia_load(struct kvm_vcpu *vcpu, int cpu)
@@ -536,7 +536,7 @@ void kvm_riscv_aia_enable(void)
if (!kvm_riscv_aia_available())
return;
- aia_set_hvictl(false);
+ csr_write(CSR_HVICTL, aia_hvictl_value(false));
csr_write(CSR_HVIPRIO1, 0x0);
csr_write(CSR_HVIPRIO2, 0x0);
#ifdef CONFIG_32BIT
@@ -572,7 +572,7 @@ void kvm_riscv_aia_disable(void)
csr_clear(CSR_HIE, BIT(IRQ_S_GEXT));
disable_percpu_irq(hgei_parent_irq);
- aia_set_hvictl(false);
+ csr_write(CSR_HVICTL, aia_hvictl_value(false));
raw_spin_lock_irqsave(&hgctrl->lock, flags);
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 06/13] RISC-V: KVM: Don't setup SGEI for zero guest external interrupts
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
` (4 preceding siblings ...)
2024-07-19 16:09 ` [PATCH 05/13] RISC-V: KVM: Replace aia_set_hvictl() with aia_hvictl_value() Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-16 21:40 ` Atish Patra
2024-07-19 16:09 ` [PATCH 07/13] RISC-V: Add defines for the SBI nested acceleration extension Anup Patel
` (6 subsequent siblings)
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
No need to setup SGEI local interrupt when there are zero guest
external interrupts (i.e. zero HW IMSIC guest files).
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/kvm/aia.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/arch/riscv/kvm/aia.c b/arch/riscv/kvm/aia.c
index 17ae4a7c0e94..8ffae0330c89 100644
--- a/arch/riscv/kvm/aia.c
+++ b/arch/riscv/kvm/aia.c
@@ -499,6 +499,10 @@ static int aia_hgei_init(void)
hgctrl->free_bitmap = 0;
}
+ /* Skip SGEI interrupt setup for zero guest external interrupts */
+ if (!kvm_riscv_aia_nr_hgei)
+ goto skip_sgei_interrupt;
+
/* Find INTC irq domain */
domain = irq_find_matching_fwnode(riscv_get_intc_hwnode(),
DOMAIN_BUS_ANY);
@@ -522,11 +526,16 @@ static int aia_hgei_init(void)
return rc;
}
+skip_sgei_interrupt:
return 0;
}
static void aia_hgei_exit(void)
{
+ /* Do nothing for zero guest external interrupts */
+ if (!kvm_riscv_aia_nr_hgei)
+ return;
+
/* Free per-CPU SGEI interrupt */
free_percpu_irq(hgei_parent_irq, &aia_hgei);
}
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 07/13] RISC-V: Add defines for the SBI nested acceleration extension
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
` (5 preceding siblings ...)
2024-07-19 16:09 ` [PATCH 06/13] RISC-V: KVM: Don't setup SGEI for zero guest external interrupts Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-17 23:55 ` Atish Patra
2024-07-19 16:09 ` [PATCH 08/13] RISC-V: KVM: Add common nested acceleration support Anup Patel
` (5 subsequent siblings)
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
Add defines for the new SBI nested acceleration extension which was
ratified as part of the SBI v2.0 specification.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/include/asm/sbi.h | 120 +++++++++++++++++++++++++++++++++++
1 file changed, 120 insertions(+)
diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h
index 1079e214fe85..7c9ec953c519 100644
--- a/arch/riscv/include/asm/sbi.h
+++ b/arch/riscv/include/asm/sbi.h
@@ -33,6 +33,7 @@ enum sbi_ext_id {
SBI_EXT_PMU = 0x504D55,
SBI_EXT_DBCN = 0x4442434E,
SBI_EXT_STA = 0x535441,
+ SBI_EXT_NACL = 0x4E41434C,
/* Experimentals extensions must lie within this range */
SBI_EXT_EXPERIMENTAL_START = 0x08000000,
@@ -279,6 +280,125 @@ struct sbi_sta_struct {
#define SBI_SHMEM_DISABLE -1
+enum sbi_ext_nacl_fid {
+ SBI_EXT_NACL_PROBE_FEATURE = 0x0,
+ SBI_EXT_NACL_SET_SHMEM = 0x1,
+ SBI_EXT_NACL_SYNC_CSR = 0x2,
+ SBI_EXT_NACL_SYNC_HFENCE = 0x3,
+ SBI_EXT_NACL_SYNC_SRET = 0x4,
+};
+
+enum sbi_ext_nacl_feature {
+ SBI_NACL_FEAT_SYNC_CSR = 0x0,
+ SBI_NACL_FEAT_SYNC_HFENCE = 0x1,
+ SBI_NACL_FEAT_SYNC_SRET = 0x2,
+ SBI_NACL_FEAT_AUTOSWAP_CSR = 0x3,
+};
+
+#define SBI_NACL_SHMEM_ADDR_SHIFT 12
+#define SBI_NACL_SHMEM_SCRATCH_OFFSET 0x0000
+#define SBI_NACL_SHMEM_SCRATCH_SIZE 0x1000
+#define SBI_NACL_SHMEM_SRET_OFFSET 0x0000
+#define SBI_NACL_SHMEM_SRET_SIZE 0x0200
+#define SBI_NACL_SHMEM_AUTOSWAP_OFFSET (SBI_NACL_SHMEM_SRET_OFFSET + \
+ SBI_NACL_SHMEM_SRET_SIZE)
+#define SBI_NACL_SHMEM_AUTOSWAP_SIZE 0x0080
+#define SBI_NACL_SHMEM_UNUSED_OFFSET (SBI_NACL_SHMEM_AUTOSWAP_OFFSET + \
+ SBI_NACL_SHMEM_AUTOSWAP_SIZE)
+#define SBI_NACL_SHMEM_UNUSED_SIZE 0x0580
+#define SBI_NACL_SHMEM_HFENCE_OFFSET (SBI_NACL_SHMEM_UNUSED_OFFSET + \
+ SBI_NACL_SHMEM_UNUSED_SIZE)
+#define SBI_NACL_SHMEM_HFENCE_SIZE 0x0780
+#define SBI_NACL_SHMEM_DBITMAP_OFFSET (SBI_NACL_SHMEM_HFENCE_OFFSET + \
+ SBI_NACL_SHMEM_HFENCE_SIZE)
+#define SBI_NACL_SHMEM_DBITMAP_SIZE 0x0080
+#define SBI_NACL_SHMEM_CSR_OFFSET (SBI_NACL_SHMEM_DBITMAP_OFFSET + \
+ SBI_NACL_SHMEM_DBITMAP_SIZE)
+#define SBI_NACL_SHMEM_CSR_SIZE ((__riscv_xlen / 8) * 1024)
+#define SBI_NACL_SHMEM_SIZE (SBI_NACL_SHMEM_CSR_OFFSET + \
+ SBI_NACL_SHMEM_CSR_SIZE)
+
+#define SBI_NACL_SHMEM_CSR_INDEX(__csr_num) \
+ ((((__csr_num) & 0xc00) >> 2) | ((__csr_num) & 0xff))
+
+#define SBI_NACL_SHMEM_HFENCE_ENTRY_SZ ((__riscv_xlen / 8) * 4)
+#define SBI_NACL_SHMEM_HFENCE_ENTRY_MAX \
+ (SBI_NACL_SHMEM_HFENCE_SIZE / \
+ SBI_NACL_SHMEM_HFENCE_ENTRY_SZ)
+#define SBI_NACL_SHMEM_HFENCE_ENTRY(__num) \
+ (SBI_NACL_SHMEM_HFENCE_OFFSET + \
+ (__num) * SBI_NACL_SHMEM_HFENCE_ENTRY_SZ)
+#define SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(__num) \
+ SBI_NACL_SHMEM_HFENCE_ENTRY(__num)
+#define SBI_NACL_SHMEM_HFENCE_ENTRY_PNUM(__num)\
+ (SBI_NACL_SHMEM_HFENCE_ENTRY(__num) + (__riscv_xlen / 8))
+#define SBI_NACL_SHMEM_HFENCE_ENTRY_PCOUNT(__num)\
+ (SBI_NACL_SHMEM_HFENCE_ENTRY(__num) + \
+ ((__riscv_xlen / 8) * 3))
+
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_BITS 1
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_SHIFT \
+ (__riscv_xlen - SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_BITS)
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_MASK \
+ ((1UL << SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_BITS) - 1)
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_PEND \
+ (SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_MASK << \
+ SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_SHIFT)
+
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD1_BITS 3
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD1_SHIFT \
+ (SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_SHIFT - \
+ SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD1_BITS)
+
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_BITS 4
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_SHIFT \
+ (SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD1_SHIFT - \
+ SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_BITS)
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_MASK \
+ ((1UL << SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_BITS) - 1)
+
+#define SBI_NACL_SHMEM_HFENCE_TYPE_GVMA 0x0
+#define SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_ALL 0x1
+#define SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID 0x2
+#define SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID_ALL 0x3
+#define SBI_NACL_SHMEM_HFENCE_TYPE_VVMA 0x4
+#define SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ALL 0x5
+#define SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID 0x6
+#define SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID_ALL 0x7
+
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD2_BITS 1
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD2_SHIFT \
+ (SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_SHIFT - \
+ SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD2_BITS)
+
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_BITS 7
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_SHIFT \
+ (SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD2_SHIFT - \
+ SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_BITS)
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_MASK \
+ ((1UL << SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_BITS) - 1)
+#define SBI_NACL_SHMEM_HFENCE_ORDER_BASE 12
+
+#if __riscv_xlen == 32
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_BITS 9
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_BITS 7
+#else
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_BITS 16
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_BITS 14
+#endif
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_SHIFT \
+ SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_BITS
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_MASK \
+ ((1UL << SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_BITS) - 1)
+#define SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_MASK \
+ ((1UL << SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_BITS) - 1)
+
+#define SBI_NACL_SHMEM_AUTOSWAP_FLAG_HSTATUS BIT(0)
+#define SBI_NACL_SHMEM_AUTOSWAP_HSTATUS ((__riscv_xlen / 8) * 1)
+
+#define SBI_NACL_SHMEM_SRET_X(__i) ((__riscv_xlen / 8) * (__i))
+#define SBI_NACL_SHMEM_SRET_X_LAST 31
+
/* SBI spec version fields */
#define SBI_SPEC_VERSION_DEFAULT 0x1
#define SBI_SPEC_VERSION_MAJOR_SHIFT 24
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 08/13] RISC-V: KVM: Add common nested acceleration support
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
` (6 preceding siblings ...)
2024-07-19 16:09 ` [PATCH 07/13] RISC-V: Add defines for the SBI nested acceleration extension Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-18 18:19 ` Atish Patra
2024-07-19 16:09 ` [PATCH 09/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing H-extension CSRs Anup Patel
` (4 subsequent siblings)
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
Add a common nested acceleration support which will be shared by
all parts of KVM RISC-V. This nested acceleration support detects
and enables SBI NACL extension usage based on static keys which
ensures minimum impact on the non-nested scenario.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/include/asm/kvm_nacl.h | 205 ++++++++++++++++++++++++++++++
arch/riscv/kvm/Makefile | 1 +
arch/riscv/kvm/main.c | 53 +++++++-
arch/riscv/kvm/nacl.c | 152 ++++++++++++++++++++++
4 files changed, 409 insertions(+), 2 deletions(-)
create mode 100644 arch/riscv/include/asm/kvm_nacl.h
create mode 100644 arch/riscv/kvm/nacl.c
diff --git a/arch/riscv/include/asm/kvm_nacl.h b/arch/riscv/include/asm/kvm_nacl.h
new file mode 100644
index 000000000000..a704e8000a58
--- /dev/null
+++ b/arch/riscv/include/asm/kvm_nacl.h
@@ -0,0 +1,205 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2024 Ventana Micro Systems Inc.
+ */
+
+#ifndef __KVM_NACL_H
+#define __KVM_NACL_H
+
+#include <linux/jump_label.h>
+#include <linux/percpu.h>
+#include <asm/byteorder.h>
+#include <asm/csr.h>
+#include <asm/sbi.h>
+
+DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_available);
+#define kvm_riscv_nacl_available() \
+ static_branch_unlikely(&kvm_riscv_nacl_available)
+
+DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_csr_available);
+#define kvm_riscv_nacl_sync_csr_available() \
+ static_branch_unlikely(&kvm_riscv_nacl_sync_csr_available)
+
+DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_hfence_available);
+#define kvm_riscv_nacl_sync_hfence_available() \
+ static_branch_unlikely(&kvm_riscv_nacl_sync_hfence_available)
+
+DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_sret_available);
+#define kvm_riscv_nacl_sync_sret_available() \
+ static_branch_unlikely(&kvm_riscv_nacl_sync_sret_available)
+
+DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_autoswap_csr_available);
+#define kvm_riscv_nacl_autoswap_csr_available() \
+ static_branch_unlikely(&kvm_riscv_nacl_autoswap_csr_available)
+
+struct kvm_riscv_nacl {
+ void *shmem;
+ phys_addr_t shmem_phys;
+};
+DECLARE_PER_CPU(struct kvm_riscv_nacl, kvm_riscv_nacl);
+
+void __kvm_riscv_nacl_hfence(void *shmem,
+ unsigned long control,
+ unsigned long page_num,
+ unsigned long page_count);
+
+int kvm_riscv_nacl_enable(void);
+
+void kvm_riscv_nacl_disable(void);
+
+void kvm_riscv_nacl_exit(void);
+
+int kvm_riscv_nacl_init(void);
+
+#ifdef CONFIG_32BIT
+#define lelong_to_cpu(__x) le32_to_cpu(__x)
+#define cpu_to_lelong(__x) cpu_to_le32(__x)
+#else
+#define lelong_to_cpu(__x) le64_to_cpu(__x)
+#define cpu_to_lelong(__x) cpu_to_le64(__x)
+#endif
+
+#define nacl_shmem() \
+ this_cpu_ptr(&kvm_riscv_nacl)->shmem
+#define nacl_shmem_fast() \
+ (kvm_riscv_nacl_available() ? nacl_shmem() : NULL)
+
+#define nacl_sync_hfence(__e) \
+ sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_HFENCE, \
+ (__e), 0, 0, 0, 0, 0)
+
+#define nacl_hfence_mkconfig(__type, __order, __vmid, __asid) \
+({ \
+ unsigned long __c = SBI_NACL_SHMEM_HFENCE_CONFIG_PEND; \
+ __c |= ((__type) & SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_MASK) \
+ << SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_SHIFT; \
+ __c |= (((__order) - SBI_NACL_SHMEM_HFENCE_ORDER_BASE) & \
+ SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_MASK) \
+ << SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_SHIFT; \
+ __c |= ((__vmid) & SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_MASK) \
+ << SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_SHIFT; \
+ __c |= ((__asid) & SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_MASK); \
+ __c; \
+})
+
+#define nacl_hfence_mkpnum(__order, __addr) \
+ ((__addr) >> (__order))
+
+#define nacl_hfence_mkpcount(__order, __size) \
+ ((__size) >> (__order))
+
+#define nacl_hfence_gvma(__shmem, __gpa, __gpsz, __order) \
+__kvm_riscv_nacl_hfence(__shmem, \
+ nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA, \
+ __order, 0, 0), \
+ nacl_hfence_mkpnum(__order, __gpa), \
+ nacl_hfence_mkpcount(__order, __gpsz))
+
+#define nacl_hfence_gvma_all(__shmem) \
+__kvm_riscv_nacl_hfence(__shmem, \
+ nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_ALL, \
+ 0, 0, 0), 0, 0)
+
+#define nacl_hfence_gvma_vmid(__shmem, __vmid, __gpa, __gpsz, __order) \
+__kvm_riscv_nacl_hfence(__shmem, \
+ nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID, \
+ __order, __vmid, 0), \
+ nacl_hfence_mkpnum(__order, __gpa), \
+ nacl_hfence_mkpcount(__order, __gpsz))
+
+#define nacl_hfence_gvma_vmid_all(__shmem, __vmid) \
+__kvm_riscv_nacl_hfence(__shmem, \
+ nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID_ALL, \
+ 0, __vmid, 0), 0, 0)
+
+#define nacl_hfence_vvma(__shmem, __vmid, __gva, __gvsz, __order) \
+__kvm_riscv_nacl_hfence(__shmem, \
+ nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA, \
+ __order, __vmid, 0), \
+ nacl_hfence_mkpnum(__order, __gva), \
+ nacl_hfence_mkpcount(__order, __gvsz))
+
+#define nacl_hfence_vvma_all(__shmem, __vmid) \
+__kvm_riscv_nacl_hfence(__shmem, \
+ nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ALL, \
+ 0, __vmid, 0), 0, 0)
+
+#define nacl_hfence_vvma_asid(__shmem, __vmid, __asid, __gva, __gvsz, __order)\
+__kvm_riscv_nacl_hfence(__shmem, \
+ nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID, \
+ __order, __vmid, __asid), \
+ nacl_hfence_mkpnum(__order, __gva), \
+ nacl_hfence_mkpcount(__order, __gvsz))
+
+#define nacl_hfence_vvma_asid_all(__shmem, __vmid, __asid) \
+__kvm_riscv_nacl_hfence(__shmem, \
+ nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID_ALL, \
+ 0, __vmid, __asid), 0, 0)
+
+#define nacl_csr_read(__shmem, __csr) \
+({ \
+ unsigned long *__a = (__shmem) + SBI_NACL_SHMEM_CSR_OFFSET; \
+ lelong_to_cpu(__a[SBI_NACL_SHMEM_CSR_INDEX(__csr)]); \
+})
+
+#define nacl_csr_write(__shmem, __csr, __val) \
+do { \
+ void *__s = (__shmem); \
+ unsigned int __i = SBI_NACL_SHMEM_CSR_INDEX(__csr); \
+ unsigned long *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \
+ u8 *__b = (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \
+ __a[__i] = cpu_to_lelong(__val); \
+ __b[__i >> 3] |= 1U << (__i & 0x7); \
+} while (0)
+
+#define nacl_csr_swap(__shmem, __csr, __val) \
+({ \
+ void *__s = (__shmem); \
+ unsigned int __i = SBI_NACL_SHMEM_CSR_INDEX(__csr); \
+ unsigned long *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \
+ u8 *__b = (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \
+ unsigned long __r = lelong_to_cpu(__a[__i]); \
+ __a[__i] = cpu_to_lelong(__val); \
+ __b[__i >> 3] |= 1U << (__i & 0x7); \
+ __r; \
+})
+
+#define nacl_sync_csr(__csr) \
+ sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_CSR, \
+ (__csr), 0, 0, 0, 0, 0)
+
+#define ncsr_read(__csr) \
+({ \
+ unsigned long __r; \
+ if (kvm_riscv_nacl_available()) \
+ __r = nacl_csr_read(nacl_shmem(), __csr); \
+ else \
+ __r = csr_read(__csr); \
+ __r; \
+})
+
+#define ncsr_write(__csr, __val) \
+do { \
+ if (kvm_riscv_nacl_sync_csr_available()) \
+ nacl_csr_write(nacl_shmem(), __csr, __val); \
+ else \
+ csr_write(__csr, __val); \
+} while (0)
+
+#define ncsr_swap(__csr, __val) \
+({ \
+ unsigned long __r; \
+ if (kvm_riscv_nacl_sync_csr_available()) \
+ __r = nacl_csr_swap(nacl_shmem(), __csr, __val); \
+ else \
+ __r = csr_swap(__csr, __val); \
+ __r; \
+})
+
+#define nsync_csr(__csr) \
+do { \
+ if (kvm_riscv_nacl_sync_csr_available()) \
+ nacl_sync_csr(__csr); \
+} while (0)
+
+#endif
diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile
index c1eac0d093de..0fb1840c3e0a 100644
--- a/arch/riscv/kvm/Makefile
+++ b/arch/riscv/kvm/Makefile
@@ -16,6 +16,7 @@ kvm-y += aia_device.o
kvm-y += aia_imsic.o
kvm-y += main.o
kvm-y += mmu.o
+kvm-y += nacl.o
kvm-y += tlb.o
kvm-y += vcpu.o
kvm-y += vcpu_exit.o
diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c
index bab2ec34cd87..fd78f40bbb04 100644
--- a/arch/riscv/kvm/main.c
+++ b/arch/riscv/kvm/main.c
@@ -10,8 +10,8 @@
#include <linux/err.h>
#include <linux/module.h>
#include <linux/kvm_host.h>
-#include <asm/csr.h>
#include <asm/cpufeature.h>
+#include <asm/kvm_nacl.h>
#include <asm/sbi.h>
long kvm_arch_dev_ioctl(struct file *filp,
@@ -22,6 +22,12 @@ long kvm_arch_dev_ioctl(struct file *filp,
int kvm_arch_hardware_enable(void)
{
+ int rc;
+
+ rc = kvm_riscv_nacl_enable();
+ if (rc)
+ return rc;
+
csr_write(CSR_HEDELEG, KVM_HEDELEG_DEFAULT);
csr_write(CSR_HIDELEG, KVM_HIDELEG_DEFAULT);
@@ -49,11 +55,14 @@ void kvm_arch_hardware_disable(void)
csr_write(CSR_HVIP, 0);
csr_write(CSR_HEDELEG, 0);
csr_write(CSR_HIDELEG, 0);
+
+ kvm_riscv_nacl_disable();
}
static int __init riscv_kvm_init(void)
{
int rc;
+ char slist[64];
const char *str;
if (!riscv_isa_extension_available(NULL, h)) {
@@ -71,16 +80,53 @@ static int __init riscv_kvm_init(void)
return -ENODEV;
}
+ rc = kvm_riscv_nacl_init();
+ if (rc && rc != -ENODEV)
+ return rc;
+
kvm_riscv_gstage_mode_detect();
kvm_riscv_gstage_vmid_detect();
rc = kvm_riscv_aia_init();
- if (rc && rc != -ENODEV)
+ if (rc && rc != -ENODEV) {
+ kvm_riscv_nacl_exit();
return rc;
+ }
kvm_info("hypervisor extension available\n");
+ if (kvm_riscv_nacl_available()) {
+ rc = 0;
+ slist[0] = '\0';
+ if (kvm_riscv_nacl_sync_csr_available()) {
+ if (rc)
+ strcat(slist, ", ");
+ strcat(slist, "sync_csr");
+ rc++;
+ }
+ if (kvm_riscv_nacl_sync_hfence_available()) {
+ if (rc)
+ strcat(slist, ", ");
+ strcat(slist, "sync_hfence");
+ rc++;
+ }
+ if (kvm_riscv_nacl_sync_sret_available()) {
+ if (rc)
+ strcat(slist, ", ");
+ strcat(slist, "sync_sret");
+ rc++;
+ }
+ if (kvm_riscv_nacl_autoswap_csr_available()) {
+ if (rc)
+ strcat(slist, ", ");
+ strcat(slist, "autoswap_csr");
+ rc++;
+ }
+ kvm_info("using SBI nested acceleration with %s\n",
+ (rc) ? slist : "no features");
+ }
+
switch (kvm_riscv_gstage_mode()) {
case HGATP_MODE_SV32X4:
str = "Sv32x4";
@@ -108,6 +154,7 @@ static int __init riscv_kvm_init(void)
rc = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
if (rc) {
kvm_riscv_aia_exit();
+ kvm_riscv_nacl_exit();
return rc;
}
@@ -119,6 +166,8 @@ static void __exit riscv_kvm_exit(void)
{
kvm_riscv_aia_exit();
+ kvm_riscv_nacl_exit();
+
kvm_exit();
}
module_exit(riscv_kvm_exit);
diff --git a/arch/riscv/kvm/nacl.c b/arch/riscv/kvm/nacl.c
new file mode 100644
index 000000000000..08a95ad9ada2
--- /dev/null
+++ b/arch/riscv/kvm/nacl.c
@@ -0,0 +1,152 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2024 Ventana Micro Systems Inc.
+ */
+
+#include <linux/kvm_host.h>
+#include <linux/vmalloc.h>
+#include <asm/kvm_nacl.h>
+
+DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_available);
+DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_csr_available);
+DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_hfence_available);
+DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_sret_available);
+DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_autoswap_csr_available);
+DEFINE_PER_CPU(struct kvm_riscv_nacl, kvm_riscv_nacl);
+
+void __kvm_riscv_nacl_hfence(void *shmem,
+ unsigned long control,
+ unsigned long page_num,
+ unsigned long page_count)
+{
+ int i, ent = -1, try_count = 5;
+ unsigned long *entp;
+
+again:
+ for (i = 0; i < SBI_NACL_SHMEM_HFENCE_ENTRY_MAX; i++) {
+ entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i);
+ if (lelong_to_cpu(*entp) & SBI_NACL_SHMEM_HFENCE_CONFIG_PEND)
+ continue;
+
+ ent = i;
+ break;
+ }
+
+ if (ent < 0) {
+ if (try_count) {
+ nacl_sync_hfence(-1UL);
+ goto again;
+ } else {
+ pr_warn("KVM: No free entry in NACL shared memory\n");
+ return;
+ }
+ }
+
+ entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i);
+ *entp = cpu_to_lelong(control);
+ entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PNUM(i);
+ *entp = cpu_to_lelong(page_num);
+ entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PCOUNT(i);
+ *entp = cpu_to_lelong(page_count);
+}
+
+int kvm_riscv_nacl_enable(void)
+{
+ int rc;
+ struct sbiret ret;
+ struct kvm_riscv_nacl *nacl;
+
+ if (!kvm_riscv_nacl_available())
+ return 0;
+ nacl = this_cpu_ptr(&kvm_riscv_nacl);
+
+ ret = sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SET_SHMEM,
+ nacl->shmem_phys, 0, 0, 0, 0, 0);
+ rc = sbi_err_map_linux_errno(ret.error);
+ if (rc)
+ return rc;
+
+ return 0;
+}
+
+void kvm_riscv_nacl_disable(void)
+{
+ if (!kvm_riscv_nacl_available())
+ return;
+
+ sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SET_SHMEM,
+ SBI_SHMEM_DISABLE, SBI_SHMEM_DISABLE, 0, 0, 0, 0);
+}
+
+void kvm_riscv_nacl_exit(void)
+{
+ int cpu;
+ struct kvm_riscv_nacl *nacl;
+
+ if (!kvm_riscv_nacl_available())
+ return;
+
+ /* Allocate per-CPU shared memory */
+ for_each_possible_cpu(cpu) {
+ nacl = per_cpu_ptr(&kvm_riscv_nacl, cpu);
+ if (!nacl->shmem)
+ continue;
+
+ free_pages((unsigned long)nacl->shmem,
+ get_order(SBI_NACL_SHMEM_SIZE));
+ nacl->shmem = NULL;
+ nacl->shmem_phys = 0;
+ }
+}
+
+static long nacl_probe_feature(long feature_id)
+{
+ struct sbiret ret;
+
+ if (!kvm_riscv_nacl_available())
+ return 0;
+
+ ret = sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_PROBE_FEATURE,
+ feature_id, 0, 0, 0, 0, 0);
+ return ret.value;
+}
+
+int kvm_riscv_nacl_init(void)
+{
+ int cpu;
+ struct page *shmem_page;
+ struct kvm_riscv_nacl *nacl;
+
+ if (sbi_spec_version < sbi_mk_version(1, 0) ||
+ sbi_probe_extension(SBI_EXT_NACL) <= 0)
+ return -ENODEV;
+
+ /* Enable NACL support */
+ static_branch_enable(&kvm_riscv_nacl_available);
+
+ /* Probe NACL features */
+ if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_CSR))
+ static_branch_enable(&kvm_riscv_nacl_sync_csr_available);
+ if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_HFENCE))
+ static_branch_enable(&kvm_riscv_nacl_sync_hfence_available);
+ if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_SRET))
+ static_branch_enable(&kvm_riscv_nacl_sync_sret_available);
+ if (nacl_probe_feature(SBI_NACL_FEAT_AUTOSWAP_CSR))
+ static_branch_enable(&kvm_riscv_nacl_autoswap_csr_available);
+
+ /* Allocate per-CPU shared memory */
+ for_each_possible_cpu(cpu) {
+ nacl = per_cpu_ptr(&kvm_riscv_nacl, cpu);
+
+ shmem_page = alloc_pages(GFP_KERNEL | __GFP_ZERO,
+ get_order(SBI_NACL_SHMEM_SIZE));
+ if (!shmem_page) {
+ kvm_riscv_nacl_exit();
+ return -ENOMEM;
+ }
+ nacl->shmem = page_to_virt(shmem_page);
+ nacl->shmem_phys = page_to_phys(shmem_page);
+ }
+
+ return 0;
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 09/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing H-extension CSRs
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
` (7 preceding siblings ...)
2024-07-19 16:09 ` [PATCH 08/13] RISC-V: KVM: Add common nested acceleration support Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-18 19:31 ` Atish Patra
2024-07-19 16:09 ` [PATCH 10/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing AIA CSRs Anup Patel
` (3 subsequent siblings)
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
When running under some other hypervisor, prefer nacl_csr_xyz()
for accessing H-extension CSRs in the run-loop. This makes CSR
access faster whenever SBI nested acceleration is available.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/kvm/mmu.c | 4 +-
arch/riscv/kvm/vcpu.c | 103 +++++++++++++++++++++++++-----------
arch/riscv/kvm/vcpu_timer.c | 28 +++++-----
3 files changed, 87 insertions(+), 48 deletions(-)
diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
index b63650f9b966..45ace9138947 100644
--- a/arch/riscv/kvm/mmu.c
+++ b/arch/riscv/kvm/mmu.c
@@ -15,7 +15,7 @@
#include <linux/vmalloc.h>
#include <linux/kvm_host.h>
#include <linux/sched/signal.h>
-#include <asm/csr.h>
+#include <asm/kvm_nacl.h>
#include <asm/page.h>
#include <asm/pgtable.h>
@@ -732,7 +732,7 @@ void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu)
hgatp |= (READ_ONCE(k->vmid.vmid) << HGATP_VMID_SHIFT) & HGATP_VMID;
hgatp |= (k->pgd_phys >> PAGE_SHIFT) & HGATP_PPN;
- csr_write(CSR_HGATP, hgatp);
+ ncsr_write(CSR_HGATP, hgatp);
if (!kvm_riscv_gstage_vmid_bits())
kvm_riscv_local_hfence_gvma_all();
diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
index 957e1a5e081b..00baaf1b0136 100644
--- a/arch/riscv/kvm/vcpu.c
+++ b/arch/riscv/kvm/vcpu.c
@@ -17,8 +17,8 @@
#include <linux/sched/signal.h>
#include <linux/fs.h>
#include <linux/kvm_host.h>
-#include <asm/csr.h>
#include <asm/cacheflush.h>
+#include <asm/kvm_nacl.h>
#include <asm/kvm_vcpu_vector.h>
#define CREATE_TRACE_POINTS
@@ -361,10 +361,10 @@ void kvm_riscv_vcpu_sync_interrupts(struct kvm_vcpu *vcpu)
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
/* Read current HVIP and VSIE CSRs */
- csr->vsie = csr_read(CSR_VSIE);
+ csr->vsie = ncsr_read(CSR_VSIE);
/* Sync-up HVIP.VSSIP bit changes does by Guest */
- hvip = csr_read(CSR_HVIP);
+ hvip = ncsr_read(CSR_HVIP);
if ((csr->hvip ^ hvip) & (1UL << IRQ_VS_SOFT)) {
if (hvip & (1UL << IRQ_VS_SOFT)) {
if (!test_and_set_bit(IRQ_VS_SOFT,
@@ -561,26 +561,49 @@ static void kvm_riscv_vcpu_setup_config(struct kvm_vcpu *vcpu)
void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
{
+ void *nsh;
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
- csr_write(CSR_VSSTATUS, csr->vsstatus);
- csr_write(CSR_VSIE, csr->vsie);
- csr_write(CSR_VSTVEC, csr->vstvec);
- csr_write(CSR_VSSCRATCH, csr->vsscratch);
- csr_write(CSR_VSEPC, csr->vsepc);
- csr_write(CSR_VSCAUSE, csr->vscause);
- csr_write(CSR_VSTVAL, csr->vstval);
- csr_write(CSR_HEDELEG, cfg->hedeleg);
- csr_write(CSR_HVIP, csr->hvip);
- csr_write(CSR_VSATP, csr->vsatp);
- csr_write(CSR_HENVCFG, cfg->henvcfg);
- if (IS_ENABLED(CONFIG_32BIT))
- csr_write(CSR_HENVCFGH, cfg->henvcfg >> 32);
- if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
- csr_write(CSR_HSTATEEN0, cfg->hstateen0);
+ if (kvm_riscv_nacl_sync_csr_available()) {
+ nsh = nacl_shmem();
+ nacl_csr_write(nsh, CSR_VSSTATUS, csr->vsstatus);
+ nacl_csr_write(nsh, CSR_VSIE, csr->vsie);
+ nacl_csr_write(nsh, CSR_VSTVEC, csr->vstvec);
+ nacl_csr_write(nsh, CSR_VSSCRATCH, csr->vsscratch);
+ nacl_csr_write(nsh, CSR_VSEPC, csr->vsepc);
+ nacl_csr_write(nsh, CSR_VSCAUSE, csr->vscause);
+ nacl_csr_write(nsh, CSR_VSTVAL, csr->vstval);
+ nacl_csr_write(nsh, CSR_HEDELEG, cfg->hedeleg);
+ nacl_csr_write(nsh, CSR_HVIP, csr->hvip);
+ nacl_csr_write(nsh, CSR_VSATP, csr->vsatp);
+ nacl_csr_write(nsh, CSR_HENVCFG, cfg->henvcfg);
+ if (IS_ENABLED(CONFIG_32BIT))
+ nacl_csr_write(nsh, CSR_HENVCFGH, cfg->henvcfg >> 32);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
+ nacl_csr_write(nsh, CSR_HSTATEEN0, cfg->hstateen0);
+ if (IS_ENABLED(CONFIG_32BIT))
+ nacl_csr_write(nsh, CSR_HSTATEEN0H, cfg->hstateen0 >> 32);
+ }
+ } else {
+ csr_write(CSR_VSSTATUS, csr->vsstatus);
+ csr_write(CSR_VSIE, csr->vsie);
+ csr_write(CSR_VSTVEC, csr->vstvec);
+ csr_write(CSR_VSSCRATCH, csr->vsscratch);
+ csr_write(CSR_VSEPC, csr->vsepc);
+ csr_write(CSR_VSCAUSE, csr->vscause);
+ csr_write(CSR_VSTVAL, csr->vstval);
+ csr_write(CSR_HEDELEG, cfg->hedeleg);
+ csr_write(CSR_HVIP, csr->hvip);
+ csr_write(CSR_VSATP, csr->vsatp);
+ csr_write(CSR_HENVCFG, cfg->henvcfg);
if (IS_ENABLED(CONFIG_32BIT))
- csr_write(CSR_HSTATEEN0H, cfg->hstateen0 >> 32);
+ csr_write(CSR_HENVCFGH, cfg->henvcfg >> 32);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
+ csr_write(CSR_HSTATEEN0, cfg->hstateen0);
+ if (IS_ENABLED(CONFIG_32BIT))
+ csr_write(CSR_HSTATEEN0H, cfg->hstateen0 >> 32);
+ }
}
kvm_riscv_gstage_update_hgatp(vcpu);
@@ -603,6 +626,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
{
+ void *nsh;
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
vcpu->cpu = -1;
@@ -618,15 +642,28 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
vcpu->arch.isa);
kvm_riscv_vcpu_host_vector_restore(&vcpu->arch.host_context);
- csr->vsstatus = csr_read(CSR_VSSTATUS);
- csr->vsie = csr_read(CSR_VSIE);
- csr->vstvec = csr_read(CSR_VSTVEC);
- csr->vsscratch = csr_read(CSR_VSSCRATCH);
- csr->vsepc = csr_read(CSR_VSEPC);
- csr->vscause = csr_read(CSR_VSCAUSE);
- csr->vstval = csr_read(CSR_VSTVAL);
- csr->hvip = csr_read(CSR_HVIP);
- csr->vsatp = csr_read(CSR_VSATP);
+ if (kvm_riscv_nacl_available()) {
+ nsh = nacl_shmem();
+ csr->vsstatus = nacl_csr_read(nsh, CSR_VSSTATUS);
+ csr->vsie = nacl_csr_read(nsh, CSR_VSIE);
+ csr->vstvec = nacl_csr_read(nsh, CSR_VSTVEC);
+ csr->vsscratch = nacl_csr_read(nsh, CSR_VSSCRATCH);
+ csr->vsepc = nacl_csr_read(nsh, CSR_VSEPC);
+ csr->vscause = nacl_csr_read(nsh, CSR_VSCAUSE);
+ csr->vstval = nacl_csr_read(nsh, CSR_VSTVAL);
+ csr->hvip = nacl_csr_read(nsh, CSR_HVIP);
+ csr->vsatp = nacl_csr_read(nsh, CSR_VSATP);
+ } else {
+ csr->vsstatus = csr_read(CSR_VSSTATUS);
+ csr->vsie = csr_read(CSR_VSIE);
+ csr->vstvec = csr_read(CSR_VSTVEC);
+ csr->vsscratch = csr_read(CSR_VSSCRATCH);
+ csr->vsepc = csr_read(CSR_VSEPC);
+ csr->vscause = csr_read(CSR_VSCAUSE);
+ csr->vstval = csr_read(CSR_VSTVAL);
+ csr->hvip = csr_read(CSR_HVIP);
+ csr->vsatp = csr_read(CSR_VSATP);
+ }
}
static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu)
@@ -681,7 +718,7 @@ static void kvm_riscv_update_hvip(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
- csr_write(CSR_HVIP, csr->hvip);
+ ncsr_write(CSR_HVIP, csr->hvip);
kvm_riscv_vcpu_aia_update_hvip(vcpu);
}
@@ -728,7 +765,9 @@ static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
kvm_riscv_vcpu_swap_in_guest_state(vcpu);
guest_state_enter_irqoff();
- hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
+ hcntx->hstatus = ncsr_swap(CSR_HSTATUS, gcntx->hstatus);
+
+ nsync_csr(-1UL);
__kvm_riscv_switch_to(&vcpu->arch);
@@ -863,8 +902,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
trap.sepc = vcpu->arch.guest_context.sepc;
trap.scause = csr_read(CSR_SCAUSE);
trap.stval = csr_read(CSR_STVAL);
- trap.htval = csr_read(CSR_HTVAL);
- trap.htinst = csr_read(CSR_HTINST);
+ trap.htval = ncsr_read(CSR_HTVAL);
+ trap.htinst = ncsr_read(CSR_HTINST);
/* Syncup interrupts state with HW */
kvm_riscv_vcpu_sync_interrupts(vcpu);
diff --git a/arch/riscv/kvm/vcpu_timer.c b/arch/riscv/kvm/vcpu_timer.c
index 75486b25ac45..96e7a4e463f7 100644
--- a/arch/riscv/kvm/vcpu_timer.c
+++ b/arch/riscv/kvm/vcpu_timer.c
@@ -11,8 +11,8 @@
#include <linux/kvm_host.h>
#include <linux/uaccess.h>
#include <clocksource/timer-riscv.h>
-#include <asm/csr.h>
#include <asm/delay.h>
+#include <asm/kvm_nacl.h>
#include <asm/kvm_vcpu_timer.h>
static u64 kvm_riscv_current_cycles(struct kvm_guest_timer *gt)
@@ -72,12 +72,12 @@ static int kvm_riscv_vcpu_timer_cancel(struct kvm_vcpu_timer *t)
static int kvm_riscv_vcpu_update_vstimecmp(struct kvm_vcpu *vcpu, u64 ncycles)
{
#if defined(CONFIG_32BIT)
- csr_write(CSR_VSTIMECMP, ncycles & 0xFFFFFFFF);
- csr_write(CSR_VSTIMECMPH, ncycles >> 32);
+ ncsr_write(CSR_VSTIMECMP, ncycles & 0xFFFFFFFF);
+ ncsr_write(CSR_VSTIMECMPH, ncycles >> 32);
#else
- csr_write(CSR_VSTIMECMP, ncycles);
+ ncsr_write(CSR_VSTIMECMP, ncycles);
#endif
- return 0;
+ return 0;
}
static int kvm_riscv_vcpu_update_hrtimer(struct kvm_vcpu *vcpu, u64 ncycles)
@@ -289,10 +289,10 @@ static void kvm_riscv_vcpu_update_timedelta(struct kvm_vcpu *vcpu)
struct kvm_guest_timer *gt = &vcpu->kvm->arch.timer;
#if defined(CONFIG_32BIT)
- csr_write(CSR_HTIMEDELTA, (u32)(gt->time_delta));
- csr_write(CSR_HTIMEDELTAH, (u32)(gt->time_delta >> 32));
+ ncsr_write(CSR_HTIMEDELTA, (u32)(gt->time_delta));
+ ncsr_write(CSR_HTIMEDELTAH, (u32)(gt->time_delta >> 32));
#else
- csr_write(CSR_HTIMEDELTA, gt->time_delta);
+ ncsr_write(CSR_HTIMEDELTA, gt->time_delta);
#endif
}
@@ -306,10 +306,10 @@ void kvm_riscv_vcpu_timer_restore(struct kvm_vcpu *vcpu)
return;
#if defined(CONFIG_32BIT)
- csr_write(CSR_VSTIMECMP, (u32)t->next_cycles);
- csr_write(CSR_VSTIMECMPH, (u32)(t->next_cycles >> 32));
+ ncsr_write(CSR_VSTIMECMP, (u32)t->next_cycles);
+ ncsr_write(CSR_VSTIMECMPH, (u32)(t->next_cycles >> 32));
#else
- csr_write(CSR_VSTIMECMP, t->next_cycles);
+ ncsr_write(CSR_VSTIMECMP, t->next_cycles);
#endif
/* timer should be enabled for the remaining operations */
@@ -327,10 +327,10 @@ void kvm_riscv_vcpu_timer_sync(struct kvm_vcpu *vcpu)
return;
#if defined(CONFIG_32BIT)
- t->next_cycles = csr_read(CSR_VSTIMECMP);
- t->next_cycles |= (u64)csr_read(CSR_VSTIMECMPH) << 32;
+ t->next_cycles = ncsr_read(CSR_VSTIMECMP);
+ t->next_cycles |= (u64)ncsr_read(CSR_VSTIMECMPH) << 32;
#else
- t->next_cycles = csr_read(CSR_VSTIMECMP);
+ t->next_cycles = ncsr_read(CSR_VSTIMECMP);
#endif
}
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 10/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing AIA CSRs
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
` (8 preceding siblings ...)
2024-07-19 16:09 ` [PATCH 09/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing H-extension CSRs Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-18 19:34 ` Atish Patra
2024-07-19 16:09 ` [PATCH 11/13] RISC-V: KVM: Use SBI sync SRET call when available Anup Patel
` (2 subsequent siblings)
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
When running under some other hypervisor, prefer nacl_csr_xyz()
for accessing AIA CSRs in the run-loop. This makes CSR access
faster whenever SBI nested acceleration is available.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/kvm/aia.c | 97 ++++++++++++++++++++++++++++----------------
1 file changed, 63 insertions(+), 34 deletions(-)
diff --git a/arch/riscv/kvm/aia.c b/arch/riscv/kvm/aia.c
index 8ffae0330c89..dcced4db7fe8 100644
--- a/arch/riscv/kvm/aia.c
+++ b/arch/riscv/kvm/aia.c
@@ -16,6 +16,7 @@
#include <linux/percpu.h>
#include <linux/spinlock.h>
#include <asm/cpufeature.h>
+#include <asm/kvm_nacl.h>
struct aia_hgei_control {
raw_spinlock_t lock;
@@ -88,7 +89,7 @@ void kvm_riscv_vcpu_aia_sync_interrupts(struct kvm_vcpu *vcpu)
struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
if (kvm_riscv_aia_available())
- csr->vsieh = csr_read(CSR_VSIEH);
+ csr->vsieh = ncsr_read(CSR_VSIEH);
}
#endif
@@ -115,7 +116,7 @@ bool kvm_riscv_vcpu_aia_has_interrupts(struct kvm_vcpu *vcpu, u64 mask)
hgei = aia_find_hgei(vcpu);
if (hgei > 0)
- return !!(csr_read(CSR_HGEIP) & BIT(hgei));
+ return !!(ncsr_read(CSR_HGEIP) & BIT(hgei));
return false;
}
@@ -128,45 +129,73 @@ void kvm_riscv_vcpu_aia_update_hvip(struct kvm_vcpu *vcpu)
return;
#ifdef CONFIG_32BIT
- csr_write(CSR_HVIPH, vcpu->arch.aia_context.guest_csr.hviph);
+ ncsr_write(CSR_HVIPH, vcpu->arch.aia_context.guest_csr.hviph);
#endif
- csr_write(CSR_HVICTL, aia_hvictl_value(!!(csr->hvip & BIT(IRQ_VS_EXT))));
+ ncsr_write(CSR_HVICTL, aia_hvictl_value(!!(csr->hvip & BIT(IRQ_VS_EXT))));
}
void kvm_riscv_vcpu_aia_load(struct kvm_vcpu *vcpu, int cpu)
{
struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
+ void *nsh;
if (!kvm_riscv_aia_available())
return;
- csr_write(CSR_VSISELECT, csr->vsiselect);
- csr_write(CSR_HVIPRIO1, csr->hviprio1);
- csr_write(CSR_HVIPRIO2, csr->hviprio2);
+ if (kvm_riscv_nacl_sync_csr_available()) {
+ nsh = nacl_shmem();
+ nacl_csr_write(nsh, CSR_VSISELECT, csr->vsiselect);
+ nacl_csr_write(nsh, CSR_HVIPRIO1, csr->hviprio1);
+ nacl_csr_write(nsh, CSR_HVIPRIO2, csr->hviprio2);
+#ifdef CONFIG_32BIT
+ nacl_csr_write(nsh, CSR_VSIEH, csr->vsieh);
+ nacl_csr_write(nsh, CSR_HVIPH, csr->hviph);
+ nacl_csr_write(nsh, CSR_HVIPRIO1H, csr->hviprio1h);
+ nacl_csr_write(nsh, CSR_HVIPRIO2H, csr->hviprio2h);
+#endif
+ } else {
+ csr_write(CSR_VSISELECT, csr->vsiselect);
+ csr_write(CSR_HVIPRIO1, csr->hviprio1);
+ csr_write(CSR_HVIPRIO2, csr->hviprio2);
#ifdef CONFIG_32BIT
- csr_write(CSR_VSIEH, csr->vsieh);
- csr_write(CSR_HVIPH, csr->hviph);
- csr_write(CSR_HVIPRIO1H, csr->hviprio1h);
- csr_write(CSR_HVIPRIO2H, csr->hviprio2h);
+ csr_write(CSR_VSIEH, csr->vsieh);
+ csr_write(CSR_HVIPH, csr->hviph);
+ csr_write(CSR_HVIPRIO1H, csr->hviprio1h);
+ csr_write(CSR_HVIPRIO2H, csr->hviprio2h);
#endif
+ }
}
void kvm_riscv_vcpu_aia_put(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
+ void *nsh;
if (!kvm_riscv_aia_available())
return;
- csr->vsiselect = csr_read(CSR_VSISELECT);
- csr->hviprio1 = csr_read(CSR_HVIPRIO1);
- csr->hviprio2 = csr_read(CSR_HVIPRIO2);
+ if (kvm_riscv_nacl_available()) {
+ nsh = nacl_shmem();
+ csr->vsiselect = nacl_csr_read(nsh, CSR_VSISELECT);
+ csr->hviprio1 = nacl_csr_read(nsh, CSR_HVIPRIO1);
+ csr->hviprio2 = nacl_csr_read(nsh, CSR_HVIPRIO2);
#ifdef CONFIG_32BIT
- csr->vsieh = csr_read(CSR_VSIEH);
- csr->hviph = csr_read(CSR_HVIPH);
- csr->hviprio1h = csr_read(CSR_HVIPRIO1H);
- csr->hviprio2h = csr_read(CSR_HVIPRIO2H);
+ csr->vsieh = nacl_csr_read(nsh, CSR_VSIEH);
+ csr->hviph = nacl_csr_read(nsh, CSR_HVIPH);
+ csr->hviprio1h = nacl_csr_read(nsh, CSR_HVIPRIO1H);
+ csr->hviprio2h = nacl_csr_read(nsh, CSR_HVIPRIO2H);
#endif
+ } else {
+ csr->vsiselect = csr_read(CSR_VSISELECT);
+ csr->hviprio1 = csr_read(CSR_HVIPRIO1);
+ csr->hviprio2 = csr_read(CSR_HVIPRIO2);
+#ifdef CONFIG_32BIT
+ csr->vsieh = csr_read(CSR_VSIEH);
+ csr->hviph = csr_read(CSR_HVIPH);
+ csr->hviprio1h = csr_read(CSR_HVIPRIO1H);
+ csr->hviprio2h = csr_read(CSR_HVIPRIO2H);
+#endif
+ }
}
int kvm_riscv_vcpu_aia_get_csr(struct kvm_vcpu *vcpu,
@@ -250,20 +279,20 @@ static u8 aia_get_iprio8(struct kvm_vcpu *vcpu, unsigned int irq)
switch (bitpos / BITS_PER_LONG) {
case 0:
- hviprio = csr_read(CSR_HVIPRIO1);
+ hviprio = ncsr_read(CSR_HVIPRIO1);
break;
case 1:
#ifndef CONFIG_32BIT
- hviprio = csr_read(CSR_HVIPRIO2);
+ hviprio = ncsr_read(CSR_HVIPRIO2);
break;
#else
- hviprio = csr_read(CSR_HVIPRIO1H);
+ hviprio = ncsr_read(CSR_HVIPRIO1H);
break;
case 2:
- hviprio = csr_read(CSR_HVIPRIO2);
+ hviprio = ncsr_read(CSR_HVIPRIO2);
break;
case 3:
- hviprio = csr_read(CSR_HVIPRIO2H);
+ hviprio = ncsr_read(CSR_HVIPRIO2H);
break;
#endif
default:
@@ -283,20 +312,20 @@ static void aia_set_iprio8(struct kvm_vcpu *vcpu, unsigned int irq, u8 prio)
switch (bitpos / BITS_PER_LONG) {
case 0:
- hviprio = csr_read(CSR_HVIPRIO1);
+ hviprio = ncsr_read(CSR_HVIPRIO1);
break;
case 1:
#ifndef CONFIG_32BIT
- hviprio = csr_read(CSR_HVIPRIO2);
+ hviprio = ncsr_read(CSR_HVIPRIO2);
break;
#else
- hviprio = csr_read(CSR_HVIPRIO1H);
+ hviprio = ncsr_read(CSR_HVIPRIO1H);
break;
case 2:
- hviprio = csr_read(CSR_HVIPRIO2);
+ hviprio = ncsr_read(CSR_HVIPRIO2);
break;
case 3:
- hviprio = csr_read(CSR_HVIPRIO2H);
+ hviprio = ncsr_read(CSR_HVIPRIO2H);
break;
#endif
default:
@@ -308,20 +337,20 @@ static void aia_set_iprio8(struct kvm_vcpu *vcpu, unsigned int irq, u8 prio)
switch (bitpos / BITS_PER_LONG) {
case 0:
- csr_write(CSR_HVIPRIO1, hviprio);
+ ncsr_write(CSR_HVIPRIO1, hviprio);
break;
case 1:
#ifndef CONFIG_32BIT
- csr_write(CSR_HVIPRIO2, hviprio);
+ ncsr_write(CSR_HVIPRIO2, hviprio);
break;
#else
- csr_write(CSR_HVIPRIO1H, hviprio);
+ ncsr_write(CSR_HVIPRIO1H, hviprio);
break;
case 2:
- csr_write(CSR_HVIPRIO2, hviprio);
+ ncsr_write(CSR_HVIPRIO2, hviprio);
break;
case 3:
- csr_write(CSR_HVIPRIO2H, hviprio);
+ ncsr_write(CSR_HVIPRIO2H, hviprio);
break;
#endif
default:
@@ -377,7 +406,7 @@ int kvm_riscv_vcpu_aia_rmw_ireg(struct kvm_vcpu *vcpu, unsigned int csr_num,
return KVM_INSN_ILLEGAL_TRAP;
/* First try to emulate in kernel space */
- isel = csr_read(CSR_VSISELECT) & ISELECT_MASK;
+ isel = ncsr_read(CSR_VSISELECT) & ISELECT_MASK;
if (isel >= ISELECT_IPRIO0 && isel <= ISELECT_IPRIO15)
return aia_rmw_iprio(vcpu, isel, val, new_val, wr_mask);
else if (isel >= IMSIC_FIRST && isel <= IMSIC_LAST &&
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 11/13] RISC-V: KVM: Use SBI sync SRET call when available
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
` (9 preceding siblings ...)
2024-07-19 16:09 ` [PATCH 10/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing AIA CSRs Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-18 20:03 ` Atish Patra
2024-07-19 16:09 ` [PATCH 12/13] RISC-V: KVM: Save trap CSRs in kvm_riscv_vcpu_enter_exit() Anup Patel
2024-07-19 16:09 ` [PATCH 13/13] RISC-V: KVM: Use NACL HFENCEs for KVM request based HFENCEs Anup Patel
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
Implement an optimized KVM world-switch using SBI sync SRET call
when SBI nested acceleration extension is available. This improves
KVM world-switch when KVM RISC-V is running as a Guest under some
other hypervisor.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/include/asm/kvm_nacl.h | 32 +++++++++++++++++++++
arch/riscv/kvm/vcpu.c | 48 ++++++++++++++++++++++++++++---
arch/riscv/kvm/vcpu_switch.S | 29 +++++++++++++++++++
3 files changed, 105 insertions(+), 4 deletions(-)
diff --git a/arch/riscv/include/asm/kvm_nacl.h b/arch/riscv/include/asm/kvm_nacl.h
index a704e8000a58..5e74238ea525 100644
--- a/arch/riscv/include/asm/kvm_nacl.h
+++ b/arch/riscv/include/asm/kvm_nacl.h
@@ -12,6 +12,8 @@
#include <asm/csr.h>
#include <asm/sbi.h>
+struct kvm_vcpu_arch;
+
DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_available);
#define kvm_riscv_nacl_available() \
static_branch_unlikely(&kvm_riscv_nacl_available)
@@ -43,6 +45,10 @@ void __kvm_riscv_nacl_hfence(void *shmem,
unsigned long page_num,
unsigned long page_count);
+void __kvm_riscv_nacl_switch_to(struct kvm_vcpu_arch *vcpu_arch,
+ unsigned long sbi_ext_id,
+ unsigned long sbi_func_id);
+
int kvm_riscv_nacl_enable(void);
void kvm_riscv_nacl_disable(void);
@@ -64,6 +70,32 @@ int kvm_riscv_nacl_init(void);
#define nacl_shmem_fast() \
(kvm_riscv_nacl_available() ? nacl_shmem() : NULL)
+#define nacl_scratch_read_long(__shmem, __offset) \
+({ \
+ unsigned long *__p = (__shmem) + \
+ SBI_NACL_SHMEM_SCRATCH_OFFSET + \
+ (__offset); \
+ lelong_to_cpu(*__p); \
+})
+
+#define nacl_scratch_write_long(__shmem, __offset, __val) \
+do { \
+ unsigned long *__p = (__shmem) + \
+ SBI_NACL_SHMEM_SCRATCH_OFFSET + \
+ (__offset); \
+ *__p = cpu_to_lelong(__val); \
+} while (0)
+
+#define nacl_scratch_write_longs(__shmem, __offset, __array, __count) \
+do { \
+ unsigned int __i; \
+ unsigned long *__p = (__shmem) + \
+ SBI_NACL_SHMEM_SCRATCH_OFFSET + \
+ (__offset); \
+ for (__i = 0; __i < (__count); __i++) \
+ __p[__i] = cpu_to_lelong((__array)[__i]); \
+} while (0)
+
#define nacl_sync_hfence(__e) \
sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_HFENCE, \
(__e), 0, 0, 0, 0, 0)
diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
index 00baaf1b0136..fe849fb1aaab 100644
--- a/arch/riscv/kvm/vcpu.c
+++ b/arch/riscv/kvm/vcpu.c
@@ -759,19 +759,59 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v
*/
static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
{
+ void *nsh;
struct kvm_cpu_context *gcntx = &vcpu->arch.guest_context;
struct kvm_cpu_context *hcntx = &vcpu->arch.host_context;
kvm_riscv_vcpu_swap_in_guest_state(vcpu);
guest_state_enter_irqoff();
- hcntx->hstatus = ncsr_swap(CSR_HSTATUS, gcntx->hstatus);
+ if (kvm_riscv_nacl_sync_sret_available()) {
+ nsh = nacl_shmem();
- nsync_csr(-1UL);
+ if (kvm_riscv_nacl_autoswap_csr_available()) {
+ hcntx->hstatus =
+ nacl_csr_read(nsh, CSR_HSTATUS);
+ nacl_scratch_write_long(nsh,
+ SBI_NACL_SHMEM_AUTOSWAP_OFFSET +
+ SBI_NACL_SHMEM_AUTOSWAP_HSTATUS,
+ gcntx->hstatus);
+ nacl_scratch_write_long(nsh,
+ SBI_NACL_SHMEM_AUTOSWAP_OFFSET,
+ SBI_NACL_SHMEM_AUTOSWAP_FLAG_HSTATUS);
+ } else if (kvm_riscv_nacl_sync_csr_available()) {
+ hcntx->hstatus = nacl_csr_swap(nsh,
+ CSR_HSTATUS, gcntx->hstatus);
+ } else {
+ hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
+ }
- __kvm_riscv_switch_to(&vcpu->arch);
+ nacl_scratch_write_longs(nsh,
+ SBI_NACL_SHMEM_SRET_OFFSET +
+ SBI_NACL_SHMEM_SRET_X(1),
+ &gcntx->ra,
+ SBI_NACL_SHMEM_SRET_X_LAST);
+
+ __kvm_riscv_nacl_switch_to(&vcpu->arch, SBI_EXT_NACL,
+ SBI_EXT_NACL_SYNC_SRET);
+
+ if (kvm_riscv_nacl_autoswap_csr_available()) {
+ nacl_scratch_write_long(nsh,
+ SBI_NACL_SHMEM_AUTOSWAP_OFFSET,
+ 0);
+ gcntx->hstatus = nacl_scratch_read_long(nsh,
+ SBI_NACL_SHMEM_AUTOSWAP_OFFSET +
+ SBI_NACL_SHMEM_AUTOSWAP_HSTATUS);
+ } else {
+ gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
+ }
+ } else {
+ hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
- gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
+ __kvm_riscv_switch_to(&vcpu->arch);
+
+ gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
+ }
vcpu->arch.last_exit_cpu = vcpu->cpu;
guest_state_exit_irqoff();
diff --git a/arch/riscv/kvm/vcpu_switch.S b/arch/riscv/kvm/vcpu_switch.S
index 9f13e5ce6a18..47686bcb21e0 100644
--- a/arch/riscv/kvm/vcpu_switch.S
+++ b/arch/riscv/kvm/vcpu_switch.S
@@ -218,6 +218,35 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
ret
SYM_FUNC_END(__kvm_riscv_switch_to)
+ /*
+ * Parameters:
+ * A0 <= Pointer to struct kvm_vcpu_arch
+ * A1 <= SBI extension ID
+ * A2 <= SBI function ID
+ */
+SYM_FUNC_START(__kvm_riscv_nacl_switch_to)
+ SAVE_HOST_GPRS
+
+ SAVE_HOST_AND_RESTORE_GUEST_CSRS .Lkvm_nacl_switch_return
+
+ /* Resume Guest using SBI nested acceleration */
+ add a6, a2, zero
+ add a7, a1, zero
+ ecall
+
+ /* Back to Host */
+ .align 2
+.Lkvm_nacl_switch_return:
+ SAVE_GUEST_GPRS
+
+ SAVE_GUEST_AND_RESTORE_HOST_CSRS
+
+ RESTORE_HOST_GPRS
+
+ /* Return to C code */
+ ret
+SYM_FUNC_END(__kvm_riscv_nacl_switch_to)
+
SYM_CODE_START(__kvm_riscv_unpriv_trap)
/*
* We assume that faulting unpriv load/store instruction is
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 12/13] RISC-V: KVM: Save trap CSRs in kvm_riscv_vcpu_enter_exit()
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
` (10 preceding siblings ...)
2024-07-19 16:09 ` [PATCH 11/13] RISC-V: KVM: Use SBI sync SRET call when available Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
2024-10-18 20:11 ` Atish Patra
2024-07-19 16:09 ` [PATCH 13/13] RISC-V: KVM: Use NACL HFENCEs for KVM request based HFENCEs Anup Patel
12 siblings, 1 reply; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
Save trap CSRs in the kvm_riscv_vcpu_enter_exit() function instead of
the kvm_arch_vcpu_ioctl_run() function so that HTVAL and HTINST CSRs
are accessed in more optimized manner while running under some other
hypervisor.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/kvm/vcpu.c | 34 +++++++++++++++++++++-------------
1 file changed, 21 insertions(+), 13 deletions(-)
diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
index fe849fb1aaab..854d98aa165e 100644
--- a/arch/riscv/kvm/vcpu.c
+++ b/arch/riscv/kvm/vcpu.c
@@ -757,12 +757,21 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v
* This must be noinstr as instrumentation may make use of RCU, and this is not
* safe during the EQS.
*/
-static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
+static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu,
+ struct kvm_cpu_trap *trap)
{
void *nsh;
struct kvm_cpu_context *gcntx = &vcpu->arch.guest_context;
struct kvm_cpu_context *hcntx = &vcpu->arch.host_context;
+ /*
+ * We save trap CSRs (such as SEPC, SCAUSE, STVAL, HTVAL, and
+ * HTINST) here because we do local_irq_enable() after this
+ * function in kvm_arch_vcpu_ioctl_run() which can result in
+ * an interrupt immediately after local_irq_enable() and can
+ * potentially change trap CSRs.
+ */
+
kvm_riscv_vcpu_swap_in_guest_state(vcpu);
guest_state_enter_irqoff();
@@ -805,14 +814,24 @@ static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
} else {
gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
}
+
+ trap->htval = nacl_csr_read(nsh, CSR_HTVAL);
+ trap->htinst = nacl_csr_read(nsh, CSR_HTINST);
} else {
hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
__kvm_riscv_switch_to(&vcpu->arch);
gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
+
+ trap->htval = csr_read(CSR_HTVAL);
+ trap->htinst = csr_read(CSR_HTINST);
}
+ trap->sepc = gcntx->sepc;
+ trap->scause = csr_read(CSR_SCAUSE);
+ trap->stval = csr_read(CSR_STVAL);
+
vcpu->arch.last_exit_cpu = vcpu->cpu;
guest_state_exit_irqoff();
kvm_riscv_vcpu_swap_in_host_state(vcpu);
@@ -929,22 +948,11 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
guest_timing_enter_irqoff();
- kvm_riscv_vcpu_enter_exit(vcpu);
+ kvm_riscv_vcpu_enter_exit(vcpu, &trap);
vcpu->mode = OUTSIDE_GUEST_MODE;
vcpu->stat.exits++;
- /*
- * Save SCAUSE, STVAL, HTVAL, and HTINST because we might
- * get an interrupt between __kvm_riscv_switch_to() and
- * local_irq_enable() which can potentially change CSRs.
- */
- trap.sepc = vcpu->arch.guest_context.sepc;
- trap.scause = csr_read(CSR_SCAUSE);
- trap.stval = csr_read(CSR_STVAL);
- trap.htval = ncsr_read(CSR_HTVAL);
- trap.htinst = ncsr_read(CSR_HTINST);
-
/* Syncup interrupts state with HW */
kvm_riscv_vcpu_sync_interrupts(vcpu);
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 13/13] RISC-V: KVM: Use NACL HFENCEs for KVM request based HFENCEs
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
` (11 preceding siblings ...)
2024-07-19 16:09 ` [PATCH 12/13] RISC-V: KVM: Save trap CSRs in kvm_riscv_vcpu_enter_exit() Anup Patel
@ 2024-07-19 16:09 ` Anup Patel
12 siblings, 0 replies; 29+ messages in thread
From: Anup Patel @ 2024-07-19 16:09 UTC (permalink / raw)
To: Palmer Dabbelt, Paul Walmsley
Cc: Atish Patra, Andrew Jones, Anup Patel, kvm, kvm-riscv,
linux-riscv, linux-kernel, Anup Patel
When running under some other hypervisor, use SBI NACL based HFENCEs
for TLB shoot-down via KVM requests. This makes HFENCEs faster whenever
SBI nested acceleration is available.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
---
arch/riscv/kvm/tlb.c | 57 +++++++++++++++++++++++++++++++-------------
1 file changed, 40 insertions(+), 17 deletions(-)
diff --git a/arch/riscv/kvm/tlb.c b/arch/riscv/kvm/tlb.c
index 23c0e82b5103..2f91ea5f8493 100644
--- a/arch/riscv/kvm/tlb.c
+++ b/arch/riscv/kvm/tlb.c
@@ -14,6 +14,7 @@
#include <asm/csr.h>
#include <asm/cpufeature.h>
#include <asm/insn-def.h>
+#include <asm/kvm_nacl.h>
#define has_svinval() riscv_has_extension_unlikely(RISCV_ISA_EXT_SVINVAL)
@@ -186,18 +187,24 @@ void kvm_riscv_fence_i_process(struct kvm_vcpu *vcpu)
void kvm_riscv_hfence_gvma_vmid_all_process(struct kvm_vcpu *vcpu)
{
- struct kvm_vmid *vmid;
+ struct kvm_vmid *v = &vcpu->kvm->arch.vmid;
+ unsigned long vmid = READ_ONCE(v->vmid);
- vmid = &vcpu->kvm->arch.vmid;
- kvm_riscv_local_hfence_gvma_vmid_all(READ_ONCE(vmid->vmid));
+ if (kvm_riscv_nacl_available())
+ nacl_hfence_gvma_vmid_all(nacl_shmem(), vmid);
+ else
+ kvm_riscv_local_hfence_gvma_vmid_all(vmid);
}
void kvm_riscv_hfence_vvma_all_process(struct kvm_vcpu *vcpu)
{
- struct kvm_vmid *vmid;
+ struct kvm_vmid *v = &vcpu->kvm->arch.vmid;
+ unsigned long vmid = READ_ONCE(v->vmid);
- vmid = &vcpu->kvm->arch.vmid;
- kvm_riscv_local_hfence_vvma_all(READ_ONCE(vmid->vmid));
+ if (kvm_riscv_nacl_available())
+ nacl_hfence_vvma_all(nacl_shmem(), vmid);
+ else
+ kvm_riscv_local_hfence_vvma_all(vmid);
}
static bool vcpu_hfence_dequeue(struct kvm_vcpu *vcpu,
@@ -251,6 +258,7 @@ static bool vcpu_hfence_enqueue(struct kvm_vcpu *vcpu,
void kvm_riscv_hfence_process(struct kvm_vcpu *vcpu)
{
+ unsigned long vmid;
struct kvm_riscv_hfence d = { 0 };
struct kvm_vmid *v = &vcpu->kvm->arch.vmid;
@@ -259,26 +267,41 @@ void kvm_riscv_hfence_process(struct kvm_vcpu *vcpu)
case KVM_RISCV_HFENCE_UNKNOWN:
break;
case KVM_RISCV_HFENCE_GVMA_VMID_GPA:
- kvm_riscv_local_hfence_gvma_vmid_gpa(
- READ_ONCE(v->vmid),
- d.addr, d.size, d.order);
+ vmid = READ_ONCE(v->vmid);
+ if (kvm_riscv_nacl_available())
+ nacl_hfence_gvma_vmid(nacl_shmem(), vmid,
+ d.addr, d.size, d.order);
+ else
+ kvm_riscv_local_hfence_gvma_vmid_gpa(vmid, d.addr,
+ d.size, d.order);
break;
case KVM_RISCV_HFENCE_VVMA_ASID_GVA:
kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_ASID_RCVD);
- kvm_riscv_local_hfence_vvma_asid_gva(
- READ_ONCE(v->vmid), d.asid,
- d.addr, d.size, d.order);
+ vmid = READ_ONCE(v->vmid);
+ if (kvm_riscv_nacl_available())
+ nacl_hfence_vvma_asid(nacl_shmem(), vmid, d.asid,
+ d.addr, d.size, d.order);
+ else
+ kvm_riscv_local_hfence_vvma_asid_gva(vmid, d.asid, d.addr,
+ d.size, d.order);
break;
case KVM_RISCV_HFENCE_VVMA_ASID_ALL:
kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_ASID_RCVD);
- kvm_riscv_local_hfence_vvma_asid_all(
- READ_ONCE(v->vmid), d.asid);
+ vmid = READ_ONCE(v->vmid);
+ if (kvm_riscv_nacl_available())
+ nacl_hfence_vvma_asid_all(nacl_shmem(), vmid, d.asid);
+ else
+ kvm_riscv_local_hfence_vvma_asid_all(vmid, d.asid);
break;
case KVM_RISCV_HFENCE_VVMA_GVA:
kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_RCVD);
- kvm_riscv_local_hfence_vvma_gva(
- READ_ONCE(v->vmid),
- d.addr, d.size, d.order);
+ vmid = READ_ONCE(v->vmid);
+ if (kvm_riscv_nacl_available())
+ nacl_hfence_vvma(nacl_shmem(), vmid,
+ d.addr, d.size, d.order);
+ else
+ kvm_riscv_local_hfence_vvma_gva(vmid, d.addr,
+ d.size, d.order);
break;
default:
break;
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [01/13] RISC-V: KVM: Order the object files alphabetically
2024-07-19 16:09 ` [PATCH 01/13] RISC-V: KVM: Order the object files alphabetically Anup Patel
@ 2024-10-16 19:03 ` Atish Patra
0 siblings, 0 replies; 29+ messages in thread
From: Atish Patra @ 2024-10-16 19:03 UTC (permalink / raw)
To: apatel
Cc: Palmer Dabbelt, Paul Walmsley, Atish Patra, Andrew Jones,
Anup Patel, kvm, kvm-riscv, linux-riscv, linux-kernel,
Atish Patra
> Order the object files alphabetically in the Makefile so that
> it is very predictable inserting new object files in the future.
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/kvm/Makefile | 26 ++++++++++++++------------
> 1 file changed, 14 insertions(+), 12 deletions(-)
> diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile
> index c2cacfbc06a0..c1eac0d093de 100644
> --- a/arch/riscv/kvm/Makefile
> +++ b/arch/riscv/kvm/Makefile
> @@ -9,27 +9,29 @@ include $(srctree)/virt/kvm/Makefile.kvm
>
> obj-$(CONFIG_KVM) += kvm.o
>
> +# Ordered alphabetically
> +kvm-y += aia.o
> +kvm-y += aia_aplic.o
> +kvm-y += aia_device.o
> +kvm-y += aia_imsic.o
> kvm-y += main.o
> -kvm-y += vm.o
> -kvm-y += vmid.o
> -kvm-y += tlb.o
> kvm-y += mmu.o
> +kvm-y += tlb.o
> kvm-y += vcpu.o
> kvm-y += vcpu_exit.o
> kvm-y += vcpu_fp.o
> -kvm-y += vcpu_vector.o
> kvm-y += vcpu_insn.o
> kvm-y += vcpu_onereg.o
> -kvm-y += vcpu_switch.o
> +kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_pmu.o
> kvm-y += vcpu_sbi.o
> -kvm-$(CONFIG_RISCV_SBI_V01) += vcpu_sbi_v01.o
> kvm-y += vcpu_sbi_base.o
> -kvm-y += vcpu_sbi_replace.o
> kvm-y += vcpu_sbi_hsm.o
> +kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_sbi_pmu.o
> +kvm-y += vcpu_sbi_replace.o
> kvm-y += vcpu_sbi_sta.o
> +kvm-$(CONFIG_RISCV_SBI_V01) += vcpu_sbi_v01.o
> +kvm-y += vcpu_switch.o
> kvm-y += vcpu_timer.o
> -kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_pmu.o vcpu_sbi_pmu.o
> -kvm-y += aia.o
> -kvm-y += aia_device.o
> -kvm-y += aia_aplic.o
> -kvm-y += aia_imsic.o
> +kvm-y += vcpu_vector.o
> +kvm-y += vm.o
> +kvm-y += vmid.o
>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [02/13] RISC-V: KVM: Save/restore HSTATUS in C source
2024-07-19 16:09 ` [PATCH 02/13] RISC-V: KVM: Save/restore HSTATUS in C source Anup Patel
@ 2024-10-16 19:27 ` Atish Patra
0 siblings, 0 replies; 29+ messages in thread
From: Atish Patra @ 2024-10-16 19:27 UTC (permalink / raw)
To: apatel
Cc: Palmer Dabbelt, Paul Walmsley, Atish Patra, Andrew Jones,
Anup Patel, kvm, kvm-riscv, linux-riscv, linux-kernel,
Atish Patra
> We will be optimizing HSTATUS CSR access via shared memory setup
> using the SBI nested acceleration extension. To facilitate this,
> we first move HSTATUS save/restore in kvm_riscv_vcpu_enter_exit().
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/kvm/vcpu.c | 9 +++++++++
> arch/riscv/kvm/vcpu_switch.S | 36 +++++++++++++-----------------------
> 2 files changed, 22 insertions(+), 23 deletions(-)
> diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
> index 449e5bb948c2..93b1ce043482 100644
> --- a/arch/riscv/kvm/vcpu.c
> +++ b/arch/riscv/kvm/vcpu.c
> @@ -720,9 +720,18 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v
> */
> static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
> {
> + struct kvm_cpu_context *gcntx = &vcpu->arch.guest_context;
> + struct kvm_cpu_context *hcntx = &vcpu->arch.host_context;
> +
> kvm_riscv_vcpu_swap_in_guest_state(vcpu);
> guest_state_enter_irqoff();
> +
> + hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
> +
> __kvm_riscv_switch_to(&vcpu->arch);
> +
> + gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
> +
> vcpu->arch.last_exit_cpu = vcpu->cpu;
> guest_state_exit_irqoff();
> kvm_riscv_vcpu_swap_in_host_state(vcpu);
> diff --git a/arch/riscv/kvm/vcpu_switch.S b/arch/riscv/kvm/vcpu_switch.S
> index 0c26189aa01c..f83643c4fdb9 100644
> --- a/arch/riscv/kvm/vcpu_switch.S
> +++ b/arch/riscv/kvm/vcpu_switch.S
> @@ -43,35 +43,30 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
>
> /* Load Guest CSR values */
> REG_L t0, (KVM_ARCH_GUEST_SSTATUS)(a0)
> - REG_L t1, (KVM_ARCH_GUEST_HSTATUS)(a0)
> - REG_L t2, (KVM_ARCH_GUEST_SCOUNTEREN)(a0)
> - la t4, .Lkvm_switch_return
> - REG_L t5, (KVM_ARCH_GUEST_SEPC)(a0)
> + REG_L t1, (KVM_ARCH_GUEST_SCOUNTEREN)(a0)
> + la t3, .Lkvm_switch_return
> + REG_L t4, (KVM_ARCH_GUEST_SEPC)(a0)
>
> /* Save Host and Restore Guest SSTATUS */
> csrrw t0, CSR_SSTATUS, t0
>
> - /* Save Host and Restore Guest HSTATUS */
> - csrrw t1, CSR_HSTATUS, t1
> -
> /* Save Host and Restore Guest SCOUNTEREN */
> - csrrw t2, CSR_SCOUNTEREN, t2
> + csrrw t1, CSR_SCOUNTEREN, t1
>
> /* Save Host STVEC and change it to return path */
> - csrrw t4, CSR_STVEC, t4
> + csrrw t3, CSR_STVEC, t3
>
> /* Save Host SSCRATCH and change it to struct kvm_vcpu_arch pointer */
> - csrrw t3, CSR_SSCRATCH, a0
> + csrrw t2, CSR_SSCRATCH, a0
>
> /* Restore Guest SEPC */
> - csrw CSR_SEPC, t5
> + csrw CSR_SEPC, t4
>
> /* Store Host CSR values */
> REG_S t0, (KVM_ARCH_HOST_SSTATUS)(a0)
> - REG_S t1, (KVM_ARCH_HOST_HSTATUS)(a0)
> - REG_S t2, (KVM_ARCH_HOST_SCOUNTEREN)(a0)
> - REG_S t3, (KVM_ARCH_HOST_SSCRATCH)(a0)
> - REG_S t4, (KVM_ARCH_HOST_STVEC)(a0)
> + REG_S t1, (KVM_ARCH_HOST_SCOUNTEREN)(a0)
> + REG_S t2, (KVM_ARCH_HOST_SSCRATCH)(a0)
> + REG_S t3, (KVM_ARCH_HOST_STVEC)(a0)
>
> /* Restore Guest GPRs (except A0) */
> REG_L ra, (KVM_ARCH_GUEST_RA)(a0)
> @@ -153,8 +148,7 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
> REG_L t1, (KVM_ARCH_HOST_STVEC)(a0)
> REG_L t2, (KVM_ARCH_HOST_SSCRATCH)(a0)
> REG_L t3, (KVM_ARCH_HOST_SCOUNTEREN)(a0)
> - REG_L t4, (KVM_ARCH_HOST_HSTATUS)(a0)
> - REG_L t5, (KVM_ARCH_HOST_SSTATUS)(a0)
> + REG_L t4, (KVM_ARCH_HOST_SSTATUS)(a0)
>
> /* Save Guest SEPC */
> csrr t0, CSR_SEPC
> @@ -168,18 +162,14 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
> /* Save Guest and Restore Host SCOUNTEREN */
> csrrw t3, CSR_SCOUNTEREN, t3
>
> - /* Save Guest and Restore Host HSTATUS */
> - csrrw t4, CSR_HSTATUS, t4
> -
> /* Save Guest and Restore Host SSTATUS */
> - csrrw t5, CSR_SSTATUS, t5
> + csrrw t4, CSR_SSTATUS, t4
>
> /* Store Guest CSR values */
> REG_S t0, (KVM_ARCH_GUEST_SEPC)(a0)
> REG_S t2, (KVM_ARCH_GUEST_A0)(a0)
> REG_S t3, (KVM_ARCH_GUEST_SCOUNTEREN)(a0)
> - REG_S t4, (KVM_ARCH_GUEST_HSTATUS)(a0)
> - REG_S t5, (KVM_ARCH_GUEST_SSTATUS)(a0)
> + REG_S t4, (KVM_ARCH_GUEST_SSTATUS)(a0)
>
> /* Restore Host GPRs (except A0 and T0-T6) */
> REG_L ra, (KVM_ARCH_HOST_RA)(a0)
>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 03/13] RISC-V: KVM: Save/restore SCOUNTEREN in C source
2024-07-19 16:09 ` [PATCH 03/13] RISC-V: KVM: Save/restore SCOUNTEREN " Anup Patel
@ 2024-10-16 21:36 ` Atish Patra
0 siblings, 0 replies; 29+ messages in thread
From: Atish Patra @ 2024-10-16 21:36 UTC (permalink / raw)
To: Anup Patel
Cc: Palmer Dabbelt, Paul Walmsley, Andrew Jones, Anup Patel, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Jul 19, 2024 at 9:09 AM Anup Patel <apatel@ventanamicro.com> wrote:
>
> The SCOUNTEREN CSR need not be saved/restored in the low-level
> __kvm_riscv_switch_to() function hence move the SCOUNTEREN CSR
> save/restore to the kvm_riscv_vcpu_swap_in_guest_state() and
> kvm_riscv_vcpu_swap_in_host_state() functions in C sources.
>
> Also, re-arrange the CSR save/restore and related GPR usage in
> the low-level __kvm_riscv_switch_to() low-level function.
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/kvm/vcpu.c | 2 ++
> arch/riscv/kvm/vcpu_switch.S | 52 +++++++++++++++---------------------
> 2 files changed, 23 insertions(+), 31 deletions(-)
>
> diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
> index 93b1ce043482..957e1a5e081b 100644
> --- a/arch/riscv/kvm/vcpu.c
> +++ b/arch/riscv/kvm/vcpu.c
> @@ -691,6 +691,7 @@ static __always_inline void kvm_riscv_vcpu_swap_in_guest_state(struct kvm_vcpu *
> struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
> struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
>
> + vcpu->arch.host_scounteren = csr_swap(CSR_SCOUNTEREN, csr->scounteren);
> vcpu->arch.host_senvcfg = csr_swap(CSR_SENVCFG, csr->senvcfg);
> if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN) &&
> (cfg->hstateen0 & SMSTATEEN0_SSTATEEN0))
> @@ -704,6 +705,7 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v
> struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
> struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
>
> + csr->scounteren = csr_swap(CSR_SCOUNTEREN, vcpu->arch.host_scounteren);
> csr->senvcfg = csr_swap(CSR_SENVCFG, vcpu->arch.host_senvcfg);
> if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN) &&
> (cfg->hstateen0 & SMSTATEEN0_SSTATEEN0))
> diff --git a/arch/riscv/kvm/vcpu_switch.S b/arch/riscv/kvm/vcpu_switch.S
> index f83643c4fdb9..3f8cbc21a644 100644
> --- a/arch/riscv/kvm/vcpu_switch.S
> +++ b/arch/riscv/kvm/vcpu_switch.S
> @@ -43,30 +43,25 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
>
> /* Load Guest CSR values */
> REG_L t0, (KVM_ARCH_GUEST_SSTATUS)(a0)
> - REG_L t1, (KVM_ARCH_GUEST_SCOUNTEREN)(a0)
> - la t3, .Lkvm_switch_return
> - REG_L t4, (KVM_ARCH_GUEST_SEPC)(a0)
> + la t1, .Lkvm_switch_return
> + REG_L t2, (KVM_ARCH_GUEST_SEPC)(a0)
>
> /* Save Host and Restore Guest SSTATUS */
> csrrw t0, CSR_SSTATUS, t0
>
> - /* Save Host and Restore Guest SCOUNTEREN */
> - csrrw t1, CSR_SCOUNTEREN, t1
> -
> /* Save Host STVEC and change it to return path */
> - csrrw t3, CSR_STVEC, t3
> -
> - /* Save Host SSCRATCH and change it to struct kvm_vcpu_arch pointer */
> - csrrw t2, CSR_SSCRATCH, a0
> + csrrw t1, CSR_STVEC, t1
>
> /* Restore Guest SEPC */
> - csrw CSR_SEPC, t4
> + csrw CSR_SEPC, t2
> +
> + /* Save Host SSCRATCH and change it to struct kvm_vcpu_arch pointer */
> + csrrw t3, CSR_SSCRATCH, a0
>
> /* Store Host CSR values */
> REG_S t0, (KVM_ARCH_HOST_SSTATUS)(a0)
> - REG_S t1, (KVM_ARCH_HOST_SCOUNTEREN)(a0)
> - REG_S t2, (KVM_ARCH_HOST_SSCRATCH)(a0)
> - REG_S t3, (KVM_ARCH_HOST_STVEC)(a0)
> + REG_S t1, (KVM_ARCH_HOST_STVEC)(a0)
> + REG_S t3, (KVM_ARCH_HOST_SSCRATCH)(a0)
>
> /* Restore Guest GPRs (except A0) */
> REG_L ra, (KVM_ARCH_GUEST_RA)(a0)
> @@ -145,31 +140,26 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
> REG_S t6, (KVM_ARCH_GUEST_T6)(a0)
>
> /* Load Host CSR values */
> - REG_L t1, (KVM_ARCH_HOST_STVEC)(a0)
> - REG_L t2, (KVM_ARCH_HOST_SSCRATCH)(a0)
> - REG_L t3, (KVM_ARCH_HOST_SCOUNTEREN)(a0)
> - REG_L t4, (KVM_ARCH_HOST_SSTATUS)(a0)
> -
> - /* Save Guest SEPC */
> - csrr t0, CSR_SEPC
> + REG_L t0, (KVM_ARCH_HOST_STVEC)(a0)
> + REG_L t1, (KVM_ARCH_HOST_SSCRATCH)(a0)
> + REG_L t2, (KVM_ARCH_HOST_SSTATUS)(a0)
>
> /* Save Guest A0 and Restore Host SSCRATCH */
> - csrrw t2, CSR_SSCRATCH, t2
> + csrrw t1, CSR_SSCRATCH, t1
>
> - /* Restore Host STVEC */
> - csrw CSR_STVEC, t1
> + /* Save Guest SEPC */
> + csrr t3, CSR_SEPC
>
> - /* Save Guest and Restore Host SCOUNTEREN */
> - csrrw t3, CSR_SCOUNTEREN, t3
> + /* Restore Host STVEC */
> + csrw CSR_STVEC, t0
>
> /* Save Guest and Restore Host SSTATUS */
> - csrrw t4, CSR_SSTATUS, t4
> + csrrw t2, CSR_SSTATUS, t2
>
> /* Store Guest CSR values */
> - REG_S t0, (KVM_ARCH_GUEST_SEPC)(a0)
> - REG_S t2, (KVM_ARCH_GUEST_A0)(a0)
> - REG_S t3, (KVM_ARCH_GUEST_SCOUNTEREN)(a0)
> - REG_S t4, (KVM_ARCH_GUEST_SSTATUS)(a0)
> + REG_S t1, (KVM_ARCH_GUEST_A0)(a0)
> + REG_S t2, (KVM_ARCH_GUEST_SSTATUS)(a0)
> + REG_S t3, (KVM_ARCH_GUEST_SEPC)(a0)
>
> /* Restore Host GPRs (except A0 and T0-T6) */
> REG_L ra, (KVM_ARCH_HOST_RA)(a0)
> --
> 2.34.1
>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
--
Regards,
Atish
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 04/13] RISC-V: KVM: Break down the __kvm_riscv_switch_to() into macros
2024-07-19 16:09 ` [PATCH 04/13] RISC-V: KVM: Break down the __kvm_riscv_switch_to() into macros Anup Patel
@ 2024-10-16 21:37 ` Atish Patra
0 siblings, 0 replies; 29+ messages in thread
From: Atish Patra @ 2024-10-16 21:37 UTC (permalink / raw)
To: Anup Patel
Cc: Palmer Dabbelt, Paul Walmsley, Andrew Jones, Anup Patel, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Jul 19, 2024 at 9:09 AM Anup Patel <apatel@ventanamicro.com> wrote:
>
> Break down the __kvm_riscv_switch_to() function into macros so that
> these macros can be later re-used by SBI NACL extension based low-level
> switch function.
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/kvm/vcpu_switch.S | 52 +++++++++++++++++++++++++++---------
> 1 file changed, 40 insertions(+), 12 deletions(-)
>
> diff --git a/arch/riscv/kvm/vcpu_switch.S b/arch/riscv/kvm/vcpu_switch.S
> index 3f8cbc21a644..9f13e5ce6a18 100644
> --- a/arch/riscv/kvm/vcpu_switch.S
> +++ b/arch/riscv/kvm/vcpu_switch.S
> @@ -11,11 +11,7 @@
> #include <asm/asm-offsets.h>
> #include <asm/csr.h>
>
> - .text
> - .altmacro
> - .option norelax
> -
> -SYM_FUNC_START(__kvm_riscv_switch_to)
> +.macro SAVE_HOST_GPRS
> /* Save Host GPRs (except A0 and T0-T6) */
> REG_S ra, (KVM_ARCH_HOST_RA)(a0)
> REG_S sp, (KVM_ARCH_HOST_SP)(a0)
> @@ -40,10 +36,12 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
> REG_S s9, (KVM_ARCH_HOST_S9)(a0)
> REG_S s10, (KVM_ARCH_HOST_S10)(a0)
> REG_S s11, (KVM_ARCH_HOST_S11)(a0)
> +.endm
>
> +.macro SAVE_HOST_AND_RESTORE_GUEST_CSRS __resume_addr
> /* Load Guest CSR values */
> REG_L t0, (KVM_ARCH_GUEST_SSTATUS)(a0)
> - la t1, .Lkvm_switch_return
> + la t1, \__resume_addr
> REG_L t2, (KVM_ARCH_GUEST_SEPC)(a0)
>
> /* Save Host and Restore Guest SSTATUS */
> @@ -62,7 +60,9 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
> REG_S t0, (KVM_ARCH_HOST_SSTATUS)(a0)
> REG_S t1, (KVM_ARCH_HOST_STVEC)(a0)
> REG_S t3, (KVM_ARCH_HOST_SSCRATCH)(a0)
> +.endm
>
> +.macro RESTORE_GUEST_GPRS
> /* Restore Guest GPRs (except A0) */
> REG_L ra, (KVM_ARCH_GUEST_RA)(a0)
> REG_L sp, (KVM_ARCH_GUEST_SP)(a0)
> @@ -97,13 +97,9 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
>
> /* Restore Guest A0 */
> REG_L a0, (KVM_ARCH_GUEST_A0)(a0)
> +.endm
>
> - /* Resume Guest */
> - sret
> -
> - /* Back to Host */
> - .align 2
> -.Lkvm_switch_return:
> +.macro SAVE_GUEST_GPRS
> /* Swap Guest A0 with SSCRATCH */
> csrrw a0, CSR_SSCRATCH, a0
>
> @@ -138,7 +134,9 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
> REG_S t4, (KVM_ARCH_GUEST_T4)(a0)
> REG_S t5, (KVM_ARCH_GUEST_T5)(a0)
> REG_S t6, (KVM_ARCH_GUEST_T6)(a0)
> +.endm
>
> +.macro SAVE_GUEST_AND_RESTORE_HOST_CSRS
> /* Load Host CSR values */
> REG_L t0, (KVM_ARCH_HOST_STVEC)(a0)
> REG_L t1, (KVM_ARCH_HOST_SSCRATCH)(a0)
> @@ -160,7 +158,9 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
> REG_S t1, (KVM_ARCH_GUEST_A0)(a0)
> REG_S t2, (KVM_ARCH_GUEST_SSTATUS)(a0)
> REG_S t3, (KVM_ARCH_GUEST_SEPC)(a0)
> +.endm
>
> +.macro RESTORE_HOST_GPRS
> /* Restore Host GPRs (except A0 and T0-T6) */
> REG_L ra, (KVM_ARCH_HOST_RA)(a0)
> REG_L sp, (KVM_ARCH_HOST_SP)(a0)
> @@ -185,6 +185,34 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
> REG_L s9, (KVM_ARCH_HOST_S9)(a0)
> REG_L s10, (KVM_ARCH_HOST_S10)(a0)
> REG_L s11, (KVM_ARCH_HOST_S11)(a0)
> +.endm
> +
> + .text
> + .altmacro
> + .option norelax
> +
> + /*
> + * Parameters:
> + * A0 <= Pointer to struct kvm_vcpu_arch
> + */
> +SYM_FUNC_START(__kvm_riscv_switch_to)
> + SAVE_HOST_GPRS
> +
> + SAVE_HOST_AND_RESTORE_GUEST_CSRS .Lkvm_switch_return
> +
> + RESTORE_GUEST_GPRS
> +
> + /* Resume Guest using SRET */
> + sret
> +
> + /* Back to Host */
> + .align 2
> +.Lkvm_switch_return:
> + SAVE_GUEST_GPRS
> +
> + SAVE_GUEST_AND_RESTORE_HOST_CSRS
> +
> + RESTORE_HOST_GPRS
>
> /* Return to C code */
> ret
> --
> 2.34.1
>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
--
Regards,
Atish
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 05/13] RISC-V: KVM: Replace aia_set_hvictl() with aia_hvictl_value()
2024-07-19 16:09 ` [PATCH 05/13] RISC-V: KVM: Replace aia_set_hvictl() with aia_hvictl_value() Anup Patel
@ 2024-10-16 21:39 ` Atish Patra
0 siblings, 0 replies; 29+ messages in thread
From: Atish Patra @ 2024-10-16 21:39 UTC (permalink / raw)
To: Anup Patel
Cc: Palmer Dabbelt, Paul Walmsley, Andrew Jones, Anup Patel, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Jul 19, 2024 at 9:09 AM Anup Patel <apatel@ventanamicro.com> wrote:
>
> The aia_set_hvictl() internally writes the HVICTL CSR which makes
> it difficult optimize the CSR write using SBI NACL extension for
> kvm_riscv_vcpu_aia_update_hvip() function so replace aia_set_hvictl()
> with new aia_hvictl_value() which only computes the HVICTL value.
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/kvm/aia.c | 10 +++++-----
> 1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/arch/riscv/kvm/aia.c b/arch/riscv/kvm/aia.c
> index 2967d305c442..17ae4a7c0e94 100644
> --- a/arch/riscv/kvm/aia.c
> +++ b/arch/riscv/kvm/aia.c
> @@ -51,7 +51,7 @@ static int aia_find_hgei(struct kvm_vcpu *owner)
> return hgei;
> }
>
> -static void aia_set_hvictl(bool ext_irq_pending)
> +static inline unsigned long aia_hvictl_value(bool ext_irq_pending)
> {
> unsigned long hvictl;
>
> @@ -62,7 +62,7 @@ static void aia_set_hvictl(bool ext_irq_pending)
>
> hvictl = (IRQ_S_EXT << HVICTL_IID_SHIFT) & HVICTL_IID;
> hvictl |= ext_irq_pending;
> - csr_write(CSR_HVICTL, hvictl);
> + return hvictl;
> }
>
> #ifdef CONFIG_32BIT
> @@ -130,7 +130,7 @@ void kvm_riscv_vcpu_aia_update_hvip(struct kvm_vcpu *vcpu)
> #ifdef CONFIG_32BIT
> csr_write(CSR_HVIPH, vcpu->arch.aia_context.guest_csr.hviph);
> #endif
> - aia_set_hvictl(!!(csr->hvip & BIT(IRQ_VS_EXT)));
> + csr_write(CSR_HVICTL, aia_hvictl_value(!!(csr->hvip & BIT(IRQ_VS_EXT))));
> }
>
> void kvm_riscv_vcpu_aia_load(struct kvm_vcpu *vcpu, int cpu)
> @@ -536,7 +536,7 @@ void kvm_riscv_aia_enable(void)
> if (!kvm_riscv_aia_available())
> return;
>
> - aia_set_hvictl(false);
> + csr_write(CSR_HVICTL, aia_hvictl_value(false));
> csr_write(CSR_HVIPRIO1, 0x0);
> csr_write(CSR_HVIPRIO2, 0x0);
> #ifdef CONFIG_32BIT
> @@ -572,7 +572,7 @@ void kvm_riscv_aia_disable(void)
> csr_clear(CSR_HIE, BIT(IRQ_S_GEXT));
> disable_percpu_irq(hgei_parent_irq);
>
> - aia_set_hvictl(false);
> + csr_write(CSR_HVICTL, aia_hvictl_value(false));
>
> raw_spin_lock_irqsave(&hgctrl->lock, flags);
>
> --
> 2.34.1
>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
--
Regards,
Atish
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 06/13] RISC-V: KVM: Don't setup SGEI for zero guest external interrupts
2024-07-19 16:09 ` [PATCH 06/13] RISC-V: KVM: Don't setup SGEI for zero guest external interrupts Anup Patel
@ 2024-10-16 21:40 ` Atish Patra
0 siblings, 0 replies; 29+ messages in thread
From: Atish Patra @ 2024-10-16 21:40 UTC (permalink / raw)
To: Anup Patel
Cc: Palmer Dabbelt, Paul Walmsley, Andrew Jones, Anup Patel, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Jul 19, 2024 at 9:09 AM Anup Patel <apatel@ventanamicro.com> wrote:
>
> No need to setup SGEI local interrupt when there are zero guest
> external interrupts (i.e. zero HW IMSIC guest files).
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/kvm/aia.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/arch/riscv/kvm/aia.c b/arch/riscv/kvm/aia.c
> index 17ae4a7c0e94..8ffae0330c89 100644
> --- a/arch/riscv/kvm/aia.c
> +++ b/arch/riscv/kvm/aia.c
> @@ -499,6 +499,10 @@ static int aia_hgei_init(void)
> hgctrl->free_bitmap = 0;
> }
>
> + /* Skip SGEI interrupt setup for zero guest external interrupts */
> + if (!kvm_riscv_aia_nr_hgei)
> + goto skip_sgei_interrupt;
> +
> /* Find INTC irq domain */
> domain = irq_find_matching_fwnode(riscv_get_intc_hwnode(),
> DOMAIN_BUS_ANY);
> @@ -522,11 +526,16 @@ static int aia_hgei_init(void)
> return rc;
> }
>
> +skip_sgei_interrupt:
> return 0;
> }
>
> static void aia_hgei_exit(void)
> {
> + /* Do nothing for zero guest external interrupts */
> + if (!kvm_riscv_aia_nr_hgei)
> + return;
> +
> /* Free per-CPU SGEI interrupt */
> free_percpu_irq(hgei_parent_irq, &aia_hgei);
> }
> --
> 2.34.1
>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
--
Regards,
Atish
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 07/13] RISC-V: Add defines for the SBI nested acceleration extension
2024-07-19 16:09 ` [PATCH 07/13] RISC-V: Add defines for the SBI nested acceleration extension Anup Patel
@ 2024-10-17 23:55 ` Atish Patra
0 siblings, 0 replies; 29+ messages in thread
From: Atish Patra @ 2024-10-17 23:55 UTC (permalink / raw)
To: Anup Patel
Cc: Palmer Dabbelt, Paul Walmsley, Andrew Jones, Anup Patel, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Jul 19, 2024 at 9:09 AM Anup Patel <apatel@ventanamicro.com> wrote:
>
> Add defines for the new SBI nested acceleration extension which was
> ratified as part of the SBI v2.0 specification.
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/include/asm/sbi.h | 120 +++++++++++++++++++++++++++++++++++
> 1 file changed, 120 insertions(+)
>
> diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h
> index 1079e214fe85..7c9ec953c519 100644
> --- a/arch/riscv/include/asm/sbi.h
> +++ b/arch/riscv/include/asm/sbi.h
> @@ -33,6 +33,7 @@ enum sbi_ext_id {
> SBI_EXT_PMU = 0x504D55,
> SBI_EXT_DBCN = 0x4442434E,
> SBI_EXT_STA = 0x535441,
> + SBI_EXT_NACL = 0x4E41434C,
>
> /* Experimentals extensions must lie within this range */
> SBI_EXT_EXPERIMENTAL_START = 0x08000000,
> @@ -279,6 +280,125 @@ struct sbi_sta_struct {
>
> #define SBI_SHMEM_DISABLE -1
>
> +enum sbi_ext_nacl_fid {
> + SBI_EXT_NACL_PROBE_FEATURE = 0x0,
> + SBI_EXT_NACL_SET_SHMEM = 0x1,
> + SBI_EXT_NACL_SYNC_CSR = 0x2,
> + SBI_EXT_NACL_SYNC_HFENCE = 0x3,
> + SBI_EXT_NACL_SYNC_SRET = 0x4,
> +};
> +
> +enum sbi_ext_nacl_feature {
> + SBI_NACL_FEAT_SYNC_CSR = 0x0,
> + SBI_NACL_FEAT_SYNC_HFENCE = 0x1,
> + SBI_NACL_FEAT_SYNC_SRET = 0x2,
> + SBI_NACL_FEAT_AUTOSWAP_CSR = 0x3,
> +};
> +
> +#define SBI_NACL_SHMEM_ADDR_SHIFT 12
> +#define SBI_NACL_SHMEM_SCRATCH_OFFSET 0x0000
> +#define SBI_NACL_SHMEM_SCRATCH_SIZE 0x1000
> +#define SBI_NACL_SHMEM_SRET_OFFSET 0x0000
> +#define SBI_NACL_SHMEM_SRET_SIZE 0x0200
> +#define SBI_NACL_SHMEM_AUTOSWAP_OFFSET (SBI_NACL_SHMEM_SRET_OFFSET + \
> + SBI_NACL_SHMEM_SRET_SIZE)
> +#define SBI_NACL_SHMEM_AUTOSWAP_SIZE 0x0080
> +#define SBI_NACL_SHMEM_UNUSED_OFFSET (SBI_NACL_SHMEM_AUTOSWAP_OFFSET + \
> + SBI_NACL_SHMEM_AUTOSWAP_SIZE)
> +#define SBI_NACL_SHMEM_UNUSED_SIZE 0x0580
> +#define SBI_NACL_SHMEM_HFENCE_OFFSET (SBI_NACL_SHMEM_UNUSED_OFFSET + \
> + SBI_NACL_SHMEM_UNUSED_SIZE)
> +#define SBI_NACL_SHMEM_HFENCE_SIZE 0x0780
> +#define SBI_NACL_SHMEM_DBITMAP_OFFSET (SBI_NACL_SHMEM_HFENCE_OFFSET + \
> + SBI_NACL_SHMEM_HFENCE_SIZE)
> +#define SBI_NACL_SHMEM_DBITMAP_SIZE 0x0080
> +#define SBI_NACL_SHMEM_CSR_OFFSET (SBI_NACL_SHMEM_DBITMAP_OFFSET + \
> + SBI_NACL_SHMEM_DBITMAP_SIZE)
> +#define SBI_NACL_SHMEM_CSR_SIZE ((__riscv_xlen / 8) * 1024)
> +#define SBI_NACL_SHMEM_SIZE (SBI_NACL_SHMEM_CSR_OFFSET + \
> + SBI_NACL_SHMEM_CSR_SIZE)
> +
> +#define SBI_NACL_SHMEM_CSR_INDEX(__csr_num) \
> + ((((__csr_num) & 0xc00) >> 2) | ((__csr_num) & 0xff))
> +
> +#define SBI_NACL_SHMEM_HFENCE_ENTRY_SZ ((__riscv_xlen / 8) * 4)
> +#define SBI_NACL_SHMEM_HFENCE_ENTRY_MAX \
> + (SBI_NACL_SHMEM_HFENCE_SIZE / \
> + SBI_NACL_SHMEM_HFENCE_ENTRY_SZ)
> +#define SBI_NACL_SHMEM_HFENCE_ENTRY(__num) \
> + (SBI_NACL_SHMEM_HFENCE_OFFSET + \
> + (__num) * SBI_NACL_SHMEM_HFENCE_ENTRY_SZ)
> +#define SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(__num) \
> + SBI_NACL_SHMEM_HFENCE_ENTRY(__num)
> +#define SBI_NACL_SHMEM_HFENCE_ENTRY_PNUM(__num)\
> + (SBI_NACL_SHMEM_HFENCE_ENTRY(__num) + (__riscv_xlen / 8))
> +#define SBI_NACL_SHMEM_HFENCE_ENTRY_PCOUNT(__num)\
> + (SBI_NACL_SHMEM_HFENCE_ENTRY(__num) + \
> + ((__riscv_xlen / 8) * 3))
> +
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_BITS 1
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_SHIFT \
> + (__riscv_xlen - SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_BITS)
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_MASK \
> + ((1UL << SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_BITS) - 1)
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_PEND \
> + (SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_MASK << \
> + SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_SHIFT)
> +
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD1_BITS 3
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD1_SHIFT \
> + (SBI_NACL_SHMEM_HFENCE_CONFIG_PEND_SHIFT - \
> + SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD1_BITS)
> +
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_BITS 4
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_SHIFT \
> + (SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD1_SHIFT - \
> + SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_BITS)
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_MASK \
> + ((1UL << SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_BITS) - 1)
> +
> +#define SBI_NACL_SHMEM_HFENCE_TYPE_GVMA 0x0
> +#define SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_ALL 0x1
> +#define SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID 0x2
> +#define SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID_ALL 0x3
> +#define SBI_NACL_SHMEM_HFENCE_TYPE_VVMA 0x4
> +#define SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ALL 0x5
> +#define SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID 0x6
> +#define SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID_ALL 0x7
> +
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD2_BITS 1
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD2_SHIFT \
> + (SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_SHIFT - \
> + SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD2_BITS)
> +
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_BITS 7
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_SHIFT \
> + (SBI_NACL_SHMEM_HFENCE_CONFIG_RSVD2_SHIFT - \
> + SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_BITS)
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_MASK \
> + ((1UL << SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_BITS) - 1)
> +#define SBI_NACL_SHMEM_HFENCE_ORDER_BASE 12
> +
> +#if __riscv_xlen == 32
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_BITS 9
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_BITS 7
> +#else
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_BITS 16
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_BITS 14
> +#endif
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_SHIFT \
> + SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_BITS
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_MASK \
> + ((1UL << SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_BITS) - 1)
> +#define SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_MASK \
> + ((1UL << SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_BITS) - 1)
> +
> +#define SBI_NACL_SHMEM_AUTOSWAP_FLAG_HSTATUS BIT(0)
> +#define SBI_NACL_SHMEM_AUTOSWAP_HSTATUS ((__riscv_xlen / 8) * 1)
> +
> +#define SBI_NACL_SHMEM_SRET_X(__i) ((__riscv_xlen / 8) * (__i))
> +#define SBI_NACL_SHMEM_SRET_X_LAST 31
> +
> /* SBI spec version fields */
> #define SBI_SPEC_VERSION_DEFAULT 0x1
> #define SBI_SPEC_VERSION_MAJOR_SHIFT 24
> --
> 2.34.1
>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
--
Regards,
Atish
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 08/13] RISC-V: KVM: Add common nested acceleration support
2024-07-19 16:09 ` [PATCH 08/13] RISC-V: KVM: Add common nested acceleration support Anup Patel
@ 2024-10-18 18:19 ` Atish Patra
2024-10-20 18:19 ` Anup Patel
0 siblings, 1 reply; 29+ messages in thread
From: Atish Patra @ 2024-10-18 18:19 UTC (permalink / raw)
To: Anup Patel
Cc: Palmer Dabbelt, Paul Walmsley, Andrew Jones, Anup Patel, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Jul 19, 2024 at 9:09 AM Anup Patel <apatel@ventanamicro.com> wrote:
>
> Add a common nested acceleration support which will be shared by
> all parts of KVM RISC-V. This nested acceleration support detects
> and enables SBI NACL extension usage based on static keys which
> ensures minimum impact on the non-nested scenario.
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/include/asm/kvm_nacl.h | 205 ++++++++++++++++++++++++++++++
> arch/riscv/kvm/Makefile | 1 +
> arch/riscv/kvm/main.c | 53 +++++++-
> arch/riscv/kvm/nacl.c | 152 ++++++++++++++++++++++
> 4 files changed, 409 insertions(+), 2 deletions(-)
> create mode 100644 arch/riscv/include/asm/kvm_nacl.h
> create mode 100644 arch/riscv/kvm/nacl.c
>
> diff --git a/arch/riscv/include/asm/kvm_nacl.h b/arch/riscv/include/asm/kvm_nacl.h
> new file mode 100644
> index 000000000000..a704e8000a58
> --- /dev/null
> +++ b/arch/riscv/include/asm/kvm_nacl.h
> @@ -0,0 +1,205 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (c) 2024 Ventana Micro Systems Inc.
> + */
> +
> +#ifndef __KVM_NACL_H
> +#define __KVM_NACL_H
> +
> +#include <linux/jump_label.h>
> +#include <linux/percpu.h>
> +#include <asm/byteorder.h>
> +#include <asm/csr.h>
> +#include <asm/sbi.h>
> +
> +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_available);
> +#define kvm_riscv_nacl_available() \
> + static_branch_unlikely(&kvm_riscv_nacl_available)
> +
> +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_csr_available);
> +#define kvm_riscv_nacl_sync_csr_available() \
> + static_branch_unlikely(&kvm_riscv_nacl_sync_csr_available)
> +
> +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_hfence_available);
> +#define kvm_riscv_nacl_sync_hfence_available() \
> + static_branch_unlikely(&kvm_riscv_nacl_sync_hfence_available)
> +
> +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_sret_available);
> +#define kvm_riscv_nacl_sync_sret_available() \
> + static_branch_unlikely(&kvm_riscv_nacl_sync_sret_available)
> +
> +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_autoswap_csr_available);
> +#define kvm_riscv_nacl_autoswap_csr_available() \
> + static_branch_unlikely(&kvm_riscv_nacl_autoswap_csr_available)
> +
> +struct kvm_riscv_nacl {
> + void *shmem;
> + phys_addr_t shmem_phys;
> +};
> +DECLARE_PER_CPU(struct kvm_riscv_nacl, kvm_riscv_nacl);
> +
> +void __kvm_riscv_nacl_hfence(void *shmem,
> + unsigned long control,
> + unsigned long page_num,
> + unsigned long page_count);
> +
> +int kvm_riscv_nacl_enable(void);
> +
> +void kvm_riscv_nacl_disable(void);
> +
> +void kvm_riscv_nacl_exit(void);
> +
> +int kvm_riscv_nacl_init(void);
> +
> +#ifdef CONFIG_32BIT
> +#define lelong_to_cpu(__x) le32_to_cpu(__x)
> +#define cpu_to_lelong(__x) cpu_to_le32(__x)
> +#else
> +#define lelong_to_cpu(__x) le64_to_cpu(__x)
> +#define cpu_to_lelong(__x) cpu_to_le64(__x)
> +#endif
> +
> +#define nacl_shmem() \
> + this_cpu_ptr(&kvm_riscv_nacl)->shmem
> +#define nacl_shmem_fast() \
> + (kvm_riscv_nacl_available() ? nacl_shmem() : NULL)
> +
I don't see any usage of this one. Most of the callers of nacl_shmem
probably require more to do if nacl is available
and need the conditional block anyways. Am I missing something ?
> +#define nacl_sync_hfence(__e) \
> + sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_HFENCE, \
> + (__e), 0, 0, 0, 0, 0)
> +
> +#define nacl_hfence_mkconfig(__type, __order, __vmid, __asid) \
> +({ \
> + unsigned long __c = SBI_NACL_SHMEM_HFENCE_CONFIG_PEND; \
> + __c |= ((__type) & SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_MASK) \
> + << SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_SHIFT; \
> + __c |= (((__order) - SBI_NACL_SHMEM_HFENCE_ORDER_BASE) & \
> + SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_MASK) \
> + << SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_SHIFT; \
> + __c |= ((__vmid) & SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_MASK) \
> + << SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_SHIFT; \
> + __c |= ((__asid) & SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_MASK); \
> + __c; \
> +})
> +
> +#define nacl_hfence_mkpnum(__order, __addr) \
> + ((__addr) >> (__order))
> +
> +#define nacl_hfence_mkpcount(__order, __size) \
> + ((__size) >> (__order))
> +
> +#define nacl_hfence_gvma(__shmem, __gpa, __gpsz, __order) \
> +__kvm_riscv_nacl_hfence(__shmem, \
> + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA, \
> + __order, 0, 0), \
> + nacl_hfence_mkpnum(__order, __gpa), \
> + nacl_hfence_mkpcount(__order, __gpsz))
> +
> +#define nacl_hfence_gvma_all(__shmem) \
> +__kvm_riscv_nacl_hfence(__shmem, \
> + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_ALL, \
> + 0, 0, 0), 0, 0)
> +
> +#define nacl_hfence_gvma_vmid(__shmem, __vmid, __gpa, __gpsz, __order) \
> +__kvm_riscv_nacl_hfence(__shmem, \
> + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID, \
> + __order, __vmid, 0), \
> + nacl_hfence_mkpnum(__order, __gpa), \
> + nacl_hfence_mkpcount(__order, __gpsz))
> +
> +#define nacl_hfence_gvma_vmid_all(__shmem, __vmid) \
> +__kvm_riscv_nacl_hfence(__shmem, \
> + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID_ALL, \
> + 0, __vmid, 0), 0, 0)
> +
> +#define nacl_hfence_vvma(__shmem, __vmid, __gva, __gvsz, __order) \
> +__kvm_riscv_nacl_hfence(__shmem, \
> + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA, \
> + __order, __vmid, 0), \
> + nacl_hfence_mkpnum(__order, __gva), \
> + nacl_hfence_mkpcount(__order, __gvsz))
> +
> +#define nacl_hfence_vvma_all(__shmem, __vmid) \
> +__kvm_riscv_nacl_hfence(__shmem, \
> + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ALL, \
> + 0, __vmid, 0), 0, 0)
> +
> +#define nacl_hfence_vvma_asid(__shmem, __vmid, __asid, __gva, __gvsz, __order)\
> +__kvm_riscv_nacl_hfence(__shmem, \
> + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID, \
> + __order, __vmid, __asid), \
> + nacl_hfence_mkpnum(__order, __gva), \
> + nacl_hfence_mkpcount(__order, __gvsz))
> +
> +#define nacl_hfence_vvma_asid_all(__shmem, __vmid, __asid) \
> +__kvm_riscv_nacl_hfence(__shmem, \
> + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID_ALL, \
> + 0, __vmid, __asid), 0, 0)
> +
> +#define nacl_csr_read(__shmem, __csr) \
> +({ \
> + unsigned long *__a = (__shmem) + SBI_NACL_SHMEM_CSR_OFFSET; \
> + lelong_to_cpu(__a[SBI_NACL_SHMEM_CSR_INDEX(__csr)]); \
> +})
> +
> +#define nacl_csr_write(__shmem, __csr, __val) \
> +do { \
> + void *__s = (__shmem); \
> + unsigned int __i = SBI_NACL_SHMEM_CSR_INDEX(__csr); \
> + unsigned long *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \
> + u8 *__b = (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \
> + __a[__i] = cpu_to_lelong(__val); \
> + __b[__i >> 3] |= 1U << (__i & 0x7); \
> +} while (0)
> +
> +#define nacl_csr_swap(__shmem, __csr, __val) \
> +({ \
> + void *__s = (__shmem); \
> + unsigned int __i = SBI_NACL_SHMEM_CSR_INDEX(__csr); \
> + unsigned long *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \
> + u8 *__b = (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \
> + unsigned long __r = lelong_to_cpu(__a[__i]); \
> + __a[__i] = cpu_to_lelong(__val); \
> + __b[__i >> 3] |= 1U << (__i & 0x7); \
> + __r; \
> +})
> +
> +#define nacl_sync_csr(__csr) \
> + sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_CSR, \
> + (__csr), 0, 0, 0, 0, 0)
> +
> +#define ncsr_read(__csr) \
> +({ \
> + unsigned long __r; \
> + if (kvm_riscv_nacl_available()) \
> + __r = nacl_csr_read(nacl_shmem(), __csr); \
> + else \
> + __r = csr_read(__csr); \
> + __r; \
> +})
> +
> +#define ncsr_write(__csr, __val) \
> +do { \
> + if (kvm_riscv_nacl_sync_csr_available()) \
> + nacl_csr_write(nacl_shmem(), __csr, __val); \
> + else \
> + csr_write(__csr, __val); \
> +} while (0)
> +
> +#define ncsr_swap(__csr, __val) \
> +({ \
> + unsigned long __r; \
> + if (kvm_riscv_nacl_sync_csr_available()) \
> + __r = nacl_csr_swap(nacl_shmem(), __csr, __val); \
> + else \
> + __r = csr_swap(__csr, __val); \
> + __r; \
> +})
> +
> +#define nsync_csr(__csr) \
> +do { \
> + if (kvm_riscv_nacl_sync_csr_available()) \
> + nacl_sync_csr(__csr); \
> +} while (0)
> +
> +#endif
> diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile
> index c1eac0d093de..0fb1840c3e0a 100644
> --- a/arch/riscv/kvm/Makefile
> +++ b/arch/riscv/kvm/Makefile
> @@ -16,6 +16,7 @@ kvm-y += aia_device.o
> kvm-y += aia_imsic.o
> kvm-y += main.o
> kvm-y += mmu.o
> +kvm-y += nacl.o
> kvm-y += tlb.o
> kvm-y += vcpu.o
> kvm-y += vcpu_exit.o
> diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c
> index bab2ec34cd87..fd78f40bbb04 100644
> --- a/arch/riscv/kvm/main.c
> +++ b/arch/riscv/kvm/main.c
> @@ -10,8 +10,8 @@
> #include <linux/err.h>
> #include <linux/module.h>
> #include <linux/kvm_host.h>
> -#include <asm/csr.h>
> #include <asm/cpufeature.h>
> +#include <asm/kvm_nacl.h>
> #include <asm/sbi.h>
>
> long kvm_arch_dev_ioctl(struct file *filp,
> @@ -22,6 +22,12 @@ long kvm_arch_dev_ioctl(struct file *filp,
>
> int kvm_arch_hardware_enable(void)
> {
> + int rc;
> +
> + rc = kvm_riscv_nacl_enable();
> + if (rc)
> + return rc;
> +
> csr_write(CSR_HEDELEG, KVM_HEDELEG_DEFAULT);
> csr_write(CSR_HIDELEG, KVM_HIDELEG_DEFAULT);
>
> @@ -49,11 +55,14 @@ void kvm_arch_hardware_disable(void)
> csr_write(CSR_HVIP, 0);
> csr_write(CSR_HEDELEG, 0);
> csr_write(CSR_HIDELEG, 0);
> +
> + kvm_riscv_nacl_disable();
> }
>
> static int __init riscv_kvm_init(void)
> {
> int rc;
> + char slist[64];
> const char *str;
>
> if (!riscv_isa_extension_available(NULL, h)) {
> @@ -71,16 +80,53 @@ static int __init riscv_kvm_init(void)
> return -ENODEV;
> }
>
> + rc = kvm_riscv_nacl_init();
> + if (rc && rc != -ENODEV)
> + return rc;
> +
> kvm_riscv_gstage_mode_detect();
>
> kvm_riscv_gstage_vmid_detect();
>
> rc = kvm_riscv_aia_init();
> - if (rc && rc != -ENODEV)
> + if (rc && rc != -ENODEV) {
> + kvm_riscv_nacl_exit();
> return rc;
> + }
>
> kvm_info("hypervisor extension available\n");
>
> + if (kvm_riscv_nacl_available()) {
> + rc = 0;
> + slist[0] = '\0';
> + if (kvm_riscv_nacl_sync_csr_available()) {
> + if (rc)
> + strcat(slist, ", ");
> + strcat(slist, "sync_csr");
> + rc++;
> + }
> + if (kvm_riscv_nacl_sync_hfence_available()) {
> + if (rc)
> + strcat(slist, ", ");
> + strcat(slist, "sync_hfence");
> + rc++;
> + }
> + if (kvm_riscv_nacl_sync_sret_available()) {
> + if (rc)
> + strcat(slist, ", ");
> + strcat(slist, "sync_sret");
> + rc++;
> + }
> + if (kvm_riscv_nacl_autoswap_csr_available()) {
> + if (rc)
> + strcat(slist, ", ");
> + strcat(slist, "autoswap_csr");
> + rc++;
> + }
> + kvm_info("using SBI nested acceleration with %s\n",
> + (rc) ? slist : "no features");
> + }
> +
> switch (kvm_riscv_gstage_mode()) {
> case HGATP_MODE_SV32X4:
> str = "Sv32x4";
> @@ -108,6 +154,7 @@ static int __init riscv_kvm_init(void)
> rc = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
> if (rc) {
> kvm_riscv_aia_exit();
> + kvm_riscv_nacl_exit();
> return rc;
> }
>
> @@ -119,6 +166,8 @@ static void __exit riscv_kvm_exit(void)
> {
> kvm_riscv_aia_exit();
>
> + kvm_riscv_nacl_exit();
> +
> kvm_exit();
> }
> module_exit(riscv_kvm_exit);
> diff --git a/arch/riscv/kvm/nacl.c b/arch/riscv/kvm/nacl.c
> new file mode 100644
> index 000000000000..08a95ad9ada2
> --- /dev/null
> +++ b/arch/riscv/kvm/nacl.c
> @@ -0,0 +1,152 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (c) 2024 Ventana Micro Systems Inc.
> + */
> +
> +#include <linux/kvm_host.h>
> +#include <linux/vmalloc.h>
> +#include <asm/kvm_nacl.h>
> +
> +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_available);
> +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_csr_available);
> +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_hfence_available);
> +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_sret_available);
> +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_autoswap_csr_available);
> +DEFINE_PER_CPU(struct kvm_riscv_nacl, kvm_riscv_nacl);
> +
> +void __kvm_riscv_nacl_hfence(void *shmem,
> + unsigned long control,
> + unsigned long page_num,
> + unsigned long page_count)
> +{
> + int i, ent = -1, try_count = 5;
> + unsigned long *entp;
> +
> +again:
> + for (i = 0; i < SBI_NACL_SHMEM_HFENCE_ENTRY_MAX; i++) {
> + entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i);
> + if (lelong_to_cpu(*entp) & SBI_NACL_SHMEM_HFENCE_CONFIG_PEND)
> + continue;
> +
> + ent = i;
> + break;
> + }
> +
> + if (ent < 0) {
> + if (try_count) {
> + nacl_sync_hfence(-1UL);
> + goto again;
> + } else {
> + pr_warn("KVM: No free entry in NACL shared memory\n");
> + return;
> + }
> + }
> +
> + entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i);
> + *entp = cpu_to_lelong(control);
> + entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PNUM(i);
> + *entp = cpu_to_lelong(page_num);
> + entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PCOUNT(i);
> + *entp = cpu_to_lelong(page_count);
> +}
> +
> +int kvm_riscv_nacl_enable(void)
> +{
> + int rc;
> + struct sbiret ret;
> + struct kvm_riscv_nacl *nacl;
> +
> + if (!kvm_riscv_nacl_available())
> + return 0;
> + nacl = this_cpu_ptr(&kvm_riscv_nacl);
> +
> + ret = sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SET_SHMEM,
> + nacl->shmem_phys, 0, 0, 0, 0, 0);
> + rc = sbi_err_map_linux_errno(ret.error);
> + if (rc)
> + return rc;
> +
> + return 0;
> +}
> +
> +void kvm_riscv_nacl_disable(void)
> +{
> + if (!kvm_riscv_nacl_available())
> + return;
> +
> + sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SET_SHMEM,
> + SBI_SHMEM_DISABLE, SBI_SHMEM_DISABLE, 0, 0, 0, 0);
> +}
> +
> +void kvm_riscv_nacl_exit(void)
> +{
> + int cpu;
> + struct kvm_riscv_nacl *nacl;
> +
> + if (!kvm_riscv_nacl_available())
> + return;
> +
> + /* Allocate per-CPU shared memory */
> + for_each_possible_cpu(cpu) {
> + nacl = per_cpu_ptr(&kvm_riscv_nacl, cpu);
> + if (!nacl->shmem)
> + continue;
> +
> + free_pages((unsigned long)nacl->shmem,
> + get_order(SBI_NACL_SHMEM_SIZE));
> + nacl->shmem = NULL;
> + nacl->shmem_phys = 0;
> + }
> +}
> +
> +static long nacl_probe_feature(long feature_id)
> +{
> + struct sbiret ret;
> +
> + if (!kvm_riscv_nacl_available())
> + return 0;
> +
> + ret = sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_PROBE_FEATURE,
> + feature_id, 0, 0, 0, 0, 0);
> + return ret.value;
> +}
> +
> +int kvm_riscv_nacl_init(void)
> +{
> + int cpu;
> + struct page *shmem_page;
> + struct kvm_riscv_nacl *nacl;
> +
> + if (sbi_spec_version < sbi_mk_version(1, 0) ||
> + sbi_probe_extension(SBI_EXT_NACL) <= 0)
> + return -ENODEV;
> +
> + /* Enable NACL support */
> + static_branch_enable(&kvm_riscv_nacl_available);
> +
> + /* Probe NACL features */
> + if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_CSR))
> + static_branch_enable(&kvm_riscv_nacl_sync_csr_available);
> + if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_HFENCE))
> + static_branch_enable(&kvm_riscv_nacl_sync_hfence_available);
> + if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_SRET))
> + static_branch_enable(&kvm_riscv_nacl_sync_sret_available);
> + if (nacl_probe_feature(SBI_NACL_FEAT_AUTOSWAP_CSR))
> + static_branch_enable(&kvm_riscv_nacl_autoswap_csr_available);
> +
> + /* Allocate per-CPU shared memory */
> + for_each_possible_cpu(cpu) {
> + nacl = per_cpu_ptr(&kvm_riscv_nacl, cpu);
> +
> + shmem_page = alloc_pages(GFP_KERNEL | __GFP_ZERO,
> + get_order(SBI_NACL_SHMEM_SIZE));
> + if (!shmem_page) {
> + kvm_riscv_nacl_exit();
> + return -ENOMEM;
> + }
> + nacl->shmem = page_to_virt(shmem_page);
> + nacl->shmem_phys = page_to_phys(shmem_page);
> + }
> +
> + return 0;
> +}
> --
> 2.34.1
>
Otherwise, it looks good to me.
Reviewed-by: Atish Patra <atishp@rivosinc.com>
--
Regards,
Atish
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 09/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing H-extension CSRs
2024-07-19 16:09 ` [PATCH 09/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing H-extension CSRs Anup Patel
@ 2024-10-18 19:31 ` Atish Patra
2024-10-20 18:43 ` Anup Patel
0 siblings, 1 reply; 29+ messages in thread
From: Atish Patra @ 2024-10-18 19:31 UTC (permalink / raw)
To: Anup Patel
Cc: Palmer Dabbelt, Paul Walmsley, Andrew Jones, Anup Patel, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Jul 19, 2024 at 9:09 AM Anup Patel <apatel@ventanamicro.com> wrote:
>
> When running under some other hypervisor, prefer nacl_csr_xyz()
> for accessing H-extension CSRs in the run-loop. This makes CSR
> access faster whenever SBI nested acceleration is available.
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/kvm/mmu.c | 4 +-
> arch/riscv/kvm/vcpu.c | 103 +++++++++++++++++++++++++-----------
> arch/riscv/kvm/vcpu_timer.c | 28 +++++-----
> 3 files changed, 87 insertions(+), 48 deletions(-)
>
> diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
> index b63650f9b966..45ace9138947 100644
> --- a/arch/riscv/kvm/mmu.c
> +++ b/arch/riscv/kvm/mmu.c
> @@ -15,7 +15,7 @@
> #include <linux/vmalloc.h>
> #include <linux/kvm_host.h>
> #include <linux/sched/signal.h>
> -#include <asm/csr.h>
> +#include <asm/kvm_nacl.h>
> #include <asm/page.h>
> #include <asm/pgtable.h>
>
> @@ -732,7 +732,7 @@ void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu)
> hgatp |= (READ_ONCE(k->vmid.vmid) << HGATP_VMID_SHIFT) & HGATP_VMID;
> hgatp |= (k->pgd_phys >> PAGE_SHIFT) & HGATP_PPN;
>
> - csr_write(CSR_HGATP, hgatp);
> + ncsr_write(CSR_HGATP, hgatp);
>
> if (!kvm_riscv_gstage_vmid_bits())
> kvm_riscv_local_hfence_gvma_all();
> diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
> index 957e1a5e081b..00baaf1b0136 100644
> --- a/arch/riscv/kvm/vcpu.c
> +++ b/arch/riscv/kvm/vcpu.c
> @@ -17,8 +17,8 @@
> #include <linux/sched/signal.h>
> #include <linux/fs.h>
> #include <linux/kvm_host.h>
> -#include <asm/csr.h>
> #include <asm/cacheflush.h>
> +#include <asm/kvm_nacl.h>
> #include <asm/kvm_vcpu_vector.h>
>
> #define CREATE_TRACE_POINTS
> @@ -361,10 +361,10 @@ void kvm_riscv_vcpu_sync_interrupts(struct kvm_vcpu *vcpu)
> struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
>
> /* Read current HVIP and VSIE CSRs */
> - csr->vsie = csr_read(CSR_VSIE);
> + csr->vsie = ncsr_read(CSR_VSIE);
>
> /* Sync-up HVIP.VSSIP bit changes does by Guest */
> - hvip = csr_read(CSR_HVIP);
> + hvip = ncsr_read(CSR_HVIP);
> if ((csr->hvip ^ hvip) & (1UL << IRQ_VS_SOFT)) {
> if (hvip & (1UL << IRQ_VS_SOFT)) {
> if (!test_and_set_bit(IRQ_VS_SOFT,
> @@ -561,26 +561,49 @@ static void kvm_riscv_vcpu_setup_config(struct kvm_vcpu *vcpu)
>
> void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
> {
> + void *nsh;
> struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
> struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
>
> - csr_write(CSR_VSSTATUS, csr->vsstatus);
> - csr_write(CSR_VSIE, csr->vsie);
> - csr_write(CSR_VSTVEC, csr->vstvec);
> - csr_write(CSR_VSSCRATCH, csr->vsscratch);
> - csr_write(CSR_VSEPC, csr->vsepc);
> - csr_write(CSR_VSCAUSE, csr->vscause);
> - csr_write(CSR_VSTVAL, csr->vstval);
> - csr_write(CSR_HEDELEG, cfg->hedeleg);
> - csr_write(CSR_HVIP, csr->hvip);
> - csr_write(CSR_VSATP, csr->vsatp);
> - csr_write(CSR_HENVCFG, cfg->henvcfg);
> - if (IS_ENABLED(CONFIG_32BIT))
> - csr_write(CSR_HENVCFGH, cfg->henvcfg >> 32);
> - if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
> - csr_write(CSR_HSTATEEN0, cfg->hstateen0);
> + if (kvm_riscv_nacl_sync_csr_available()) {
> + nsh = nacl_shmem();
> + nacl_csr_write(nsh, CSR_VSSTATUS, csr->vsstatus);
> + nacl_csr_write(nsh, CSR_VSIE, csr->vsie);
> + nacl_csr_write(nsh, CSR_VSTVEC, csr->vstvec);
> + nacl_csr_write(nsh, CSR_VSSCRATCH, csr->vsscratch);
> + nacl_csr_write(nsh, CSR_VSEPC, csr->vsepc);
> + nacl_csr_write(nsh, CSR_VSCAUSE, csr->vscause);
> + nacl_csr_write(nsh, CSR_VSTVAL, csr->vstval);
> + nacl_csr_write(nsh, CSR_HEDELEG, cfg->hedeleg);
> + nacl_csr_write(nsh, CSR_HVIP, csr->hvip);
> + nacl_csr_write(nsh, CSR_VSATP, csr->vsatp);
> + nacl_csr_write(nsh, CSR_HENVCFG, cfg->henvcfg);
> + if (IS_ENABLED(CONFIG_32BIT))
> + nacl_csr_write(nsh, CSR_HENVCFGH, cfg->henvcfg >> 32);
> + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
> + nacl_csr_write(nsh, CSR_HSTATEEN0, cfg->hstateen0);
> + if (IS_ENABLED(CONFIG_32BIT))
> + nacl_csr_write(nsh, CSR_HSTATEEN0H, cfg->hstateen0 >> 32);
> + }
> + } else {
> + csr_write(CSR_VSSTATUS, csr->vsstatus);
> + csr_write(CSR_VSIE, csr->vsie);
> + csr_write(CSR_VSTVEC, csr->vstvec);
> + csr_write(CSR_VSSCRATCH, csr->vsscratch);
> + csr_write(CSR_VSEPC, csr->vsepc);
> + csr_write(CSR_VSCAUSE, csr->vscause);
> + csr_write(CSR_VSTVAL, csr->vstval);
> + csr_write(CSR_HEDELEG, cfg->hedeleg);
> + csr_write(CSR_HVIP, csr->hvip);
> + csr_write(CSR_VSATP, csr->vsatp);
> + csr_write(CSR_HENVCFG, cfg->henvcfg);
> if (IS_ENABLED(CONFIG_32BIT))
> - csr_write(CSR_HSTATEEN0H, cfg->hstateen0 >> 32);
> + csr_write(CSR_HENVCFGH, cfg->henvcfg >> 32);
> + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
> + csr_write(CSR_HSTATEEN0, cfg->hstateen0);
> + if (IS_ENABLED(CONFIG_32BIT))
> + csr_write(CSR_HSTATEEN0H, cfg->hstateen0 >> 32);
> + }
> }
>
> kvm_riscv_gstage_update_hgatp(vcpu);
> @@ -603,6 +626,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
>
> void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
> {
> + void *nsh;
> struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
>
> vcpu->cpu = -1;
> @@ -618,15 +642,28 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
> vcpu->arch.isa);
> kvm_riscv_vcpu_host_vector_restore(&vcpu->arch.host_context);
>
> - csr->vsstatus = csr_read(CSR_VSSTATUS);
> - csr->vsie = csr_read(CSR_VSIE);
> - csr->vstvec = csr_read(CSR_VSTVEC);
> - csr->vsscratch = csr_read(CSR_VSSCRATCH);
> - csr->vsepc = csr_read(CSR_VSEPC);
> - csr->vscause = csr_read(CSR_VSCAUSE);
> - csr->vstval = csr_read(CSR_VSTVAL);
> - csr->hvip = csr_read(CSR_HVIP);
> - csr->vsatp = csr_read(CSR_VSATP);
> + if (kvm_riscv_nacl_available()) {
Should we leave a comment here why ncsr_read is not efficient here
i.e. due to block access ?
> + nsh = nacl_shmem();
> + csr->vsstatus = nacl_csr_read(nsh, CSR_VSSTATUS);
> + csr->vsie = nacl_csr_read(nsh, CSR_VSIE);
> + csr->vstvec = nacl_csr_read(nsh, CSR_VSTVEC);
> + csr->vsscratch = nacl_csr_read(nsh, CSR_VSSCRATCH);
> + csr->vsepc = nacl_csr_read(nsh, CSR_VSEPC);
> + csr->vscause = nacl_csr_read(nsh, CSR_VSCAUSE);
> + csr->vstval = nacl_csr_read(nsh, CSR_VSTVAL);
> + csr->hvip = nacl_csr_read(nsh, CSR_HVIP);
> + csr->vsatp = nacl_csr_read(nsh, CSR_VSATP);
> + } else {
> + csr->vsstatus = csr_read(CSR_VSSTATUS);
> + csr->vsie = csr_read(CSR_VSIE);
> + csr->vstvec = csr_read(CSR_VSTVEC);
> + csr->vsscratch = csr_read(CSR_VSSCRATCH);
> + csr->vsepc = csr_read(CSR_VSEPC);
> + csr->vscause = csr_read(CSR_VSCAUSE);
> + csr->vstval = csr_read(CSR_VSTVAL);
> + csr->hvip = csr_read(CSR_HVIP);
> + csr->vsatp = csr_read(CSR_VSATP);
> + }
> }
>
> static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu)
> @@ -681,7 +718,7 @@ static void kvm_riscv_update_hvip(struct kvm_vcpu *vcpu)
> {
> struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
>
> - csr_write(CSR_HVIP, csr->hvip);
> + ncsr_write(CSR_HVIP, csr->hvip);
> kvm_riscv_vcpu_aia_update_hvip(vcpu);
> }
>
> @@ -728,7 +765,9 @@ static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
> kvm_riscv_vcpu_swap_in_guest_state(vcpu);
> guest_state_enter_irqoff();
>
> - hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
> + hcntx->hstatus = ncsr_swap(CSR_HSTATUS, gcntx->hstatus);
> +
> + nsync_csr(-1UL);
>
> __kvm_riscv_switch_to(&vcpu->arch);
>
> @@ -863,8 +902,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
> trap.sepc = vcpu->arch.guest_context.sepc;
> trap.scause = csr_read(CSR_SCAUSE);
> trap.stval = csr_read(CSR_STVAL);
> - trap.htval = csr_read(CSR_HTVAL);
> - trap.htinst = csr_read(CSR_HTINST);
> + trap.htval = ncsr_read(CSR_HTVAL);
> + trap.htinst = ncsr_read(CSR_HTINST);
>
> /* Syncup interrupts state with HW */
> kvm_riscv_vcpu_sync_interrupts(vcpu);
> diff --git a/arch/riscv/kvm/vcpu_timer.c b/arch/riscv/kvm/vcpu_timer.c
> index 75486b25ac45..96e7a4e463f7 100644
> --- a/arch/riscv/kvm/vcpu_timer.c
> +++ b/arch/riscv/kvm/vcpu_timer.c
> @@ -11,8 +11,8 @@
> #include <linux/kvm_host.h>
> #include <linux/uaccess.h>
> #include <clocksource/timer-riscv.h>
> -#include <asm/csr.h>
> #include <asm/delay.h>
> +#include <asm/kvm_nacl.h>
> #include <asm/kvm_vcpu_timer.h>
>
> static u64 kvm_riscv_current_cycles(struct kvm_guest_timer *gt)
> @@ -72,12 +72,12 @@ static int kvm_riscv_vcpu_timer_cancel(struct kvm_vcpu_timer *t)
> static int kvm_riscv_vcpu_update_vstimecmp(struct kvm_vcpu *vcpu, u64 ncycles)
> {
> #if defined(CONFIG_32BIT)
> - csr_write(CSR_VSTIMECMP, ncycles & 0xFFFFFFFF);
> - csr_write(CSR_VSTIMECMPH, ncycles >> 32);
> + ncsr_write(CSR_VSTIMECMP, ncycles & 0xFFFFFFFF);
> + ncsr_write(CSR_VSTIMECMPH, ncycles >> 32);
> #else
> - csr_write(CSR_VSTIMECMP, ncycles);
> + ncsr_write(CSR_VSTIMECMP, ncycles);
> #endif
> - return 0;
> + return 0;
> }
>
> static int kvm_riscv_vcpu_update_hrtimer(struct kvm_vcpu *vcpu, u64 ncycles)
> @@ -289,10 +289,10 @@ static void kvm_riscv_vcpu_update_timedelta(struct kvm_vcpu *vcpu)
> struct kvm_guest_timer *gt = &vcpu->kvm->arch.timer;
>
> #if defined(CONFIG_32BIT)
> - csr_write(CSR_HTIMEDELTA, (u32)(gt->time_delta));
> - csr_write(CSR_HTIMEDELTAH, (u32)(gt->time_delta >> 32));
> + ncsr_write(CSR_HTIMEDELTA, (u32)(gt->time_delta));
> + ncsr_write(CSR_HTIMEDELTAH, (u32)(gt->time_delta >> 32));
> #else
> - csr_write(CSR_HTIMEDELTA, gt->time_delta);
> + ncsr_write(CSR_HTIMEDELTA, gt->time_delta);
> #endif
> }
>
> @@ -306,10 +306,10 @@ void kvm_riscv_vcpu_timer_restore(struct kvm_vcpu *vcpu)
> return;
>
> #if defined(CONFIG_32BIT)
> - csr_write(CSR_VSTIMECMP, (u32)t->next_cycles);
> - csr_write(CSR_VSTIMECMPH, (u32)(t->next_cycles >> 32));
> + ncsr_write(CSR_VSTIMECMP, (u32)t->next_cycles);
> + ncsr_write(CSR_VSTIMECMPH, (u32)(t->next_cycles >> 32));
> #else
> - csr_write(CSR_VSTIMECMP, t->next_cycles);
> + ncsr_write(CSR_VSTIMECMP, t->next_cycles);
> #endif
>
> /* timer should be enabled for the remaining operations */
> @@ -327,10 +327,10 @@ void kvm_riscv_vcpu_timer_sync(struct kvm_vcpu *vcpu)
> return;
>
> #if defined(CONFIG_32BIT)
> - t->next_cycles = csr_read(CSR_VSTIMECMP);
> - t->next_cycles |= (u64)csr_read(CSR_VSTIMECMPH) << 32;
> + t->next_cycles = ncsr_read(CSR_VSTIMECMP);
> + t->next_cycles |= (u64)ncsr_read(CSR_VSTIMECMPH) << 32;
> #else
> - t->next_cycles = csr_read(CSR_VSTIMECMP);
> + t->next_cycles = ncsr_read(CSR_VSTIMECMP);
> #endif
> }
>
> --
> 2.34.1
>
Otherwise, LGTM.
Reviewed-by: Atish Patra <atishp@rivosinc.com>
--
Regards,
Atish
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 10/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing AIA CSRs
2024-07-19 16:09 ` [PATCH 10/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing AIA CSRs Anup Patel
@ 2024-10-18 19:34 ` Atish Patra
0 siblings, 0 replies; 29+ messages in thread
From: Atish Patra @ 2024-10-18 19:34 UTC (permalink / raw)
To: Anup Patel
Cc: Palmer Dabbelt, Paul Walmsley, Andrew Jones, Anup Patel, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Jul 19, 2024 at 9:10 AM Anup Patel <apatel@ventanamicro.com> wrote:
>
> When running under some other hypervisor, prefer nacl_csr_xyz()
> for accessing AIA CSRs in the run-loop. This makes CSR access
> faster whenever SBI nested acceleration is available.
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/kvm/aia.c | 97 ++++++++++++++++++++++++++++----------------
> 1 file changed, 63 insertions(+), 34 deletions(-)
>
> diff --git a/arch/riscv/kvm/aia.c b/arch/riscv/kvm/aia.c
> index 8ffae0330c89..dcced4db7fe8 100644
> --- a/arch/riscv/kvm/aia.c
> +++ b/arch/riscv/kvm/aia.c
> @@ -16,6 +16,7 @@
> #include <linux/percpu.h>
> #include <linux/spinlock.h>
> #include <asm/cpufeature.h>
> +#include <asm/kvm_nacl.h>
>
> struct aia_hgei_control {
> raw_spinlock_t lock;
> @@ -88,7 +89,7 @@ void kvm_riscv_vcpu_aia_sync_interrupts(struct kvm_vcpu *vcpu)
> struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
>
> if (kvm_riscv_aia_available())
> - csr->vsieh = csr_read(CSR_VSIEH);
> + csr->vsieh = ncsr_read(CSR_VSIEH);
> }
> #endif
>
> @@ -115,7 +116,7 @@ bool kvm_riscv_vcpu_aia_has_interrupts(struct kvm_vcpu *vcpu, u64 mask)
>
> hgei = aia_find_hgei(vcpu);
> if (hgei > 0)
> - return !!(csr_read(CSR_HGEIP) & BIT(hgei));
> + return !!(ncsr_read(CSR_HGEIP) & BIT(hgei));
>
> return false;
> }
> @@ -128,45 +129,73 @@ void kvm_riscv_vcpu_aia_update_hvip(struct kvm_vcpu *vcpu)
> return;
>
> #ifdef CONFIG_32BIT
> - csr_write(CSR_HVIPH, vcpu->arch.aia_context.guest_csr.hviph);
> + ncsr_write(CSR_HVIPH, vcpu->arch.aia_context.guest_csr.hviph);
> #endif
> - csr_write(CSR_HVICTL, aia_hvictl_value(!!(csr->hvip & BIT(IRQ_VS_EXT))));
> + ncsr_write(CSR_HVICTL, aia_hvictl_value(!!(csr->hvip & BIT(IRQ_VS_EXT))));
> }
>
> void kvm_riscv_vcpu_aia_load(struct kvm_vcpu *vcpu, int cpu)
> {
> struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
> + void *nsh;
>
> if (!kvm_riscv_aia_available())
> return;
>
> - csr_write(CSR_VSISELECT, csr->vsiselect);
> - csr_write(CSR_HVIPRIO1, csr->hviprio1);
> - csr_write(CSR_HVIPRIO2, csr->hviprio2);
> + if (kvm_riscv_nacl_sync_csr_available()) {
> + nsh = nacl_shmem();
> + nacl_csr_write(nsh, CSR_VSISELECT, csr->vsiselect);
> + nacl_csr_write(nsh, CSR_HVIPRIO1, csr->hviprio1);
> + nacl_csr_write(nsh, CSR_HVIPRIO2, csr->hviprio2);
> +#ifdef CONFIG_32BIT
> + nacl_csr_write(nsh, CSR_VSIEH, csr->vsieh);
> + nacl_csr_write(nsh, CSR_HVIPH, csr->hviph);
> + nacl_csr_write(nsh, CSR_HVIPRIO1H, csr->hviprio1h);
> + nacl_csr_write(nsh, CSR_HVIPRIO2H, csr->hviprio2h);
> +#endif
> + } else {
> + csr_write(CSR_VSISELECT, csr->vsiselect);
> + csr_write(CSR_HVIPRIO1, csr->hviprio1);
> + csr_write(CSR_HVIPRIO2, csr->hviprio2);
> #ifdef CONFIG_32BIT
> - csr_write(CSR_VSIEH, csr->vsieh);
> - csr_write(CSR_HVIPH, csr->hviph);
> - csr_write(CSR_HVIPRIO1H, csr->hviprio1h);
> - csr_write(CSR_HVIPRIO2H, csr->hviprio2h);
> + csr_write(CSR_VSIEH, csr->vsieh);
> + csr_write(CSR_HVIPH, csr->hviph);
> + csr_write(CSR_HVIPRIO1H, csr->hviprio1h);
> + csr_write(CSR_HVIPRIO2H, csr->hviprio2h);
> #endif
> + }
> }
>
> void kvm_riscv_vcpu_aia_put(struct kvm_vcpu *vcpu)
> {
> struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr;
> + void *nsh;
>
> if (!kvm_riscv_aia_available())
> return;
>
> - csr->vsiselect = csr_read(CSR_VSISELECT);
> - csr->hviprio1 = csr_read(CSR_HVIPRIO1);
> - csr->hviprio2 = csr_read(CSR_HVIPRIO2);
> + if (kvm_riscv_nacl_available()) {
> + nsh = nacl_shmem();
> + csr->vsiselect = nacl_csr_read(nsh, CSR_VSISELECT);
> + csr->hviprio1 = nacl_csr_read(nsh, CSR_HVIPRIO1);
> + csr->hviprio2 = nacl_csr_read(nsh, CSR_HVIPRIO2);
> #ifdef CONFIG_32BIT
> - csr->vsieh = csr_read(CSR_VSIEH);
> - csr->hviph = csr_read(CSR_HVIPH);
> - csr->hviprio1h = csr_read(CSR_HVIPRIO1H);
> - csr->hviprio2h = csr_read(CSR_HVIPRIO2H);
> + csr->vsieh = nacl_csr_read(nsh, CSR_VSIEH);
> + csr->hviph = nacl_csr_read(nsh, CSR_HVIPH);
> + csr->hviprio1h = nacl_csr_read(nsh, CSR_HVIPRIO1H);
> + csr->hviprio2h = nacl_csr_read(nsh, CSR_HVIPRIO2H);
> #endif
> + } else {
> + csr->vsiselect = csr_read(CSR_VSISELECT);
> + csr->hviprio1 = csr_read(CSR_HVIPRIO1);
> + csr->hviprio2 = csr_read(CSR_HVIPRIO2);
> +#ifdef CONFIG_32BIT
> + csr->vsieh = csr_read(CSR_VSIEH);
> + csr->hviph = csr_read(CSR_HVIPH);
> + csr->hviprio1h = csr_read(CSR_HVIPRIO1H);
> + csr->hviprio2h = csr_read(CSR_HVIPRIO2H);
> +#endif
> + }
> }
>
> int kvm_riscv_vcpu_aia_get_csr(struct kvm_vcpu *vcpu,
> @@ -250,20 +279,20 @@ static u8 aia_get_iprio8(struct kvm_vcpu *vcpu, unsigned int irq)
>
> switch (bitpos / BITS_PER_LONG) {
> case 0:
> - hviprio = csr_read(CSR_HVIPRIO1);
> + hviprio = ncsr_read(CSR_HVIPRIO1);
> break;
> case 1:
> #ifndef CONFIG_32BIT
> - hviprio = csr_read(CSR_HVIPRIO2);
> + hviprio = ncsr_read(CSR_HVIPRIO2);
> break;
> #else
> - hviprio = csr_read(CSR_HVIPRIO1H);
> + hviprio = ncsr_read(CSR_HVIPRIO1H);
> break;
> case 2:
> - hviprio = csr_read(CSR_HVIPRIO2);
> + hviprio = ncsr_read(CSR_HVIPRIO2);
> break;
> case 3:
> - hviprio = csr_read(CSR_HVIPRIO2H);
> + hviprio = ncsr_read(CSR_HVIPRIO2H);
> break;
> #endif
> default:
> @@ -283,20 +312,20 @@ static void aia_set_iprio8(struct kvm_vcpu *vcpu, unsigned int irq, u8 prio)
>
> switch (bitpos / BITS_PER_LONG) {
> case 0:
> - hviprio = csr_read(CSR_HVIPRIO1);
> + hviprio = ncsr_read(CSR_HVIPRIO1);
> break;
> case 1:
> #ifndef CONFIG_32BIT
> - hviprio = csr_read(CSR_HVIPRIO2);
> + hviprio = ncsr_read(CSR_HVIPRIO2);
> break;
> #else
> - hviprio = csr_read(CSR_HVIPRIO1H);
> + hviprio = ncsr_read(CSR_HVIPRIO1H);
> break;
> case 2:
> - hviprio = csr_read(CSR_HVIPRIO2);
> + hviprio = ncsr_read(CSR_HVIPRIO2);
> break;
> case 3:
> - hviprio = csr_read(CSR_HVIPRIO2H);
> + hviprio = ncsr_read(CSR_HVIPRIO2H);
> break;
> #endif
> default:
> @@ -308,20 +337,20 @@ static void aia_set_iprio8(struct kvm_vcpu *vcpu, unsigned int irq, u8 prio)
>
> switch (bitpos / BITS_PER_LONG) {
> case 0:
> - csr_write(CSR_HVIPRIO1, hviprio);
> + ncsr_write(CSR_HVIPRIO1, hviprio);
> break;
> case 1:
> #ifndef CONFIG_32BIT
> - csr_write(CSR_HVIPRIO2, hviprio);
> + ncsr_write(CSR_HVIPRIO2, hviprio);
> break;
> #else
> - csr_write(CSR_HVIPRIO1H, hviprio);
> + ncsr_write(CSR_HVIPRIO1H, hviprio);
> break;
> case 2:
> - csr_write(CSR_HVIPRIO2, hviprio);
> + ncsr_write(CSR_HVIPRIO2, hviprio);
> break;
> case 3:
> - csr_write(CSR_HVIPRIO2H, hviprio);
> + ncsr_write(CSR_HVIPRIO2H, hviprio);
> break;
> #endif
> default:
> @@ -377,7 +406,7 @@ int kvm_riscv_vcpu_aia_rmw_ireg(struct kvm_vcpu *vcpu, unsigned int csr_num,
> return KVM_INSN_ILLEGAL_TRAP;
>
> /* First try to emulate in kernel space */
> - isel = csr_read(CSR_VSISELECT) & ISELECT_MASK;
> + isel = ncsr_read(CSR_VSISELECT) & ISELECT_MASK;
> if (isel >= ISELECT_IPRIO0 && isel <= ISELECT_IPRIO15)
> return aia_rmw_iprio(vcpu, isel, val, new_val, wr_mask);
> else if (isel >= IMSIC_FIRST && isel <= IMSIC_LAST &&
> --
> 2.34.1
>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
--
Regards,
Atish
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 11/13] RISC-V: KVM: Use SBI sync SRET call when available
2024-07-19 16:09 ` [PATCH 11/13] RISC-V: KVM: Use SBI sync SRET call when available Anup Patel
@ 2024-10-18 20:03 ` Atish Patra
2024-10-20 19:28 ` Anup Patel
0 siblings, 1 reply; 29+ messages in thread
From: Atish Patra @ 2024-10-18 20:03 UTC (permalink / raw)
To: Anup Patel
Cc: Palmer Dabbelt, Paul Walmsley, Andrew Jones, Anup Patel, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Jul 19, 2024 at 9:10 AM Anup Patel <apatel@ventanamicro.com> wrote:
>
> Implement an optimized KVM world-switch using SBI sync SRET call
> when SBI nested acceleration extension is available. This improves
> KVM world-switch when KVM RISC-V is running as a Guest under some
> other hypervisor.
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/include/asm/kvm_nacl.h | 32 +++++++++++++++++++++
> arch/riscv/kvm/vcpu.c | 48 ++++++++++++++++++++++++++++---
> arch/riscv/kvm/vcpu_switch.S | 29 +++++++++++++++++++
> 3 files changed, 105 insertions(+), 4 deletions(-)
>
> diff --git a/arch/riscv/include/asm/kvm_nacl.h b/arch/riscv/include/asm/kvm_nacl.h
> index a704e8000a58..5e74238ea525 100644
> --- a/arch/riscv/include/asm/kvm_nacl.h
> +++ b/arch/riscv/include/asm/kvm_nacl.h
> @@ -12,6 +12,8 @@
> #include <asm/csr.h>
> #include <asm/sbi.h>
>
> +struct kvm_vcpu_arch;
> +
> DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_available);
> #define kvm_riscv_nacl_available() \
> static_branch_unlikely(&kvm_riscv_nacl_available)
> @@ -43,6 +45,10 @@ void __kvm_riscv_nacl_hfence(void *shmem,
> unsigned long page_num,
> unsigned long page_count);
>
> +void __kvm_riscv_nacl_switch_to(struct kvm_vcpu_arch *vcpu_arch,
> + unsigned long sbi_ext_id,
> + unsigned long sbi_func_id);
> +
> int kvm_riscv_nacl_enable(void);
>
> void kvm_riscv_nacl_disable(void);
> @@ -64,6 +70,32 @@ int kvm_riscv_nacl_init(void);
> #define nacl_shmem_fast() \
> (kvm_riscv_nacl_available() ? nacl_shmem() : NULL)
>
> +#define nacl_scratch_read_long(__shmem, __offset) \
> +({ \
> + unsigned long *__p = (__shmem) + \
> + SBI_NACL_SHMEM_SCRATCH_OFFSET + \
> + (__offset); \
> + lelong_to_cpu(*__p); \
> +})
> +
> +#define nacl_scratch_write_long(__shmem, __offset, __val) \
> +do { \
> + unsigned long *__p = (__shmem) + \
> + SBI_NACL_SHMEM_SCRATCH_OFFSET + \
> + (__offset); \
> + *__p = cpu_to_lelong(__val); \
> +} while (0)
> +
> +#define nacl_scratch_write_longs(__shmem, __offset, __array, __count) \
> +do { \
> + unsigned int __i; \
> + unsigned long *__p = (__shmem) + \
> + SBI_NACL_SHMEM_SCRATCH_OFFSET + \
> + (__offset); \
> + for (__i = 0; __i < (__count); __i++) \
> + __p[__i] = cpu_to_lelong((__array)[__i]); \
> +} while (0)
> +
This should be in a separate patch along with other helpers ?
> #define nacl_sync_hfence(__e) \
> sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_HFENCE, \
> (__e), 0, 0, 0, 0, 0)
> diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
> index 00baaf1b0136..fe849fb1aaab 100644
> --- a/arch/riscv/kvm/vcpu.c
> +++ b/arch/riscv/kvm/vcpu.c
> @@ -759,19 +759,59 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v
> */
> static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
> {
> + void *nsh;
> struct kvm_cpu_context *gcntx = &vcpu->arch.guest_context;
> struct kvm_cpu_context *hcntx = &vcpu->arch.host_context;
>
> kvm_riscv_vcpu_swap_in_guest_state(vcpu);
> guest_state_enter_irqoff();
>
> - hcntx->hstatus = ncsr_swap(CSR_HSTATUS, gcntx->hstatus);
> + if (kvm_riscv_nacl_sync_sret_available()) {
> + nsh = nacl_shmem();
>
> - nsync_csr(-1UL);
> + if (kvm_riscv_nacl_autoswap_csr_available()) {
> + hcntx->hstatus =
> + nacl_csr_read(nsh, CSR_HSTATUS);
> + nacl_scratch_write_long(nsh,
> + SBI_NACL_SHMEM_AUTOSWAP_OFFSET +
> + SBI_NACL_SHMEM_AUTOSWAP_HSTATUS,
> + gcntx->hstatus);
> + nacl_scratch_write_long(nsh,
> + SBI_NACL_SHMEM_AUTOSWAP_OFFSET,
> + SBI_NACL_SHMEM_AUTOSWAP_FLAG_HSTATUS);
> + } else if (kvm_riscv_nacl_sync_csr_available()) {
> + hcntx->hstatus = nacl_csr_swap(nsh,
> + CSR_HSTATUS, gcntx->hstatus);
> + } else {
> + hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
> + }
>
> - __kvm_riscv_switch_to(&vcpu->arch);
> + nacl_scratch_write_longs(nsh,
> + SBI_NACL_SHMEM_SRET_OFFSET +
> + SBI_NACL_SHMEM_SRET_X(1),
> + &gcntx->ra,
> + SBI_NACL_SHMEM_SRET_X_LAST);
> +
> + __kvm_riscv_nacl_switch_to(&vcpu->arch, SBI_EXT_NACL,
> + SBI_EXT_NACL_SYNC_SRET);
> +
> + if (kvm_riscv_nacl_autoswap_csr_available()) {
> + nacl_scratch_write_long(nsh,
> + SBI_NACL_SHMEM_AUTOSWAP_OFFSET,
> + 0);
> + gcntx->hstatus = nacl_scratch_read_long(nsh,
> + SBI_NACL_SHMEM_AUTOSWAP_OFFSET +
> + SBI_NACL_SHMEM_AUTOSWAP_HSTATUS);
> + } else {
> + gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
> + }
> + } else {
> + hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
>
> - gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
> + __kvm_riscv_switch_to(&vcpu->arch);
> +
> + gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
> + }
>
> vcpu->arch.last_exit_cpu = vcpu->cpu;
> guest_state_exit_irqoff();
> diff --git a/arch/riscv/kvm/vcpu_switch.S b/arch/riscv/kvm/vcpu_switch.S
> index 9f13e5ce6a18..47686bcb21e0 100644
> --- a/arch/riscv/kvm/vcpu_switch.S
> +++ b/arch/riscv/kvm/vcpu_switch.S
> @@ -218,6 +218,35 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
> ret
> SYM_FUNC_END(__kvm_riscv_switch_to)
>
> + /*
> + * Parameters:
> + * A0 <= Pointer to struct kvm_vcpu_arch
> + * A1 <= SBI extension ID
> + * A2 <= SBI function ID
> + */
> +SYM_FUNC_START(__kvm_riscv_nacl_switch_to)
> + SAVE_HOST_GPRS
> +
> + SAVE_HOST_AND_RESTORE_GUEST_CSRS .Lkvm_nacl_switch_return
> +
> + /* Resume Guest using SBI nested acceleration */
> + add a6, a2, zero
> + add a7, a1, zero
> + ecall
> +
> + /* Back to Host */
> + .align 2
> +.Lkvm_nacl_switch_return:
> + SAVE_GUEST_GPRS
> +
> + SAVE_GUEST_AND_RESTORE_HOST_CSRS
> +
> + RESTORE_HOST_GPRS
> +
> + /* Return to C code */
> + ret
> +SYM_FUNC_END(__kvm_riscv_nacl_switch_to)
> +
> SYM_CODE_START(__kvm_riscv_unpriv_trap)
> /*
> * We assume that faulting unpriv load/store instruction is
> --
> 2.34.1
>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
--
Regards,
Atish
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 12/13] RISC-V: KVM: Save trap CSRs in kvm_riscv_vcpu_enter_exit()
2024-07-19 16:09 ` [PATCH 12/13] RISC-V: KVM: Save trap CSRs in kvm_riscv_vcpu_enter_exit() Anup Patel
@ 2024-10-18 20:11 ` Atish Patra
0 siblings, 0 replies; 29+ messages in thread
From: Atish Patra @ 2024-10-18 20:11 UTC (permalink / raw)
To: Anup Patel
Cc: Palmer Dabbelt, Paul Walmsley, Andrew Jones, Anup Patel, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Jul 19, 2024 at 9:10 AM Anup Patel <apatel@ventanamicro.com> wrote:
>
> Save trap CSRs in the kvm_riscv_vcpu_enter_exit() function instead of
> the kvm_arch_vcpu_ioctl_run() function so that HTVAL and HTINST CSRs
> are accessed in more optimized manner while running under some other
> hypervisor.
>
> Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> ---
> arch/riscv/kvm/vcpu.c | 34 +++++++++++++++++++++-------------
> 1 file changed, 21 insertions(+), 13 deletions(-)
>
> diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
> index fe849fb1aaab..854d98aa165e 100644
> --- a/arch/riscv/kvm/vcpu.c
> +++ b/arch/riscv/kvm/vcpu.c
> @@ -757,12 +757,21 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v
> * This must be noinstr as instrumentation may make use of RCU, and this is not
> * safe during the EQS.
> */
> -static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
> +static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu,
> + struct kvm_cpu_trap *trap)
> {
> void *nsh;
> struct kvm_cpu_context *gcntx = &vcpu->arch.guest_context;
> struct kvm_cpu_context *hcntx = &vcpu->arch.host_context;
>
> + /*
> + * We save trap CSRs (such as SEPC, SCAUSE, STVAL, HTVAL, and
> + * HTINST) here because we do local_irq_enable() after this
> + * function in kvm_arch_vcpu_ioctl_run() which can result in
> + * an interrupt immediately after local_irq_enable() and can
> + * potentially change trap CSRs.
> + */
> +
> kvm_riscv_vcpu_swap_in_guest_state(vcpu);
> guest_state_enter_irqoff();
>
> @@ -805,14 +814,24 @@ static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
> } else {
> gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
> }
> +
> + trap->htval = nacl_csr_read(nsh, CSR_HTVAL);
> + trap->htinst = nacl_csr_read(nsh, CSR_HTINST);
> } else {
> hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
>
> __kvm_riscv_switch_to(&vcpu->arch);
>
> gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
> +
> + trap->htval = csr_read(CSR_HTVAL);
> + trap->htinst = csr_read(CSR_HTINST);
> }
>
> + trap->sepc = gcntx->sepc;
> + trap->scause = csr_read(CSR_SCAUSE);
> + trap->stval = csr_read(CSR_STVAL);
> +
> vcpu->arch.last_exit_cpu = vcpu->cpu;
> guest_state_exit_irqoff();
> kvm_riscv_vcpu_swap_in_host_state(vcpu);
> @@ -929,22 +948,11 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
>
> guest_timing_enter_irqoff();
>
> - kvm_riscv_vcpu_enter_exit(vcpu);
> + kvm_riscv_vcpu_enter_exit(vcpu, &trap);
>
> vcpu->mode = OUTSIDE_GUEST_MODE;
> vcpu->stat.exits++;
>
> - /*
> - * Save SCAUSE, STVAL, HTVAL, and HTINST because we might
> - * get an interrupt between __kvm_riscv_switch_to() and
> - * local_irq_enable() which can potentially change CSRs.
> - */
> - trap.sepc = vcpu->arch.guest_context.sepc;
> - trap.scause = csr_read(CSR_SCAUSE);
> - trap.stval = csr_read(CSR_STVAL);
> - trap.htval = ncsr_read(CSR_HTVAL);
> - trap.htinst = ncsr_read(CSR_HTINST);
> -
> /* Syncup interrupts state with HW */
> kvm_riscv_vcpu_sync_interrupts(vcpu);
>
> --
> 2.34.1
>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
--
Regards,
Atish
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 08/13] RISC-V: KVM: Add common nested acceleration support
2024-10-18 18:19 ` Atish Patra
@ 2024-10-20 18:19 ` Anup Patel
0 siblings, 0 replies; 29+ messages in thread
From: Anup Patel @ 2024-10-20 18:19 UTC (permalink / raw)
To: Atish Patra
Cc: Anup Patel, Palmer Dabbelt, Paul Walmsley, Andrew Jones, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Fri, Oct 18, 2024 at 11:49 PM Atish Patra <atishp@atishpatra.org> wrote:
>
> On Fri, Jul 19, 2024 at 9:09 AM Anup Patel <apatel@ventanamicro.com> wrote:
> >
> > Add a common nested acceleration support which will be shared by
> > all parts of KVM RISC-V. This nested acceleration support detects
> > and enables SBI NACL extension usage based on static keys which
> > ensures minimum impact on the non-nested scenario.
> >
> > Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> > ---
> > arch/riscv/include/asm/kvm_nacl.h | 205 ++++++++++++++++++++++++++++++
> > arch/riscv/kvm/Makefile | 1 +
> > arch/riscv/kvm/main.c | 53 +++++++-
> > arch/riscv/kvm/nacl.c | 152 ++++++++++++++++++++++
> > 4 files changed, 409 insertions(+), 2 deletions(-)
> > create mode 100644 arch/riscv/include/asm/kvm_nacl.h
> > create mode 100644 arch/riscv/kvm/nacl.c
> >
> > diff --git a/arch/riscv/include/asm/kvm_nacl.h b/arch/riscv/include/asm/kvm_nacl.h
> > new file mode 100644
> > index 000000000000..a704e8000a58
> > --- /dev/null
> > +++ b/arch/riscv/include/asm/kvm_nacl.h
> > @@ -0,0 +1,205 @@
> > +/* SPDX-License-Identifier: GPL-2.0-only */
> > +/*
> > + * Copyright (c) 2024 Ventana Micro Systems Inc.
> > + */
> > +
> > +#ifndef __KVM_NACL_H
> > +#define __KVM_NACL_H
> > +
> > +#include <linux/jump_label.h>
> > +#include <linux/percpu.h>
> > +#include <asm/byteorder.h>
> > +#include <asm/csr.h>
> > +#include <asm/sbi.h>
> > +
> > +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_available);
> > +#define kvm_riscv_nacl_available() \
> > + static_branch_unlikely(&kvm_riscv_nacl_available)
> > +
> > +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_csr_available);
> > +#define kvm_riscv_nacl_sync_csr_available() \
> > + static_branch_unlikely(&kvm_riscv_nacl_sync_csr_available)
> > +
> > +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_hfence_available);
> > +#define kvm_riscv_nacl_sync_hfence_available() \
> > + static_branch_unlikely(&kvm_riscv_nacl_sync_hfence_available)
> > +
> > +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_sret_available);
> > +#define kvm_riscv_nacl_sync_sret_available() \
> > + static_branch_unlikely(&kvm_riscv_nacl_sync_sret_available)
> > +
> > +DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_autoswap_csr_available);
> > +#define kvm_riscv_nacl_autoswap_csr_available() \
> > + static_branch_unlikely(&kvm_riscv_nacl_autoswap_csr_available)
> > +
> > +struct kvm_riscv_nacl {
> > + void *shmem;
> > + phys_addr_t shmem_phys;
> > +};
> > +DECLARE_PER_CPU(struct kvm_riscv_nacl, kvm_riscv_nacl);
> > +
> > +void __kvm_riscv_nacl_hfence(void *shmem,
> > + unsigned long control,
> > + unsigned long page_num,
> > + unsigned long page_count);
> > +
> > +int kvm_riscv_nacl_enable(void);
> > +
> > +void kvm_riscv_nacl_disable(void);
> > +
> > +void kvm_riscv_nacl_exit(void);
> > +
> > +int kvm_riscv_nacl_init(void);
> > +
> > +#ifdef CONFIG_32BIT
> > +#define lelong_to_cpu(__x) le32_to_cpu(__x)
> > +#define cpu_to_lelong(__x) cpu_to_le32(__x)
> > +#else
> > +#define lelong_to_cpu(__x) le64_to_cpu(__x)
> > +#define cpu_to_lelong(__x) cpu_to_le64(__x)
> > +#endif
> > +
> > +#define nacl_shmem() \
> > + this_cpu_ptr(&kvm_riscv_nacl)->shmem
> > +#define nacl_shmem_fast() \
> > + (kvm_riscv_nacl_available() ? nacl_shmem() : NULL)
> > +
>
> I don't see any usage of this one. Most of the callers of nacl_shmem
> probably require more to do if nacl is available
> and need the conditional block anyways. Am I missing something ?
Make sense, I will drop nacl_shmem_fast().
>
> > +#define nacl_sync_hfence(__e) \
> > + sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_HFENCE, \
> > + (__e), 0, 0, 0, 0, 0)
> > +
> > +#define nacl_hfence_mkconfig(__type, __order, __vmid, __asid) \
> > +({ \
> > + unsigned long __c = SBI_NACL_SHMEM_HFENCE_CONFIG_PEND; \
> > + __c |= ((__type) & SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_MASK) \
> > + << SBI_NACL_SHMEM_HFENCE_CONFIG_TYPE_SHIFT; \
> > + __c |= (((__order) - SBI_NACL_SHMEM_HFENCE_ORDER_BASE) & \
> > + SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_MASK) \
> > + << SBI_NACL_SHMEM_HFENCE_CONFIG_ORDER_SHIFT; \
> > + __c |= ((__vmid) & SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_MASK) \
> > + << SBI_NACL_SHMEM_HFENCE_CONFIG_VMID_SHIFT; \
> > + __c |= ((__asid) & SBI_NACL_SHMEM_HFENCE_CONFIG_ASID_MASK); \
> > + __c; \
> > +})
> > +
> > +#define nacl_hfence_mkpnum(__order, __addr) \
> > + ((__addr) >> (__order))
> > +
> > +#define nacl_hfence_mkpcount(__order, __size) \
> > + ((__size) >> (__order))
> > +
> > +#define nacl_hfence_gvma(__shmem, __gpa, __gpsz, __order) \
> > +__kvm_riscv_nacl_hfence(__shmem, \
> > + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA, \
> > + __order, 0, 0), \
> > + nacl_hfence_mkpnum(__order, __gpa), \
> > + nacl_hfence_mkpcount(__order, __gpsz))
> > +
> > +#define nacl_hfence_gvma_all(__shmem) \
> > +__kvm_riscv_nacl_hfence(__shmem, \
> > + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_ALL, \
> > + 0, 0, 0), 0, 0)
> > +
> > +#define nacl_hfence_gvma_vmid(__shmem, __vmid, __gpa, __gpsz, __order) \
> > +__kvm_riscv_nacl_hfence(__shmem, \
> > + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID, \
> > + __order, __vmid, 0), \
> > + nacl_hfence_mkpnum(__order, __gpa), \
> > + nacl_hfence_mkpcount(__order, __gpsz))
> > +
> > +#define nacl_hfence_gvma_vmid_all(__shmem, __vmid) \
> > +__kvm_riscv_nacl_hfence(__shmem, \
> > + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_GVMA_VMID_ALL, \
> > + 0, __vmid, 0), 0, 0)
> > +
> > +#define nacl_hfence_vvma(__shmem, __vmid, __gva, __gvsz, __order) \
> > +__kvm_riscv_nacl_hfence(__shmem, \
> > + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA, \
> > + __order, __vmid, 0), \
> > + nacl_hfence_mkpnum(__order, __gva), \
> > + nacl_hfence_mkpcount(__order, __gvsz))
> > +
> > +#define nacl_hfence_vvma_all(__shmem, __vmid) \
> > +__kvm_riscv_nacl_hfence(__shmem, \
> > + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ALL, \
> > + 0, __vmid, 0), 0, 0)
> > +
> > +#define nacl_hfence_vvma_asid(__shmem, __vmid, __asid, __gva, __gvsz, __order)\
> > +__kvm_riscv_nacl_hfence(__shmem, \
> > + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID, \
> > + __order, __vmid, __asid), \
> > + nacl_hfence_mkpnum(__order, __gva), \
> > + nacl_hfence_mkpcount(__order, __gvsz))
> > +
> > +#define nacl_hfence_vvma_asid_all(__shmem, __vmid, __asid) \
> > +__kvm_riscv_nacl_hfence(__shmem, \
> > + nacl_hfence_mkconfig(SBI_NACL_SHMEM_HFENCE_TYPE_VVMA_ASID_ALL, \
> > + 0, __vmid, __asid), 0, 0)
> > +
> > +#define nacl_csr_read(__shmem, __csr) \
> > +({ \
> > + unsigned long *__a = (__shmem) + SBI_NACL_SHMEM_CSR_OFFSET; \
> > + lelong_to_cpu(__a[SBI_NACL_SHMEM_CSR_INDEX(__csr)]); \
> > +})
> > +
> > +#define nacl_csr_write(__shmem, __csr, __val) \
> > +do { \
> > + void *__s = (__shmem); \
> > + unsigned int __i = SBI_NACL_SHMEM_CSR_INDEX(__csr); \
> > + unsigned long *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \
> > + u8 *__b = (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \
> > + __a[__i] = cpu_to_lelong(__val); \
> > + __b[__i >> 3] |= 1U << (__i & 0x7); \
> > +} while (0)
> > +
> > +#define nacl_csr_swap(__shmem, __csr, __val) \
> > +({ \
> > + void *__s = (__shmem); \
> > + unsigned int __i = SBI_NACL_SHMEM_CSR_INDEX(__csr); \
> > + unsigned long *__a = (__s) + SBI_NACL_SHMEM_CSR_OFFSET; \
> > + u8 *__b = (__s) + SBI_NACL_SHMEM_DBITMAP_OFFSET; \
> > + unsigned long __r = lelong_to_cpu(__a[__i]); \
> > + __a[__i] = cpu_to_lelong(__val); \
> > + __b[__i >> 3] |= 1U << (__i & 0x7); \
> > + __r; \
> > +})
> > +
> > +#define nacl_sync_csr(__csr) \
> > + sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_CSR, \
> > + (__csr), 0, 0, 0, 0, 0)
> > +
> > +#define ncsr_read(__csr) \
> > +({ \
> > + unsigned long __r; \
> > + if (kvm_riscv_nacl_available()) \
> > + __r = nacl_csr_read(nacl_shmem(), __csr); \
> > + else \
> > + __r = csr_read(__csr); \
> > + __r; \
> > +})
> > +
> > +#define ncsr_write(__csr, __val) \
> > +do { \
> > + if (kvm_riscv_nacl_sync_csr_available()) \
> > + nacl_csr_write(nacl_shmem(), __csr, __val); \
> > + else \
> > + csr_write(__csr, __val); \
> > +} while (0)
> > +
> > +#define ncsr_swap(__csr, __val) \
> > +({ \
> > + unsigned long __r; \
> > + if (kvm_riscv_nacl_sync_csr_available()) \
> > + __r = nacl_csr_swap(nacl_shmem(), __csr, __val); \
> > + else \
> > + __r = csr_swap(__csr, __val); \
> > + __r; \
> > +})
> > +
> > +#define nsync_csr(__csr) \
> > +do { \
> > + if (kvm_riscv_nacl_sync_csr_available()) \
> > + nacl_sync_csr(__csr); \
> > +} while (0)
> > +
> > +#endif
> > diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile
> > index c1eac0d093de..0fb1840c3e0a 100644
> > --- a/arch/riscv/kvm/Makefile
> > +++ b/arch/riscv/kvm/Makefile
> > @@ -16,6 +16,7 @@ kvm-y += aia_device.o
> > kvm-y += aia_imsic.o
> > kvm-y += main.o
> > kvm-y += mmu.o
> > +kvm-y += nacl.o
> > kvm-y += tlb.o
> > kvm-y += vcpu.o
> > kvm-y += vcpu_exit.o
> > diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c
> > index bab2ec34cd87..fd78f40bbb04 100644
> > --- a/arch/riscv/kvm/main.c
> > +++ b/arch/riscv/kvm/main.c
> > @@ -10,8 +10,8 @@
> > #include <linux/err.h>
> > #include <linux/module.h>
> > #include <linux/kvm_host.h>
> > -#include <asm/csr.h>
> > #include <asm/cpufeature.h>
> > +#include <asm/kvm_nacl.h>
> > #include <asm/sbi.h>
> >
> > long kvm_arch_dev_ioctl(struct file *filp,
> > @@ -22,6 +22,12 @@ long kvm_arch_dev_ioctl(struct file *filp,
> >
> > int kvm_arch_hardware_enable(void)
> > {
> > + int rc;
> > +
> > + rc = kvm_riscv_nacl_enable();
> > + if (rc)
> > + return rc;
> > +
> > csr_write(CSR_HEDELEG, KVM_HEDELEG_DEFAULT);
> > csr_write(CSR_HIDELEG, KVM_HIDELEG_DEFAULT);
> >
> > @@ -49,11 +55,14 @@ void kvm_arch_hardware_disable(void)
> > csr_write(CSR_HVIP, 0);
> > csr_write(CSR_HEDELEG, 0);
> > csr_write(CSR_HIDELEG, 0);
> > +
> > + kvm_riscv_nacl_disable();
> > }
> >
> > static int __init riscv_kvm_init(void)
> > {
> > int rc;
> > + char slist[64];
> > const char *str;
> >
> > if (!riscv_isa_extension_available(NULL, h)) {
> > @@ -71,16 +80,53 @@ static int __init riscv_kvm_init(void)
> > return -ENODEV;
> > }
> >
> > + rc = kvm_riscv_nacl_init();
> > + if (rc && rc != -ENODEV)
> > + return rc;
> > +
> > kvm_riscv_gstage_mode_detect();
> >
> > kvm_riscv_gstage_vmid_detect();
> >
> > rc = kvm_riscv_aia_init();
> > - if (rc && rc != -ENODEV)
> > + if (rc && rc != -ENODEV) {
> > + kvm_riscv_nacl_exit();
> > return rc;
> > + }
> >
> > kvm_info("hypervisor extension available\n");
> >
> > + if (kvm_riscv_nacl_available()) {
> > + rc = 0;
> > + slist[0] = '\0';
> > + if (kvm_riscv_nacl_sync_csr_available()) {
> > + if (rc)
> > + strcat(slist, ", ");
> > + strcat(slist, "sync_csr");
> > + rc++;
> > + }
> > + if (kvm_riscv_nacl_sync_hfence_available()) {
> > + if (rc)
> > + strcat(slist, ", ");
> > + strcat(slist, "sync_hfence");
> > + rc++;
> > + }
> > + if (kvm_riscv_nacl_sync_sret_available()) {
> > + if (rc)
> > + strcat(slist, ", ");
> > + strcat(slist, "sync_sret");
> > + rc++;
> > + }
> > + if (kvm_riscv_nacl_autoswap_csr_available()) {
> > + if (rc)
> > + strcat(slist, ", ");
> > + strcat(slist, "autoswap_csr");
> > + rc++;
> > + }
> > + kvm_info("using SBI nested acceleration with %s\n",
> > + (rc) ? slist : "no features");
> > + }
> > +
> > switch (kvm_riscv_gstage_mode()) {
> > case HGATP_MODE_SV32X4:
> > str = "Sv32x4";
> > @@ -108,6 +154,7 @@ static int __init riscv_kvm_init(void)
> > rc = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
> > if (rc) {
> > kvm_riscv_aia_exit();
> > + kvm_riscv_nacl_exit();
> > return rc;
> > }
> >
> > @@ -119,6 +166,8 @@ static void __exit riscv_kvm_exit(void)
> > {
> > kvm_riscv_aia_exit();
> >
> > + kvm_riscv_nacl_exit();
> > +
> > kvm_exit();
> > }
> > module_exit(riscv_kvm_exit);
> > diff --git a/arch/riscv/kvm/nacl.c b/arch/riscv/kvm/nacl.c
> > new file mode 100644
> > index 000000000000..08a95ad9ada2
> > --- /dev/null
> > +++ b/arch/riscv/kvm/nacl.c
> > @@ -0,0 +1,152 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Copyright (c) 2024 Ventana Micro Systems Inc.
> > + */
> > +
> > +#include <linux/kvm_host.h>
> > +#include <linux/vmalloc.h>
> > +#include <asm/kvm_nacl.h>
> > +
> > +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_available);
> > +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_csr_available);
> > +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_hfence_available);
> > +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_sync_sret_available);
> > +DEFINE_STATIC_KEY_FALSE(kvm_riscv_nacl_autoswap_csr_available);
> > +DEFINE_PER_CPU(struct kvm_riscv_nacl, kvm_riscv_nacl);
> > +
> > +void __kvm_riscv_nacl_hfence(void *shmem,
> > + unsigned long control,
> > + unsigned long page_num,
> > + unsigned long page_count)
> > +{
> > + int i, ent = -1, try_count = 5;
> > + unsigned long *entp;
> > +
> > +again:
> > + for (i = 0; i < SBI_NACL_SHMEM_HFENCE_ENTRY_MAX; i++) {
> > + entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i);
> > + if (lelong_to_cpu(*entp) & SBI_NACL_SHMEM_HFENCE_CONFIG_PEND)
> > + continue;
> > +
> > + ent = i;
> > + break;
> > + }
> > +
> > + if (ent < 0) {
> > + if (try_count) {
> > + nacl_sync_hfence(-1UL);
> > + goto again;
> > + } else {
> > + pr_warn("KVM: No free entry in NACL shared memory\n");
> > + return;
> > + }
> > + }
> > +
> > + entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_CONFIG(i);
> > + *entp = cpu_to_lelong(control);
> > + entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PNUM(i);
> > + *entp = cpu_to_lelong(page_num);
> > + entp = shmem + SBI_NACL_SHMEM_HFENCE_ENTRY_PCOUNT(i);
> > + *entp = cpu_to_lelong(page_count);
> > +}
> > +
> > +int kvm_riscv_nacl_enable(void)
> > +{
> > + int rc;
> > + struct sbiret ret;
> > + struct kvm_riscv_nacl *nacl;
> > +
> > + if (!kvm_riscv_nacl_available())
> > + return 0;
> > + nacl = this_cpu_ptr(&kvm_riscv_nacl);
> > +
> > + ret = sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SET_SHMEM,
> > + nacl->shmem_phys, 0, 0, 0, 0, 0);
> > + rc = sbi_err_map_linux_errno(ret.error);
> > + if (rc)
> > + return rc;
> > +
> > + return 0;
> > +}
> > +
> > +void kvm_riscv_nacl_disable(void)
> > +{
> > + if (!kvm_riscv_nacl_available())
> > + return;
> > +
> > + sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SET_SHMEM,
> > + SBI_SHMEM_DISABLE, SBI_SHMEM_DISABLE, 0, 0, 0, 0);
> > +}
> > +
> > +void kvm_riscv_nacl_exit(void)
> > +{
> > + int cpu;
> > + struct kvm_riscv_nacl *nacl;
> > +
> > + if (!kvm_riscv_nacl_available())
> > + return;
> > +
> > + /* Allocate per-CPU shared memory */
> > + for_each_possible_cpu(cpu) {
> > + nacl = per_cpu_ptr(&kvm_riscv_nacl, cpu);
> > + if (!nacl->shmem)
> > + continue;
> > +
> > + free_pages((unsigned long)nacl->shmem,
> > + get_order(SBI_NACL_SHMEM_SIZE));
> > + nacl->shmem = NULL;
> > + nacl->shmem_phys = 0;
> > + }
> > +}
> > +
> > +static long nacl_probe_feature(long feature_id)
> > +{
> > + struct sbiret ret;
> > +
> > + if (!kvm_riscv_nacl_available())
> > + return 0;
> > +
> > + ret = sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_PROBE_FEATURE,
> > + feature_id, 0, 0, 0, 0, 0);
> > + return ret.value;
> > +}
> > +
> > +int kvm_riscv_nacl_init(void)
> > +{
> > + int cpu;
> > + struct page *shmem_page;
> > + struct kvm_riscv_nacl *nacl;
> > +
> > + if (sbi_spec_version < sbi_mk_version(1, 0) ||
> > + sbi_probe_extension(SBI_EXT_NACL) <= 0)
> > + return -ENODEV;
> > +
> > + /* Enable NACL support */
> > + static_branch_enable(&kvm_riscv_nacl_available);
> > +
> > + /* Probe NACL features */
> > + if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_CSR))
> > + static_branch_enable(&kvm_riscv_nacl_sync_csr_available);
> > + if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_HFENCE))
> > + static_branch_enable(&kvm_riscv_nacl_sync_hfence_available);
> > + if (nacl_probe_feature(SBI_NACL_FEAT_SYNC_SRET))
> > + static_branch_enable(&kvm_riscv_nacl_sync_sret_available);
> > + if (nacl_probe_feature(SBI_NACL_FEAT_AUTOSWAP_CSR))
> > + static_branch_enable(&kvm_riscv_nacl_autoswap_csr_available);
> > +
> > + /* Allocate per-CPU shared memory */
> > + for_each_possible_cpu(cpu) {
> > + nacl = per_cpu_ptr(&kvm_riscv_nacl, cpu);
> > +
> > + shmem_page = alloc_pages(GFP_KERNEL | __GFP_ZERO,
> > + get_order(SBI_NACL_SHMEM_SIZE));
> > + if (!shmem_page) {
> > + kvm_riscv_nacl_exit();
> > + return -ENOMEM;
> > + }
> > + nacl->shmem = page_to_virt(shmem_page);
> > + nacl->shmem_phys = page_to_phys(shmem_page);
> > + }
> > +
> > + return 0;
> > +}
> > --
> > 2.34.1
> >
>
> Otherwise, it looks good to me.
>
> Reviewed-by: Atish Patra <atishp@rivosinc.com>
>
> --
> Regards,
> Atish
Regards,
Anup
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 09/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing H-extension CSRs
2024-10-18 19:31 ` Atish Patra
@ 2024-10-20 18:43 ` Anup Patel
0 siblings, 0 replies; 29+ messages in thread
From: Anup Patel @ 2024-10-20 18:43 UTC (permalink / raw)
To: Atish Patra
Cc: Anup Patel, Palmer Dabbelt, Paul Walmsley, Andrew Jones, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Sat, Oct 19, 2024 at 1:01 AM Atish Patra <atishp@atishpatra.org> wrote:
>
> On Fri, Jul 19, 2024 at 9:09 AM Anup Patel <apatel@ventanamicro.com> wrote:
> >
> > When running under some other hypervisor, prefer nacl_csr_xyz()
> > for accessing H-extension CSRs in the run-loop. This makes CSR
> > access faster whenever SBI nested acceleration is available.
> >
> > Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> > ---
> > arch/riscv/kvm/mmu.c | 4 +-
> > arch/riscv/kvm/vcpu.c | 103 +++++++++++++++++++++++++-----------
> > arch/riscv/kvm/vcpu_timer.c | 28 +++++-----
> > 3 files changed, 87 insertions(+), 48 deletions(-)
> >
> > diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
> > index b63650f9b966..45ace9138947 100644
> > --- a/arch/riscv/kvm/mmu.c
> > +++ b/arch/riscv/kvm/mmu.c
> > @@ -15,7 +15,7 @@
> > #include <linux/vmalloc.h>
> > #include <linux/kvm_host.h>
> > #include <linux/sched/signal.h>
> > -#include <asm/csr.h>
> > +#include <asm/kvm_nacl.h>
> > #include <asm/page.h>
> > #include <asm/pgtable.h>
> >
> > @@ -732,7 +732,7 @@ void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu)
> > hgatp |= (READ_ONCE(k->vmid.vmid) << HGATP_VMID_SHIFT) & HGATP_VMID;
> > hgatp |= (k->pgd_phys >> PAGE_SHIFT) & HGATP_PPN;
> >
> > - csr_write(CSR_HGATP, hgatp);
> > + ncsr_write(CSR_HGATP, hgatp);
> >
> > if (!kvm_riscv_gstage_vmid_bits())
> > kvm_riscv_local_hfence_gvma_all();
> > diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
> > index 957e1a5e081b..00baaf1b0136 100644
> > --- a/arch/riscv/kvm/vcpu.c
> > +++ b/arch/riscv/kvm/vcpu.c
> > @@ -17,8 +17,8 @@
> > #include <linux/sched/signal.h>
> > #include <linux/fs.h>
> > #include <linux/kvm_host.h>
> > -#include <asm/csr.h>
> > #include <asm/cacheflush.h>
> > +#include <asm/kvm_nacl.h>
> > #include <asm/kvm_vcpu_vector.h>
> >
> > #define CREATE_TRACE_POINTS
> > @@ -361,10 +361,10 @@ void kvm_riscv_vcpu_sync_interrupts(struct kvm_vcpu *vcpu)
> > struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
> >
> > /* Read current HVIP and VSIE CSRs */
> > - csr->vsie = csr_read(CSR_VSIE);
> > + csr->vsie = ncsr_read(CSR_VSIE);
> >
> > /* Sync-up HVIP.VSSIP bit changes does by Guest */
> > - hvip = csr_read(CSR_HVIP);
> > + hvip = ncsr_read(CSR_HVIP);
> > if ((csr->hvip ^ hvip) & (1UL << IRQ_VS_SOFT)) {
> > if (hvip & (1UL << IRQ_VS_SOFT)) {
> > if (!test_and_set_bit(IRQ_VS_SOFT,
> > @@ -561,26 +561,49 @@ static void kvm_riscv_vcpu_setup_config(struct kvm_vcpu *vcpu)
> >
> > void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
> > {
> > + void *nsh;
> > struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
> > struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
> >
> > - csr_write(CSR_VSSTATUS, csr->vsstatus);
> > - csr_write(CSR_VSIE, csr->vsie);
> > - csr_write(CSR_VSTVEC, csr->vstvec);
> > - csr_write(CSR_VSSCRATCH, csr->vsscratch);
> > - csr_write(CSR_VSEPC, csr->vsepc);
> > - csr_write(CSR_VSCAUSE, csr->vscause);
> > - csr_write(CSR_VSTVAL, csr->vstval);
> > - csr_write(CSR_HEDELEG, cfg->hedeleg);
> > - csr_write(CSR_HVIP, csr->hvip);
> > - csr_write(CSR_VSATP, csr->vsatp);
> > - csr_write(CSR_HENVCFG, cfg->henvcfg);
> > - if (IS_ENABLED(CONFIG_32BIT))
> > - csr_write(CSR_HENVCFGH, cfg->henvcfg >> 32);
> > - if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
> > - csr_write(CSR_HSTATEEN0, cfg->hstateen0);
> > + if (kvm_riscv_nacl_sync_csr_available()) {
> > + nsh = nacl_shmem();
> > + nacl_csr_write(nsh, CSR_VSSTATUS, csr->vsstatus);
> > + nacl_csr_write(nsh, CSR_VSIE, csr->vsie);
> > + nacl_csr_write(nsh, CSR_VSTVEC, csr->vstvec);
> > + nacl_csr_write(nsh, CSR_VSSCRATCH, csr->vsscratch);
> > + nacl_csr_write(nsh, CSR_VSEPC, csr->vsepc);
> > + nacl_csr_write(nsh, CSR_VSCAUSE, csr->vscause);
> > + nacl_csr_write(nsh, CSR_VSTVAL, csr->vstval);
> > + nacl_csr_write(nsh, CSR_HEDELEG, cfg->hedeleg);
> > + nacl_csr_write(nsh, CSR_HVIP, csr->hvip);
> > + nacl_csr_write(nsh, CSR_VSATP, csr->vsatp);
> > + nacl_csr_write(nsh, CSR_HENVCFG, cfg->henvcfg);
> > + if (IS_ENABLED(CONFIG_32BIT))
> > + nacl_csr_write(nsh, CSR_HENVCFGH, cfg->henvcfg >> 32);
> > + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
> > + nacl_csr_write(nsh, CSR_HSTATEEN0, cfg->hstateen0);
> > + if (IS_ENABLED(CONFIG_32BIT))
> > + nacl_csr_write(nsh, CSR_HSTATEEN0H, cfg->hstateen0 >> 32);
> > + }
> > + } else {
> > + csr_write(CSR_VSSTATUS, csr->vsstatus);
> > + csr_write(CSR_VSIE, csr->vsie);
> > + csr_write(CSR_VSTVEC, csr->vstvec);
> > + csr_write(CSR_VSSCRATCH, csr->vsscratch);
> > + csr_write(CSR_VSEPC, csr->vsepc);
> > + csr_write(CSR_VSCAUSE, csr->vscause);
> > + csr_write(CSR_VSTVAL, csr->vstval);
> > + csr_write(CSR_HEDELEG, cfg->hedeleg);
> > + csr_write(CSR_HVIP, csr->hvip);
> > + csr_write(CSR_VSATP, csr->vsatp);
> > + csr_write(CSR_HENVCFG, cfg->henvcfg);
> > if (IS_ENABLED(CONFIG_32BIT))
> > - csr_write(CSR_HSTATEEN0H, cfg->hstateen0 >> 32);
> > + csr_write(CSR_HENVCFGH, cfg->henvcfg >> 32);
> > + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
> > + csr_write(CSR_HSTATEEN0, cfg->hstateen0);
> > + if (IS_ENABLED(CONFIG_32BIT))
> > + csr_write(CSR_HSTATEEN0H, cfg->hstateen0 >> 32);
> > + }
> > }
> >
> > kvm_riscv_gstage_update_hgatp(vcpu);
> > @@ -603,6 +626,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
> >
> > void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
> > {
> > + void *nsh;
> > struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
> >
> > vcpu->cpu = -1;
> > @@ -618,15 +642,28 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
> > vcpu->arch.isa);
> > kvm_riscv_vcpu_host_vector_restore(&vcpu->arch.host_context);
> >
> > - csr->vsstatus = csr_read(CSR_VSSTATUS);
> > - csr->vsie = csr_read(CSR_VSIE);
> > - csr->vstvec = csr_read(CSR_VSTVEC);
> > - csr->vsscratch = csr_read(CSR_VSSCRATCH);
> > - csr->vsepc = csr_read(CSR_VSEPC);
> > - csr->vscause = csr_read(CSR_VSCAUSE);
> > - csr->vstval = csr_read(CSR_VSTVAL);
> > - csr->hvip = csr_read(CSR_HVIP);
> > - csr->vsatp = csr_read(CSR_VSATP);
> > + if (kvm_riscv_nacl_available()) {
>
> Should we leave a comment here why ncsr_read is not efficient here
> i.e. due to block access ?
Each nacl_read/write() has its own static-branch so we will have one
direct (patchable) jump emitted for one nacl_read/write(). This means
multiple nacl_read/write() results in many direct (patchable) jumps.
Let me add a comment-block in kvm_nacl.h in the next revision.
>
> > + nsh = nacl_shmem();
> > + csr->vsstatus = nacl_csr_read(nsh, CSR_VSSTATUS);
> > + csr->vsie = nacl_csr_read(nsh, CSR_VSIE);
> > + csr->vstvec = nacl_csr_read(nsh, CSR_VSTVEC);
> > + csr->vsscratch = nacl_csr_read(nsh, CSR_VSSCRATCH);
> > + csr->vsepc = nacl_csr_read(nsh, CSR_VSEPC);
> > + csr->vscause = nacl_csr_read(nsh, CSR_VSCAUSE);
> > + csr->vstval = nacl_csr_read(nsh, CSR_VSTVAL);
> > + csr->hvip = nacl_csr_read(nsh, CSR_HVIP);
> > + csr->vsatp = nacl_csr_read(nsh, CSR_VSATP);
> > + } else {
> > + csr->vsstatus = csr_read(CSR_VSSTATUS);
> > + csr->vsie = csr_read(CSR_VSIE);
> > + csr->vstvec = csr_read(CSR_VSTVEC);
> > + csr->vsscratch = csr_read(CSR_VSSCRATCH);
> > + csr->vsepc = csr_read(CSR_VSEPC);
> > + csr->vscause = csr_read(CSR_VSCAUSE);
> > + csr->vstval = csr_read(CSR_VSTVAL);
> > + csr->hvip = csr_read(CSR_HVIP);
> > + csr->vsatp = csr_read(CSR_VSATP);
> > + }
> > }
> >
> > static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu)
> > @@ -681,7 +718,7 @@ static void kvm_riscv_update_hvip(struct kvm_vcpu *vcpu)
> > {
> > struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
> >
> > - csr_write(CSR_HVIP, csr->hvip);
> > + ncsr_write(CSR_HVIP, csr->hvip);
> > kvm_riscv_vcpu_aia_update_hvip(vcpu);
> > }
> >
> > @@ -728,7 +765,9 @@ static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
> > kvm_riscv_vcpu_swap_in_guest_state(vcpu);
> > guest_state_enter_irqoff();
> >
> > - hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
> > + hcntx->hstatus = ncsr_swap(CSR_HSTATUS, gcntx->hstatus);
> > +
> > + nsync_csr(-1UL);
> >
> > __kvm_riscv_switch_to(&vcpu->arch);
> >
> > @@ -863,8 +902,8 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
> > trap.sepc = vcpu->arch.guest_context.sepc;
> > trap.scause = csr_read(CSR_SCAUSE);
> > trap.stval = csr_read(CSR_STVAL);
> > - trap.htval = csr_read(CSR_HTVAL);
> > - trap.htinst = csr_read(CSR_HTINST);
> > + trap.htval = ncsr_read(CSR_HTVAL);
> > + trap.htinst = ncsr_read(CSR_HTINST);
> >
> > /* Syncup interrupts state with HW */
> > kvm_riscv_vcpu_sync_interrupts(vcpu);
> > diff --git a/arch/riscv/kvm/vcpu_timer.c b/arch/riscv/kvm/vcpu_timer.c
> > index 75486b25ac45..96e7a4e463f7 100644
> > --- a/arch/riscv/kvm/vcpu_timer.c
> > +++ b/arch/riscv/kvm/vcpu_timer.c
> > @@ -11,8 +11,8 @@
> > #include <linux/kvm_host.h>
> > #include <linux/uaccess.h>
> > #include <clocksource/timer-riscv.h>
> > -#include <asm/csr.h>
> > #include <asm/delay.h>
> > +#include <asm/kvm_nacl.h>
> > #include <asm/kvm_vcpu_timer.h>
> >
> > static u64 kvm_riscv_current_cycles(struct kvm_guest_timer *gt)
> > @@ -72,12 +72,12 @@ static int kvm_riscv_vcpu_timer_cancel(struct kvm_vcpu_timer *t)
> > static int kvm_riscv_vcpu_update_vstimecmp(struct kvm_vcpu *vcpu, u64 ncycles)
> > {
> > #if defined(CONFIG_32BIT)
> > - csr_write(CSR_VSTIMECMP, ncycles & 0xFFFFFFFF);
> > - csr_write(CSR_VSTIMECMPH, ncycles >> 32);
> > + ncsr_write(CSR_VSTIMECMP, ncycles & 0xFFFFFFFF);
> > + ncsr_write(CSR_VSTIMECMPH, ncycles >> 32);
> > #else
> > - csr_write(CSR_VSTIMECMP, ncycles);
> > + ncsr_write(CSR_VSTIMECMP, ncycles);
> > #endif
> > - return 0;
> > + return 0;
> > }
> >
> > static int kvm_riscv_vcpu_update_hrtimer(struct kvm_vcpu *vcpu, u64 ncycles)
> > @@ -289,10 +289,10 @@ static void kvm_riscv_vcpu_update_timedelta(struct kvm_vcpu *vcpu)
> > struct kvm_guest_timer *gt = &vcpu->kvm->arch.timer;
> >
> > #if defined(CONFIG_32BIT)
> > - csr_write(CSR_HTIMEDELTA, (u32)(gt->time_delta));
> > - csr_write(CSR_HTIMEDELTAH, (u32)(gt->time_delta >> 32));
> > + ncsr_write(CSR_HTIMEDELTA, (u32)(gt->time_delta));
> > + ncsr_write(CSR_HTIMEDELTAH, (u32)(gt->time_delta >> 32));
> > #else
> > - csr_write(CSR_HTIMEDELTA, gt->time_delta);
> > + ncsr_write(CSR_HTIMEDELTA, gt->time_delta);
> > #endif
> > }
> >
> > @@ -306,10 +306,10 @@ void kvm_riscv_vcpu_timer_restore(struct kvm_vcpu *vcpu)
> > return;
> >
> > #if defined(CONFIG_32BIT)
> > - csr_write(CSR_VSTIMECMP, (u32)t->next_cycles);
> > - csr_write(CSR_VSTIMECMPH, (u32)(t->next_cycles >> 32));
> > + ncsr_write(CSR_VSTIMECMP, (u32)t->next_cycles);
> > + ncsr_write(CSR_VSTIMECMPH, (u32)(t->next_cycles >> 32));
> > #else
> > - csr_write(CSR_VSTIMECMP, t->next_cycles);
> > + ncsr_write(CSR_VSTIMECMP, t->next_cycles);
> > #endif
> >
> > /* timer should be enabled for the remaining operations */
> > @@ -327,10 +327,10 @@ void kvm_riscv_vcpu_timer_sync(struct kvm_vcpu *vcpu)
> > return;
> >
> > #if defined(CONFIG_32BIT)
> > - t->next_cycles = csr_read(CSR_VSTIMECMP);
> > - t->next_cycles |= (u64)csr_read(CSR_VSTIMECMPH) << 32;
> > + t->next_cycles = ncsr_read(CSR_VSTIMECMP);
> > + t->next_cycles |= (u64)ncsr_read(CSR_VSTIMECMPH) << 32;
> > #else
> > - t->next_cycles = csr_read(CSR_VSTIMECMP);
> > + t->next_cycles = ncsr_read(CSR_VSTIMECMP);
> > #endif
> > }
> >
> > --
> > 2.34.1
> >
>
> Otherwise, LGTM.
>
> Reviewed-by: Atish Patra <atishp@rivosinc.com>
>
> --
> Regards,
> Atish
Regards,
Anup
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 11/13] RISC-V: KVM: Use SBI sync SRET call when available
2024-10-18 20:03 ` Atish Patra
@ 2024-10-20 19:28 ` Anup Patel
0 siblings, 0 replies; 29+ messages in thread
From: Anup Patel @ 2024-10-20 19:28 UTC (permalink / raw)
To: Atish Patra
Cc: Anup Patel, Palmer Dabbelt, Paul Walmsley, Andrew Jones, kvm,
kvm-riscv, linux-riscv, linux-kernel
On Sat, Oct 19, 2024 at 1:33 AM Atish Patra <atishp@atishpatra.org> wrote:
>
> On Fri, Jul 19, 2024 at 9:10 AM Anup Patel <apatel@ventanamicro.com> wrote:
> >
> > Implement an optimized KVM world-switch using SBI sync SRET call
> > when SBI nested acceleration extension is available. This improves
> > KVM world-switch when KVM RISC-V is running as a Guest under some
> > other hypervisor.
> >
> > Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> > ---
> > arch/riscv/include/asm/kvm_nacl.h | 32 +++++++++++++++++++++
> > arch/riscv/kvm/vcpu.c | 48 ++++++++++++++++++++++++++++---
> > arch/riscv/kvm/vcpu_switch.S | 29 +++++++++++++++++++
> > 3 files changed, 105 insertions(+), 4 deletions(-)
> >
> > diff --git a/arch/riscv/include/asm/kvm_nacl.h b/arch/riscv/include/asm/kvm_nacl.h
> > index a704e8000a58..5e74238ea525 100644
> > --- a/arch/riscv/include/asm/kvm_nacl.h
> > +++ b/arch/riscv/include/asm/kvm_nacl.h
> > @@ -12,6 +12,8 @@
> > #include <asm/csr.h>
> > #include <asm/sbi.h>
> >
> > +struct kvm_vcpu_arch;
> > +
> > DECLARE_STATIC_KEY_FALSE(kvm_riscv_nacl_available);
> > #define kvm_riscv_nacl_available() \
> > static_branch_unlikely(&kvm_riscv_nacl_available)
> > @@ -43,6 +45,10 @@ void __kvm_riscv_nacl_hfence(void *shmem,
> > unsigned long page_num,
> > unsigned long page_count);
> >
> > +void __kvm_riscv_nacl_switch_to(struct kvm_vcpu_arch *vcpu_arch,
> > + unsigned long sbi_ext_id,
> > + unsigned long sbi_func_id);
> > +
> > int kvm_riscv_nacl_enable(void);
> >
> > void kvm_riscv_nacl_disable(void);
> > @@ -64,6 +70,32 @@ int kvm_riscv_nacl_init(void);
> > #define nacl_shmem_fast() \
> > (kvm_riscv_nacl_available() ? nacl_shmem() : NULL)
> >
> > +#define nacl_scratch_read_long(__shmem, __offset) \
> > +({ \
> > + unsigned long *__p = (__shmem) + \
> > + SBI_NACL_SHMEM_SCRATCH_OFFSET + \
> > + (__offset); \
> > + lelong_to_cpu(*__p); \
> > +})
> > +
> > +#define nacl_scratch_write_long(__shmem, __offset, __val) \
> > +do { \
> > + unsigned long *__p = (__shmem) + \
> > + SBI_NACL_SHMEM_SCRATCH_OFFSET + \
> > + (__offset); \
> > + *__p = cpu_to_lelong(__val); \
> > +} while (0)
> > +
> > +#define nacl_scratch_write_longs(__shmem, __offset, __array, __count) \
> > +do { \
> > + unsigned int __i; \
> > + unsigned long *__p = (__shmem) + \
> > + SBI_NACL_SHMEM_SCRATCH_OFFSET + \
> > + (__offset); \
> > + for (__i = 0; __i < (__count); __i++) \
> > + __p[__i] = cpu_to_lelong((__array)[__i]); \
> > +} while (0)
> > +
>
> This should be in a separate patch along with other helpers ?
Okay, I will move these macros to PATCH8.
>
> > #define nacl_sync_hfence(__e) \
> > sbi_ecall(SBI_EXT_NACL, SBI_EXT_NACL_SYNC_HFENCE, \
> > (__e), 0, 0, 0, 0, 0)
> > diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c
> > index 00baaf1b0136..fe849fb1aaab 100644
> > --- a/arch/riscv/kvm/vcpu.c
> > +++ b/arch/riscv/kvm/vcpu.c
> > @@ -759,19 +759,59 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v
> > */
> > static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
> > {
> > + void *nsh;
> > struct kvm_cpu_context *gcntx = &vcpu->arch.guest_context;
> > struct kvm_cpu_context *hcntx = &vcpu->arch.host_context;
> >
> > kvm_riscv_vcpu_swap_in_guest_state(vcpu);
> > guest_state_enter_irqoff();
> >
> > - hcntx->hstatus = ncsr_swap(CSR_HSTATUS, gcntx->hstatus);
> > + if (kvm_riscv_nacl_sync_sret_available()) {
> > + nsh = nacl_shmem();
> >
> > - nsync_csr(-1UL);
> > + if (kvm_riscv_nacl_autoswap_csr_available()) {
> > + hcntx->hstatus =
> > + nacl_csr_read(nsh, CSR_HSTATUS);
> > + nacl_scratch_write_long(nsh,
> > + SBI_NACL_SHMEM_AUTOSWAP_OFFSET +
> > + SBI_NACL_SHMEM_AUTOSWAP_HSTATUS,
> > + gcntx->hstatus);
> > + nacl_scratch_write_long(nsh,
> > + SBI_NACL_SHMEM_AUTOSWAP_OFFSET,
> > + SBI_NACL_SHMEM_AUTOSWAP_FLAG_HSTATUS);
> > + } else if (kvm_riscv_nacl_sync_csr_available()) {
> > + hcntx->hstatus = nacl_csr_swap(nsh,
> > + CSR_HSTATUS, gcntx->hstatus);
> > + } else {
> > + hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
> > + }
> >
> > - __kvm_riscv_switch_to(&vcpu->arch);
> > + nacl_scratch_write_longs(nsh,
> > + SBI_NACL_SHMEM_SRET_OFFSET +
> > + SBI_NACL_SHMEM_SRET_X(1),
> > + &gcntx->ra,
> > + SBI_NACL_SHMEM_SRET_X_LAST);
> > +
> > + __kvm_riscv_nacl_switch_to(&vcpu->arch, SBI_EXT_NACL,
> > + SBI_EXT_NACL_SYNC_SRET);
> > +
> > + if (kvm_riscv_nacl_autoswap_csr_available()) {
> > + nacl_scratch_write_long(nsh,
> > + SBI_NACL_SHMEM_AUTOSWAP_OFFSET,
> > + 0);
> > + gcntx->hstatus = nacl_scratch_read_long(nsh,
> > + SBI_NACL_SHMEM_AUTOSWAP_OFFSET +
> > + SBI_NACL_SHMEM_AUTOSWAP_HSTATUS);
> > + } else {
> > + gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
> > + }
> > + } else {
> > + hcntx->hstatus = csr_swap(CSR_HSTATUS, gcntx->hstatus);
> >
> > - gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
> > + __kvm_riscv_switch_to(&vcpu->arch);
> > +
> > + gcntx->hstatus = csr_swap(CSR_HSTATUS, hcntx->hstatus);
> > + }
> >
> > vcpu->arch.last_exit_cpu = vcpu->cpu;
> > guest_state_exit_irqoff();
> > diff --git a/arch/riscv/kvm/vcpu_switch.S b/arch/riscv/kvm/vcpu_switch.S
> > index 9f13e5ce6a18..47686bcb21e0 100644
> > --- a/arch/riscv/kvm/vcpu_switch.S
> > +++ b/arch/riscv/kvm/vcpu_switch.S
> > @@ -218,6 +218,35 @@ SYM_FUNC_START(__kvm_riscv_switch_to)
> > ret
> > SYM_FUNC_END(__kvm_riscv_switch_to)
> >
> > + /*
> > + * Parameters:
> > + * A0 <= Pointer to struct kvm_vcpu_arch
> > + * A1 <= SBI extension ID
> > + * A2 <= SBI function ID
> > + */
> > +SYM_FUNC_START(__kvm_riscv_nacl_switch_to)
> > + SAVE_HOST_GPRS
> > +
> > + SAVE_HOST_AND_RESTORE_GUEST_CSRS .Lkvm_nacl_switch_return
> > +
> > + /* Resume Guest using SBI nested acceleration */
> > + add a6, a2, zero
> > + add a7, a1, zero
> > + ecall
> > +
> > + /* Back to Host */
> > + .align 2
> > +.Lkvm_nacl_switch_return:
> > + SAVE_GUEST_GPRS
> > +
> > + SAVE_GUEST_AND_RESTORE_HOST_CSRS
> > +
> > + RESTORE_HOST_GPRS
> > +
> > + /* Return to C code */
> > + ret
> > +SYM_FUNC_END(__kvm_riscv_nacl_switch_to)
> > +
> > SYM_CODE_START(__kvm_riscv_unpriv_trap)
> > /*
> > * We assume that faulting unpriv load/store instruction is
> > --
> > 2.34.1
> >
>
>
> Reviewed-by: Atish Patra <atishp@rivosinc.com>
> --
> Regards,
> Atish
Regards,
Anup
^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2024-10-20 19:28 UTC | newest]
Thread overview: 29+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-07-19 16:09 [PATCH 00/13] Accelerate KVM RISC-V when running as a guest Anup Patel
2024-07-19 16:09 ` [PATCH 01/13] RISC-V: KVM: Order the object files alphabetically Anup Patel
2024-10-16 19:03 ` [01/13] " Atish Patra
2024-07-19 16:09 ` [PATCH 02/13] RISC-V: KVM: Save/restore HSTATUS in C source Anup Patel
2024-10-16 19:27 ` [02/13] " Atish Patra
2024-07-19 16:09 ` [PATCH 03/13] RISC-V: KVM: Save/restore SCOUNTEREN " Anup Patel
2024-10-16 21:36 ` Atish Patra
2024-07-19 16:09 ` [PATCH 04/13] RISC-V: KVM: Break down the __kvm_riscv_switch_to() into macros Anup Patel
2024-10-16 21:37 ` Atish Patra
2024-07-19 16:09 ` [PATCH 05/13] RISC-V: KVM: Replace aia_set_hvictl() with aia_hvictl_value() Anup Patel
2024-10-16 21:39 ` Atish Patra
2024-07-19 16:09 ` [PATCH 06/13] RISC-V: KVM: Don't setup SGEI for zero guest external interrupts Anup Patel
2024-10-16 21:40 ` Atish Patra
2024-07-19 16:09 ` [PATCH 07/13] RISC-V: Add defines for the SBI nested acceleration extension Anup Patel
2024-10-17 23:55 ` Atish Patra
2024-07-19 16:09 ` [PATCH 08/13] RISC-V: KVM: Add common nested acceleration support Anup Patel
2024-10-18 18:19 ` Atish Patra
2024-10-20 18:19 ` Anup Patel
2024-07-19 16:09 ` [PATCH 09/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing H-extension CSRs Anup Patel
2024-10-18 19:31 ` Atish Patra
2024-10-20 18:43 ` Anup Patel
2024-07-19 16:09 ` [PATCH 10/13] RISC-V: KVM: Use nacl_csr_xyz() for accessing AIA CSRs Anup Patel
2024-10-18 19:34 ` Atish Patra
2024-07-19 16:09 ` [PATCH 11/13] RISC-V: KVM: Use SBI sync SRET call when available Anup Patel
2024-10-18 20:03 ` Atish Patra
2024-10-20 19:28 ` Anup Patel
2024-07-19 16:09 ` [PATCH 12/13] RISC-V: KVM: Save trap CSRs in kvm_riscv_vcpu_enter_exit() Anup Patel
2024-10-18 20:11 ` Atish Patra
2024-07-19 16:09 ` [PATCH 13/13] RISC-V: KVM: Use NACL HFENCEs for KVM request based HFENCEs Anup Patel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox