[PATCH v2] RISC-V: KVM: Delegate kvm unhandled faults to VS mode

[PATCH v2] RISC-V: KVM: Delegate kvm unhandled faults to VS mode

[Xu Lu]

Delegate faults which are not handled by kvm to VS mode to avoid
unnecessary traps to HS mode. These faults include illegal instruction
fault, instruction access fault, load access fault and store access
fault.

The delegation of illegal instruction fault is particularly important
to guest applications that use vector instructions frequently. In such
cases, an illegal instruction fault will be raised when guest user thread
uses vector instruction the first time and then guest kernel will enable
user thread to execute following vector instructions.

The fw pmu event counters remain undeleted so that guest can still get
these events via sbi call. Guest will only see zero count on these
events and know 'firmware' has delegated these faults.

Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
---
 arch/riscv/include/asm/kvm_host.h |  4 ++++
 arch/riscv/kvm/vcpu_exit.c        | 18 ------------------
 2 files changed, 4 insertions(+), 18 deletions(-)

diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
index 85cfebc32e4cf..e04851cf0115c 100644
--- a/arch/riscv/include/asm/kvm_host.h
+++ b/arch/riscv/include/asm/kvm_host.h
@@ -44,7 +44,11 @@
 #define KVM_REQ_STEAL_UPDATE		KVM_ARCH_REQ(6)
 
 #define KVM_HEDELEG_DEFAULT		(BIT(EXC_INST_MISALIGNED) | \
+					 BIT(EXC_INST_ACCESS)     | \
+					 BIT(EXC_INST_ILLEGAL)    | \
 					 BIT(EXC_BREAKPOINT)      | \
+					 BIT(EXC_LOAD_ACCESS)     | \
+					 BIT(EXC_STORE_ACCESS)    | \
 					 BIT(EXC_SYSCALL)         | \
 					 BIT(EXC_INST_PAGE_FAULT) | \
 					 BIT(EXC_LOAD_PAGE_FAULT) | \
diff --git a/arch/riscv/kvm/vcpu_exit.c b/arch/riscv/kvm/vcpu_exit.c
index 6e0c184127956..6e2302c65e193 100644
--- a/arch/riscv/kvm/vcpu_exit.c
+++ b/arch/riscv/kvm/vcpu_exit.c
@@ -193,11 +193,6 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
 	ret = -EFAULT;
 	run->exit_reason = KVM_EXIT_UNKNOWN;
 	switch (trap->scause) {
-	case EXC_INST_ILLEGAL:
-		kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_ILLEGAL_INSN);
-		vcpu->stat.instr_illegal_exits++;
-		ret = vcpu_redirect(vcpu, trap);
-		break;
 	case EXC_LOAD_MISALIGNED:
 		kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_MISALIGNED_LOAD);
 		vcpu->stat.load_misaligned_exits++;
@@ -208,19 +203,6 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
 		vcpu->stat.store_misaligned_exits++;
 		ret = vcpu_redirect(vcpu, trap);
 		break;
-	case EXC_LOAD_ACCESS:
-		kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_ACCESS_LOAD);
-		vcpu->stat.load_access_exits++;
-		ret = vcpu_redirect(vcpu, trap);
-		break;
-	case EXC_STORE_ACCESS:
-		kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_ACCESS_STORE);
-		vcpu->stat.store_access_exits++;
-		ret = vcpu_redirect(vcpu, trap);
-		break;
-	case EXC_INST_ACCESS:
-		ret = vcpu_redirect(vcpu, trap);
-		break;
 	case EXC_VIRTUAL_INST_FAULT:
 		if (vcpu->arch.guest_context.hstatus & HSTATUS_SPV)
 			ret = kvm_riscv_vcpu_virtual_insn(vcpu, run, trap);
-- 
2.20.1

Re: [PATCH v2] RISC-V: KVM: Delegate kvm unhandled faults to VS mode

[Radim Krčmář]

2025-07-10T21:30:30+08:00, Xu Lu <luxu.kernel@bytedance.com>:
> Delegate faults which are not handled by kvm to VS mode to avoid
> unnecessary traps to HS mode. These faults include illegal instruction
> fault, instruction access fault, load access fault and store access
> fault.
>
> The delegation of illegal instruction fault is particularly important
> to guest applications that use vector instructions frequently. In such
> cases, an illegal instruction fault will be raised when guest user thread
> uses vector instruction the first time and then guest kernel will enable
> user thread to execute following vector instructions.

(This optimization will be even more significant when nesting, where it
 would currently go -> HS0 -> HS1 -> HS0 -> VS1, instead of -> VS1.)

> The fw pmu event counters remain undeleted so that guest can still get
> these events via sbi call. Guest will only see zero count on these
> events and know 'firmware' has delegated these faults.
>
> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
> ---

Reviewed-by: Radim Krčmář <rkrcmar@ventanamicro.com>

Re: [External] Re: [PATCH v2] RISC-V: KVM: Delegate kvm unhandled faults to VS mode

[Xu Lu]

On Fri, Jul 11, 2025 at 2:16 PM Radim Krčmář <rkrcmar@ventanamicro.com> wrote:
>
> 2025-07-10T21:30:30+08:00, Xu Lu <luxu.kernel@bytedance.com>:
> > Delegate faults which are not handled by kvm to VS mode to avoid
> > unnecessary traps to HS mode. These faults include illegal instruction
> > fault, instruction access fault, load access fault and store access
> > fault.
> >
> > The delegation of illegal instruction fault is particularly important
> > to guest applications that use vector instructions frequently. In such
> > cases, an illegal instruction fault will be raised when guest user thread
> > uses vector instruction the first time and then guest kernel will enable
> > user thread to execute following vector instructions.
>
> (This optimization will be even more significant when nesting, where it
>  would currently go -> HS0 -> HS1 -> HS0 -> VS1, instead of -> VS1.)

Nice supplement! Thanks.

>
> > The fw pmu event counters remain undeleted so that guest can still get
> > these events via sbi call. Guest will only see zero count on these
> > events and know 'firmware' has delegated these faults.
> >
> > Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
> > ---
>
> Reviewed-by: Radim Krčmář <rkrcmar@ventanamicro.com>

Re: [PATCH v2] RISC-V: KVM: Delegate kvm unhandled faults to VS mode

[Anup Patel]

On Thu, Jul 10, 2025 at 7:00 PM Xu Lu <luxu.kernel@bytedance.com> wrote:
>
> Delegate faults which are not handled by kvm to VS mode to avoid
> unnecessary traps to HS mode. These faults include illegal instruction
> fault, instruction access fault, load access fault and store access
> fault.
>
> The delegation of illegal instruction fault is particularly important
> to guest applications that use vector instructions frequently. In such
> cases, an illegal instruction fault will be raised when guest user thread
> uses vector instruction the first time and then guest kernel will enable
> user thread to execute following vector instructions.
>
> The fw pmu event counters remain undeleted so that guest can still get
> these events via sbi call. Guest will only see zero count on these
> events and know 'firmware' has delegated these faults.

Currently, we don't delegate illegal instruction faults and various
access faults to Guest because we allow Guest to count PMU
firmware events. Refer, [1] and [2] for past discussions.

[1] http://lists.infradead.org/pipermail/linux-riscv/2024-August/059658.html
[2] https://lore.kernel.org/all/20241224-kvm_guest_stat-v2-0-08a77ac36b02@rivosinc.com/

I do understand that additional redirection hoop can slow down
lazy enabling of vector state so drop delegating various access
faults.

Regards,
Anup

>
> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
> ---
>  arch/riscv/include/asm/kvm_host.h |  4 ++++
>  arch/riscv/kvm/vcpu_exit.c        | 18 ------------------
>  2 files changed, 4 insertions(+), 18 deletions(-)
>
> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
> index 85cfebc32e4cf..e04851cf0115c 100644
> --- a/arch/riscv/include/asm/kvm_host.h
> +++ b/arch/riscv/include/asm/kvm_host.h
> @@ -44,7 +44,11 @@
>  #define KVM_REQ_STEAL_UPDATE           KVM_ARCH_REQ(6)
>
>  #define KVM_HEDELEG_DEFAULT            (BIT(EXC_INST_MISALIGNED) | \
> +                                        BIT(EXC_INST_ACCESS)     | \
> +                                        BIT(EXC_INST_ILLEGAL)    | \
>                                          BIT(EXC_BREAKPOINT)      | \
> +                                        BIT(EXC_LOAD_ACCESS)     | \
> +                                        BIT(EXC_STORE_ACCESS)    | \
>                                          BIT(EXC_SYSCALL)         | \
>                                          BIT(EXC_INST_PAGE_FAULT) | \
>                                          BIT(EXC_LOAD_PAGE_FAULT) | \
> diff --git a/arch/riscv/kvm/vcpu_exit.c b/arch/riscv/kvm/vcpu_exit.c
> index 6e0c184127956..6e2302c65e193 100644
> --- a/arch/riscv/kvm/vcpu_exit.c
> +++ b/arch/riscv/kvm/vcpu_exit.c
> @@ -193,11 +193,6 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
>         ret = -EFAULT;
>         run->exit_reason = KVM_EXIT_UNKNOWN;
>         switch (trap->scause) {
> -       case EXC_INST_ILLEGAL:
> -               kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_ILLEGAL_INSN);
> -               vcpu->stat.instr_illegal_exits++;
> -               ret = vcpu_redirect(vcpu, trap);
> -               break;
>         case EXC_LOAD_MISALIGNED:
>                 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_MISALIGNED_LOAD);
>                 vcpu->stat.load_misaligned_exits++;
> @@ -208,19 +203,6 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
>                 vcpu->stat.store_misaligned_exits++;
>                 ret = vcpu_redirect(vcpu, trap);
>                 break;
> -       case EXC_LOAD_ACCESS:
> -               kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_ACCESS_LOAD);
> -               vcpu->stat.load_access_exits++;
> -               ret = vcpu_redirect(vcpu, trap);
> -               break;
> -       case EXC_STORE_ACCESS:
> -               kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_ACCESS_STORE);
> -               vcpu->stat.store_access_exits++;
> -               ret = vcpu_redirect(vcpu, trap);
> -               break;
> -       case EXC_INST_ACCESS:
> -               ret = vcpu_redirect(vcpu, trap);
> -               break;
>         case EXC_VIRTUAL_INST_FAULT:
>                 if (vcpu->arch.guest_context.hstatus & HSTATUS_SPV)
>                         ret = kvm_riscv_vcpu_virtual_insn(vcpu, run, trap);
> --
> 2.20.1
>
>

Re: [External] Re: [PATCH v2] RISC-V: KVM: Delegate kvm unhandled faults to VS mode

[Xu Lu]

On Fri, Jul 11, 2025 at 4:28 PM Anup Patel <apatel@ventanamicro.com> wrote:
>
> On Thu, Jul 10, 2025 at 7:00 PM Xu Lu <luxu.kernel@bytedance.com> wrote:
> >
> > Delegate faults which are not handled by kvm to VS mode to avoid
> > unnecessary traps to HS mode. These faults include illegal instruction
> > fault, instruction access fault, load access fault and store access
> > fault.
> >
> > The delegation of illegal instruction fault is particularly important
> > to guest applications that use vector instructions frequently. In such
> > cases, an illegal instruction fault will be raised when guest user thread
> > uses vector instruction the first time and then guest kernel will enable
> > user thread to execute following vector instructions.
> >
> > The fw pmu event counters remain undeleted so that guest can still get
> > these events via sbi call. Guest will only see zero count on these
> > events and know 'firmware' has delegated these faults.
>
> Currently, we don't delegate illegal instruction faults and various
> access faults to Guest because we allow Guest to count PMU
> firmware events. Refer, [1] and [2] for past discussions.
>
> [1] http://lists.infradead.org/pipermail/linux-riscv/2024-August/059658.html
> [2] https://lore.kernel.org/all/20241224-kvm_guest_stat-v2-0-08a77ac36b02@rivosinc.com/
>
> I do understand that additional redirection hoop can slow down
> lazy enabling of vector state so drop delegating various access
> faults.

Roger that. I will resend the patch and only delegate illegal insn fault.

>
> Regards,
> Anup
>
> >
> > Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
> > ---
> >  arch/riscv/include/asm/kvm_host.h |  4 ++++
> >  arch/riscv/kvm/vcpu_exit.c        | 18 ------------------
> >  2 files changed, 4 insertions(+), 18 deletions(-)
> >
> > diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
> > index 85cfebc32e4cf..e04851cf0115c 100644
> > --- a/arch/riscv/include/asm/kvm_host.h
> > +++ b/arch/riscv/include/asm/kvm_host.h
> > @@ -44,7 +44,11 @@
> >  #define KVM_REQ_STEAL_UPDATE           KVM_ARCH_REQ(6)
> >
> >  #define KVM_HEDELEG_DEFAULT            (BIT(EXC_INST_MISALIGNED) | \
> > +                                        BIT(EXC_INST_ACCESS)     | \
> > +                                        BIT(EXC_INST_ILLEGAL)    | \
> >                                          BIT(EXC_BREAKPOINT)      | \
> > +                                        BIT(EXC_LOAD_ACCESS)     | \
> > +                                        BIT(EXC_STORE_ACCESS)    | \
> >                                          BIT(EXC_SYSCALL)         | \
> >                                          BIT(EXC_INST_PAGE_FAULT) | \
> >                                          BIT(EXC_LOAD_PAGE_FAULT) | \
> > diff --git a/arch/riscv/kvm/vcpu_exit.c b/arch/riscv/kvm/vcpu_exit.c
> > index 6e0c184127956..6e2302c65e193 100644
> > --- a/arch/riscv/kvm/vcpu_exit.c
> > +++ b/arch/riscv/kvm/vcpu_exit.c
> > @@ -193,11 +193,6 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
> >         ret = -EFAULT;
> >         run->exit_reason = KVM_EXIT_UNKNOWN;
> >         switch (trap->scause) {
> > -       case EXC_INST_ILLEGAL:
> > -               kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_ILLEGAL_INSN);
> > -               vcpu->stat.instr_illegal_exits++;
> > -               ret = vcpu_redirect(vcpu, trap);
> > -               break;
> >         case EXC_LOAD_MISALIGNED:
> >                 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_MISALIGNED_LOAD);
> >                 vcpu->stat.load_misaligned_exits++;
> > @@ -208,19 +203,6 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
> >                 vcpu->stat.store_misaligned_exits++;
> >                 ret = vcpu_redirect(vcpu, trap);
> >                 break;
> > -       case EXC_LOAD_ACCESS:
> > -               kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_ACCESS_LOAD);
> > -               vcpu->stat.load_access_exits++;
> > -               ret = vcpu_redirect(vcpu, trap);
> > -               break;
> > -       case EXC_STORE_ACCESS:
> > -               kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_ACCESS_STORE);
> > -               vcpu->stat.store_access_exits++;
> > -               ret = vcpu_redirect(vcpu, trap);
> > -               break;
> > -       case EXC_INST_ACCESS:
> > -               ret = vcpu_redirect(vcpu, trap);
> > -               break;
> >         case EXC_VIRTUAL_INST_FAULT:
> >                 if (vcpu->arch.guest_context.hstatus & HSTATUS_SPV)
> >                         ret = kvm_riscv_vcpu_virtual_insn(vcpu, run, trap);
> > --
> > 2.20.1
> >
> >