[PATCH] RISC-V: KVM: Delegate illegal instruction fault

[PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Xu Lu]

Delegate illegal instruction fault to VS mode in default to avoid such
exceptions being trapped to HS and redirected back to VS.

Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
---
 arch/riscv/include/asm/kvm_host.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
index 85cfebc32e4cf..97cc2c0dba73a 100644
--- a/arch/riscv/include/asm/kvm_host.h
+++ b/arch/riscv/include/asm/kvm_host.h
@@ -48,6 +48,7 @@
 					 BIT(EXC_SYSCALL)         | \
 					 BIT(EXC_INST_PAGE_FAULT) | \
 					 BIT(EXC_LOAD_PAGE_FAULT) | \
+					 BIT(EXC_INST_ILLEGAL)    | \
 					 BIT(EXC_STORE_PAGE_FAULT))
 
 #define KVM_HIDELEG_DEFAULT		(BIT(IRQ_VS_SOFT)  | \
-- 
2.20.1

Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Radim Krčmář]

2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
> Delegate illegal instruction fault to VS mode in default to avoid such
> exceptions being trapped to HS and redirected back to VS.
>
> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
> ---
> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
> @@ -48,6 +48,7 @@
> +					 BIT(EXC_INST_ILLEGAL)    | \

You should also remove the dead code in kvm_riscv_vcpu_exit.

And why not delegate the others as well?
(EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
 EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Thanks.

Re: [External] Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Xu Lu]

Hi Radim,

On Fri, Jun 20, 2025 at 8:04 PM Radim Krčmář <rkrcmar@ventanamicro.com> wrote:
>
> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
> > Delegate illegal instruction fault to VS mode in default to avoid such
> > exceptions being trapped to HS and redirected back to VS.
> >
> > Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
> > ---
> > diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
> > @@ -48,6 +48,7 @@
> > +                                      BIT(EXC_INST_ILLEGAL)    | \
>
> You should also remove the dead code in kvm_riscv_vcpu_exit.

I only want to delegate it by default. And KVM may still want to
delegate different exceptions for different VMs like what it does for
EXC_BREAKPOINT. So maybe it is better to reserve these codes?

>
> And why not delegate the others as well?
> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
>  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Thanks for the reminder. I will have a test and resend the patch if it works.

>
> Thanks.

Best Regards,
Xu Lu

Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Clément Léger]

On 20/06/2025 14:04, Radim Krčmář wrote:
> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
>> Delegate illegal instruction fault to VS mode in default to avoid such
>> exceptions being trapped to HS and redirected back to VS.
>>
>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
>> ---
>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
>> @@ -48,6 +48,7 @@
>> +					 BIT(EXC_INST_ILLEGAL)    | \
> 
> You should also remove the dead code in kvm_riscv_vcpu_exit.
> 
> And why not delegate the others as well?
> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
>  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)

Currently, OpenSBI does not delegate misaligned exception by default and
handles misaligned access by itself, this is (partially) why we added
the FWFT SBI extension to request such delegation. Since some supervisor
software expect that default, they do not have code to handle misaligned
accesses emulation. So they should not be delegated by default.

Thanks,

Clément

> 
> Thanks.
> 
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Radim Krčmář]

2025-06-23T10:04:45+02:00, Clément Léger <cleger@rivosinc.com>:
> On 20/06/2025 14:04, Radim Krčmář wrote:
>> And why not delegate the others as well?
>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
>>  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)
>
> Currently, OpenSBI does not delegate misaligned exception by default and
> handles misaligned access by itself, this is (partially) why we added
> the FWFT SBI extension to request such delegation. Since some supervisor
> software expect that default, they do not have code to handle misaligned
> accesses emulation. So they should not be delegated by default.

Yeah, I forgot about your patches that conflict with the change, thanks.

(The current KVM exception handler only forwards all the listed
 exceptions, so the only observable difference this change would make is
 that the KVM SBI PMU event is not counted.)

Re: [External] Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Radim Krčmář]

2025-06-22T18:11:49+08:00, Xu Lu <luxu.kernel@bytedance.com>:
> Hi Radim,
>
> On Fri, Jun 20, 2025 at 8:04 PM Radim Krčmář <rkrcmar@ventanamicro.com> wrote:
>>
>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
>> > Delegate illegal instruction fault to VS mode in default to avoid such
>> > exceptions being trapped to HS and redirected back to VS.
>> >
>> > Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
>> > ---
>> > diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
>> > @@ -48,6 +48,7 @@
>> > +                                      BIT(EXC_INST_ILLEGAL)    | \
>>
>> You should also remove the dead code in kvm_riscv_vcpu_exit.
>
> I only want to delegate it by default. And KVM may still want to
> delegate different exceptions for different VMs like what it does for
> EXC_BREAKPOINT.

(I think we could easily reintroduce the code if KVM wants to do that in
 the future.  I also think that it's bad that this patch is doing an
 observable change without userspace involvement -- the counting of KVM
 SBI PMU events, but others will probably disagree with me on this.)

>                 So maybe it is better to reserve these codes?

Possibly, the current is acceptable if you have considered the
implications on PMU events.

>> And why not delegate the others as well?
>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
>>  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)
>
> Thanks for the reminder. I will have a test and resend the patch if it works.

The misaligned exceptions are already being worked on, so don't waste
your time on them, sorry.

Re: [External] Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Xu Lu]

Hi Radim,

On Mon, Jun 23, 2025 at 6:04 PM Radim Krčmář <rkrcmar@ventanamicro.com> wrote:
>
> 2025-06-22T18:11:49+08:00, Xu Lu <luxu.kernel@bytedance.com>:
> > Hi Radim,
> >
> > On Fri, Jun 20, 2025 at 8:04 PM Radim Krčmář <rkrcmar@ventanamicro.com> wrote:
> >>
> >> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
> >> > Delegate illegal instruction fault to VS mode in default to avoid such
> >> > exceptions being trapped to HS and redirected back to VS.
> >> >
> >> > Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
> >> > ---
> >> > diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
> >> > @@ -48,6 +48,7 @@
> >> > +                                      BIT(EXC_INST_ILLEGAL)    | \
> >>
> >> You should also remove the dead code in kvm_riscv_vcpu_exit.
> >
> > I only want to delegate it by default. And KVM may still want to
> > delegate different exceptions for different VMs like what it does for
> > EXC_BREAKPOINT.
>
> (I think we could easily reintroduce the code if KVM wants to do that in
>  the future.  I also think that it's bad that this patch is doing an
>  observable change without userspace involvement -- the counting of KVM
>  SBI PMU events, but others will probably disagree with me on this.)
>
> >                 So maybe it is better to reserve these codes?
>
> Possibly, the current is acceptable if you have considered the
> implications on PMU events.

So maybe it comes back to our discussion on the difference between vs
insn fault and illegal insn fault again~ In my personal opinion, it
seems to be a waste of CPU resources to trap illegal instruction to
HS-mode hypervisor, which does nothing but redirect it back to VS-mode
guest kernel. I think it is OK (and maybe it should) to record 0
illegal instruction exits in KVM PMU. If someone wants illegal insn to
trigger an vcpu exit, then an ioctl can be provided to remove the
delegation like what KVM_SET_GUEST_DEBUG does.

>
> >> And why not delegate the others as well?
> >> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
> >>  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)
> >
> > Thanks for the reminder. I will have a test and resend the patch if it works.
>
> The misaligned exceptions are already being worked on, so don't waste
> your time on them, sorry.

Thanks for the reminder too. I did not consider this before. I will
leave the MISALIGNED faults alone.

Best Regards,

Xu Lu

Re: [External] Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Xu Lu]

Hi Clément,

On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote:
>
>
>
> On 20/06/2025 14:04, Radim Krčmář wrote:
> > 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
> >> Delegate illegal instruction fault to VS mode in default to avoid such
> >> exceptions being trapped to HS and redirected back to VS.
> >>
> >> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
> >> ---
> >> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
> >> @@ -48,6 +48,7 @@
> >> +                                     BIT(EXC_INST_ILLEGAL)    | \
> >
> > You should also remove the dead code in kvm_riscv_vcpu_exit.
> >
> > And why not delegate the others as well?
> > (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
> >  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)
>
> Currently, OpenSBI does not delegate misaligned exception by default and
> handles misaligned access by itself, this is (partially) why we added
> the FWFT SBI extension to request such delegation. Since some supervisor
> software expect that default, they do not have code to handle misaligned
> accesses emulation. So they should not be delegated by default.

It doesn't matter whether these exceptions are delegated in medeleg.
KVM in HS-mode does not handle illegal instruction or misaligned
access and only redirects them back to VS-mode. Delegating such
exceptions in hedeleg helps save CPU usage even when they are not
delegated in medeleg: opensbi will check whether these exceptions are
delegated to VS-mode and redirect them to VS-mode if possible. There
seems to be no conflicts with SSE implementation. Please correct me if
I missed anything.

Best Regards,
Xu Lu

>
> Thanks,
>
> Clément
>
> >
> > Thanks.
> >
> > _______________________________________________
> > linux-riscv mailing list
> > linux-riscv@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-riscv
>

Re: [External] Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Clément Léger]

On 23/06/2025 14:12, Xu Lu wrote:
> Hi Clément,
> 
> On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote:
>>
>>
>>
>> On 20/06/2025 14:04, Radim Krčmář wrote:
>>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
>>>> Delegate illegal instruction fault to VS mode in default to avoid such
>>>> exceptions being trapped to HS and redirected back to VS.
>>>>
>>>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
>>>> ---
>>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
>>>> @@ -48,6 +48,7 @@
>>>> +                                     BIT(EXC_INST_ILLEGAL)    | \
>>>
>>> You should also remove the dead code in kvm_riscv_vcpu_exit.
>>>
>>> And why not delegate the others as well?
>>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
>>>  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)
>>
>> Currently, OpenSBI does not delegate misaligned exception by default and
>> handles misaligned access by itself, this is (partially) why we added
>> the FWFT SBI extension to request such delegation. Since some supervisor
>> software expect that default, they do not have code to handle misaligned
>> accesses emulation. So they should not be delegated by default.
> 
> It doesn't matter whether these exceptions are delegated in medeleg.

Not sure to totally understand, but if the exceptions are not delegated
in medeleg, they won't be delegated to VS-mode even though hedeleg bit
is set right ? The spec says:

A synchronous trap that has been delegated to HS-mode (using medeleg)
is further delegated to VS-mode if V=1 before the trap and the
corresponding hedeleg bit is set.



> KVM in HS-mode does not handle illegal instruction or misaligned
> access and only redirects them back to VS-mode. Delegating such
> exceptions in hedeleg helps save CPU usage even when they are not
> delegated in medeleg: opensbi will check whether these exceptions are
> delegated to VS-mode and redirect them to VS-mode if possible. There
> seems to be no conflicts with SSE implementation. Please correct me if
> I missed anything.

AFAIU, this means that since medeleg bit for misaligned accesses were
not delegated up to the introduction of the FWFT extension, VS-mode
generated misaligned accesses were handled by OpenSBI right ? Now that
we are requesting openSBI to delegate misaligned accesses, HS-mode
handles it's own misaligned accesses through the trap handler. With that
configuration, if VS-mode generate a misaligned access, it will end up
being redirected to VS-mode and won't be handle by HS-mode.

To summarize, prior to FWFT, medeleg wasn't delegating misaligned
accesses to S-mode:

- VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it ->
Back to VS-mode, misaligned access fixed up by OpenSBI

Now that Linux uses SBI FWFT to delegates misaligned accesses (without
hedeleg being set for misaligned delegation, but that doesn't really
matter, the outcome is the same):

- VS-mode misaligned access -> trap to HS-mode -> redirection to
VS-mode, needs to handle the misaligned access by itself


This means that previously, misaligned access were silently fixed up by
OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't
true anymore. So, old kernel or sueprvisor software that  included code
to handle misaligned accesses will crash. Did I missed something ?

Note: this is not directly related to your series but my introduction of
FWFT !

Thanks,

Clément

> 
> Best Regards,
> Xu Lu
> 
>>
>> Thanks,
>>
>> Clément
>>
>>>
>>> Thanks.
>>>
>>> _______________________________________________
>>> linux-riscv mailing list
>>> linux-riscv@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-riscv
>>

Re: [External] Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Xu Lu]

Hi Clément,

On Mon, Jun 23, 2025 at 8:35 PM Clément Léger <cleger@rivosinc.com> wrote:
>
>
>
> On 23/06/2025 14:12, Xu Lu wrote:
> > Hi Clément,
> >
> > On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote:
> >>
> >>
> >>
> >> On 20/06/2025 14:04, Radim Krčmář wrote:
> >>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
> >>>> Delegate illegal instruction fault to VS mode in default to avoid such
> >>>> exceptions being trapped to HS and redirected back to VS.
> >>>>
> >>>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
> >>>> ---
> >>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
> >>>> @@ -48,6 +48,7 @@
> >>>> +                                     BIT(EXC_INST_ILLEGAL)    | \
> >>>
> >>> You should also remove the dead code in kvm_riscv_vcpu_exit.
> >>>
> >>> And why not delegate the others as well?
> >>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
> >>>  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)
> >>
> >> Currently, OpenSBI does not delegate misaligned exception by default and
> >> handles misaligned access by itself, this is (partially) why we added
> >> the FWFT SBI extension to request such delegation. Since some supervisor
> >> software expect that default, they do not have code to handle misaligned
> >> accesses emulation. So they should not be delegated by default.
> >
> > It doesn't matter whether these exceptions are delegated in medeleg.
>
> Not sure to totally understand, but if the exceptions are not delegated
> in medeleg, they won't be delegated to VS-mode even though hedeleg bit
> is set right ? The spec says:
>
> A synchronous trap that has been delegated to HS-mode (using medeleg)
> is further delegated to VS-mode if V=1 before the trap and the
> corresponding hedeleg bit is set.

Yes, you are right. The illegal insn exception is still trapped in M
mode if it is not delegated in medeleg. But delegating it in hedeleg
is still useful. The opensbi will check CSR_HEDELEG in the function
sbi_trap_redirect. If the exception has been delegated to VS-mode in
CSR_HEDLEG, opensbi can directly return back to VS-mode, without the
overhead of going back to HS-mode and then going back to VS-mode.

>
>
>
> > KVM in HS-mode does not handle illegal instruction or misaligned
> > access and only redirects them back to VS-mode. Delegating such
> > exceptions in hedeleg helps save CPU usage even when they are not
> > delegated in medeleg: opensbi will check whether these exceptions are
> > delegated to VS-mode and redirect them to VS-mode if possible. There
> > seems to be no conflicts with SSE implementation. Please correct me if
> > I missed anything.
>
> AFAIU, this means that since medeleg bit for misaligned accesses were
> not delegated up to the introduction of the FWFT extension, VS-mode
> generated misaligned accesses were handled by OpenSBI right ? Now that
> we are requesting openSBI to delegate misaligned accesses, HS-mode
> handles it's own misaligned accesses through the trap handler. With that
> configuration, if VS-mode generate a misaligned access, it will end up
> being redirected to VS-mode and won't be handle by HS-mode.
>
> To summarize, prior to FWFT, medeleg wasn't delegating misaligned
> accesses to S-mode:
>
> - VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it ->
> Back to VS-mode, misaligned access fixed up by OpenSBI

Yes, this is what I want the procedure of handling illegal insn
exceptions to be. Actually it now behaves as:

VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it
-> Back to HS-mode, does nothing -> Back to VS-mode.

I want to avoid going through HS-mode.

>
> Now that Linux uses SBI FWFT to delegates misaligned accesses (without
> hedeleg being set for misaligned delegation, but that doesn't really
> matter, the outcome is the same):
>
> - VS-mode misaligned access -> trap to HS-mode -> redirection to
> VS-mode, needs to handle the misaligned access by itself
>
>
> This means that previously, misaligned access were silently fixed up by
> OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't
> true anymore. So, old kernel or sueprvisor software that  included code
> to handle misaligned accesses will crash. Did I missed something ?

Great! You make it very clear! Thanks for your explanation. But even
when misalign exceptions are delegated to HS-mode, KVM seems to do
nothing but redirect to VS-mode when VM get trapped due to misalign
exceptions. So maybe we can directly delegate the misaligned
exceptions in hedeleg too before running VCPU and retrieve them after
VCPU exists. And then the handling procedure will be:

VS-mode misaligned exception -> trap to VS-mode -> VS handles it ->
Back to VU-mode.

Please correct me if I missed anything.

Best Regards,

Xu Lu

>
> Note: this is not directly related to your series but my introduction of
> FWFT !
>
> Thanks,
>
> Clément
>
> >
> > Best Regards,
> > Xu Lu
> >
> >>
> >> Thanks,
> >>
> >> Clément
> >>
> >>>
> >>> Thanks.
> >>>
> >>> _______________________________________________
> >>> linux-riscv mailing list
> >>> linux-riscv@lists.infradead.org
> >>> http://lists.infradead.org/mailman/listinfo/linux-riscv
> >>
>

Re: [External] Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Clément Léger]

On 23/06/2025 15:30, Xu Lu wrote:
> Hi Clément,
> 
> On Mon, Jun 23, 2025 at 8:35 PM Clément Léger <cleger@rivosinc.com> wrote:
>>
>>
>>
>> On 23/06/2025 14:12, Xu Lu wrote:
>>> Hi Clément,
>>>
>>> On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 20/06/2025 14:04, Radim Krčmář wrote:
>>>>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
>>>>>> Delegate illegal instruction fault to VS mode in default to avoid such
>>>>>> exceptions being trapped to HS and redirected back to VS.
>>>>>>
>>>>>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
>>>>>> ---
>>>>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
>>>>>> @@ -48,6 +48,7 @@
>>>>>> +                                     BIT(EXC_INST_ILLEGAL)    | \
>>>>>
>>>>> You should also remove the dead code in kvm_riscv_vcpu_exit.
>>>>>
>>>>> And why not delegate the others as well?
>>>>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
>>>>>  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)
>>>>
>>>> Currently, OpenSBI does not delegate misaligned exception by default and
>>>> handles misaligned access by itself, this is (partially) why we added
>>>> the FWFT SBI extension to request such delegation. Since some supervisor
>>>> software expect that default, they do not have code to handle misaligned
>>>> accesses emulation. So they should not be delegated by default.
>>>
>>> It doesn't matter whether these exceptions are delegated in medeleg.
>>
>> Not sure to totally understand, but if the exceptions are not delegated
>> in medeleg, they won't be delegated to VS-mode even though hedeleg bit
>> is set right ? The spec says:
>>
>> A synchronous trap that has been delegated to HS-mode (using medeleg)
>> is further delegated to VS-mode if V=1 before the trap and the
>> corresponding hedeleg bit is set.
> 
> Yes, you are right. The illegal insn exception is still trapped in M
> mode if it is not delegated in medeleg. But delegating it in hedeleg
> is still useful. The opensbi will check CSR_HEDELEG in the function
> sbi_trap_redirect. If the exception has been delegated to VS-mode in
> CSR_HEDLEG, opensbi can directly return back to VS-mode, without the
> overhead of going back to HS-mode and then going back to VS-mode.
> 
>>
>>
>>
>>> KVM in HS-mode does not handle illegal instruction or misaligned
>>> access and only redirects them back to VS-mode. Delegating such
>>> exceptions in hedeleg helps save CPU usage even when they are not
>>> delegated in medeleg: opensbi will check whether these exceptions are
>>> delegated to VS-mode and redirect them to VS-mode if possible. There
>>> seems to be no conflicts with SSE implementation. Please correct me if
>>> I missed anything.
>>
>> AFAIU, this means that since medeleg bit for misaligned accesses were
>> not delegated up to the introduction of the FWFT extension, VS-mode
>> generated misaligned accesses were handled by OpenSBI right ? Now that
>> we are requesting openSBI to delegate misaligned accesses, HS-mode
>> handles it's own misaligned accesses through the trap handler. With that
>> configuration, if VS-mode generate a misaligned access, it will end up
>> being redirected to VS-mode and won't be handle by HS-mode.
>>
>> To summarize, prior to FWFT, medeleg wasn't delegating misaligned
>> accesses to S-mode:
>>
>> - VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it ->
>> Back to VS-mode, misaligned access fixed up by OpenSBI
> 
> Yes, this is what I want the procedure of handling illegal insn
> exceptions to be. Actually it now behaves as:
> 
> VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it
> -> Back to HS-mode, does nothing -> Back to VS-mode.
> 
> I want to avoid going through HS-mode.

Hi Xu,

Yeah, that make sense as well but that should only happen if the VS-mode
requested misaligned access delegation via KVM SBI FWFT interface. I
know that currently KVM does do anything useful from the misaligned
exception except redirecting it to VS-mode but IMHO, that's a regression
I introduced with FWFT misaligned requested delegation...

> 
>>
>> Now that Linux uses SBI FWFT to delegates misaligned accesses (without
>> hedeleg being set for misaligned delegation, but that doesn't really
>> matter, the outcome is the same):
>>
>> - VS-mode misaligned access -> trap to HS-mode -> redirection to
>> VS-mode, needs to handle the misaligned access by itself
>>
>>
>> This means that previously, misaligned access were silently fixed up by
>> OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't
>> true anymore. So, old kernel or sueprvisor software that  included code
>> to handle misaligned accesses will crash. Did I missed something ?
> 
> Great! You make it very clear! Thanks for your explanation. But even
> when misalign exceptions are delegated to HS-mode, KVM seems to do
> nothing but redirect to VS-mode when VM get trapped due to misalign
> exceptions. 

Exactly, which is why I said that either setting hedeleg by default or
not will lead to the same outcome, ie: VS-mode needs to handle access by
itself (which is a regression introduced by FWFT usage).


> So maybe we can directly delegate the misaligned
> exceptions in hedeleg too before running VCPU and retrieve them after
> VCPU exists. And then the handling procedure will be:
> 
> VS-mode misaligned exception -> trap to VS-mode -> VS handles it ->
> Back to VU-mode.

I'd better want to let the HS-mode handle the misaligned accesses if not
requested via the KVM SBI FWFT interface by VS-mode to keep HS-mode
expected behavior. As you pointed out, this is not currently the case
and the misaligned exceptions are directly redirected to VS-mode, this
differs from what was actually done previously without FWFT (ie OpenSBI
handles the misaligned access).

To summarize, I think HS-mode should fixup VS-mode misaligned accesses
unless requested via KVM SBI FWFT interface, in which case it will
delegates them (which is done by the FWFT series). This would match the
HS-mode <-> OpenSBI behavior.

Thanks,

Clément

> 
> Please correct me if I missed anything.
> 
> Best Regards,
> 
> Xu Lu
> 
>>
>> Note: this is not directly related to your series but my introduction of
>> FWFT !
>>
>> Thanks,
>>
>> Clément
>>
>>>
>>> Best Regards,
>>> Xu Lu
>>>
>>>>
>>>> Thanks,
>>>>
>>>> Clément
>>>>
>>>>>
>>>>> Thanks.
>>>>>
>>>>> _______________________________________________
>>>>> linux-riscv mailing list
>>>>> linux-riscv@lists.infradead.org
>>>>> http://lists.infradead.org/mailman/listinfo/linux-riscv
>>>>
>>

Re: [External] Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Xu Lu]

On Mon, Jun 23, 2025 at 9:42 PM Clément Léger <cleger@rivosinc.com> wrote:
>
>
>
> On 23/06/2025 15:30, Xu Lu wrote:
> > Hi Clément,
> >
> > On Mon, Jun 23, 2025 at 8:35 PM Clément Léger <cleger@rivosinc.com> wrote:
> >>
> >>
> >>
> >> On 23/06/2025 14:12, Xu Lu wrote:
> >>> Hi Clément,
> >>>
> >>> On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 20/06/2025 14:04, Radim Krčmář wrote:
> >>>>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
> >>>>>> Delegate illegal instruction fault to VS mode in default to avoid such
> >>>>>> exceptions being trapped to HS and redirected back to VS.
> >>>>>>
> >>>>>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
> >>>>>> ---
> >>>>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
> >>>>>> @@ -48,6 +48,7 @@
> >>>>>> +                                     BIT(EXC_INST_ILLEGAL)    | \
> >>>>>
> >>>>> You should also remove the dead code in kvm_riscv_vcpu_exit.
> >>>>>
> >>>>> And why not delegate the others as well?
> >>>>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
> >>>>>  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)
> >>>>
> >>>> Currently, OpenSBI does not delegate misaligned exception by default and
> >>>> handles misaligned access by itself, this is (partially) why we added
> >>>> the FWFT SBI extension to request such delegation. Since some supervisor
> >>>> software expect that default, they do not have code to handle misaligned
> >>>> accesses emulation. So they should not be delegated by default.
> >>>
> >>> It doesn't matter whether these exceptions are delegated in medeleg.
> >>
> >> Not sure to totally understand, but if the exceptions are not delegated
> >> in medeleg, they won't be delegated to VS-mode even though hedeleg bit
> >> is set right ? The spec says:
> >>
> >> A synchronous trap that has been delegated to HS-mode (using medeleg)
> >> is further delegated to VS-mode if V=1 before the trap and the
> >> corresponding hedeleg bit is set.
> >
> > Yes, you are right. The illegal insn exception is still trapped in M
> > mode if it is not delegated in medeleg. But delegating it in hedeleg
> > is still useful. The opensbi will check CSR_HEDELEG in the function
> > sbi_trap_redirect. If the exception has been delegated to VS-mode in
> > CSR_HEDLEG, opensbi can directly return back to VS-mode, without the
> > overhead of going back to HS-mode and then going back to VS-mode.
> >
> >>
> >>
> >>
> >>> KVM in HS-mode does not handle illegal instruction or misaligned
> >>> access and only redirects them back to VS-mode. Delegating such
> >>> exceptions in hedeleg helps save CPU usage even when they are not
> >>> delegated in medeleg: opensbi will check whether these exceptions are
> >>> delegated to VS-mode and redirect them to VS-mode if possible. There
> >>> seems to be no conflicts with SSE implementation. Please correct me if
> >>> I missed anything.
> >>
> >> AFAIU, this means that since medeleg bit for misaligned accesses were
> >> not delegated up to the introduction of the FWFT extension, VS-mode
> >> generated misaligned accesses were handled by OpenSBI right ? Now that
> >> we are requesting openSBI to delegate misaligned accesses, HS-mode
> >> handles it's own misaligned accesses through the trap handler. With that
> >> configuration, if VS-mode generate a misaligned access, it will end up
> >> being redirected to VS-mode and won't be handle by HS-mode.
> >>
> >> To summarize, prior to FWFT, medeleg wasn't delegating misaligned
> >> accesses to S-mode:
> >>
> >> - VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it ->
> >> Back to VS-mode, misaligned access fixed up by OpenSBI
> >
> > Yes, this is what I want the procedure of handling illegal insn
> > exceptions to be. Actually it now behaves as:
> >
> > VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it
> > -> Back to HS-mode, does nothing -> Back to VS-mode.
> >
> > I want to avoid going through HS-mode.
>
> Hi Xu,
>
> Yeah, that make sense as well but that should only happen if the VS-mode
> requested misaligned access delegation via KVM SBI FWFT interface. I
> know that currently KVM does do anything useful from the misaligned
> exception except redirecting it to VS-mode but IMHO, that's a regression
> I introduced with FWFT misaligned requested delegation...
>
> >
> >>
> >> Now that Linux uses SBI FWFT to delegates misaligned accesses (without
> >> hedeleg being set for misaligned delegation, but that doesn't really
> >> matter, the outcome is the same):
> >>
> >> - VS-mode misaligned access -> trap to HS-mode -> redirection to
> >> VS-mode, needs to handle the misaligned access by itself
> >>
> >>
> >> This means that previously, misaligned access were silently fixed up by
> >> OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't
> >> true anymore. So, old kernel or sueprvisor software that  included code
> >> to handle misaligned accesses will crash. Did I missed something ?
> >
> > Great! You make it very clear! Thanks for your explanation. But even
> > when misalign exceptions are delegated to HS-mode, KVM seems to do
> > nothing but redirect to VS-mode when VM get trapped due to misalign
> > exceptions.
>
> Exactly, which is why I said that either setting hedeleg by default or
> not will lead to the same outcome, ie: VS-mode needs to handle access by
> itself (which is a regression introduced by FWFT usage).
>
>
> > So maybe we can directly delegate the misaligned
> > exceptions in hedeleg too before running VCPU and retrieve them after
> > VCPU exists. And then the handling procedure will be:
> >
> > VS-mode misaligned exception -> trap to VS-mode -> VS handles it ->
> > Back to VU-mode.
>
> I'd better want to let the HS-mode handle the misaligned accesses if not
> requested via the KVM SBI FWFT interface by VS-mode to keep HS-mode
> expected behavior. As you pointed out, this is not currently the case
> and the misaligned exceptions are directly redirected to VS-mode, this
> differs from what was actually done previously without FWFT (ie OpenSBI
> handles the misaligned access).
>
> To summarize, I think HS-mode should fixup VS-mode misaligned accesses
> unless requested via KVM SBI FWFT interface, in which case it will
> delegates them (which is done by the FWFT series). This would match the
> HS-mode <-> OpenSBI behavior.

Great! Roger that. Hope it can be fixed in the future.

>
> Thanks,
>
> Clément
>
> >
> > Please correct me if I missed anything.
> >
> > Best Regards,
> >
> > Xu Lu
> >
> >>
> >> Note: this is not directly related to your series but my introduction of
> >> FWFT !
> >>
> >> Thanks,
> >>
> >> Clément
> >>
> >>>
> >>> Best Regards,
> >>> Xu Lu
> >>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Clément
> >>>>
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> _______________________________________________
> >>>>> linux-riscv mailing list
> >>>>> linux-riscv@lists.infradead.org
> >>>>> http://lists.infradead.org/mailman/listinfo/linux-riscv
> >>>>
> >>
>

Re: [External] Re: [PATCH] RISC-V: KVM: Delegate illegal instruction fault

[Clément Léger]

On 23/06/2025 16:09, Xu Lu wrote:
> On Mon, Jun 23, 2025 at 9:42 PM Clément Léger <cleger@rivosinc.com> wrote:
>>
>>
>>
>> On 23/06/2025 15:30, Xu Lu wrote:
>>> Hi Clément,
>>>
>>> On Mon, Jun 23, 2025 at 8:35 PM Clément Léger <cleger@rivosinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 23/06/2025 14:12, Xu Lu wrote:
>>>>> Hi Clément,
>>>>>
>>>>> On Mon, Jun 23, 2025 at 4:05 PM Clément Léger <cleger@rivosinc.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 20/06/2025 14:04, Radim Krčmář wrote:
>>>>>>> 2025-06-20T17:17:20+08:00, Xu Lu <luxu.kernel@bytedance.com>:
>>>>>>>> Delegate illegal instruction fault to VS mode in default to avoid such
>>>>>>>> exceptions being trapped to HS and redirected back to VS.
>>>>>>>>
>>>>>>>> Signed-off-by: Xu Lu <luxu.kernel@bytedance.com>
>>>>>>>> ---
>>>>>>>> diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h
>>>>>>>> @@ -48,6 +48,7 @@
>>>>>>>> +                                     BIT(EXC_INST_ILLEGAL)    | \
>>>>>>>
>>>>>>> You should also remove the dead code in kvm_riscv_vcpu_exit.
>>>>>>>
>>>>>>> And why not delegate the others as well?
>>>>>>> (EXC_LOAD_MISALIGNED, EXC_STORE_MISALIGNED, EXC_LOAD_ACCESS,
>>>>>>>  EXC_STORE_ACCESS, and EXC_INST_ACCESS.)
>>>>>>
>>>>>> Currently, OpenSBI does not delegate misaligned exception by default and
>>>>>> handles misaligned access by itself, this is (partially) why we added
>>>>>> the FWFT SBI extension to request such delegation. Since some supervisor
>>>>>> software expect that default, they do not have code to handle misaligned
>>>>>> accesses emulation. So they should not be delegated by default.
>>>>>
>>>>> It doesn't matter whether these exceptions are delegated in medeleg.
>>>>
>>>> Not sure to totally understand, but if the exceptions are not delegated
>>>> in medeleg, they won't be delegated to VS-mode even though hedeleg bit
>>>> is set right ? The spec says:
>>>>
>>>> A synchronous trap that has been delegated to HS-mode (using medeleg)
>>>> is further delegated to VS-mode if V=1 before the trap and the
>>>> corresponding hedeleg bit is set.
>>>
>>> Yes, you are right. The illegal insn exception is still trapped in M
>>> mode if it is not delegated in medeleg. But delegating it in hedeleg
>>> is still useful. The opensbi will check CSR_HEDELEG in the function
>>> sbi_trap_redirect. If the exception has been delegated to VS-mode in
>>> CSR_HEDLEG, opensbi can directly return back to VS-mode, without the
>>> overhead of going back to HS-mode and then going back to VS-mode.
>>>
>>>>
>>>>
>>>>
>>>>> KVM in HS-mode does not handle illegal instruction or misaligned
>>>>> access and only redirects them back to VS-mode. Delegating such
>>>>> exceptions in hedeleg helps save CPU usage even when they are not
>>>>> delegated in medeleg: opensbi will check whether these exceptions are
>>>>> delegated to VS-mode and redirect them to VS-mode if possible. There
>>>>> seems to be no conflicts with SSE implementation. Please correct me if
>>>>> I missed anything.
>>>>
>>>> AFAIU, this means that since medeleg bit for misaligned accesses were
>>>> not delegated up to the introduction of the FWFT extension, VS-mode
>>>> generated misaligned accesses were handled by OpenSBI right ? Now that
>>>> we are requesting openSBI to delegate misaligned accesses, HS-mode
>>>> handles it's own misaligned accesses through the trap handler. With that
>>>> configuration, if VS-mode generate a misaligned access, it will end up
>>>> being redirected to VS-mode and won't be handle by HS-mode.
>>>>
>>>> To summarize, prior to FWFT, medeleg wasn't delegating misaligned
>>>> accesses to S-mode:
>>>>
>>>> - VS-mode misaligned access -> trap to M-mode -> OpenSBI handle it ->
>>>> Back to VS-mode, misaligned access fixed up by OpenSBI
>>>
>>> Yes, this is what I want the procedure of handling illegal insn
>>> exceptions to be. Actually it now behaves as:
>>>
>>> VS-mode illegal insn exception -> trap to M-mode -> OpenSBI handles it
>>> -> Back to HS-mode, does nothing -> Back to VS-mode.
>>>
>>> I want to avoid going through HS-mode.
>>
>> Hi Xu,
>>
>> Yeah, that make sense as well but that should only happen if the VS-mode
>> requested misaligned access delegation via KVM SBI FWFT interface. I
>> know that currently KVM does do anything useful from the misaligned
>> exception except redirecting it to VS-mode but IMHO, that's a regression
>> I introduced with FWFT misaligned requested delegation...
>>
>>>
>>>>
>>>> Now that Linux uses SBI FWFT to delegates misaligned accesses (without
>>>> hedeleg being set for misaligned delegation, but that doesn't really
>>>> matter, the outcome is the same):
>>>>
>>>> - VS-mode misaligned access -> trap to HS-mode -> redirection to
>>>> VS-mode, needs to handle the misaligned access by itself
>>>>
>>>>
>>>> This means that previously, misaligned access were silently fixed up by
>>>> OpenSBI for VS-mode and now that FWFT is used for delegation, this isn't
>>>> true anymore. So, old kernel or sueprvisor software that  included code
>>>> to handle misaligned accesses will crash. Did I missed something ?
>>>
>>> Great! You make it very clear! Thanks for your explanation. But even
>>> when misalign exceptions are delegated to HS-mode, KVM seems to do
>>> nothing but redirect to VS-mode when VM get trapped due to misalign
>>> exceptions.
>>
>> Exactly, which is why I said that either setting hedeleg by default or
>> not will lead to the same outcome, ie: VS-mode needs to handle access by
>> itself (which is a regression introduced by FWFT usage).
>>
>>
>>> So maybe we can directly delegate the misaligned
>>> exceptions in hedeleg too before running VCPU and retrieve them after
>>> VCPU exists. And then the handling procedure will be:
>>>
>>> VS-mode misaligned exception -> trap to VS-mode -> VS handles it ->
>>> Back to VU-mode.
>>
>> I'd better want to let the HS-mode handle the misaligned accesses if not
>> requested via the KVM SBI FWFT interface by VS-mode to keep HS-mode
>> expected behavior. As you pointed out, this is not currently the case
>> and the misaligned exceptions are directly redirected to VS-mode, this
>> differs from what was actually done previously without FWFT (ie OpenSBI
>> handles the misaligned access).
>>
>> To summarize, I think HS-mode should fixup VS-mode misaligned accesses
>> unless requested via KVM SBI FWFT interface, in which case it will
>> delegates them (which is done by the FWFT series). This would match the
>> HS-mode <-> OpenSBI behavior.
> 
> Great! Roger that. Hope it can be fixed in the future.

Ahah ! I'll take a look at it if everyone agrees that it's a regression.

Thanks,

Clément

> 
>>
>> Thanks,
>>
>> Clément
>>
>>>
>>> Please correct me if I missed anything.
>>>
>>> Best Regards,
>>>
>>> Xu Lu
>>>
>>>>
>>>> Note: this is not directly related to your series but my introduction of
>>>> FWFT !
>>>>
>>>> Thanks,
>>>>
>>>> Clément
>>>>
>>>>>
>>>>> Best Regards,
>>>>> Xu Lu
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Clément
>>>>>>
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> linux-riscv mailing list
>>>>>>> linux-riscv@lists.infradead.org
>>>>>>> http://lists.infradead.org/mailman/listinfo/linux-riscv
>>>>>>
>>>>
>>