Random corruption on SpacemiT K1 with RVV and THP

Random corruption on SpacemiT K1 with RVV and THP

[Aurelien Jarno]

Dear all,

For the last weeks, I have been tracking a random memory corruption and
relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
started upgrading to glibc 2.43, which does memset() through vector 
instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
kernel, using a similar configuration to the Debian kernel. The board 
uses OpenSBI 1.9 and the vendor U-Boot.

Typically it manifests itself with the following kind of error, when 
running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
Blender, Dolfin, Qt6):   

Assembler messages:
{standard input}:284588: Error: unknown pseudo-op: `.uleb1'
{standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'

The broken chars are 0xff and it seems there are always 240 (but with 
poor statistics). Sometimes it instead causes a GCC ICE instead.

Using glibc 2.44 instead of glibc 2.43, which does a lot of string 
operations through vector instructions, increases the probability to 
have corruption, makes it a bit more reproducible, but it always seems 
to manifest as a GCC ICE. It typically happens withing one hour when 
running on the 8 cores instead of every 1 or 2 days. From there I have 
been able to determine the following things:

- Disabling vector instructions (by patching the DTB to remove "v", 
  "zvhf" and "zvtk") fixes the issue

- Disabling THP (by setting /sys/kernel/mm/transparent_hugepage/enabled 
  to never instead of always) also seems to fix the issue. Keeping it 
  enabled with defrag=never or use_zero_page=0 doesn't change anything.

- The issue is reproducible with or without swap enabled.

I have not been able to reproduce the issue on other non RVV boards 
(Unmatched, VF2) nor on a SpacemiT K3 board.

I am not really sure how to debug that further. I tried a few ways to 
reproduce the issue with a small C code around the glibc memset code 
(including triggering unaligned accesses and page faults), but failed to 
do so. I therefore welcome any idea about the issue or how to debug it 
further.

Thanks
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 with RVV

[Aurelien Jarno]

Dear all,

I have done some small progress on that issue. Help is still wanted and 
would be appreciated.

On 2026-08-30 22:52, Aurelien Jarno wrote:
> Dear all,
> 
> For the last weeks, I have been tracking a random memory corruption and
> relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
> started upgrading to glibc 2.43, which does memset() through vector 
> instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
> kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
> kernel, using a similar configuration to the Debian kernel. The board 
> uses OpenSBI 1.9 and the vendor U-Boot.

I have been able to rule out OpenSBI from the issue, I have checked 
there is not trap top OpenSBI when the problem happens.

> Typically it manifests itself with the following kind of error, when 
> running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
> Blender, Dolfin, Qt6):   
> 
> Assembler messages:
> {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> 
> The broken chars are 0xff and it seems there are always 240 (but with 
> poor statistics). Sometimes it instead causes a GCC ICE instead.

I have identified that the 0xff comes from the poisoning done for v0 in
__riscv_v_vstate_discard(), which should only happen for a syscall. 
Indeed changing the value to different ones propagates to the above 
error message.

Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
changes the number of broken bytes.

Disabling the vectored user code entirely with the following patch seems
to prevent the issue (or make it sufficiently rare that I have not 
encountered it):

--- a/arch/riscv/lib/riscv_v_helpers.c
+++ b/arch/riscv/lib/riscv_v_helpers.c
@@ -25,7 +25,7 @@ asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n,
        size_t remain, copied;
 
        /* skip has_vector() check because it has been done by the asm  */
-       if (!may_use_simd())
+       if (!may_use_simd() || true)
                goto fallback;
 
        kernel_vector_begin();

Overall, while the corruption happens randomly, the corruption itself 
seems to be deterministic, and the data source of the corruption is 
identified. But I don't understand how those values propagate to user 
space. It is not clear to me if it is a bug somewhere in the context 
switching code, or a bug in __asm_vector_usercopy_sum_enabled fixup 
code, with the values just being the one at the entry of the function.
Also I can't completely rule out a CPU bug.

> Using glibc 2.44 instead of glibc 2.43, which does a lot of string 
> operations through vector instructions, increases the probability to 
> have corruption, makes it a bit more reproducible, but it always seems 
> to manifest as a GCC ICE. It typically happens withing one hour when 
> running on the 8 cores instead of every 1 or 2 days. From there I have 
> been able to determine the following things:

I have stopped using glibc 2.44 as a reproducer, and it generate 
different type of crashes, and there could be another set of bugs on top 
of the one I am trying to identify. This however means it takes a lot of 
time to test any change.
 
> - Disabling vector instructions (by patching the DTB to remove "v", 
>   "zvhf" and "zvtk") fixes the issue
> 
> - Disabling THP (by setting /sys/kernel/mm/transparent_hugepage/enabled 
>   to never instead of always) also seems to fix the issue. Keeping it 
>   enabled with defrag=never or use_zero_page=0 doesn't change anything.

I don't think this is a good direction, my impression is that it just 
changes timing a bit, which just hide the issue, or require different 
condition to trigger it.

Regards
Aurelien 

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Aurelien Jarno]

[Added Andy and Karl in Cc: as they have been involved in the vectored 
user copy code and fighting similar issues]

Hi,

Some more progress on that topic.

On 2026-09-08 06:42, Aurelien Jarno wrote:
> Dear all,
> 
> I have done some small progress on that issue. Help is still wanted and 
> would be appreciated.
> 
> On 2026-08-30 22:52, Aurelien Jarno wrote:
> > Dear all,
> > 
> > For the last weeks, I have been tracking a random memory corruption and
> > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
> > started upgrading to glibc 2.43, which does memset() through vector 
> > instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
> > kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
> > kernel, using a similar configuration to the Debian kernel. The board 
> > uses OpenSBI 1.9 and the vendor U-Boot.
> 
> I have been able to rule out OpenSBI from the issue, I have checked 
> there is not trap top OpenSBI when the problem happens.
> 
> > Typically it manifests itself with the following kind of error, when 
> > running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
> > Blender, Dolfin, Qt6):   
> > 
> > Assembler messages:
> > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> > 
> > The broken chars are 0xff and it seems there are always 240 (but with 
> > poor statistics). Sometimes it instead causes a GCC ICE instead.
> 
> I have identified that the 0xff comes from the poisoning done for v0 in
> __riscv_v_vstate_discard(), which should only happen for a syscall. 
> Indeed changing the value to different ones propagates to the above 
> error message.
> 
> Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
> doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
> changes the number of broken bytes.

The values from __riscv_v_vstate_discard() end-up there because they are 
the last values written to the vector registers. I have added some 
poisoning in __asm_vector_usercopy_sum_enabled before the loop, and 
those values appear instead. Using different values per vector register, 
I have found that the corruption comes from a partial load of the vle8.v 
instruction, while the result of the poisoning and the partial load are
then both written by the vse8.v:

 loop:
        vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
        fixup vle8.v vData, (pSrc), 10f
        sub iNum, iNum, iVL
        add pSrc, pSrc, iVL
        fixup vse8.v vData, (pDst), 11f
        add pDst, pDst, iVL
        bnez iNum, loop

One hypothesis could be that for some reason, an exception (page table 
fault, irq, ...), the vle8.v instruction is interrupted, and only 
partially loads the vector register from memory. This stops at a 16-byte 
boundary (at least on the K1), and the remaining part of the v0-v7 
register is left with the previous value (with the original kernel that 
is the poisoning done in __riscv_v_vstate_discard). In theory when the 
instruction is interrupted, vstart is set to a non-zero value (for 
instance 16), and it should get re-executed after the exception. In 
practice, in some very rare cases, it doesn't happen and the next 
executed instruction is the following sub.

That said with exceptions and vector context switches, the explanation 
is likely way more complex. [1] gives a possible different scenario, 
involving an interrupt or a fault between the vsetvli and vle8.v/vse8.v 
instructions. However it doesn't fix the issue, neither patch [2]
(in that case tested with CONFIG_RISCV_ISA_V_PREEMPTIVE=y).


> Disabling the vectored user code entirely with the following patch seems
> to prevent the issue (or make it sufficiently rare that I have not 
> encountered it):

A much better way to do that is setting RISCV_ISA_V_UCOPY_THRESHOLD=-1.  

I also tried the same kernel on a K3, and it appears to improve 
stability, getting rid of issues that I attributed to thermal issues.  
(due to the absence of fan driver, I run the fan as a fixed speed 
~4000rpm). The symptoms are however quite different, it's GCC crashes 
with "The bug is not reproducible, so it is likely a hardware or OS 
problem.".

Regards
Aurelien

[1] https://lore.kernel.org/20260806193241.10552-1-kmehltretter@gmail.com/
[2] https://lore.kernel.org/all/20260810172255.1532787-2-tchiu@tenstorrent.com/

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Aurelien Jarno]

Hi,

Some more update, even if the progress is quite low.

On 2026-09-09 18:45, Aurelien Jarno wrote:
> [Added Andy and Karl in Cc: as they have been involved in the vectored 
> user copy code and fighting similar issues]
> 
> Hi,
> 
> Some more progress on that topic.
> 
> On 2026-09-08 06:42, Aurelien Jarno wrote:
> > Dear all,
> > 
> > I have done some small progress on that issue. Help is still wanted and 
> > would be appreciated.
> > 
> > On 2026-08-30 22:52, Aurelien Jarno wrote:
> > > Dear all,
> > > 
> > > For the last weeks, I have been tracking a random memory corruption and
> > > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
> > > started upgrading to glibc 2.43, which does memset() through vector 
> > > instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
> > > kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
> > > kernel, using a similar configuration to the Debian kernel. The board 
> > > uses OpenSBI 1.9 and the vendor U-Boot.

I have been able to also reproduce the issue with the vendor OpenSBI. It 
is also reproducible with linux 7.2.6.

> > I have been able to rule out OpenSBI from the issue, I have checked 
> > there is not trap to OpenSBI when the problem happens.
> > 
> > > Typically it manifests itself with the following kind of error, when 
> > > running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
> > > Blender, Dolfin, Qt6):   
> > > 
> > > Assembler messages:
> > > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> > > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> > > 
> > > The broken chars are 0xff and it seems there are always 240 (but with 
> > > poor statistics). Sometimes it instead causes a GCC ICE instead.
> > 
> > I have identified that the 0xff comes from the poisoning done for v0 in
> > __riscv_v_vstate_discard(), which should only happen for a syscall. 
> > Indeed changing the value to different ones propagates to the above 
> > error message.
> > 
> > Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
> > doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
> > changes the number of broken bytes.
> 
> The values from __riscv_v_vstate_discard() end-up there because they are 
> the last values written to the vector registers. I have added some 
> poisoning in __asm_vector_usercopy_sum_enabled before the loop, and 
> those values appear instead. Using different values per vector register, 
> I have found that the corruption comes from a partial load of the vle8.v 
> instruction, while the result of the poisoning and the partial load are
> then both written by the vse8.v:
> 
>  loop:
>         vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
>         fixup vle8.v vData, (pSrc), 10f
>         sub iNum, iNum, iVL
>         add pSrc, pSrc, iVL
>         fixup vse8.v vData, (pDst), 11f
>         add pDst, pDst, iVL
>         bnez iNum, loop

I have instrumented that code, basically adding a function to check the 
copy afterwards (only using scalar instructions), and report values in 
case of errors. So far I have been able to catch the issue 4 times that 
way (including one where I haven't seen any visible consequence on the 
userland). 

This is what I observed:
- The issue always happens for the first bytes of the user copy, which 
  in all cases were supposed to be up to 4096 bytes long.
- On the first iteration of the loop only the first 16 bytes are loaded 
  by the vle8.v instruction but the full 256 bytes are written to the 
  destination by the vse8.v instrution.
- On the second iteration of the loop, the vle8.v causes a trap, which 
  ends up in the 10f fixup. At fixup level, vl=0x100, vlenb=0x20, 
  vstart=0x0 and vtype=0xc3.
- Considering the 2 first iterations of the loop (i.e. 512 bytes), 
  neither the source nor destination addresses cross a page.

All that said, it is what I can *observe*. I have no idea if the first 
iteration of the loop also goes through fixup or not. Also in addition 
to precise traps (which definitely happen for the second iteration), IRQ 
could happen at any moment. I use CONFIG_RISCV_ISA_V_PREEMPTIVE=n for my 
test, in order to limit the code paths to review, so during that copy 
preempt is disabled, but IRQ is not.

Regards
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Palmer Dabbelt]

On Tue, 15 Sep 2026 14:14:10 PDT (-0700), aurelien@aurel32.net wrote:
> Hi,
>
> Some more update, even if the progress is quite low.

Well, thanks for digging into this.

>
> On 2026-09-09 18:45, Aurelien Jarno wrote:
>> [Added Andy and Karl in Cc: as they have been involved in the vectored 
>> user copy code and fighting similar issues]
>> 
>> Hi,
>> 
>> Some more progress on that topic.
>> 
>> On 2026-09-08 06:42, Aurelien Jarno wrote:
>> > Dear all,
>> > 
>> > I have done some small progress on that issue. Help is still wanted and 
>> > would be appreciated.
>> > 
>> > On 2026-08-30 22:52, Aurelien Jarno wrote:
>> > > Dear all,
>> > > 
>> > > For the last weeks, I have been tracking a random memory corruption and
>> > > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
>> > > started upgrading to glibc 2.43, which does memset() through vector 
>> > > instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
>> > > kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
>> > > kernel, using a similar configuration to the Debian kernel. The board 
>> > > uses OpenSBI 1.9 and the vendor U-Boot.
>
> I have been able to also reproduce the issue with the vendor OpenSBI. It 
> is also reproducible with linux 7.2.6.
>
>> > I have been able to rule out OpenSBI from the issue, I have checked 
>> > there is not trap to OpenSBI when the problem happens.
>> > 
>> > > Typically it manifests itself with the following kind of error, when 
>> > > running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
>> > > Blender, Dolfin, Qt6):   
>> > > 
>> > > Assembler messages:
>> > > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
>> > > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
>> > > 
>> > > The broken chars are 0xff and it seems there are always 240 (but with 
>> > > poor statistics). Sometimes it instead causes a GCC ICE instead.
>> > 
>> > I have identified that the 0xff comes from the poisoning done for v0 in
>> > __riscv_v_vstate_discard(), which should only happen for a syscall. 
>> > Indeed changing the value to different ones propagates to the above 
>> > error message.
>> > 
>> > Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
>> > doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
>> > changes the number of broken bytes.
>> 
>> The values from __riscv_v_vstate_discard() end-up there because they are 
>> the last values written to the vector registers. I have added some 
>> poisoning in __asm_vector_usercopy_sum_enabled before the loop, and 
>> those values appear instead. Using different values per vector register, 
>> I have found that the corruption comes from a partial load of the vle8.v 
>> instruction, while the result of the poisoning and the partial load are
>> then both written by the vse8.v:
>> 
>>  loop:
>>         vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
>>         fixup vle8.v vData, (pSrc), 10f
>>         sub iNum, iNum, iVL
>>         add pSrc, pSrc, iVL
>>         fixup vse8.v vData, (pDst), 11f
>>         add pDst, pDst, iVL
>>         bnez iNum, loop
>
> I have instrumented that code, basically adding a function to check the 
> copy afterwards (only using scalar instructions), and report values in 
> case of errors. So far I have been able to catch the issue 4 times that 
> way (including one where I haven't seen any visible consequence on the 
> userland). 
>
> This is what I observed:
> - The issue always happens for the first bytes of the user copy, which 
>   in all cases were supposed to be up to 4096 bytes long.
> - On the first iteration of the loop only the first 16 bytes are loaded 
>   by the vle8.v instruction but the full 256 bytes are written to the 
>   destination by the vse8.v instrution.

That sounds like some sort of trap handling issue: something's filling 
out the first word of the vector, but then not the rest, but then not 
triggering a restart of the instruction.  We talked some on IRC and it 
sounds like that trap isn't getting into Linux, so it's possible 
something in M-mode is mangling it -- this is where we start to get into 
vstart/idempotent instruction territory and that's not well tested by 
running on QEMU, so it wouldn't be super surprising we're just stumbling 
into a bug there.

I have a K3 that I've been meaning to play around with, I'm going to go 
try to see if I can get it running and reproduce this one.

> - On the second iteration of the loop, the vle8.v causes a trap, which 
>   ends up in the 10f fixup. At fixup level, vl=0x100, vlenb=0x20, 
>   vstart=0x0 and vtype=0xc3.
> - Considering the 2 first iterations of the loop (i.e. 512 bytes), 
>   neither the source nor destination addresses cross a page.
>
> All that said, it is what I can *observe*. I have no idea if the first 
> iteration of the loop also goes through fixup or not. Also in addition 
> to precise traps (which definitely happen for the second iteration), IRQ 
> could happen at any moment. I use CONFIG_RISCV_ISA_V_PREEMPTIVE=n for my 
> test, in order to limit the code paths to review, so during that copy 
> preempt is disabled, but IRQ is not.
>
> Regards
> Aurelien
>
> -- 
> Aurelien Jarno                          GPG: 4096R/1DDD8C9B
> aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Aurelien Jarno]

Hi Palmer,

Thanks for your answer.

On 2026-09-17 12:18, Palmer Dabbelt wrote:
> On Tue, 15 Sep 2026 14:14:10 PDT (-0700), aurelien@aurel32.net wrote:
> > Hi,
> > 
> > Some more update, even if the progress is quite low.
> 
> Well, thanks for digging into this.
> 
> > 
> > On 2026-09-09 18:45, Aurelien Jarno wrote:
> > > [Added Andy and Karl in Cc: as they have been involved in the
> > > vectored user copy code and fighting similar issues]
> > > 
> > > Hi,
> > > 
> > > Some more progress on that topic.
> > > 
> > > On 2026-09-08 06:42, Aurelien Jarno wrote:
> > > > Dear all,
> > > > > I have done some small progress on that issue. Help is still
> > > wanted and > would be appreciated.
> > > > > On 2026-08-30 22:52, Aurelien Jarno wrote:
> > > > > Dear all,
> > > > > > > For the last weeks, I have been tracking a random memory
> > > corruption and
> > > > > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V
> > > Jupiter). It > > started upgrading to glibc 2.43, which does
> > > memset() through vector > > instructions.  It is reproducible using
> > > the Debian 7.1.7-1~bpo13+1 > > kernel, but I have also been able to
> > > reproduce it with a vanilla 7.2.2 > > kernel, using a similar
> > > configuration to the Debian kernel. The board > > uses OpenSBI 1.9
> > > and the vendor U-Boot.
> > 
> > I have been able to also reproduce the issue with the vendor OpenSBI. It
> > is also reproducible with linux 7.2.6.
> > 
> > > > I have been able to rule out OpenSBI from the issue, I have
> > > checked > there is not trap to OpenSBI when the problem happens.
> > > > > > Typically it manifests itself with the following kind of
> > > error, when > > running g++ from GCC 16 as part of building software
> > > (e.g. OpenJDK, > > Blender, Dolfin, Qt6):   > > > > Assembler
> > > messages:
> > > > > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> > > > > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> > > > > > > The broken chars are 0xff and it seems there are always 240
> > > (but with > > poor statistics). Sometimes it instead causes a GCC
> > > ICE instead.
> > > > > I have identified that the 0xff comes from the poisoning done
> > > for v0 in
> > > > __riscv_v_vstate_discard(), which should only happen for a
> > > syscall. > Indeed changing the value to different ones propagates to
> > > the above > error message.
> > > > > Furthermore I have found that changing
> > > CONFIG_RISCV_ISA_V_PREEMPTIVE > doesn't change anything, and that
> > > changing RISCV_ISA_V_UCOPY_THRESHOLD > changes the number of broken
> > > bytes.
> > > 
> > > The values from __riscv_v_vstate_discard() end-up there because they
> > > are the last values written to the vector registers. I have added
> > > some poisoning in __asm_vector_usercopy_sum_enabled before the loop,
> > > and those values appear instead. Using different values per vector
> > > register, I have found that the corruption comes from a partial load
> > > of the vle8.v instruction, while the result of the poisoning and the
> > > partial load are
> > > then both written by the vse8.v:
> > > 
> > >  loop:
> > >         vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
> > >         fixup vle8.v vData, (pSrc), 10f
> > >         sub iNum, iNum, iVL
> > >         add pSrc, pSrc, iVL
> > >         fixup vse8.v vData, (pDst), 11f
> > >         add pDst, pDst, iVL
> > >         bnez iNum, loop
> > 
> > I have instrumented that code, basically adding a function to check the
> > copy afterwards (only using scalar instructions), and report values in
> > case of errors. So far I have been able to catch the issue 4 times that
> > way (including one where I haven't seen any visible consequence on the
> > userland).
> > 
> > This is what I observed:
> > - The issue always happens for the first bytes of the user copy, which
> > in all cases were supposed to be up to 4096 bytes long.
> > - On the first iteration of the loop only the first 16 bytes are loaded
> > by the vle8.v instruction but the full 256 bytes are written to the
> > destination by the vse8.v instrution.
> 
> That sounds like some sort of trap handling issue: something's filling out
> the first word of the vector, but then not the rest, but then not triggering
> a restart of the instruction.  We talked some on IRC and it sounds like that

Yes, it's really looks like the instruction is interrupted but not 
restarted, instead the following instruction is executed.

> trap isn't getting into Linux, so it's possible something in M-mode is
> mangling it -- this is where we start to get into vstart/idempotent

If you mean sbi_misaligned_v_ld_emulator() and 
sbi_misaligned_v_st_emulator(), I have already ruled the mout. They are 
not called because both userland (glibc 2.43) and kernel (vector user 
copy) are using e8, so there are no possible alignment issues.


> instruction territory and that's not well tested by running on QEMU, so it
> wouldn't be super surprising we're just stumbling into a bug there.

I don't think it's a vstart issue. If vstart were wrongly restored to 0, 
the whole vector would be loaded again, inefficient but correct. If 
vstart were restored to another wrong value, the end of vector would 
also be loaded, overriding the values from riscv_v_vstate_discard(). The 
only vstart case that could explain that is vstart >= vl, but I don't 
think this is allowed.

> I have a K3 that I've been meaning to play around with, I'm going to go try
> to see if I can get it running and reproduce this one.

Just to be clear, I only observed corruption in the vectored user copy 
code on the K1. But the K3 is affected by similar random corruption than 
the K1 that disappear when disabling the vector unit. I believe that the 
corruption happens at multiple places, and that by chance on the K1 it 
can be observed on the vectored user copy.

To reproduce the issue, I use on a Debian installation (or on the 
original Bianbu installation) the following commands in a loop:
- sbuild -d sid blender
- sbuild -d sid --build-dep-resolver=aptitude --extra-repository="deb http://deb.debian.org/debian experimental main" --add-depends "libc6 (>= 2.44)" blender

The second command uses glibc 2.44, which uses more vector instructions 
via ifunc, so it crashes faster. Still it can takes a few hours until it 
happens. On the K1 it takes on average 5 hours, sometimes it takes up to 
18 hours before I can reproduce the issue.

> > - On the second iteration of the loop, the vle8.v causes a trap, which
> > ends up in the 10f fixup. At fixup level, vl=0x100, vlenb=0x20,
> > vstart=0x0 and vtype=0xc3.
> > - Considering the 2 first iterations of the loop (i.e. 512 bytes),
> > neither the source nor destination addresses cross a page.
> > 
> > All that said, it is what I can *observe*. I have no idea if the first
> > iteration of the loop also goes through fixup or not. Also in addition
> > to precise traps (which definitely happen for the second iteration), IRQ
> > could happen at any moment. I use CONFIG_RISCV_ISA_V_PREEMPTIVE=n for my
> > test, in order to limit the code paths to review, so during that copy
> > preempt is disabled, but IRQ is not.

I tried disabling the IRQ around the vectored user copy, but it didn't 
change anything.

Regards
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Andy Chiu]

Hi Aurelien,

Sorry for replying late. I did run a compilation test with vectorized
glibc in qemu after seeing this thread shortly, but then focus on
something else as it didn't catch any corruption we've seen here.

On Wed, Sep 09, 2026 at 06:45:14PM +0200, Aurelien Jarno wrote:
> [Added Andy and Karl in Cc: as they have been involved in the vectored 
> user copy code and fighting similar issues]
> 
> Hi,
> 
> Some more progress on that topic.
> 
> On 2026-09-08 06:42, Aurelien Jarno wrote:
> > Dear all,
> > 
> > I have done some small progress on that issue. Help is still wanted and 
> > would be appreciated.
> > 
> > On 2026-08-30 22:52, Aurelien Jarno wrote:
> > > Dear all,
> > > 
> > > For the last weeks, I have been tracking a random memory corruption and
> > > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
> > > started upgrading to glibc 2.43, which does memset() through vector 
> > > instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
> > > kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
> > > kernel, using a similar configuration to the Debian kernel. The board 
> > > uses OpenSBI 1.9 and the vendor U-Boot.

The KVM fix, which is in 7.3 now, is irrelavant to this bug (see below),
because the fix targets preemptible kernel-mode vector, but we can
trigger this bug with CONFIG_RISCV_ISA_V_PREEMPTIVE unset. Nontheless,
if we want to test on the latest code, feel free to grab the series at
[3] and boot with riscv_novstateopt to preseve the context poisoning
behavior.

> > 
> > I have been able to rule out OpenSBI from the issue, I have checked 
> > there is not trap top OpenSBI when the problem happens.
> > 
> > > Typically it manifests itself with the following kind of error, when 
> > > running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
> > > Blender, Dolfin, Qt6):   
> > > 
> > > Assembler messages:
> > > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> > > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> > > 
> > > The broken chars are 0xff and it seems there are always 240 (but with 
> > > poor statistics). Sometimes it instead causes a GCC ICE instead.
> > 
> > I have identified that the 0xff comes from the poisoning done for v0 in
> > __riscv_v_vstate_discard(), which should only happen for a syscall. 
> > Indeed changing the value to different ones propagates to the above 
> > error message.
> > 
> > Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
> > doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
> > changes the number of broken bytes.

Turnning off CONFIG_RISCV_ISA_V_PREEMPTIVE makes the ucopy fall back to
the scalar copy on a page fault, because faulthandler_disabled() would
return true after kernel_vector_begin(). So if we are suspecting a
corruption at vector restart, then it is not, there is no restarting of
such instruction.

The only possible way to have a restarted vector ld/st here under
!CONFIG_RISCV_ISA_V_PREEMPTIVE is then an irq restart. I don't know if
you test this code with !CONFIG_RISCV_ISA_V_PREEMPTIVE **and** with
irqs_disabled() == true in the user copy code. If we still observed
a corruption under this case, then it suggests that the corruption
happens even eariler. It is then either the faulting instruction itself,
or the corruption happens even eariler.

> 
> The values from __riscv_v_vstate_discard() end-up there because they are 
> the last values written to the vector registers. I have added some 
> poisoning in __asm_vector_usercopy_sum_enabled before the loop, and 
> those values appear instead. Using different values per vector register, 

Do you know what is the address of corrupted data? If it always starts at
a page boundary (e.g offset 0x00000 for THP) then it suggests a strong
correlation with fault handling.

> I have found that the corruption comes from a partial load of the vle8.v 
> instruction, while the result of the poisoning and the partial load are
> then both written by the vse8.v:
> 
>  loop:
>         vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
>         fixup vle8.v vData, (pSrc), 10f
>         sub iNum, iNum, iVL
>         add pSrc, pSrc, iVL
>         fixup vse8.v vData, (pDst), 11f
>         add pDst, pDst, iVL
>         bnez iNum, loop
> 
> One hypothesis could be that for some reason, an exception (page table 
> fault, irq, ...), the vle8.v instruction is interrupted, and only 
> partially loads the vector register from memory. This stops at a 16-byte 
> boundary (at least on the K1), and the remaining part of the v0-v7 
> register is left with the previous value (with the original kernel that 
> is the poisoning done in __riscv_v_vstate_discard). In theory when the 
> instruction is interrupted, vstart is set to a non-zero value (for 
> instance 16), and it should get re-executed after the exception. In 
> practice, in some very rare cases, it doesn't happen and the next 
> executed instruction is the following sub.
> 
> That said with exceptions and vector context switches, the explanation 
> is likely way more complex. [1] gives a possible different scenario, 
> involving an interrupt or a fault between the vsetvli and vle8.v/vse8.v 
> instructions. However it doesn't fix the issue, neither patch [2]
> (in that case tested with CONFIG_RISCV_ISA_V_PREEMPTIVE=y).
> 
> 
> > Disabling the vectored user code entirely with the following patch seems
> > to prevent the issue (or make it sufficiently rare that I have not 
> > encountered it):
> 
> A much better way to do that is setting RISCV_ISA_V_UCOPY_THRESHOLD=-1.  
> 
> I also tried the same kernel on a K3, and it appears to improve 
> stability, getting rid of issues that I attributed to thermal issues.  
> (due to the absence of fan driver, I run the fan as a fixed speed 
> ~4000rpm). The symptoms are however quite different, it's GCC crashes 
> with "The bug is not reproducible, so it is likely a hardware or OS 
> problem.".
> 
> Regards
> Aurelien
> 
> [1] https://lore.kernel.org/20260806193241.10552-1-kmehltretter@gmail.com/
> [2] https://lore.kernel.org/all/20260810172255.1532787-2-tchiu@tenstorrent.com/
> 

[3] https://lore.kernel.org/all/20260918215154.2481482-1-tchiu@tenstorrent.com/

> -- 
> Aurelien Jarno                          GPG: 4096R/1DDD8C9B
> aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Aurelien Jarno]

Hi Andy,

On 2026-09-22 16:33, Andy Chiu wrote:
> Hi Aurelien,
> 
> Sorry for replying late. I did run a compilation test with vectorized
> glibc in qemu after seeing this thread shortly, but then focus on
> something else as it didn't catch any corruption we've seen here.

Thanks for your feedback. Note that at this stage I have not been able 
to reproduce the issue with QEMU. I guess it's very timing dependent, 
also I am not sure if QEMU simulates partially executed instructions 
(outside of page faults).

> On Wed, Sep 09, 2026 at 06:45:14PM +0200, Aurelien Jarno wrote:
> > [Added Andy and Karl in Cc: as they have been involved in the vectored 
> > user copy code and fighting similar issues]
> > 
> > Hi,
> > 
> > Some more progress on that topic.
> > 
> > On 2026-09-08 06:42, Aurelien Jarno wrote:
> > > Dear all,
> > > 
> > > I have done some small progress on that issue. Help is still wanted and 
> > > would be appreciated.
> > > 
> > > On 2026-08-30 22:52, Aurelien Jarno wrote:
> > > > Dear all,
> > > > 
> > > > For the last weeks, I have been tracking a random memory corruption and
> > > > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
> > > > started upgrading to glibc 2.43, which does memset() through vector 
> > > > instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
> > > > kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
> > > > kernel, using a similar configuration to the Debian kernel. The board 
> > > > uses OpenSBI 1.9 and the vendor U-Boot.
> 
> The KVM fix, which is in 7.3 now, is irrelavant to this bug (see below),
> because the fix targets preemptible kernel-mode vector, but we can
> trigger this bug with CONFIG_RISCV_ISA_V_PREEMPTIVE unset. Nontheless,
> if we want to test on the latest code, feel free to grab the series at
> [3] and boot with riscv_novstateopt to preseve the context poisoning
> behavior.

Thanks, I'll try that.

> > > 
> > > I have been able to rule out OpenSBI from the issue, I have checked 
> > > there is not trap top OpenSBI when the problem happens.
> > > 
> > > > Typically it manifests itself with the following kind of error, when 
> > > > running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
> > > > Blender, Dolfin, Qt6):   
> > > > 
> > > > Assembler messages:
> > > > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> > > > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> > > > 
> > > > The broken chars are 0xff and it seems there are always 240 (but with 
> > > > poor statistics). Sometimes it instead causes a GCC ICE instead.
> > > 
> > > I have identified that the 0xff comes from the poisoning done for v0 in
> > > __riscv_v_vstate_discard(), which should only happen for a syscall. 
> > > Indeed changing the value to different ones propagates to the above 
> > > error message.
> > > 
> > > Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
> > > doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
> > > changes the number of broken bytes.
> 
> Turnning off CONFIG_RISCV_ISA_V_PREEMPTIVE makes the ucopy fall back to
> the scalar copy on a page fault, because faulthandler_disabled() would
> return true after kernel_vector_begin(). So if we are suspecting a
> corruption at vector restart, then it is not, there is no restarting of
> such instruction.
> 
> The only possible way to have a restarted vector ld/st here under
> !CONFIG_RISCV_ISA_V_PREEMPTIVE is then an irq restart. I don't know if
> you test this code with !CONFIG_RISCV_ISA_V_PREEMPTIVE **and** with
> irqs_disabled() == true in the user copy code. If we still observed
> a corruption under this case, then it suggests that the corruption
> happens even eariler. It is then either the faulting instruction itself,
> or the corruption happens even eariler.

I have already tried with both CONFIG_RISCV_ISA_V_PREEMPTIVE=n and 
wrapping the vectored user copy call in between local_irq_save(flags) 
and local_irq_restore(flags). The problem is still reproducible.

I have also tried to instrument the vectored user copy code by saving 
the insret crs before and after the vle8.v instruction to detect a 
possible interrupted instruction (as suggested on IRC), but adding this 
code seems to have "fixed" the issue.

> > The values from __riscv_v_vstate_discard() end-up there because they are 
> > the last values written to the vector registers. I have added some 
> > poisoning in __asm_vector_usercopy_sum_enabled before the loop, and 
> > those values appear instead. Using different values per vector register, 
> 
> Do you know what is the address of corrupted data? If it always starts at
> a page boundary (e.g offset 0x00000 for THP) then it suggests a strong
> correlation with fault handling.

On the K1, for the few cases I captured, none of them involved 4k page 
crossing on either source or destination. The only invariant I have 
found is that for some reason the first iteration of the loop only 
loads 16 bytes, and write 256 bytes. The second iteration of the loop 
then goes to the load fixup and exits.

On the K3, I have never observed a corruption at the vector user copy, 
but I still observed rare memory corruptions causing gcc to crash, and 
which disappeared when disabling vector instructions at the device tree
level, like on the K1. But that could be two different issues linked to 
vector instructions.

Regards
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Karl Mehltretter]

On Thu, Sep 24, 2026 at 06:52:21AM +0100, Aurelien Jarno wrote:
> Thanks for your feedback. Note that at this stage I have not been able 
> to reproduce the issue with QEMU. I guess it's very timing dependent, 
> also I am not sure if QEMU simulates partially executed instructions 
> (outside of page faults).
> 

Hello Aurelien, Andy,

I tested this with a local TCG diagnostic change. It did not reproduce
the K1 failure, but it answers the partial-instruction question.

My LLM agent helped me running these tests.

In stock TCG at 7074591d7954, vle8.v can leave partial state on a memory
fault, but the vector helper runs atomically with respect to guest
interrupts [1,2]. Stock QEMU therefore cannot take an asynchronous
interrupt partway through this load.

In a bare-metal test at VLEN=256 and LMUL=8, a page fault after element
16 left vstart=16 and a snapshot containing 16 source bytes followed by
240 poison bytes. After the missing page was mapped, QEMU resumed the
load and completed the copy correctly.

I then added a hook to QEMU vector-load which performs 16 elements, sets
vstart=16, raises a timer interrupt, and resumes at the same vle8.v. The
bare-metal test completed 262,145 such restarts and 67,108,864 copied
bytes without a mismatch.

I also booted Linux 388b607d107c with:

  CONFIG_RISCV_ISA_V=y
  CONFIG_RISCV_ISA_V_UCOPY_THRESHOLD=1
  CONFIG_RISCV_ISA_V_PREEMPTIVE=n

With the hook restricted to S-mode loads with SUM and SIE set, a checked
pipe test completed 90,308,608 bytes across copy_from_user() and
copy_to_user(), including demand-faulting source and destination pages.
The hook logged at least 327,680 forced interruptions at vstart=16,
without a byte mismatch.

As a negative control, I made one load read 16 elements and then retire
as if all 256 had completed. That produced the 16-source/240-poison
signature in bare metal, and the Linux checker reported exactly 240
differing bytes. This is an injected symptom, but confirms that the test
detects the reported failure shape.

Correct QEMU fault recovery and forced interrupt restart therefore did
not produce the corruption. Reaching the 16/240 result required
deliberately modelling a load that completed early without a trap. That
fits the observed first load/store pair, but does not establish its
cause. Your IRQ-disabled result also makes a normal asynchronous restart
a poor fit.

The normal Linux load-fault path exits before vse8.v and falls back to
the scalar copy [3,4]. Do you have the original trap PC, cause and fault
address for the second-iteration fault? Those values could show whether
an unexpected synchronous trap is involved.

Thanks,
Karl

[1] https://gitlab.com/qemu-project/qemu/-/blob/7074591d7954876951f84c15b994a43251d5a3c1/target/riscv/tcg/vector_helper.c#L403
[2] https://gitlab.com/qemu-project/qemu/-/blob/7074591d7954876951f84c15b994a43251d5a3c1/accel/tcg/cpu-exec.c#L930
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/riscv/lib/uaccess_vector.S?h=v7.2#n38
[4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/riscv/lib/riscv_v_helpers.c?h=v7.2#n23

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Aurelien Jarno]

Hi Karl,

On 2026-09-24 08:15, Karl Mehltretter wrote:
> On Thu, Sep 24, 2026 at 06:52:21AM +0100, Aurelien Jarno wrote:
> > Thanks for your feedback. Note that at this stage I have not been able 
> > to reproduce the issue with QEMU. I guess it's very timing dependent, 
> > also I am not sure if QEMU simulates partially executed instructions 
> > (outside of page faults).
> > 
> 
> Hello Aurelien, Andy,
> 
> I tested this with a local TCG diagnostic change. It did not reproduce
> the K1 failure, but it answers the partial-instruction question.
> 
> My LLM agent helped me running these tests.
> 
> In stock TCG at 7074591d7954, vle8.v can leave partial state on a memory
> fault, but the vector helper runs atomically with respect to guest
> interrupts [1,2]. Stock QEMU therefore cannot take an asynchronous
> interrupt partway through this load.
> 
> In a bare-metal test at VLEN=256 and LMUL=8, a page fault after element
> 16 left vstart=16 and a snapshot containing 16 source bytes followed by
> 240 poison bytes. After the missing page was mapped, QEMU resumed the
> load and completed the copy correctly.
> 
> I then added a hook to QEMU vector-load which performs 16 elements, sets
> vstart=16, raises a timer interrupt, and resumes at the same vle8.v. The
> bare-metal test completed 262,145 such restarts and 67,108,864 copied
> bytes without a mismatch.
> 
> I also booted Linux 388b607d107c with:
> 
>   CONFIG_RISCV_ISA_V=y
>   CONFIG_RISCV_ISA_V_UCOPY_THRESHOLD=1
>   CONFIG_RISCV_ISA_V_PREEMPTIVE=n
> 
> With the hook restricted to S-mode loads with SUM and SIE set, a checked
> pipe test completed 90,308,608 bytes across copy_from_user() and
> copy_to_user(), including demand-faulting source and destination pages.
> The hook logged at least 327,680 forced interruptions at vstart=16,
> without a byte mismatch.
> 
> As a negative control, I made one load read 16 elements and then retire
> as if all 256 had completed. That produced the 16-source/240-poison
> signature in bare metal, and the Linux checker reported exactly 240
> differing bytes. This is an injected symptom, but confirms that the test
> detects the reported failure shape.
> 
> Correct QEMU fault recovery and forced interrupt restart therefore did
> not produce the corruption. Reaching the 16/240 result required
> deliberately modelling a load that completed early without a trap. That
> fits the observed first load/store pair, but does not establish its
> cause. Your IRQ-disabled result also makes a normal asynchronous restart
> a poor fit.

Thanks for all those extensive tests.

> The normal Linux load-fault path exits before vse8.v and falls back to
> the scalar copy [3,4]. Do you have the original trap PC, cause and fault
> address for the second-iteration fault? Those values could show whether
> an unexpected synchronous trap is involved.

Unfortunately the issue is quite rare, so I have not found a way to 
trace all that information when the issue happens.

That said Han Gao pointed me to this patch:
https://lore.kernel.org/linux-riscv/20260807-vector_fpu_regs_status_rmw_fix-v1-1-0c16848b60db@intel.com/

I have been testing it, and so far it seems to fix my issue with both 
CONFIG_RISCV_ISA_V_PREEMPTIVE enable and disabled. Or maybe it just 
hides the issue by changing the timing, as at this point, I haven't 
fully understood how the bug fixed by this patch can completely explain 
my observations.

I still observe a few GCC crashes, on both the K3, but more rarely, but 
without this corruption pattern, just as non reproducible ICE in GCC. It 
is likely a different issue.

Regards
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Aurelien Jarno]

Hi,

On 2026-09-25 06:40, Aurelien Jarno wrote:
> Hi Karl,
> 
> On 2026-09-24 08:15, Karl Mehltretter wrote:
> > On Thu, Sep 24, 2026 at 06:52:21AM +0100, Aurelien Jarno wrote:
> > > Thanks for your feedback. Note that at this stage I have not been able 
> > > to reproduce the issue with QEMU. I guess it's very timing dependent, 
> > > also I am not sure if QEMU simulates partially executed instructions 
> > > (outside of page faults).
> > > 
> > 
> > Hello Aurelien, Andy,
> > 
> > I tested this with a local TCG diagnostic change. It did not reproduce
> > the K1 failure, but it answers the partial-instruction question.
> > 
> > My LLM agent helped me running these tests.
> > 
> > In stock TCG at 7074591d7954, vle8.v can leave partial state on a memory
> > fault, but the vector helper runs atomically with respect to guest
> > interrupts [1,2]. Stock QEMU therefore cannot take an asynchronous
> > interrupt partway through this load.
> > 
> > In a bare-metal test at VLEN=256 and LMUL=8, a page fault after element
> > 16 left vstart=16 and a snapshot containing 16 source bytes followed by
> > 240 poison bytes. After the missing page was mapped, QEMU resumed the
> > load and completed the copy correctly.
> > 
> > I then added a hook to QEMU vector-load which performs 16 elements, sets
> > vstart=16, raises a timer interrupt, and resumes at the same vle8.v. The
> > bare-metal test completed 262,145 such restarts and 67,108,864 copied
> > bytes without a mismatch.
> > 
> > I also booted Linux 388b607d107c with:
> > 
> >   CONFIG_RISCV_ISA_V=y
> >   CONFIG_RISCV_ISA_V_UCOPY_THRESHOLD=1
> >   CONFIG_RISCV_ISA_V_PREEMPTIVE=n
> > 
> > With the hook restricted to S-mode loads with SUM and SIE set, a checked
> > pipe test completed 90,308,608 bytes across copy_from_user() and
> > copy_to_user(), including demand-faulting source and destination pages.
> > The hook logged at least 327,680 forced interruptions at vstart=16,
> > without a byte mismatch.
> > 
> > As a negative control, I made one load read 16 elements and then retire
> > as if all 256 had completed. That produced the 16-source/240-poison
> > signature in bare metal, and the Linux checker reported exactly 240
> > differing bytes. This is an injected symptom, but confirms that the test
> > detects the reported failure shape.
> > 
> > Correct QEMU fault recovery and forced interrupt restart therefore did
> > not produce the corruption. Reaching the 16/240 result required
> > deliberately modelling a load that completed early without a trap. That
> > fits the observed first load/store pair, but does not establish its
> > cause. Your IRQ-disabled result also makes a normal asynchronous restart
> > a poor fit.
> 
> Thanks for all those extensive tests.
> 
> > The normal Linux load-fault path exits before vse8.v and falls back to
> > the scalar copy [3,4]. Do you have the original trap PC, cause and fault
> > address for the second-iteration fault? Those values could show whether
> > an unexpected synchronous trap is involved.
> 
> Unfortunately the issue is quite rare, so I have not found a way to 
> trace all that information when the issue happens.
> 
> That said Han Gao pointed me to this patch:
> https://lore.kernel.org/linux-riscv/20260807-vector_fpu_regs_status_rmw_fix-v1-1-0c16848b60db@intel.com/
> 
> I have been testing it, and so far it seems to fix my issue with both 
> CONFIG_RISCV_ISA_V_PREEMPTIVE enable and disabled. Or maybe it just 
> hides the issue by changing the timing, as at this point, I haven't 
> fully understood how the bug fixed by this patch can completely explain 
> my observations.

Unfortunately, I have been able to reproduce the issue, even with this 
patch applied. It just significantly reduces the frequency of the issue.

Regards
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Vivian Wang]

On 9/28/26 12:32, Aurelien Jarno wrote:
> Hi,
>
> On 2026-09-25 06:40, Aurelien Jarno wrote:
>
> [...]
>
> That said Han Gao pointed me to this patch:
> https://lore.kernel.org/linux-riscv/20260807-vector_fpu_regs_status_rmw_fix-v1-1-0c16848b60db@intel.com/
>
> I have been testing it, and so far it seems to fix my issue with both 
> CONFIG_RISCV_ISA_V_PREEMPTIVE enable and disabled. Or maybe it just 
> hides the issue by changing the timing, as at this point, I haven't 
> fully understood how the bug fixed by this patch can completely explain 
> my observations.

I don't know how that would cause what you have observed either, but it
does superficially appears to fix a real bug.

> Unfortunately, I have been able to reproduce the issue, even with this 
> patch applied. It just significantly reduces the frequency of the issue.
>
But this, I suppose, is still useful information that we just *might* be
on the right track.

Can you perhaps try this hacked together patch below and see if it does
anything? I just put guard(preempt)() or something around a bunch of
places I could find that access regs->status. It should apply to 7.1 and
7.2 as is. (The coding style is deliberately horrible to minimize diff
lines and discourage others from shipping it)

(This adds some more widely-scoped guards, and AFAICT /replaces/ the
above linked patch from Guobin Zhang.)

vvv DO NOT USE EXCEPT FOR TESTING vvv

diff --git a/arch/riscv/include/asm/switch_to.h b/arch/riscv/include/asm/switch_to.h
index 04f10a949066..6486f9fe0ac5 100644
--- a/arch/riscv/include/asm/switch_to.h
+++ b/arch/riscv/include/asm/switch_to.h
@@ -6,6 +6,7 @@
 #ifndef _ASM_RISCV_SWITCH_TO_H
 #define _ASM_RISCV_SWITCH_TO_H
 
+#include <linux/cleanup.h>
 #include <linux/jump_label.h>
 #include <linux/sched/task_stack.h>
 #include <linux/mm_types.h>
@@ -28,12 +29,14 @@ static inline void __fstate_clean(struct pt_regs *regs)
 static inline void fstate_off(struct task_struct *task,
 			      struct pt_regs *regs)
 {
+	guard(preempt)();
 	regs->status = (regs->status & ~SR_FS) | SR_FS_OFF;
 }
 
 static inline void fstate_save(struct task_struct *task,
 			       struct pt_regs *regs)
 {
+	guard(preempt)();
 	if ((regs->status & SR_FS) == SR_FS_DIRTY) {
 		__fstate_save(task);
 		__fstate_clean(regs);
@@ -43,6 +46,7 @@ static inline void fstate_save(struct task_struct *task,
 static inline void fstate_restore(struct task_struct *task,
 				  struct pt_regs *regs)
 {
+	guard(preempt)();
 	if ((regs->status & SR_FS) != SR_FS_OFF) {
 		__fstate_restore(task);
 		__fstate_clean(regs);
diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h
index fffe72a77208..30fc51713542 100644
--- a/arch/riscv/include/asm/vector.h
+++ b/arch/riscv/include/asm/vector.h
@@ -97,11 +97,13 @@ static inline void __riscv_v_vstate_dirty(struct pt_regs *regs)
 
 static inline void riscv_v_vstate_off(struct pt_regs *regs)
 {
+	guard(preempt)();
 	regs->status = __riscv_v_vstate_or(regs->status, OFF);
 }
 
 static inline void riscv_v_vstate_on(struct pt_regs *regs)
 {
+	guard(preempt)();
 	regs->status = __riscv_v_vstate_or(regs->status, INITIAL);
 }
 
@@ -302,6 +304,7 @@ static inline void __riscv_v_vstate_discard(void)
 
 static inline void riscv_v_vstate_discard(struct pt_regs *regs)
 {
+	guard(preempt)();
 	if (riscv_v_vstate_query(regs)) {
 		__riscv_v_vstate_discard();
 		__riscv_v_vstate_dirty(regs);
@@ -311,6 +314,7 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs)
 static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate,
 				       struct pt_regs *regs)
 {
+	guard(preempt)();
 	if (__riscv_v_vstate_check(regs->status, DIRTY)) {
 		__riscv_v_vstate_save(vstate, vstate->datap);
 		__riscv_v_vstate_clean(regs);
@@ -320,6 +324,7 @@ static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate,
 static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate,
 					  struct pt_regs *regs)
 {
+	guard(preempt)();
 	if (riscv_v_vstate_query(regs)) {
 		__riscv_v_vstate_restore(vstate, vstate->datap);
 		__riscv_v_vstate_clean(regs);
@@ -329,6 +334,7 @@ static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate,
 static inline void riscv_v_vstate_set_restore(struct task_struct *task,
 					      struct pt_regs *regs)
 {
+	guard(preempt)();
 	if (riscv_v_vstate_query(regs)) {
 		set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE);
 		riscv_v_vstate_on(regs);
diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c
index b2df7f72241a..45963c7baebe 100644
--- a/arch/riscv/kernel/process.c
+++ b/arch/riscv/kernel/process.c
@@ -144,6 +144,7 @@ early_initcall(compat_mode_detect);
 void start_thread(struct pt_regs *regs, unsigned long pc,
 	unsigned long sp)
 {
+	guard(preempt)();
 	regs->status = SR_PIE;
 	if (has_fpu()) {
 		regs->status |= SR_FS_INITIAL;
diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c
index 59784dc117e4..623cc35d567c 100644
--- a/arch/riscv/kernel/signal.c
+++ b/arch/riscv/kernel/signal.c
@@ -6,6 +6,7 @@
  * Copyright (C) 2012 Regents of the University of California
  */
 
+#include <linux/cleanup.h>
 #include <linux/compat.h>
 #include <linux/signal.h>
 #include <linux/uaccess.h>
@@ -76,6 +77,7 @@ static long save_v_state(struct pt_regs *regs, void __user *sc_vec)
 	void __user *datap;
 	long err;
 
+	scoped_guard(preempt)
 	if (!IS_ENABLED(CONFIG_RISCV_ISA_V) ||
 	    !((has_vector() || has_xtheadvector()) &&
 	    riscv_v_vstate_query(regs)))
@@ -287,6 +289,7 @@ static size_t get_rt_frame_size(bool cal_all)
 	frame_size = sizeof(*frame);
 
 	if (has_vector() || has_xtheadvector()) {
+		scoped_guard(preempt)
 		if (cal_all || riscv_v_vstate_query(task_pt_regs(current)))
 			total_context_size += riscv_v_sc_size;
 	}
diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c
index b112166d51e9..27078b70b65b 100644
--- a/arch/riscv/kernel/vector.c
+++ b/arch/riscv/kernel/vector.c
@@ -3,6 +3,7 @@
  * Copyright (C) 2023 SiFive
  * Author: Andy Chiu <andy.chiu@sifive.com>
  */
+#include <linux/cleanup.h>
 #include <linux/export.h>
 #include <linux/sched/signal.h>
 #include <linux/types.h>
@@ -194,6 +195,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs)
 	if (!riscv_v_vstate_ctrl_user_allowed())
 		return false;
 
+	scoped_guard(preempt)
 	/* If V has been enabled then it is not the first-use trap */
 	if (riscv_v_vstate_query(regs))
 		return false;

^^^ DO NOT USE EXCEPT FOR TESTING ^^^


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv