Random corruption on SpacemiT K1 with RVV and THP

Random corruption on SpacemiT K1 with RVV and THP

[Aurelien Jarno]

Dear all,

For the last weeks, I have been tracking a random memory corruption and
relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
started upgrading to glibc 2.43, which does memset() through vector 
instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
kernel, using a similar configuration to the Debian kernel. The board 
uses OpenSBI 1.9 and the vendor U-Boot.

Typically it manifests itself with the following kind of error, when 
running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
Blender, Dolfin, Qt6):   

Assembler messages:
{standard input}:284588: Error: unknown pseudo-op: `.uleb1'
{standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'

The broken chars are 0xff and it seems there are always 240 (but with 
poor statistics). Sometimes it instead causes a GCC ICE instead.

Using glibc 2.44 instead of glibc 2.43, which does a lot of string 
operations through vector instructions, increases the probability to 
have corruption, makes it a bit more reproducible, but it always seems 
to manifest as a GCC ICE. It typically happens withing one hour when 
running on the 8 cores instead of every 1 or 2 days. From there I have 
been able to determine the following things:

- Disabling vector instructions (by patching the DTB to remove "v", 
  "zvhf" and "zvtk") fixes the issue

- Disabling THP (by setting /sys/kernel/mm/transparent_hugepage/enabled 
  to never instead of always) also seems to fix the issue. Keeping it 
  enabled with defrag=never or use_zero_page=0 doesn't change anything.

- The issue is reproducible with or without swap enabled.

I have not been able to reproduce the issue on other non RVV boards 
(Unmatched, VF2) nor on a SpacemiT K3 board.

I am not really sure how to debug that further. I tried a few ways to 
reproduce the issue with a small C code around the glibc memset code 
(including triggering unaligned accesses and page faults), but failed to 
do so. I therefore welcome any idea about the issue or how to debug it 
further.

Thanks
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 with RVV

[Aurelien Jarno]

Dear all,

I have done some small progress on that issue. Help is still wanted and 
would be appreciated.

On 2026-08-30 22:52, Aurelien Jarno wrote:
> Dear all,
> 
> For the last weeks, I have been tracking a random memory corruption and
> relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
> started upgrading to glibc 2.43, which does memset() through vector 
> instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
> kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
> kernel, using a similar configuration to the Debian kernel. The board 
> uses OpenSBI 1.9 and the vendor U-Boot.

I have been able to rule out OpenSBI from the issue, I have checked 
there is not trap top OpenSBI when the problem happens.

> Typically it manifests itself with the following kind of error, when 
> running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
> Blender, Dolfin, Qt6):   
> 
> Assembler messages:
> {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> 
> The broken chars are 0xff and it seems there are always 240 (but with 
> poor statistics). Sometimes it instead causes a GCC ICE instead.

I have identified that the 0xff comes from the poisoning done for v0 in
__riscv_v_vstate_discard(), which should only happen for a syscall. 
Indeed changing the value to different ones propagates to the above 
error message.

Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
changes the number of broken bytes.

Disabling the vectored user code entirely with the following patch seems
to prevent the issue (or make it sufficiently rare that I have not 
encountered it):

--- a/arch/riscv/lib/riscv_v_helpers.c
+++ b/arch/riscv/lib/riscv_v_helpers.c
@@ -25,7 +25,7 @@ asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n,
        size_t remain, copied;
 
        /* skip has_vector() check because it has been done by the asm  */
-       if (!may_use_simd())
+       if (!may_use_simd() || true)
                goto fallback;
 
        kernel_vector_begin();

Overall, while the corruption happens randomly, the corruption itself 
seems to be deterministic, and the data source of the corruption is 
identified. But I don't understand how those values propagate to user 
space. It is not clear to me if it is a bug somewhere in the context 
switching code, or a bug in __asm_vector_usercopy_sum_enabled fixup 
code, with the values just being the one at the entry of the function.
Also I can't completely rule out a CPU bug.

> Using glibc 2.44 instead of glibc 2.43, which does a lot of string 
> operations through vector instructions, increases the probability to 
> have corruption, makes it a bit more reproducible, but it always seems 
> to manifest as a GCC ICE. It typically happens withing one hour when 
> running on the 8 cores instead of every 1 or 2 days. From there I have 
> been able to determine the following things:

I have stopped using glibc 2.44 as a reproducer, and it generate 
different type of crashes, and there could be another set of bugs on top 
of the one I am trying to identify. This however means it takes a lot of 
time to test any change.
 
> - Disabling vector instructions (by patching the DTB to remove "v", 
>   "zvhf" and "zvtk") fixes the issue
> 
> - Disabling THP (by setting /sys/kernel/mm/transparent_hugepage/enabled 
>   to never instead of always) also seems to fix the issue. Keeping it 
>   enabled with defrag=never or use_zero_page=0 doesn't change anything.

I don't think this is a good direction, my impression is that it just 
changes timing a bit, which just hide the issue, or require different 
condition to trigger it.

Regards
Aurelien 

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Aurelien Jarno]

[Added Andy and Karl in Cc: as they have been involved in the vectored 
user copy code and fighting similar issues]

Hi,

Some more progress on that topic.

On 2026-09-08 06:42, Aurelien Jarno wrote:
> Dear all,
> 
> I have done some small progress on that issue. Help is still wanted and 
> would be appreciated.
> 
> On 2026-08-30 22:52, Aurelien Jarno wrote:
> > Dear all,
> > 
> > For the last weeks, I have been tracking a random memory corruption and
> > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
> > started upgrading to glibc 2.43, which does memset() through vector 
> > instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
> > kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
> > kernel, using a similar configuration to the Debian kernel. The board 
> > uses OpenSBI 1.9 and the vendor U-Boot.
> 
> I have been able to rule out OpenSBI from the issue, I have checked 
> there is not trap top OpenSBI when the problem happens.
> 
> > Typically it manifests itself with the following kind of error, when 
> > running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
> > Blender, Dolfin, Qt6):   
> > 
> > Assembler messages:
> > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> > 
> > The broken chars are 0xff and it seems there are always 240 (but with 
> > poor statistics). Sometimes it instead causes a GCC ICE instead.
> 
> I have identified that the 0xff comes from the poisoning done for v0 in
> __riscv_v_vstate_discard(), which should only happen for a syscall. 
> Indeed changing the value to different ones propagates to the above 
> error message.
> 
> Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
> doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
> changes the number of broken bytes.

The values from __riscv_v_vstate_discard() end-up there because they are 
the last values written to the vector registers. I have added some 
poisoning in __asm_vector_usercopy_sum_enabled before the loop, and 
those values appear instead. Using different values per vector register, 
I have found that the corruption comes from a partial load of the vle8.v 
instruction, while the result of the poisoning and the partial load are
then both written by the vse8.v:

 loop:
        vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
        fixup vle8.v vData, (pSrc), 10f
        sub iNum, iNum, iVL
        add pSrc, pSrc, iVL
        fixup vse8.v vData, (pDst), 11f
        add pDst, pDst, iVL
        bnez iNum, loop

One hypothesis could be that for some reason, an exception (page table 
fault, irq, ...), the vle8.v instruction is interrupted, and only 
partially loads the vector register from memory. This stops at a 16-byte 
boundary (at least on the K1), and the remaining part of the v0-v7 
register is left with the previous value (with the original kernel that 
is the poisoning done in __riscv_v_vstate_discard). In theory when the 
instruction is interrupted, vstart is set to a non-zero value (for 
instance 16), and it should get re-executed after the exception. In 
practice, in some very rare cases, it doesn't happen and the next 
executed instruction is the following sub.

That said with exceptions and vector context switches, the explanation 
is likely way more complex. [1] gives a possible different scenario, 
involving an interrupt or a fault between the vsetvli and vle8.v/vse8.v 
instructions. However it doesn't fix the issue, neither patch [2]
(in that case tested with CONFIG_RISCV_ISA_V_PREEMPTIVE=y).


> Disabling the vectored user code entirely with the following patch seems
> to prevent the issue (or make it sufficiently rare that I have not 
> encountered it):

A much better way to do that is setting RISCV_ISA_V_UCOPY_THRESHOLD=-1.  

I also tried the same kernel on a K3, and it appears to improve 
stability, getting rid of issues that I attributed to thermal issues.  
(due to the absence of fan driver, I run the fan as a fixed speed 
~4000rpm). The symptoms are however quite different, it's GCC crashes 
with "The bug is not reproducible, so it is likely a hardware or OS 
problem.".

Regards
Aurelien

[1] https://lore.kernel.org/20260806193241.10552-1-kmehltretter@gmail.com/
[2] https://lore.kernel.org/all/20260810172255.1532787-2-tchiu@tenstorrent.com/

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv