Re: Random corruption on SpacemiT K1 with RVV

[Aurelien Jarno <aurelien@aurel32.net>]

Dear all,

I have done some small progress on that issue. Help is still wanted and 
would be appreciated.

On 2026-08-30 22:52, Aurelien Jarno wrote:
> Dear all,
> 
> For the last weeks, I have been tracking a random memory corruption and
> relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
> started upgrading to glibc 2.43, which does memset() through vector 
> instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
> kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
> kernel, using a similar configuration to the Debian kernel. The board 
> uses OpenSBI 1.9 and the vendor U-Boot.

I have been able to rule out OpenSBI from the issue, I have checked 
there is not trap top OpenSBI when the problem happens.

> Typically it manifests itself with the following kind of error, when 
> running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
> Blender, Dolfin, Qt6):   
> 
> Assembler messages:
> {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> 
> The broken chars are 0xff and it seems there are always 240 (but with 
> poor statistics). Sometimes it instead causes a GCC ICE instead.

I have identified that the 0xff comes from the poisoning done for v0 in
__riscv_v_vstate_discard(), which should only happen for a syscall. 
Indeed changing the value to different ones propagates to the above 
error message.

Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
changes the number of broken bytes.

Disabling the vectored user code entirely with the following patch seems
to prevent the issue (or make it sufficiently rare that I have not 
encountered it):

--- a/arch/riscv/lib/riscv_v_helpers.c
+++ b/arch/riscv/lib/riscv_v_helpers.c
@@ -25,7 +25,7 @@ asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n,
        size_t remain, copied;
 
        /* skip has_vector() check because it has been done by the asm  */
-       if (!may_use_simd())
+       if (!may_use_simd() || true)
                goto fallback;
 
        kernel_vector_begin();

Overall, while the corruption happens randomly, the corruption itself 
seems to be deterministic, and the data source of the corruption is 
identified. But I don't understand how those values propagate to user 
space. It is not clear to me if it is a bug somewhere in the context 
switching code, or a bug in __asm_vector_usercopy_sum_enabled fixup 
code, with the values just being the one at the entry of the function.
Also I can't completely rule out a CPU bug.

> Using glibc 2.44 instead of glibc 2.43, which does a lot of string 
> operations through vector instructions, increases the probability to 
> have corruption, makes it a bit more reproducible, but it always seems 
> to manifest as a GCC ICE. It typically happens withing one hour when 
> running on the 8 cores instead of every 1 or 2 days. From there I have 
> been able to determine the following things:

I have stopped using glibc 2.44 as a reproducer, and it generate 
different type of crashes, and there could be another set of bugs on top 
of the one I am trying to identify. This however means it takes a lot of 
time to test any change.
 
> - Disabling vector instructions (by patching the DTB to remove "v", 
>   "zvhf" and "zvtk") fixes the issue
> 
> - Disabling THP (by setting /sys/kernel/mm/transparent_hugepage/enabled 
>   to never instead of always) also seems to fix the issue. Keeping it 
>   enabled with defrag=never or use_zero_page=0 doesn't change anything.

I don't think this is a good direction, my impression is that it just 
changes timing a bit, which just hide the issue, or require different 
condition to trigger it.

Regards
Aurelien 

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv