Re: Random corruption on SpacemiT K1 (and K3) with RVV

[Aurelien Jarno <aurelien@aurel32.net>]

Hi,

Some more update, even if the progress is quite low.

On 2026-09-09 18:45, Aurelien Jarno wrote:
> [Added Andy and Karl in Cc: as they have been involved in the vectored 
> user copy code and fighting similar issues]
> 
> Hi,
> 
> Some more progress on that topic.
> 
> On 2026-09-08 06:42, Aurelien Jarno wrote:
> > Dear all,
> > 
> > I have done some small progress on that issue. Help is still wanted and 
> > would be appreciated.
> > 
> > On 2026-08-30 22:52, Aurelien Jarno wrote:
> > > Dear all,
> > > 
> > > For the last weeks, I have been tracking a random memory corruption and
> > > relatively rare on SpacemiT K1 (Banana Pi F3 and Milk-V Jupiter). It 
> > > started upgrading to glibc 2.43, which does memset() through vector 
> > > instructions.  It is reproducible using the Debian 7.1.7-1~bpo13+1 
> > > kernel, but I have also been able to reproduce it with a vanilla 7.2.2 
> > > kernel, using a similar configuration to the Debian kernel. The board 
> > > uses OpenSBI 1.9 and the vendor U-Boot.

I have been able to also reproduce the issue with the vendor OpenSBI. It 
is also reproducible with linux 7.2.6.

> > I have been able to rule out OpenSBI from the issue, I have checked 
> > there is not trap to OpenSBI when the problem happens.
> > 
> > > Typically it manifests itself with the following kind of error, when 
> > > running g++ from GCC 16 as part of building software (e.g. OpenJDK, 
> > > Blender, Dolfin, Qt6):   
> > > 
> > > Assembler messages:
> > > {standard input}:284588: Error: unknown pseudo-op: `.uleb1'
> > > {standard input}:284588: Error: unrecognized opcode `ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvl874'
> > > 
> > > The broken chars are 0xff and it seems there are always 240 (but with 
> > > poor statistics). Sometimes it instead causes a GCC ICE instead.
> > 
> > I have identified that the 0xff comes from the poisoning done for v0 in
> > __riscv_v_vstate_discard(), which should only happen for a syscall. 
> > Indeed changing the value to different ones propagates to the above 
> > error message.
> > 
> > Furthermore I have found that changing CONFIG_RISCV_ISA_V_PREEMPTIVE 
> > doesn't change anything, and that changing RISCV_ISA_V_UCOPY_THRESHOLD 
> > changes the number of broken bytes.
> 
> The values from __riscv_v_vstate_discard() end-up there because they are 
> the last values written to the vector registers. I have added some 
> poisoning in __asm_vector_usercopy_sum_enabled before the loop, and 
> those values appear instead. Using different values per vector register, 
> I have found that the corruption comes from a partial load of the vle8.v 
> instruction, while the result of the poisoning and the partial load are
> then both written by the vse8.v:
> 
>  loop:
>         vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
>         fixup vle8.v vData, (pSrc), 10f
>         sub iNum, iNum, iVL
>         add pSrc, pSrc, iVL
>         fixup vse8.v vData, (pDst), 11f
>         add pDst, pDst, iVL
>         bnez iNum, loop

I have instrumented that code, basically adding a function to check the 
copy afterwards (only using scalar instructions), and report values in 
case of errors. So far I have been able to catch the issue 4 times that 
way (including one where I haven't seen any visible consequence on the 
userland). 

This is what I observed:
- The issue always happens for the first bytes of the user copy, which 
  in all cases were supposed to be up to 4096 bytes long.
- On the first iteration of the loop only the first 16 bytes are loaded 
  by the vle8.v instruction but the full 256 bytes are written to the 
  destination by the vse8.v instrution.
- On the second iteration of the loop, the vle8.v causes a trap, which 
  ends up in the 10f fixup. At fixup level, vl=0x100, vlenb=0x20, 
  vstart=0x0 and vtype=0xc3.
- Considering the 2 first iterations of the loop (i.e. 512 bytes), 
  neither the source nor destination addresses cross a page.

All that said, it is what I can *observe*. I have no idea if the first 
iteration of the loop also goes through fixup or not. Also in addition 
to precise traps (which definitely happen for the second iteration), IRQ 
could happen at any moment. I use CONFIG_RISCV_ISA_V_PREEMPTIVE=n for my 
test, in order to limit the code paths to review, so during that copy 
preempt is disabled, but IRQ is not.

Regards
Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv