Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Lukas Wunner <lukas@wunner.de>]

On Thu, Feb 26, 2026 at 11:41:56AM +0000, Kepplinger-Novakovic Martin wrote:
> [    2.272135] PKEY: ==>public_key_verify_signature()
> [    2.272165] CAAM rsa init start
> [    2.272180] CAAM rsa init done
> [    2.272191] caam_rsa_pub_key: free old key in ctx
> [    2.272201] caam_rsa_pub_key: write rsa_key->e
> [    2.272210] caam_rsa_pub_key: write rsa_key->n
> [    2.272220] start rsassa_pkcs1_verify
> [    2.272228] slen: 256
> [    2.272238] child_req address: 1d64b62a full size: 64 + 48 + 256 = 368
> [    2.272274] out_buf1:00000000: 00000000 00000000 00000000 00000000  ................
> [    2.272298] out_buf1:00000010: 00000000 00000000 00000000 00000000  ................
> [    2.272322] SRC BUF in out_buf1 CRC: 969ee858
> [    2.272335] start caam_rsa_enc
> [    2.272352] key:00000000: cf60a600 cf4d1240 00000000 00000000  ..`.@.M.........
> [    2.272377] key:00000010: 00000000 00000000 00000000 00000000  ................
> [    2.272413] edesc:00000000: 00000001 00000001 00000000 00000000  ................
> [    2.272438] edesc:00000010: 00000000 00000000 00000000 cf533d6c  ............l=S.
> [    2.272466] req:00000000: 00000000 00000000 c02e2f68 d083dcb4  ........h/......
> [    2.272491] req:00000010: cf60a540 00000200 d083dc94 d083dca4  @.`.............
> [    2.272509] CAAM: calling caam_jr_enqueue
> [    2.272524] key:00000000: cf60a600 cf4d1240 00000000 00000000  ..`.@.M.........
> [    2.272546] key:00000010: 00000000 00000000 00000000 00000000  ................
> [    2.277444] CAAM: completion callback
> [    2.424765] OUT BUF in out_buf2 CRC: fd0eef11
> [    2.424799] out_buf2:00000000: 00000000 00000000 00000000 00000000  ................
> [    2.424827] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424853] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424878] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424902] out_buf2:00000040: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424926] out_buf2:00000050: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424949] out_buf2:00000060: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424973] out_buf2:00000070: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424996] out_buf2:00000080: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.425020] out_buf2:00000090: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.425043] out_buf2:000000a0: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.425068] out_buf2:000000b0: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.425095] out_buf2:000000c0: ffffffff ffffffff ffffffff 30313000  .............010
> [    2.425123] out_buf2:000000d0: 6009060d 65014886 01020403 20040005  ...`.H.e....... 
> [    2.425148] out_buf2:000000e0: 6155a84e 7aa089cb 7540e613 f28b9a30  N.Ua...z..@u0...
> [    2.425172] out_buf2:000000f0: 1e98ec34 cecb0e0f 9ee8951a ad8baec3  4...............

There's an endianness issue here:  30313000 is the zero byte prescribed
by EMSA-PKCS1-v1_5 ("in_buf[ps_end] = 0x00;" in rsassa_pkcs1_sign()),
followed by the first three bytes of hash_prefix_sha256[] in reverse order.

Then 6009060d are the next four bytes of hash_prefix_sha256[], again
in reverse order.  And so on until 20040005, which are the last four
bytes of the prefix in reverse order.

How are you generating that hexdump?  What's the CPU's endianness?
Is the caam RSA accelerator using a different endianness?

Thanks,

Lukas