[BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Kepplinger-Novakovic Martin]

hi Horia, Pankaj, Gaurav and all interested,

I run imx6ul with FSL_CAAM* enabled and simply add ca.pem 
( openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem -nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com )
to CONFIG_SYSTEM_TRUSTED_KEYS in order to use it to verify a squashfs rootfs via dm-verity
(but forget that for a moment, the failure is early during boot, rsa verify()).

This works until v6.6 and fails after ("crypto: ahash - optimize performance when wrapping shash")
but too much has happened that I could revert one and I might be wrong with that commit even.

I run v6.18 now where this works when I *disable* FSL_CAAM* completely and use the rsa-generic code.
Also it works, when I generate from elliptic curve key.

Only the CAAM+RSA case is failing in a weird way:

During boot, already the RSA .encrypt triggered from preparse() / verify() for all built-in keys/certs
(my own+the wireless regdb keys from mainline) fails for *some* and for *some not*, but *not*
always the same keys fail. They all sometimes fail and on a subsequent reboot a previously failing one *may* succeed.
If a verify() fails, it *always* fails here:

https://elixir.bootlin.com/linux/v6.18.12/source/crypto/rsassa-pkcs1.c#L266

but why? Because after the CAAM completion callback returning to the caller, out_buf
(that was set to the key-part inside of the request earlier) still holds *old* data,
and NOT the "encyption block" with 00 01 ff ff.... 

out_buf1 is (I assume valid because never changing) input data "out_buf" inside of "child_req".
Directly when caam jr dequeue() returns with the user-callback, I see 64 bytes (all?)
old data. SOMEHOW the cryto-wait needs 100ms or longer (no idea why) and after THAT,
out_buf has only the first 16 bytes old data (out_buf2 in the logs):


[    2.267849] start rsassa_pkcs1_verify
[    2.267857] child_req address: 0e805627 full size: 64 + 48 + 256 = 368
[    2.267898] out_buf1:00000000: f2da0387 afddc282 862f447c 934c5fd3  ........|D/.._L.
[    2.267923] out_buf1:00000010: 07feb948 f721bb17 aa4e2325 b9160c22  H.....!.%#N."...
[    2.267947] out_buf1:00000020: 469dae73 c3d9757c bf475749 ec97b733  s..F|u..IWG.3...
[    2.267970] out_buf1:00000030: c07540f5 a0f02246 13799c5d a3b8ffa1  .@u.F"..].y.....
[    2.267993] SRC BUF in out_buf1 CRC: 60791b87
[    2.268007] start caam_rsa_enc
[    2.268172] CAAM: calling caam_jr_enqueue
[    2.268238] jr areq+112:00000000: f2da0387 afddc282 862f447c 934c5fd3  ........|D/.._L.
[    2.268264] jr areq+112:00000010: 07feb948 f721bb17 aa4e2325 b9160c22  H.....!.%#N."...
[    2.268288] jr areq+112:00000020: 469dae73 c3d9757c bf475749 ec97b733  s..F|u..IWG.3...
[    2.268311] jr areq+112:00000030: c07540f5 a0f02246 13799c5d a3b8ffa1  .@u.F"..].y.....
[    2.275958] CAAM: completion callback
  here only old data!?
[    2.275986] jr deq userarg + 112:00000000: f2da0387 afddc282 862f447c 934c5fd3  ........|D/.._L.
[    2.276012] jr deq userarg + 112:00000010: 07feb948 f721bb17 aa4e2325 b9160c22  H.....!.%#N."...
[    2.276038] jr deq userarg + 112:00000020: 469dae73 c3d9757c bf475749 ec97b733  s..F|u..IWG.3...
[    2.276063] jr deq userarg + 112:00000030: c07540f5 a0f02246 13799c5d a3b8ffa1  .@u.F"..].y.....
[    2.276079] rsa_pub_done start
[    2.276092] calling akcipher_request_complete
[    2.276100] crypto_req_done calling complete
   why the delay here?
[    2.416791] OUT BUF in out_buf2 CRC: 12298efd
[    2.416810] mapping pa 8f551870 to va 1775c16c
   now only the first 16 bytes are old??
[    2.416843] out_buf2:00000000: f2da0387 afddc282 862f447c 934c5fd3  ........|D/.._L.
[    2.416868] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff  ................
[    2.416892] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff  ................
[    2.416915] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff  ................
[    2.416930] Encrypted value had no leading 0 byte.
[    2.416941] PKEY: crypto_sig_verify error: -22
[    2.416964] PKEY: <==public_key_verify_signature() = -22
[    2.416979] X.509: public_key_verify_signature error: -22


compare it to a successful run. Note that NO visible delay after "crypto_req_done calling complete".
New data immediately in caam jr dequeue().

[    2.148544] jr areq+112:00000000: a9fe4420 ea9bdd9e 087525ce f7532bf0   D.......%u..+S.
[    2.148569] jr areq+112:00000010: 4a1c365a 41d07f23 b92b123c 158a4e80  Z6.J#..A<.+..N..
[    2.148592] jr areq+112:00000020: a7401f5d c3322826 2d28065b 1e09083d  ].@.&(2.[.(-=...
[    2.148614] jr areq+112:00000030: e367e901 4515e633 8317ee39 7fff42db  ..g.3..E9....B..
[    2.152208] CAAM: completion callback
[    2.152265] jr deq userarg + 112:00000000: ffff0100 ffffffff ffffffff ffffffff  ................
[    2.152291] jr deq userarg + 112:00000010: ffffffff ffffffff ffffffff ffffffff  ................
[    2.152316] jr deq userarg + 112:00000020: ffffffff ffffffff ffffffff ffffffff  ................
[    2.152339] jr deq userarg + 112:00000030: ffffffff ffffffff ffffffff ffffffff  ................
[    2.152354] rsa_pub_done start
[    2.152372] calling akcipher_request_complete
[    2.152380] crypto_req_done calling complete
[    2.152701] OUT BUF in out_buf2 CRC: 136f61c9
[    2.152719] mapping pa 8f551670 to va 3952caaf
[    2.152753] out_buf2:00000000: ffff0100 ffffffff ffffffff ffffffff  ................
[    2.152777] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff  ................
[    2.152801] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff  ................
[    2.152823] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff  ................
[    2.152860] PKEY: <==public_key_verify_signature() = 0
[    2.152874] X.509: Cert Self-signature verified
[    2.152880] X.509: <==x509_check_for_self_signed() = 0
[    2.152898] X.509: Cert Issuerrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr: benh@debian.org
[    2.152910] X.509: Cert Subject: benh@debian.org
[    2.152921] X.509: Cert Key Algo: rsa
[    2.152930] X.509: Cert Valid period: 1580390773-4733990773
[    2.152947] X.509: Cert Signature: rsa + sha256

msleep(500) before checking out_buf[0] in crypto/rsassa-pkcs1.c doesn't help. 

Again, if I reboot as-is, a previously failing cert can succeed, so there is some kind of race-condition
involved here. Again, for rsa-generic without FSL_CAAM*, this never happens.

Can you tell what might go wrong? I'm kind of stuck but will keep debugging. I don't see
significant changes in the nxp/lf-6.12* tree either.

There has been quite some changes, most notably ("crypto: ahash - optimize performance when wrapping shash")
or ("crypto: rsassa-pkcs1 - Migrate to sig_alg backend") which squashes differnt changes into
one commit...

thank you in case you have time to have a look,

                                            martin

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Lukas Wunner]

On Tue, Feb 24, 2026 at 02:17:22PM +0000, Kepplinger-Novakovic Martin wrote:
> I run imx6ul with FSL_CAAM* enabled and simply add ca.pem 
> ( openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem \
> -nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com )
> to CONFIG_SYSTEM_TRUSTED_KEYS in order to use it to verify a squashfs
> rootfs via dm-verity (but forget that for a moment, the failure is early
> during boot, rsa verify()).

The issue might be easier to debug if you could come up with a
simpler reproducer.  E.g. you could use keyctl(1) to add a key to
a keyring in the kernel and subsequently have the kernel verify
a signature with it.

> This works until v6.6 and fails after ("crypto: ahash - optimize
> performance when wrapping shash")
> but too much has happened that I could revert one and I might be wrong
> with that commit even.

It would be good if you could bisect to exactly pinpoint the offending
commit.

> I run v6.18 now where this works when I *disable* FSL_CAAM* completely
> and use the rsa-generic code.
> Also it works, when I generate from elliptic curve key.
> 
> Only the CAAM+RSA case is failing in a weird way:
[...]
> but why? Because after the CAAM completion callback returning to the
> caller, out_buf (that was set to the key-part inside of the request
> earlier) still holds *old* data,
> and NOT the "encyption block" with 00 01 ff ff.... 
> 
> out_buf1 is (I assume valid because never changing) input data "out_buf"
> inside of "child_req".

"git grep out_buf1" finds nothing in a v6.19 source tree, so I'm not sure
what you're referring to?

> Directly when caam jr dequeue() returns with the user-callback, I see 64 bytes (all?)
> old data. SOMEHOW the cryto-wait needs 100ms or longer (no idea why) and after THAT,
> out_buf has only the first 16 bytes old data (out_buf2 in the logs):

I assume the caam device writes to memory via DMA.  Perhaps the CPU isn't
aware that the memory has been changed and is using data in its cache?
That could be caused by an incorrect dma_map_*() call in caampkc.c.

> There has been quite some changes, most notably ("crypto: ahash - optimize
> performance when wrapping shash")
> or ("crypto: rsassa-pkcs1 - Migrate to sig_alg backend") which squashes
> differnt changes into one commit...

Being the author of the latter, your initial report scared me that
I might have broken something.  However I looked through the code
and nothing stuck out.

Thanks,

Lukas

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Kepplinger-Novakovic Martin]

Am Dienstag, dem 24.02.2026 um 16:04 +0100 schrieb Lukas Wunner:
> On Tue, Feb 24, 2026 at 02:17:22PM +0000, Kepplinger-Novakovic Martin wrote:
> > I run imx6ul with FSL_CAAM* enabled and simply add ca.pem 
> > ( openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem \
> > -nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com )
> > to CONFIG_SYSTEM_TRUSTED_KEYS in order to use it to verify a squashfs
> > rootfs via dm-verity (but forget that for a moment, the failure is early
> > during boot, rsa verify()).
> 
> The issue might be easier to debug if you could come up with a
> simpler reproducer.  E.g. you could use keyctl(1) to add a key to
> a keyring in the kernel and subsequently have the kernel verify
> a signature with it.

I can try that, but I think CONFIG_CFG80211_REQUIRE_SIGNED_REGDB should be enough
to trigger this after rebooting.

> 
> > This works until v6.6 and fails after ("crypto: ahash - optimize
> > performance when wrapping shash")
> > but too much has happened that I could revert one and I might be wrong
> > with that commit even.
> 
> It would be good if you could bisect to exactly pinpoint the offending
> commit.

I know v6.6 worked. v6.7 showed
[    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx 0: Header Error. Invalid length or parity, or certain other problems.
and I never found one commit I could semantically revert as a fix.
I can try to narrow this down a bit later.

> 
> > I run v6.18 now where this works when I *disable* FSL_CAAM* completely
> > and use the rsa-generic code.
> > Also it works, when I generate from elliptic curve key.
> > 
> > Only the CAAM+RSA case is failing in a weird way:
> [...]
> > but why? Because after the CAAM completion callback returning to the
> > caller, out_buf (that was set to the key-part inside of the request
> > earlier) still holds *old* data,
> > and NOT the "encyption block" with 00 01 ff ff.... 
> > 
> > out_buf1 is (I assume valid because never changing) input data "out_buf"
> > inside of "child_req".
> 
> "git grep out_buf1" finds nothing in a v6.19 source tree, so I'm not sure
> what you're referring to?

sorry; out_buf here https://elixir.bootlin.com/linux/v6.19.3/source/crypto/rsassa-pkcs1.c#L246
is what I called out_buf1 and ~20 lines below I called it out_buf2 in my lost.

> 
> > Directly when caam jr dequeue() returns with the user-callback, I see 64 bytes (all?)
> > old data. SOMEHOW the cryto-wait needs 100ms or longer (no idea why) and after THAT,
> > out_buf has only the first 16 bytes old data (out_buf2 in the logs):
> 
> I assume the caam device writes to memory via DMA.  Perhaps the CPU isn't
> aware that the memory has been changed and is using data in its cache?
> That could be caused by an incorrect dma_map_*() call in caampkc.c.
> 
> > There has been quite some changes, most notably ("crypto: ahash - optimize
> > performance when wrapping shash")
> > or ("crypto: rsassa-pkcs1 - Migrate to sig_alg backend") which squashes
> > differnt changes into one commit...
> 
> Being the author of the latter, your initial report scared me that
> I might have broken something.  However I looked through the code
> and nothing stuck out.
> 
> Thanks,
> 
> Lukas

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Lukas Wunner]

On Tue, Feb 24, 2026 at 04:09:51PM +0000, Kepplinger-Novakovic Martin wrote:
> Am Dienstag, dem 24.02.2026 um 16:04 +0100 schrieb Lukas Wunner:
> > On Tue, Feb 24, 2026 at 02:17:22PM +0000, Kepplinger-Novakovic Martin wrote:
> > > This works until v6.6 and fails after ("crypto: ahash - optimize
> > > performance when wrapping shash")
> > > but too much has happened that I could revert one and I might be wrong
> > > with that commit even.
> > 
> > It would be good if you could bisect to exactly pinpoint the offending
> > commit.
> 
> I know v6.6 worked. v6.7 showed
> [    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx 0: Header Error.
> Invalid length or parity, or certain other problems.

Well there are 18404 commits between v6.6 and v6.7, so 14 or 15 steps
should be sufficient to find the culprit with git bisect.

It's quite doubtful that 2f1f34c1bf7b ("crypto: ahash - optimize
performance when wrapping shash") has anything to do with it.
It doesn't touch asymmetric crypto code.  If you absolutely
positively think it's the culprit, "git checkout 2f1f34c1bf7b^"
plus compile would confirm that.

> I can try to narrow this down a bit later.

I really recommend starting with git bisect now, not doing it later.
It's the most efficient use of your time.

Thanks,

Lukas

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Kepplinger-Novakovic Martin]

Am Dienstag, dem 24.02.2026 um 17:41 +0100 schrieb Lukas Wunner:
> On Tue, Feb 24, 2026 at 04:09:51PM +0000, Kepplinger-Novakovic Martin wrote:
> > Am Dienstag, dem 24.02.2026 um 16:04 +0100 schrieb Lukas Wunner:
> > > On Tue, Feb 24, 2026 at 02:17:22PM +0000, Kepplinger-Novakovic Martin wrote:
> > > > This works until v6.6 and fails after ("crypto: ahash - optimize
> > > > performance when wrapping shash")
> > > > but too much has happened that I could revert one and I might be wrong
> > > > with that commit even.
> > > 
> > > It would be good if you could bisect to exactly pinpoint the offending
> > > commit.
> > 
> > I know v6.6 worked. v6.7 showed
> > [    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx 0: Header Error.
> > Invalid length or parity, or certain other problems.
> 
> Well there are 18404 commits between v6.6 and v6.7, so 14 or 15 steps
> should be sufficient to find the culprit with git bisect.
> 
> It's quite doubtful that 2f1f34c1bf7b ("crypto: ahash - optimize
> performance when wrapping shash") has anything to do with it.
> It doesn't touch asymmetric crypto code.  If you absolutely
> positively think it's the culprit, "git checkout 2f1f34c1bf7b^"
> plus compile would confirm that.
> 
> > I can try to narrow this down a bit later.
> 
> I really recommend starting with git bisect now, not doing it later.
> It's the most efficient use of your time.
> 

ok I can confirm: "git checkout 2f1f34c1bf7b^" indeed is ok and 2f1f34c1bf7b is bad.

It's not the same behaviour I described (from v6.18/v6.19. that could be a combination of bugs) because on 2f1f34c1bf7b regdb cert verify succeeds,
only dm-verity fails, starting with this commit
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2f1f34c1bf7b309b296bc04321a09e6b5dba0673


    2.956222] caam_jr 2142000.jr: 40000013: DECO: desc idx 0: Header Error. Invalid length or parity, or certain other problems.
[    2.967918] SQUASHFS error: Failed to read block 0x0: -5
[    2.973269] unable to read squashfs_super_block
kinit: Cannot open root device dev(254,0)
kinit: init not found!
[    3.009360] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200
[    3.017063] CPU: 0 PID: 1 Comm: init Not tainted 6.6.0-rc1-ge-26.02.1-ge_pc_cc_gw-fronius-gbec79294ad24 #215
[    3.026918] Hardware name: Freescale i.MX6 Ultralite (Device Tree)
[    3.033124]  unwind_backtrace from show_stack+0x10/0x14
[    3.038396]  show_stack from dump_stack_lvl+0x24/0x2c
[    3.043494]  dump_stack_lvl from panic+0xf4/0x2d0
[    3.048252]  panic from do_exit+0x224/0x874
[    3.052492]  do_exit from do_group_exit+0x0/0xa4
[    3.057150]  do_group_exit from ret_fast_syscall+0x0/0x54


what could I test for you in order to help?

thank you!

                                   martin

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Lukas Wunner]

On Wed, Feb 25, 2026 at 08:02:08AM +0000, Kepplinger-Novakovic Martin wrote:
> ok I can confirm: "git checkout 2f1f34c1bf7b^" indeed is ok and
> 2f1f34c1bf7b is bad.
> 
> It's not the same behaviour I described (from v6.18/v6.19. that could be
> a combination of bugs) because on 2f1f34c1bf7b regdb cert verify succeeds,
> only dm-verity fails

Hm, I assume CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n magically
makes the issue go away?

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Kepplinger-Novakovic Martin]

Am Mittwoch, dem 25.02.2026 um 09:13 +0100 schrieb Lukas Wunner:
> On Wed, Feb 25, 2026 at 08:02:08AM +0000, Kepplinger-Novakovic Martin wrote:
> > ok I can confirm: "git checkout 2f1f34c1bf7b^" indeed is ok and
> > 2f1f34c1bf7b is bad.
> > 
> > It's not the same behaviour I described (from v6.18/v6.19. that could be
> > a combination of bugs) because on 2f1f34c1bf7b regdb cert verify succeeds,
> > only dm-verity fails
> 
> Hm, I assume CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n magically
> makes the issue go away?

correct. where I see that specific issue (on 2f1f34c1bf7b and v6.7)
"caam_jr 2142000.jr: 40000013: DECO: desc idx 0: Header Error. Invalid length or parity, or certain other problems."
it then goes away.

on v6.18 CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n doesn't seem to help and I see the bugreport's behaviour.

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Lukas Wunner]

On Wed, Feb 25, 2026 at 08:47:07AM +0000, Kepplinger-Novakovic Martin wrote:
> Am Mittwoch, dem 25.02.2026 um 09:13 +0100 schrieb Lukas Wunner:
> > On Wed, Feb 25, 2026 at 08:02:08AM +0000, Kepplinger-Novakovic Martin wrote:
> > > ok I can confirm: "git checkout 2f1f34c1bf7b^" indeed is ok and
> > > 2f1f34c1bf7b is bad.
> > > 
> > > It's not the same behaviour I described (from v6.18/v6.19. that could be
> > > a combination of bugs) because on 2f1f34c1bf7b regdb cert verify succeeds,
> > > only dm-verity fails
> > 
> > Hm, I assume CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n magically
> > makes the issue go away?
> 
> correct. where I see that specific issue (on 2f1f34c1bf7b and v6.7)
> "caam_jr 2142000.jr: 40000013: DECO: desc idx 0: Header Error.
> Invalid length or parity, or certain other problems."
> it then goes away.
> 
> on v6.18 CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n doesn't seem to help
> and I see the bugreport's behaviour.

I note that for the RSA verification, since 8552cb04e083 the same buffer
in memory is used for source and destination of RSA encrypt operation
invoked by crypto/rsassa-pkcs1.c.

That's fine for the RSA software implementation in crypto/rsa.c but
I could very well imagine it causes problems with an RSA accelerator,
particularly because rsa_edesc_alloc() in drivers/crypto/caam/caampkc.c
now maps the same buffer with DMA_TO_DEVICE and then DMA_FROM_DEVICE.

On v6.19, 8552cb04e083 seems to revert cleanly.  Could you try that
and see if it helps?  Be sure to set CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n
and CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API=y so that we focus on the RSA
issue for now.  We can look at the ahash one afterwards.

Thanks,

Lukas

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Kepplinger-Novakovic Martin]

Am Donnerstag, dem 26.02.2026 um 08:17 +0100 schrieb Lukas Wunner:
> On Wed, Feb 25, 2026 at 08:47:07AM +0000, Kepplinger-Novakovic Martin wrote:
> > Am Mittwoch, dem 25.02.2026 um 09:13 +0100 schrieb Lukas Wunner:
> > > On Wed, Feb 25, 2026 at 08:02:08AM +0000, Kepplinger-Novakovic Martin wrote:
> > > > ok I can confirm: "git checkout 2f1f34c1bf7b^" indeed is ok and
> > > > 2f1f34c1bf7b is bad.
> > > > 
> > > > It's not the same behaviour I described (from v6.18/v6.19. that could be
> > > > a combination of bugs) because on 2f1f34c1bf7b regdb cert verify succeeds,
> > > > only dm-verity fails
> > > 
> > > Hm, I assume CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n magically
> > > makes the issue go away?
> > 
> > correct. where I see that specific issue (on 2f1f34c1bf7b and v6.7)
> > "caam_jr 2142000.jr: 40000013: DECO: desc idx 0: Header Error.
> > Invalid length or parity, or certain other problems."
> > it then goes away.
> > 
> > on v6.18 CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n doesn't seem to help
> > and I see the bugreport's behaviour.
> 
> I note that for the RSA verification, since 8552cb04e083 the same buffer
> in memory is used for source and destination of RSA encrypt operation
> invoked by crypto/rsassa-pkcs1.c.
> 
> That's fine for the RSA software implementation in crypto/rsa.c but
> I could very well imagine it causes problems with an RSA accelerator,
> particularly because rsa_edesc_alloc() in drivers/crypto/caam/caampkc.c
> now maps the same buffer with DMA_TO_DEVICE and then DMA_FROM_DEVICE.
> 
> On v6.19, 8552cb04e083 seems to revert cleanly.  Could you try that
> and see if it helps?  Be sure to set CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n
> and CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API=y so that we focus on the RSA
> issue for now.  We can look at the ahash one afterwards.


hi Lukas, I'm happy you have a look here but that does not seem to be it:

out_buf after caam (out_buf2 is what I print) still holds old data, it now is zeroes (not exactly sure why though) and thus the final check fails one
byte later ("leading zero" is now part of the old data), see one typical case:


[    2.272135] PKEY: ==>public_key_verify_signature()
[    2.272165] CAAM rsa init start
[    2.272180] CAAM rsa init done
[    2.272191] caam_rsa_pub_key: free old key in ctx
[    2.272201] caam_rsa_pub_key: write rsa_key->e
[    2.272210] caam_rsa_pub_key: write rsa_key->n
[    2.272220] start rsassa_pkcs1_verify
[    2.272228] slen: 256
[    2.272238] child_req address: 1d64b62a full size: 64 + 48 + 256 = 368
[    2.272274] out_buf1:00000000: 00000000 00000000 00000000 00000000  ................
[    2.272298] out_buf1:00000010: 00000000 00000000 00000000 00000000  ................
[    2.272322] SRC BUF in out_buf1 CRC: 969ee858
[    2.272335] start caam_rsa_enc
[    2.272352] key:00000000: cf60a600 cf4d1240 00000000 00000000  ..`.@.M.........
[    2.272377] key:00000010: 00000000 00000000 00000000 00000000  ................
[    2.272413] edesc:00000000: 00000001 00000001 00000000 00000000  ................
[    2.272438] edesc:00000010: 00000000 00000000 00000000 cf533d6c  ............l=S.
[    2.272466] req:00000000: 00000000 00000000 c02e2f68 d083dcb4  ........h/......
[    2.272491] req:00000010: cf60a540 00000200 d083dc94 d083dca4  @.`.............
[    2.272509] CAAM: calling caam_jr_enqueue
[    2.272524] key:00000000: cf60a600 cf4d1240 00000000 00000000  ..`.@.M.........
[    2.272546] key:00000010: 00000000 00000000 00000000 00000000  ................
[    2.277444] CAAM: completion callback
[    2.424765] OUT BUF in out_buf2 CRC: fd0eef11
[    2.424799] out_buf2:00000000: 00000000 00000000 00000000 00000000  ................
[    2.424827] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff  ................
[    2.424853] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff  ................
[    2.424878] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff  ................
[    2.424902] out_buf2:00000040: ffffffff ffffffff ffffffff ffffffff  ................
[    2.424926] out_buf2:00000050: ffffffff ffffffff ffffffff ffffffff  ................
[    2.424949] out_buf2:00000060: ffffffff ffffffff ffffffff ffffffff  ................
[    2.424973] out_buf2:00000070: ffffffff ffffffff ffffffff ffffffff  ................
[    2.424996] out_buf2:00000080: ffffffff ffffffff ffffffff ffffffff  ................
[    2.425020] out_buf2:00000090: ffffffff ffffffff ffffffff ffffffff  ................
[    2.425043] out_buf2:000000a0: ffffffff ffffffff ffffffff ffffffff  ................
[    2.425068] out_buf2:000000b0: ffffffff ffffffff ffffffff ffffffff  ................
[    2.425095] out_buf2:000000c0: ffffffff ffffffff ffffffff 30313000  .............010
[    2.425123] out_buf2:000000d0: 6009060d 65014886 01020403 20040005  ...`.H.e....... 
[    2.425148] out_buf2:000000e0: 6155a84e 7aa089cb 7540e613 f28b9a30  N.Ua...z..@u0...
[    2.425172] out_buf2:000000f0: 1e98ec34 cecb0e0f 9ee8951a ad8baec3  4...............
[    2.425196] digest (in):00000000: 6155a84e 7aa089cb 7540e613 f28b9a30  N.Ua...z..@u0...
[    2.425220] digest (in):00000010: 1e98ec34 cecb0e0f 9ee8951a ad8baec3  4...............
[    2.425239] PKEY: crypto_sig_verify error: -74
[    2.425262] PKEY: <==public_key_verify_signature() = -74


so long,

                                             martin

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Lukas Wunner]

On Thu, Feb 26, 2026 at 11:41:56AM +0000, Kepplinger-Novakovic Martin wrote:
> [    2.272135] PKEY: ==>public_key_verify_signature()
> [    2.272165] CAAM rsa init start
> [    2.272180] CAAM rsa init done
> [    2.272191] caam_rsa_pub_key: free old key in ctx
> [    2.272201] caam_rsa_pub_key: write rsa_key->e
> [    2.272210] caam_rsa_pub_key: write rsa_key->n
> [    2.272220] start rsassa_pkcs1_verify
> [    2.272228] slen: 256
> [    2.272238] child_req address: 1d64b62a full size: 64 + 48 + 256 = 368
> [    2.272274] out_buf1:00000000: 00000000 00000000 00000000 00000000  ................
> [    2.272298] out_buf1:00000010: 00000000 00000000 00000000 00000000  ................
> [    2.272322] SRC BUF in out_buf1 CRC: 969ee858
> [    2.272335] start caam_rsa_enc
> [    2.272352] key:00000000: cf60a600 cf4d1240 00000000 00000000  ..`.@.M.........
> [    2.272377] key:00000010: 00000000 00000000 00000000 00000000  ................
> [    2.272413] edesc:00000000: 00000001 00000001 00000000 00000000  ................
> [    2.272438] edesc:00000010: 00000000 00000000 00000000 cf533d6c  ............l=S.
> [    2.272466] req:00000000: 00000000 00000000 c02e2f68 d083dcb4  ........h/......
> [    2.272491] req:00000010: cf60a540 00000200 d083dc94 d083dca4  @.`.............
> [    2.272509] CAAM: calling caam_jr_enqueue
> [    2.272524] key:00000000: cf60a600 cf4d1240 00000000 00000000  ..`.@.M.........
> [    2.272546] key:00000010: 00000000 00000000 00000000 00000000  ................
> [    2.277444] CAAM: completion callback
> [    2.424765] OUT BUF in out_buf2 CRC: fd0eef11
> [    2.424799] out_buf2:00000000: 00000000 00000000 00000000 00000000  ................
> [    2.424827] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424853] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424878] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424902] out_buf2:00000040: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424926] out_buf2:00000050: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424949] out_buf2:00000060: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424973] out_buf2:00000070: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.424996] out_buf2:00000080: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.425020] out_buf2:00000090: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.425043] out_buf2:000000a0: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.425068] out_buf2:000000b0: ffffffff ffffffff ffffffff ffffffff  ................
> [    2.425095] out_buf2:000000c0: ffffffff ffffffff ffffffff 30313000  .............010
> [    2.425123] out_buf2:000000d0: 6009060d 65014886 01020403 20040005  ...`.H.e....... 
> [    2.425148] out_buf2:000000e0: 6155a84e 7aa089cb 7540e613 f28b9a30  N.Ua...z..@u0...
> [    2.425172] out_buf2:000000f0: 1e98ec34 cecb0e0f 9ee8951a ad8baec3  4...............

There's an endianness issue here:  30313000 is the zero byte prescribed
by EMSA-PKCS1-v1_5 ("in_buf[ps_end] = 0x00;" in rsassa_pkcs1_sign()),
followed by the first three bytes of hash_prefix_sha256[] in reverse order.

Then 6009060d are the next four bytes of hash_prefix_sha256[], again
in reverse order.  And so on until 20040005, which are the last four
bytes of the prefix in reverse order.

How are you generating that hexdump?  What's the CPU's endianness?
Is the caam RSA accelerator using a different endianness?

Thanks,

Lukas

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Kepplinger-Novakovic Martin]

Am Donnerstag, dem 26.02.2026 um 14:27 +0100 schrieb Lukas Wunner:
> On Thu, Feb 26, 2026 at 11:41:56AM +0000, Kepplinger-Novakovic Martin wrote:
> > [    2.272135] PKEY: ==>public_key_verify_signature()
> > [    2.272165] CAAM rsa init start
> > [    2.272180] CAAM rsa init done
> > [    2.272191] caam_rsa_pub_key: free old key in ctx
> > [    2.272201] caam_rsa_pub_key: write rsa_key->e
> > [    2.272210] caam_rsa_pub_key: write rsa_key->n
> > [    2.272220] start rsassa_pkcs1_verify
> > [    2.272228] slen: 256
> > [    2.272238] child_req address: 1d64b62a full size: 64 + 48 + 256 = 368
> > [    2.272274] out_buf1:00000000: 00000000 00000000 00000000 00000000  ................
> > [    2.272298] out_buf1:00000010: 00000000 00000000 00000000 00000000  ................
> > [    2.272322] SRC BUF in out_buf1 CRC: 969ee858
> > [    2.272335] start caam_rsa_enc
> > [    2.272352] key:00000000: cf60a600 cf4d1240 00000000 00000000  ..`.@.M.........
> > [    2.272377] key:00000010: 00000000 00000000 00000000 00000000  ................
> > [    2.272413] edesc:00000000: 00000001 00000001 00000000 00000000  ................
> > [    2.272438] edesc:00000010: 00000000 00000000 00000000 cf533d6c  ............l=S.
> > [    2.272466] req:00000000: 00000000 00000000 c02e2f68 d083dcb4  ........h/......
> > [    2.272491] req:00000010: cf60a540 00000200 d083dc94 d083dca4  @.`.............
> > [    2.272509] CAAM: calling caam_jr_enqueue
> > [    2.272524] key:00000000: cf60a600 cf4d1240 00000000 00000000  ..`.@.M.........
> > [    2.272546] key:00000010: 00000000 00000000 00000000 00000000  ................
> > [    2.277444] CAAM: completion callback
> > [    2.424765] OUT BUF in out_buf2 CRC: fd0eef11
> > [    2.424799] out_buf2:00000000: 00000000 00000000 00000000 00000000  ................
> > [    2.424827] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424853] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424878] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424902] out_buf2:00000040: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424926] out_buf2:00000050: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424949] out_buf2:00000060: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424973] out_buf2:00000070: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424996] out_buf2:00000080: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.425020] out_buf2:00000090: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.425043] out_buf2:000000a0: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.425068] out_buf2:000000b0: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.425095] out_buf2:000000c0: ffffffff ffffffff ffffffff 30313000  .............010
> > [    2.425123] out_buf2:000000d0: 6009060d 65014886 01020403 20040005  ...`.H.e....... 
> > [    2.425148] out_buf2:000000e0: 6155a84e 7aa089cb 7540e613 f28b9a30  N.Ua...z..@u0...
> > [    2.425172] out_buf2:000000f0: 1e98ec34 cecb0e0f 9ee8951a ad8baec3  4...............
> 
> There's an endianness issue here:  30313000 is the zero byte prescribed
> by EMSA-PKCS1-v1_5 ("in_buf[ps_end] = 0x00;" in rsassa_pkcs1_sign()),
> followed by the first three bytes of hash_prefix_sha256[] in reverse order.
> 
> Then 6009060d are the next four bytes of hash_prefix_sha256[], again
> in reverse order.  And so on until 20040005, which are the last four
> bytes of the prefix in reverse order.
> 
> How are you generating that hexdump?  What's the CPU's endianness?
> Is the caam RSA accelerator using a different endianness?


imx6ul is armv7, little endian byte order and the following returns 1 which supports that:
echo -n I | od -o | head -n1 | cut -f2 -d" " | cut -c6

I always print the hex dump in the following way (here "out_buf" at line
https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L247 )
print_hex_dump(KERN_ERR, "out_buf1:", DUMP_PREFIX_OFFSET, 16, 4, out_buf, 32, true);

I don't find anything about imx6ul's CAAM internal endianness in the ref.man.
and can't say much about that.

I simply run mainline, CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n and "crypto: rsassa-pkcs1 - Copy source data for SG list"
reverted. 

Again, with this revert, the problem seems to be the same, only that the data that rsassa_pkcs1_verify() is
starting to check here https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L266 is still
"old" but now zeroes, not the input-data, thus failing with -EBADMSG instead of -EINVAL.

My feeling is that endianness is not the issue here. I see what you mean, kind of, but let's look at a success-case.
"out_buf" printed here: https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L264

[    3.863485] out_buf2:00000000: ffff0100 ffffffff ffffffff ffffffff  ................
[    3.863516] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff  ................
[    3.863542] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff  ................
[    3.863567] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff  ................

so out_buf[0] is 0x00, out_buf[1] is 0x01, "seeking" forward until !0xff, all succeeds in the lines below.

and sorry for my late response here. I'd be *very* happy to test any ideas!

                               martin

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Lukas Wunner]

On Tue, Mar 10, 2026 at 08:57:36AM +0000, Kepplinger-Novakovic Martin wrote:
> Am Donnerstag, dem 26.02.2026 um 14:27 +0100 schrieb Lukas Wunner:
> > There's an endianness issue here:  30313000 is the zero byte prescribed
> > by EMSA-PKCS1-v1_5 ("in_buf[ps_end] = 0x00;" in rsassa_pkcs1_sign()),
> > followed by the first three bytes of hash_prefix_sha256[] in reverse order.
> > 
> > Then 6009060d are the next four bytes of hash_prefix_sha256[], again
> > in reverse order.  And so on until 20040005, which are the last four
> > bytes of the prefix in reverse order.
> > 
> > How are you generating that hexdump?  What's the CPU's endianness?
> > Is the caam RSA accelerator using a different endianness?
> 
> imx6ul is armv7, little endian byte order and the following returns 1
> which supports that:
> echo -n I | od -o | head -n1 | cut -f2 -d" " | cut -c6

Please double-check whether your .config enables CONFIG_CPU_BIG_ENDIAN
or CONFIG_CPU_LITTLE_ENDIAN, just to cover all bases.

> I always print the hex dump in the following way (here "out_buf" at line
> https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L247 )
> print_hex_dump(KERN_ERR, "out_buf1:", DUMP_PREFIX_OFFSET, 16, 4, out_buf, 32, true);

Please use 1 instead of 4 as 5th parameter of print_hex_dump().
Using 4 only makes sense if the memory location you want to dump
contains 32-bit values.  That's not the case here as the signature
is a bytestream.

I guess if you use 4, print_hex_dump() dumps the 32-bit values
in big endian order for human readability, but that's confusing
if the memory location actually contains a bytestream.

> Again, with this revert, the problem seems to be the same, only that
> the data that rsassa_pkcs1_verify() is starting to check here
> https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L266
> is still "old" but now zeroes, not the input-data, thus failing with
> -EBADMSG instead of -EINVAL.

Actually the "out_buf2" that you've included in this message...

https://lore.kernel.org/all/1a65ac92579fadb4bfc76b32a3a4f1c6df022801.camel@ginzinger.com/

...looks like a valid verified (i.e. encrypted) signature,
the only thing that's weird is the endianness issue and
that there's a bunch of zero bytes at the beginning of
the buffer.

Please re-generate the hexdump of "out_buf" after the call
to crypto_wait_req(), once with a stock kernel and once with
8552cb04e083 reverted, and use 1 as 5th argument to
print_hex_dump().

Thanks,

Lukas

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Kepplinger-Novakovic Martin]

Am Freitag, dem 13.03.2026 um 10:18 +0100 schrieb Lukas Wunner:
> On Tue, Mar 10, 2026 at 08:57:36AM +0000, Kepplinger-Novakovic Martin wrote:
> > Am Donnerstag, dem 26.02.2026 um 14:27 +0100 schrieb Lukas Wunner:
> > > There's an endianness issue here:  30313000 is the zero byte prescribed
> > > by EMSA-PKCS1-v1_5 ("in_buf[ps_end] = 0x00;" in rsassa_pkcs1_sign()),
> > > followed by the first three bytes of hash_prefix_sha256[] in reverse order.
> > >
> > > Then 6009060d are the next four bytes of hash_prefix_sha256[], again
> > > in reverse order.  And so on until 20040005, which are the last four
> > > bytes of the prefix in reverse order.
> > >
> > > How are you generating that hexdump?  What's the CPU's endianness?
> > > Is the caam RSA accelerator using a different endianness?
> >
> > imx6ul is armv7, little endian byte order and the following returns 1
> > which supports that:
> > echo -n I | od -o | head -n1 | cut -f2 -d" " | cut -c6
>
> Please double-check whether your .config enables CONFIG_CPU_BIG_ENDIAN
> or CONFIG_CPU_LITTLE_ENDIAN, just to cover all bases.

CONFIG_CPU_LITTLE_ENDIAN is enabled, not CPU_BIG_ENDIAN. thanks for the hint.

>
> > I always print the hex dump in the following way (here "out_buf" at line
> > https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L247
> >  )
> > print_hex_dump(KERN_ERR, "out_buf1:", DUMP_PREFIX_OFFSET, 16, 4, out_buf, 32, true);
>
> Please use 1 instead of 4 as 5th parameter of print_hex_dump().
> Using 4 only makes sense if the memory location you want to dump
> contains 32-bit values.  That's not the case here as the signature
> is a bytestream.
>
> I guess if you use 4, print_hex_dump() dumps the 32-bit values
> in big endian order for human readability, but that's confusing
> if the memory location actually contains a bytestream.

true, thanks, there would have been other debug examples I should have looked at. Changed this, see below.

>
> > Again, with this revert, the problem seems to be the same, only that
> > the data that rsassa_pkcs1_verify() is starting to check here
> > https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L266
> > is still "old" but now zeroes, not the input-data, thus failing with
> > -EBADMSG instead of -EINVAL.
>
> Actually the "out_buf2" that you've included in this message...
>
> https://lore.kernel.org/all/1a65ac92579fadb4bfc76b32a3a4f1c6df022801.camel@ginzinger.com/
>
> ...looks like a valid verified (i.e. encrypted) signature,
> the only thing that's weird is the endianness issue and
> that there's a bunch of zero bytes at the beginning of
> the buffer.

yes. As mentioned in this thread's very first email, always only the first 16 bytes of out_buf stay old, thus wrong, after crypto_req_done(). With
commit 8552cb04e083 reverted, it's 16 bytes of zeroes, but otherwise no change, see the dumps below.

>
> Please re-generate the hexdump of "out_buf" after the call
> to crypto_wait_req(), once with a stock kernel and once with
> 8552cb04e083 reverted, and use 1 as 5th argument to
> print_hex_dump().

so, nothing new, but a failure-case dumped as byte-stream running a stock kernel (the "usual" caam config without ahash api...)

[    2.339406] start rsassa_pkcs1_verify
[    2.339413] slen: 256
[    2.339425] child_req address: bc6b492f full size: 64 + 48 + 256 = 368
[    2.339455] out_buf1:00000000: 87 03 da f2 82 c2 dd af 7c 44 2f 86 d3 5f 4c 93  ........|D/.._L.
[    2.339476] out_buf1:00000010: 48 b9 fe 07 17 bb 21 f7 25 23 4e aa 22 0c 16 b9  H.....!.%#N."...
[    2.339500] SRC BUF in out_buf1 CRC: 60791b87
[    2.339513] start caam_rsa_enc
[    2.339523] key:00000000: 00 c8 62 cf 40 74 62 cf 00 00 00 00 00 00 00 00  ..b.@tb.........
[    2.339540] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[    2.339567] edesc:00000000: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00  ................
[    2.339584] edesc:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 ec 7e 49 cf  .............~I.
[    2.722145] req:00000000: 00 00 00 00 00 00 00 00 88 22 2e c0 a4 dc 83 d0  ........."......
[    2.722178] req:00000010: 40 c7 62 cf 00 02 00 00 b4 dc 83 d0 b4 dc 83 d0  @.b.............
[    2.722200] CAAM: calling caam_jr_enqueue
[    2.722210] key:00000000: 00 c8 62 cf 40 74 62 cf 00 00 00 00 00 00 00 00  ..b.@tb.........
[    2.722227] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[    2.727096] CAAM: completion callback
[    2.742603] OUT BUF in out_buf2 CRC: 12298efd
[    2.742622] out_buf2:00000000: 87 03 da f2 82 c2 dd af 7c 44 2f 86 d3 5f 4c 93  ........|D/.._L.
[    2.742643] out_buf2:00000010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742659] out_buf2:00000020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742675] out_buf2:00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742691] out_buf2:00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742706] out_buf2:00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742722] out_buf2:00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742737] out_buf2:00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742753] out_buf2:00000080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742769] out_buf2:00000090: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742785] out_buf2:000000a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742801] out_buf2:000000b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.742818] out_buf2:000000c0: ff ff ff ff ff ff ff ff ff ff ff ff 00 30 31 30  .............010
[    2.742833] out_buf2:000000d0: 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20  ...`.H.e.......
[    2.742850] out_buf2:000000e0: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e  i..Y.K.......).>
[    2.742866] out_buf2:000000f0: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa  %[7.....U1..wq..
[    2.742883] digest (in):00000000: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e  i..Y.K.......).>
[    2.742900] digest (in):00000010: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa  %[7.....U1..wq..
[    2.742915] Encrypted value had no leading 0 byte.
[    2.742927] PKEY: crypto_sig_verify error: -22
[    2.742947] PKEY: <==public_key_verify_signature() = -22
[    2.742961] X.509: public_key_verify_signature error: -22
[    2.742970] X.509: <==x509_check_for_self_signed() = -22
[    2.742983] X.509: x509_check_for_self_signed failed: -22
[    2.743000] X.509: x509_cert_parse failed: -22
[    2.743009] Parser recognised the format (ret -22)
[    2.743022] <==asymmetric_key_preparse() = -22
[    2.743035] preparse: -22
[    2.743643] Problem loading in-kernel X.509 certificate (-22)


a failure-case dumped running 8552cb04e083 reverted:


[    2.307228] start rsassa_pkcs1_verify
[    2.307236] slen: 256
[    2.307249] child_req address: f56af4d7 full size: 64 + 48 + 256 = 368
[    2.307285] out_buf1:00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[    2.307306] out_buf1:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[    2.307330] SRC BUF in out_buf1 CRC: 3e37f250
[    2.307346] start caam_rsa_enc
[    2.307355] key:00000000: 00 f8 5f cf 80 a3 5f cf 00 00 00 00 00 00 00 00  .._..._.........
[    2.307372] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[    2.307401] edesc:00000000: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00  ................
[    2.307419] edesc:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 6c 7e 3d c1  ............l~=.
[    2.307440] req:00000000: 00 00 00 00 00 00 00 00 88 32 2e c0 b4 dc 83 d0  .........2......
[    2.307458] req:00000010: 40 f7 5f cf 00 02 00 00 94 dc 83 d0 a4 dc 83 d0  @._.............
[    2.307476] CAAM: calling caam_jr_enqueue
[    2.307485] key:00000000: 00 f8 5f cf 80 a3 5f cf 00 00 00 00 00 00 00 00  .._..._.........
[    2.307502] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[    2.308771] CAAM: completion callback
[    2.308840] OUT BUF in out_buf2 CRC: 86440841
[    2.308860] out_buf2:00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[    2.308881] out_buf2:00000010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.308898] out_buf2:00000020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.308914] out_buf2:00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.308930] out_buf2:00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.308947] out_buf2:00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.308963] out_buf2:00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.308980] out_buf2:00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.308997] out_buf2:00000080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.309013] out_buf2:00000090: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.309031] out_buf2:000000a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.309048] out_buf2:000000b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[    2.309064] out_buf2:000000c0: ff ff ff ff ff ff ff ff ff ff ff ff 00 30 31 30  .............010
[    2.309082] out_buf2:000000d0: 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20  ...`.H.e.......
[    2.309099] out_buf2:000000e0: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e  i..Y.K.......).>
[    2.309116] out_buf2:000000f0: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa  %[7.....U1..wq..
[    2.309133] digest (in):00000000: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e  i..Y.K.......).>
[    2.309150] digest (in):00000010: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa  %[7.....U1..wq..
[    2.309168] PKEY: crypto_sig_verify error: -74
[    2.309188] PKEY: <==public_key_verify_signature() = -74
[    2.309201] X.509: public_key_verify_signature error: -74
[    2.309212] X.509: <==x509_check_for_self_signed() = -74
[    2.309225] X.509: x509_check_for_self_signed failed: -74
[    2.309240] X.509: x509_cert_parse failed: -74
[    2.309252] <==asymmetric_key_preparse() = -74
[    2.309268] preparse: -74
[    2.413845] Problem loading in-kernel X.509 certificate (-74)


thank you very much for having a look! I'd be happy to test any thoughts you might have.

                                    martin

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Herbert Xu]

On Thu, Feb 26, 2026 at 08:17:18AM +0100, Lukas Wunner wrote:
>
> That's fine for the RSA software implementation in crypto/rsa.c but
> I could very well imagine it causes problems with an RSA accelerator,
> particularly because rsa_edesc_alloc() in drivers/crypto/caam/caampkc.c
> now maps the same buffer with DMA_TO_DEVICE and then DMA_FROM_DEVICE.

That's definitely not good.  It needs to handle in-place encryption
by using DMA_BIDIRECTIONAL.  If the hardware is not capable of that
then an extra copy must be performed by the driver.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Lukas Wunner]

On Sat, Mar 07, 2026 at 02:32:53PM +0900, Herbert Xu wrote:
> On Thu, Feb 26, 2026 at 08:17:18AM +0100, Lukas Wunner wrote:
> > That's fine for the RSA software implementation in crypto/rsa.c but
> > I could very well imagine it causes problems with an RSA accelerator,
> > particularly because rsa_edesc_alloc() in drivers/crypto/caam/caampkc.c
> > now maps the same buffer with DMA_TO_DEVICE and then DMA_FROM_DEVICE.
> 
> That's definitely not good.  It needs to handle in-place encryption
> by using DMA_BIDIRECTIONAL.  If the hardware is not capable of that
> then an extra copy must be performed by the driver.

Okay.  However in the latest hexdump provided by Martin (on Feb 26),
the data returned from the caam RSA accelerator is byte-swapped.
i.MX6 does seem to be capable of big endian mode, but I guess
peripherals still use little endian and so it would be necessary
to postprocess (i.e. byte-swap) the buffer in caampkc.c if the CPU
is running in big endian mode.

It's unclear to me whether the byte swap issue is the only problem
or whether there's a DMA issue on top.  Martin hasn't responded to
my latest e-mail yet and without someone willing to answer questions
or test patches, I can't help any further here as I don't have such a
device myself.

Thanks,

Lukas