[PATCH] crypto/authenc: don't return -EBUSY when enqueuing the hash request

Mikulas Patocka
2025-09-22T19:09:01Z

On Thu, 18 Sep 2025, Harald Freudenberger wrote:

> On 2025-09-11 17:58, Mikulas Patocka wrote:
> > On Thu, 11 Sep 2025, Ingo Franzki wrote:
> >
> > > >> So, it looks like a dm-crypt bug.
> > > >>
> > > >> Please, revert my patches and run the same test on a clean 6.17.0-rc5
> > > just
> > > >> to verify that the patches do not introduce the bug.
> > > >
> > > > With your patches reverted the combined mode fails the same way as with
> > > your patches.
> > > > So they did not introduce the bug.
> > >
> > > Mikulas, do you have any idea what could be causing this errors?
> > > Is it that dm-crypt is not properly dealing with async-only HMAC ciphers?
> > > Async-only encryption ciphers seem to work fine in dm-crypt, since LUKS
> > > with PAES (but no integrity) works fine, and PAES is an async-onky cipher.
> > > LUKS with sync-HMAC ciphers (e.g. clear key HMAC) also works fine, even in
> > > combination with PAES.
> >
> > Yes, I think that it's a problem with async HMAC. The bug is probably
> > either in dm-crypt or in the crypto library.
> >
> > Do you have some other (non-dm-crypt-related) workload that uses the
> > async authentication, so that we can determine whether the bug is in
> > dm-crypt or crypto?
> >
> > Otherwise, would it be possible to give us a virtual machine on the
> > mainframe to debug this issue?
> >
> > Mikulas
>
> So here is now an out-of-tree kernel module build which offers a pseudo
> phmac-sha256
> for testing and debugging purpose. In the end this is just a asynch (ahash)
> wrapper
> around the hmac-sha256 shash crypto subsystem implementation. It should
> compile and
> be usable on all platforms (s390, x64, arm, ...).
>
> I ran dm-integrity tests with this and all worked fine. Ingo ran dm-crypt
> tests
> where he combined aes-cbc encryption with phmac-sha256 integrity and saw hangs
> on cryptsetup open. He also reported that these issues are different to what
> he
> saw with the 'real' phmac in combination with aes encryption. A short glimpse
> gives
> me the impression that there is a job blocking the system's workqueue.
> However, I
> could not find any indication that the pseudo phmac is not working properly.
>
> For instructions on how to build and use the module see the README in the tgz
> archive.
>
> Thanks to all
> Harald Freudenberger

Hi

I analyzed the dm-crypt deadlock - in crypto_authenc_decrypt, there's
err = crypto_ahash_digest(ahreq);
if (err)
return err;
- here we get -EBUSY, that -EBUSY is propagated to dm-crypt. When dm-crypt
gets -EBUSY, it waits in "wait_for_completion(&ctx->restart);" in
crypt_convert.

dm-crypt wakes up when kcryptd_async_done gets -EINPROGRESS, but the
crypto API never calls kcryptd_async_done with -EINPROGRESS, so dm-crypt
deadlocks.

I've made this patch for the bug, it is tested only lightly, please review
it.

Mikulas

From: Mikulas Patocka <mpatocka@redhat.com>

When we return -EBUSY from encryption routines, the caller is supposed to
sleep until it receives -EINPROGRESS in the callback method.

However when using authenc with asynchronous hash, the hash function may
return -EBUSY. In this case, the crypto API never calls the caller with
-EINPROGRESS and it causes deadlock in dm-crypt.

Fix this by turning -EBUSY into -EINPROGRESS.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Cc: stable@vger.kernel.org

---
crypto/authenc.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)

Index: linux-2.6/crypto/authenc.c
===================================================================
--- linux-2.6.orig/crypto/authenc.c 2025-09-22 20:32:02.000000000 +0200
+++ linux-2.6/crypto/authenc.c 2025-09-22 20:35:38.000000000 +0200
@@ -146,8 +146,11 @@ static int crypto_authenc_genicv(struct
authenc_geniv_ahash_done, req);

err = crypto_ahash_digest(ahreq);
- if (err)
+ if (err) {
+ if (err == -EBUSY)
+ err = -EINPROGRESS;
return err;
+ }

scatterwalk_map_and_copy(hash, req->dst, req->assoclen + req->cryptlen,
crypto_aead_authsize(authenc), 1);
@@ -270,8 +273,11 @@ static int crypto_authenc_decrypt(struct
authenc_verify_ahash_done, req);

err = crypto_ahash_digest(ahreq);
- if (err)
+ if (err) {
+ if (err == -EBUSY)
+ err = -EINPROGRESS;
return err;
+ }

return crypto_authenc_decrypt_tail(req, aead_request_flags(req));
}