Re: [PATCH 1/4] lib/crypto: Add ML-DSA verification support

[Eric Biggers <ebiggers@kernel.org>]

On Fri, Nov 21, 2025 at 09:39:31PM +0000, David Howells wrote:
> Eric Biggers <ebiggers@kernel.org> wrote:
> 
> > > > +	if (memcmp(ws->ctildeprime, ctilde, params->ctilde_len) != 0)
> > > > +		return -EBADMSG;
> > > 
> > > Actually, this should return -EKEYREJECTED, not -EBADMSG.
> > 
> > Who/what decided that?
> 
> I did.  When I added RSA support in 2012 for module signing.  Note that it
> was originally added as part of crypto/asymmetric_keys/ and was not covered by
> a crypto API.  The RSA code has since been moved to crypto/ and is now
> accessed through the crypto API, but it has retained this error code and this
> is also used by other public key algos.
> 
> > A lot of the crypto code uses -EBADMSG already.
> > crypto_aead uses it, for example.
> 
> ecdsa.c:60:	return -EKEYREJECTED;
> ecrdsa.c:111:		return -EKEYREJECTED;
> ecrdsa.c:139:		return -EKEYREJECTED;
> ecrdsa.c:239:		return -EKEYREJECTED;
> rsassa-pkcs1.c:293:		return -EKEYREJECTED;
> rsassa-pkcs1.c:295:		return -EKEYREJECTED;

crypto/aegis128-core.c:442:             return -EBADMSG;
crypto/aegis128-core.c:499:             return -EBADMSG;
crypto/algif_aead.c:313:                        if (err == -EIOCBQUEUED || err == -EBADMSG || !ret)
crypto/authenc.c:223:           return -EBADMSG;
crypto/authencesn.c:220:                return -EBADMSG;
crypto/ccm.c:336:                       err = -EBADMSG;
crypto/ccm.c:384:               return -EBADMSG;
crypto/chacha20poly1305.c:90:           return -EBADMSG;
crypto/dh.c:207:                                ret = -EBADMSG;
crypto/dh.c:221:                                ret = -EBADMSG;
crypto/dh.c:242:                ret = -EBADMSG;
crypto/ecdsa.c:37:              return -EBADMSG;
crypto/ecrdsa.c:101:            return -EBADMSG;
crypto/gcm.c:471:       return crypto_memneq(iauth_tag, auth_tag, authsize) ? -EBADMSG : 0;
crypto/krb5enc.c:259:           return -EBADMSG;
crypto/rsa.c:150:               ret = -EBADMSG;
crypto/rsa.c:189:               ret = -EBADMSG;
crypto/rsassa-pkcs1.c:275:              return -EBADMSG;
crypto/rsassa-pkcs1.c:282:              return -EBADMSG;
crypto/rsassa-pkcs1.c:286:              return -EBADMSG;
crypto/rsassa-pkcs1.c:288:              return -EBADMSG;
crypto/testmgr.c:90:     * algorithm might result in EINVAL rather than EBADMSG, due to other
crypto/testmgr.c:2179:      (err != vec->crypt_error && !(err == -EBADMSG && vec->novrfy))) {
crypto/testmgr.c:2183:              vec->crypt_error != 0 && vec->crypt_error != -EBADMSG)
crypto/testmgr.c:2184:                  sprintf(expected_error, "-EBADMSG or %d",
crypto/testmgr.c:2187:                  sprintf(expected_error, "-EBADMSG");
include/crypto/aead.h:37: * operation is that the caller should explicitly check for -EBADMSG of the
include/crypto/aead.h:39: * a breach in the integrity of the message. In essence, that -EBADMSG error
include/crypto/aead.h:375: * Return: 0 if the cipher operation was successful; -EBADMSG: The AEAD

That list actually includes the same three files that use -EKEYREJECTED.
It looks like if the signature verification fails "early" it's -EBADMSG,
whereas if it fails "late" it's -EKEYREJECTED?  I'm skeptical that
that's a meaningful difference.  And it's not like this is documented
either; crypto_sig_verify() just says "error code in case of error".

- Eric