[PATCH 0/2] crypto: rsassa-pkcs1: fix undersized key handling

[PATCH 0/2] crypto: rsassa-pkcs1: fix undersized key handling

[Jérémy Jean]

PKCS#1 v1.5 requires an encoded message of at least 11 bytes. The
signing and verification functions do not reject smaller RSA modulus
sizes reported by the child implementation.

This series include two patches due to integer underflows:
* one in the signing function that can write OOB,
* one in the verification function that can read 1 byte OOB.

Jérémy Jean (2):
  crypto: rsassa-pkcs1: reject undersized keys when signing
  crypto: rsassa-pkcs1: reject undersized keys when verifying

 crypto/rsassa-pkcs1.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

-- 
2.47.3

[PATCH 1/2] crypto: rsassa-pkcs1: reject undersized keys when signing

[Jérémy Jean]

rsassa_pkcs1_sign() subtracts 11 from the unsigned key size before
checking that the key is large enough for PKCS#1 v1.5 padding:

	if (slen + hash_prefix->size > ctx->key_size - 11)
		return -EOVERFLOW;

If the RSA modulus is shorter than 11 bytes, the subtraction wraps.
With a one-byte key and hash=none, the padding memset() writes past
the output buffer.

KASAN reports:

    BUG: KASAN: slab-out-of-bounds in rsassa_pkcs1_sign+0x1ad/0x3b0
    Write of size 4294967294 at addr ...
    The buggy address is located 0 bytes to the right of
     allocated 1-byte region [...]

Reject keys shorter than the minimum encoded message size.

Fixes: 3d5b1ecdea6f ("crypto: rsa - RSA padding algorithm")
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
---
 crypto/rsassa-pkcs1.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
index 94fa5e9600e7..d0e4a885397f 100644
--- a/crypto/rsassa-pkcs1.c
+++ b/crypto/rsassa-pkcs1.c
@@ -169,7 +169,7 @@ static int rsassa_pkcs1_sign(struct crypto_sig *tfm,
 	u8 *in_buf;
 	int err;
 
-	if (!ctx->key_size)
+	if (ctx->key_size < 11)
 		return -EINVAL;
 
 	if (dlen < ctx->key_size)
-- 
2.47.3

Re: [PATCH 1/2] crypto: rsassa-pkcs1: reject undersized keys when signing

[Sudhakar Kuppusamy]

> On 26 Aug 2026, at 4:07 PM, Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr> wrote:
> 
> rsassa_pkcs1_sign() subtracts 11 from the unsigned key size before
> checking that the key is large enough for PKCS#1 v1.5 padding:
> 
> if (slen + hash_prefix->size > ctx->key_size - 11)
> return -EOVERFLOW;
> 
> If the RSA modulus is shorter than 11 bytes, the subtraction wraps.
> With a one-byte key and hash=none, the padding memset() writes past
> the output buffer.
> 
> KASAN reports:
> 
>    BUG: KASAN: slab-out-of-bounds in rsassa_pkcs1_sign+0x1ad/0x3b0
>    Write of size 4294967294 at addr ...
>    The buggy address is located 0 bytes to the right of
>     allocated 1-byte region [...]
> 
> Reject keys shorter than the minimum encoded message size.
> 
> Fixes: 3d5b1ecdea6f ("crypto: rsa - RSA padding algorithm")
> Assisted-by: Codex:gpt-5
> Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>


Reviewed-by: Sudhakar Kuppusamy <sudhakar@linux.ibm.com>


Thanks,
Sudhakar

> ---
> crypto/rsassa-pkcs1.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
> index 94fa5e9600e7..d0e4a885397f 100644
> --- a/crypto/rsassa-pkcs1.c
> +++ b/crypto/rsassa-pkcs1.c
> @@ -169,7 +169,7 @@ static int rsassa_pkcs1_sign(struct crypto_sig *tfm,
> u8 *in_buf;
> int err;
> 
> - if (!ctx->key_size)
> + if (ctx->key_size < 11)
> return -EINVAL;
> 
> if (dlen < ctx->key_size)
> -- 
> 2.47.3
> 
> 

[PATCH 2/2] crypto: rsassa-pkcs1: reject undersized keys when verifying

[Jérémy Jean]

rsassa_pkcs1_verify() accepts any nonzero key size.

With a one-byte key and a zero RSA result, the leading-zero handling
decrements dst_len to zero and advances out_buf past the allocation.
The next out_buf[0] access reads out of bounds.

KASAN reports:

    BUG: KASAN: slab-out-of-bounds in rsassa_pkcs1_verify+0x79d/0x900
    Read of size 1 at addr ...
    The buggy address is located 0 bytes to the right of
     allocated 73-byte region [...]

Reject keys shorter than the minimum PKCS#1 v1.5 encoded message size.

Fixes: 3d5b1ecdea6f ("crypto: rsa - RSA padding algorithm")
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
---
 crypto/rsassa-pkcs1.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
index d0e4a885397f..39137476ce81 100644
--- a/crypto/rsassa-pkcs1.c
+++ b/crypto/rsassa-pkcs1.c
@@ -231,7 +231,7 @@ static int rsassa_pkcs1_verify(struct crypto_sig *tfm,
 	int err;
 
 	/* RFC 8017 sec 8.2.2 step 1 - length checking */
-	if (!ctx->key_size ||
+	if (ctx->key_size < 11 ||
 	    slen != ctx->key_size ||
 	    rsassa_pkcs1_invalid_hash_len(dlen, hash_prefix))
 		return -EINVAL;
-- 
2.47.3

Re: [PATCH 2/2] crypto: rsassa-pkcs1: reject undersized keys when verifying

[Sudhakar Kuppusamy]

> On 26 Aug 2026, at 4:07 PM, Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr> wrote:
> 
> rsassa_pkcs1_verify() accepts any nonzero key size.
> 
> With a one-byte key and a zero RSA result, the leading-zero handling
> decrements dst_len to zero and advances out_buf past the allocation.
> The next out_buf[0] access reads out of bounds.
> 
> KASAN reports:
> 
>    BUG: KASAN: slab-out-of-bounds in rsassa_pkcs1_verify+0x79d/0x900
>    Read of size 1 at addr ...
>    The buggy address is located 0 bytes to the right of
>     allocated 73-byte region [...]
> 
> Reject keys shorter than the minimum PKCS#1 v1.5 encoded message size.
> 
> Fixes: 3d5b1ecdea6f ("crypto: rsa - RSA padding algorithm")
> Assisted-by: Codex:gpt-5
> Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>


Reviewed-by: Sudhakar Kuppusamy <sudhakar@linux.ibm.com>


Thanks,
Sudhakar

> ---
> crypto/rsassa-pkcs1.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
> index d0e4a885397f..39137476ce81 100644
> --- a/crypto/rsassa-pkcs1.c
> +++ b/crypto/rsassa-pkcs1.c
> @@ -231,7 +231,7 @@ static int rsassa_pkcs1_verify(struct crypto_sig *tfm,
> int err;
> 
> /* RFC 8017 sec 8.2.2 step 1 - length checking */
> - if (!ctx->key_size ||
> + if (ctx->key_size < 11 ||
>    slen != ctx->key_size ||
>    rsassa_pkcs1_invalid_hash_len(dlen, hash_prefix))
> return -EINVAL;
> -- 
> 2.47.3
> 
>