[bug report] crypto: rsassa-pkcs1 - Migrate to sig_alg backend

[Dan Carpenter <error27@gmail.com>]

Hello Lukas Wunner,

Commit 1e562deacecc ("crypto: rsassa-pkcs1 - Migrate to sig_alg
backend") from Sep 10, 2024 (linux-next), leads to the following
Smatch static checker warning:

	crypto/rsassa-pkcs1.c:193 rsassa_pkcs1_sign()
	warn: check that subtract can't underflow 'ps_end - 1' '0-4294967293'

crypto/rsassa-pkcs1.c
    158 static int rsassa_pkcs1_sign(struct crypto_sig *tfm,
    159                              const void *src, unsigned int slen,
    160                              void *dst, unsigned int dlen)
    161 {
    162         struct sig_instance *inst = sig_alg_instance(tfm);
    163         struct rsassa_pkcs1_inst_ctx *ictx = sig_instance_ctx(inst);
    164         const struct hash_prefix *hash_prefix = ictx->hash_prefix;
    165         struct rsassa_pkcs1_ctx *ctx = crypto_sig_ctx(tfm);
    166         unsigned int pad_len;
    167         unsigned int ps_end;
    168         unsigned int len;
    169         u8 *in_buf;
    170         int err;
    171 
    172         if (!ctx->key_size)

Could this be a check for if (ctx->key_size < 11) instead?

    173                 return -EINVAL;
    174 
    175         if (dlen < ctx->key_size)
    176                 return -EOVERFLOW;
    177 
    178         if (rsassa_pkcs1_invalid_hash_len(slen, hash_prefix))
    179                 return -EINVAL;
    180 
    181         if (slen + hash_prefix->size > ctx->key_size - 11)
    182                 return -EOVERFLOW;
    183 
    184         pad_len = ctx->key_size - slen - hash_prefix->size - 1;
    185 
    186         /* RFC 8017 sec 8.2.1 step 1 - EMSA-PKCS1-v1_5 encoding generation */
    187         in_buf = dst;
    188         memmove(in_buf + pad_len + hash_prefix->size, src, slen);
    189         memcpy(in_buf + pad_len, hash_prefix->data, hash_prefix->size);
    190 
    191         ps_end = pad_len - 1;
    192         in_buf[0] = 0x01;
--> 193         memset(in_buf + 1, 0xff, ps_end - 1);

Smatch thinks ps_end could be zero.

    194         in_buf[ps_end] = 0x00;
    195 
    196 

This email is a free service from the Smatch-CI project [smatch.sf.net].

regards,
dan carpenter