From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Rini Date: Wed, 8 Jul 2020 20:24:27 -0400 Subject: [PATCH v7 1/1] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop() In-Reply-To: <20200707205726.45988-1-xypron.glpk@gmx.de> References: <20200707205726.45988-1-xypron.glpk@gmx.de> Message-ID: <20200709002427.GS1306@bill-the-cat> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: u-boot@lists.denx.de On Tue, Jul 07, 2020 at 10:57:26PM +0200, Heinrich Schuchardt wrote: > From: Heiko Stuebner > > When calculating rrtmp/rr rsa_gen_key_prop() tries to make > (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and > (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[] > with rlen being num_bits * 2 > > On a 4096bit key this comes down to to 257 uint32_t elements > in rr and 256 elements in rrtmp but with the current allocation > rr and rrtmp only have 129 uint32_t elements. > > On 2048bit keys this works by chance as the defined max_rsa_size=4096 > allocates a suitable number of elements, but with an actual 4096bit key > this results in other memory parts getting overwritten. > > So as suggested by Heinrich Schuchardt just use the actual bit-size > of the key as base for the size calculation, in turn making the code > compatible to any future keysizes. > > Suggested-by: Heinrich Schuchardt > Signed-off-by: Heiko Stuebner > Reviewed-by: Simon Glass > > rrtmp needs 2 + (((*prop)->num_bits * 2) >> 5) array elements. > > Reviewed-by: Heinrich Schuchardt Applied to u-boot/master, thanks! -- Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: