[PATCH 0/2] lib/bch: fix undefined behavior from signed left-shifts

[PATCH 0/2] lib/bch: fix undefined behavior from signed left-shifts

[Josh Law]

Fix two instances of undefined behavior in lib/bch.c caused by
left-shifting signed integers into or past the sign bit.

While the kernel's -fno-strict-overflow flag prevents miscompilation
today, these are formally UB per C11 6.5.7p4 and trivial to fix.

Josh Law (2):
  lib/bch: fix signed left-shift undefined behavior
  lib/bch: fix signed shift overflow in build_mod8_tables

 lib/bch.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

-- 
2.34.1

[PATCH 1/2] lib/bch: fix signed left-shift undefined behavior

[Josh Law]

Use 1u instead of 1 to avoid undefined behavior when left-shifting
into the sign bit of a signed int. deg() can return up to 31, and
1 << 31 is UB per C11.

Signed-off-by: Josh Law <objecting@objecting.org>
---
 lib/bch.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/bch.c b/lib/bch.c
index 9561c0828802..ef733f08082f 100644
--- a/lib/bch.c
+++ b/lib/bch.c
@@ -392,7 +392,7 @@ static void compute_syndromes(struct bch_control *bch, uint32_t *ecc,
 			for (j = 0; j < 2*t; j += 2)
 				syn[j] ^= a_pow(bch, (j+1)*(i+s));
 
-			poly ^= (1 << i);
+			poly ^= (1u << i);
 		}
 	} while (s > 0);
 
@@ -612,7 +612,7 @@ static int find_poly_deg2_roots(struct bch_control *bch, struct gf_poly *poly,
 		while (v) {
 			i = deg(v);
 			r ^= bch->xi_tab[i];
-			v ^= (1 << i);
+			v ^= (1u << i);
 		}
 		/* verify root */
 		if ((gf_sqr(bch, r)^r) == u) {
-- 
2.34.1

[PATCH 2/2] lib/bch: fix signed shift overflow in build_mod8_tables

[Josh Law]

Cast loop variable to unsigned int before left-shifting to avoid
undefined behavior when i >= 128 and b == 3 (i << 24 overflows
signed int).

Signed-off-by: Josh Law <objecting@objecting.org>
---
 lib/bch.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bch.c b/lib/bch.c
index ef733f08082f..c991c71c4cbd 100644
--- a/lib/bch.c
+++ b/lib/bch.c
@@ -1116,7 +1116,7 @@ static void build_mod8_tables(struct bch_control *bch, const uint32_t *g)
 		for (b = 0; b < 4; b++) {
 			/* we want to compute (p(X).X^(8*b+deg(g))) mod g(X) */
 			tab = bch->mod8_tab + (b*256+i)*l;
-			data = i << (8*b);
+			data = (unsigned int)i << (8*b);
 			while (data) {
 				d = deg(data);
 				/* subtract X^d.g(X) from p(X).X^(8*b+deg(g)) */
-- 
2.34.1