[PATCH 0/3] mtd: rawnand: sunxi: harden protected OOB handling

[PATCH 0/3] mtd: rawnand: sunxi: harden protected OOB handling

[James Hilliard]

The H6/H616 variable protected-OOB implementation has three related
failure cases around its per-step user-data lengths.

First, propagate failures while allocating the per-step length array.
Then reserve an encodable four-byte first section for the bad block
marker and reject layouts which cannot provide it. Finally, replace the
per-write heap allocation used for BBM compensation with the
controller's bounded 32-byte stack buffer.

These fixes keep the calculated, advertised and programmed OOB layouts
consistent and prevent zero-sized sections from reaching the BBM path.

Assisted-by: OpenAI Codex (gpt-5.6-sol, max)
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
James Hilliard (3):
      mtd: rawnand: sunxi: propagate user-data allocation errors
      mtd: rawnand: sunxi: reserve a full user-data word for BBM
      mtd: rawnand: sunxi: use a stack buffer for BBM randomization

 drivers/mtd/nand/raw/sunxi_nand.c | 32 +++++++++++++++++++++-----------
 1 file changed, 21 insertions(+), 11 deletions(-)
---
base-commit: 15a3cbce32994141252bb4ecfe3ff3a5d22d0b4f
change-id: 20260810-sunxi-nand-protected-oob-fixes-15db4368a8ae

Best regards,
--  
James Hilliard <james.hilliard1@gmail.com>

[PATCH 1/3] mtd: rawnand: sunxi: propagate user-data allocation errors

[James Hilliard]

sunxi_nfc_maximize_user_data() returns -ENOMEM when its per-step
length array cannot be allocated, but the caller ignores that error.
The NULL array then makes sunxi_nfc_user_data_sz() report the fixed
four-byte fallback, which is not the layout that the variable-length
setup calculated.

Return the setup error so an allocation failure cannot silently select
and validate a different OOB layout.

Fixes: 54dcd6aa69db ("mtd: rawnand: sunxi: introduce maximize variable user data length")
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 drivers/mtd/nand/raw/sunxi_nand.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/mtd/nand/raw/sunxi_nand.c b/drivers/mtd/nand/raw/sunxi_nand.c
index 45ccbce91551..108161fe10cb 100644
--- a/drivers/mtd/nand/raw/sunxi_nand.c
+++ b/drivers/mtd/nand/raw/sunxi_nand.c
@@ -2072,6 +2072,7 @@ static int sunxi_nand_hw_ecc_ctrl_init(struct nand_chip *nand,
 	int total_user_data_sz = 0;
 	int nsectors;
 	int ecc_mode;
+	int ret;
 	int i;
 
 	if (nanddev->ecc.user_conf.flags & NAND_ECC_MAXIMIZE_STRENGTH) {
@@ -2169,9 +2170,12 @@ static int sunxi_nand_hw_ecc_ctrl_init(struct nand_chip *nand,
 	 * The rationale for variable data length is to prioritize maximum ECC
 	 * strength, and then use the remaining space for user data.
 	 */
-	if (nfc->caps->reg_user_data_len)
-		sunxi_nfc_maximize_user_data(nand, mtd->oobsize, ecc->bytes,
-					     nsectors);
+	if (nfc->caps->reg_user_data_len) {
+		ret = sunxi_nfc_maximize_user_data(nand, mtd->oobsize,
+						   ecc->bytes, nsectors);
+		if (ret)
+			return ret;
+	}
 
 	if (total_user_data_sz == 0)
 		for (i = 0; i < nsectors; i++)

-- 
2.53.0

[PATCH 2/3] mtd: rawnand: sunxi: reserve a full user-data word for BBM

[James Hilliard]

H6/H616 protected user-data lengths are encoded in four-byte units,
but ECC maximization reserves only the two bad block marker bytes. A
strength leaving fewer than four bytes therefore selects a zero-byte
first user-data section.

The OOB layout then subtracts the two marker bytes from that unsigned
zero length, and the controller has no protected user-data word in which
to store the marker.

Reserve one complete user-data word while maximizing ECC, reject
configurations which produce no ECC sectors or a shorter first section,
and make the OOB iterator bounds-safe.

Fixes: 54dcd6aa69db ("mtd: rawnand: sunxi: introduce maximize variable user data length")
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 drivers/mtd/nand/raw/sunxi_nand.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/drivers/mtd/nand/raw/sunxi_nand.c b/drivers/mtd/nand/raw/sunxi_nand.c
index 108161fe10cb..01a0d0fa7b62 100644
--- a/drivers/mtd/nand/raw/sunxi_nand.c
+++ b/drivers/mtd/nand/raw/sunxi_nand.c
@@ -1991,7 +1991,7 @@ static int sunxi_nand_ooblayout_free(struct mtd_info *mtd, int section,
 	struct nand_chip *nand = mtd_to_nand(mtd);
 	struct nand_ecc_ctrl *ecc = &nand->ecc;
 	struct sunxi_nand_chip *sunxi_nand = to_sunxi_nand(nand);
-	unsigned int user_data_sz = sunxi_nfc_user_data_sz(sunxi_nand, section);
+	unsigned int user_data_sz;
 
 	/*
 	 * The controller does not provide access to OOB bytes
@@ -2000,6 +2000,8 @@ static int sunxi_nand_ooblayout_free(struct mtd_info *mtd, int section,
 	if (section >= ecc->steps)
 		return -ERANGE;
 
+	user_data_sz = sunxi_nfc_user_data_sz(sunxi_nand, section);
+
 	/*
 	 * The first 2 bytes are used for BB markers, hence we
 	 * only have user_data_sz - 2 bytes available in the first user data
@@ -2007,7 +2009,7 @@ static int sunxi_nand_ooblayout_free(struct mtd_info *mtd, int section,
 	 */
 	if (section == 0) {
 		oobregion->offset = 2;
-		oobregion->length = user_data_sz - 2;
+		oobregion->length = user_data_sz > 2 ? user_data_sz - 2 : 0;
 
 		return 0;
 	}
@@ -2041,6 +2043,9 @@ static int sunxi_nfc_maximize_user_data(struct nand_chip *nand, uint32_t oobsize
 	int remaining_bytes = oobsize - (ecc_bytes * nsectors);
 	int i, step;
 
+	if (nsectors <= 0)
+		return -EINVAL;
+
 	sunxi_nand->user_data_bytes = devm_kzalloc(nfc->dev, nsectors,
 						   GFP_KERNEL);
 	if (!sunxi_nand->user_data_bytes)
@@ -2056,6 +2061,8 @@ static int sunxi_nfc_maximize_user_data(struct nand_chip *nand, uint32_t oobsize
 		if (sunxi_nand->user_data_bytes[step] == 0)
 			break;
 	}
+	if (sunxi_nand->user_data_bytes[0] < USER_DATA_SZ)
+		return -EINVAL;
 
 	return 0;
 }
@@ -2103,10 +2110,10 @@ static int sunxi_nand_hw_ecc_ctrl_init(struct nand_chip *nand,
 			bytes -= total_user_data_sz;
 		} else {
 			/*
-			 * remove at least the BBM size before computing the
-			 * max ECC
+			 * User-data lengths are encoded in four-byte units. Reserve
+			 * the first word because it contains the two BBM bytes.
 			 */
-			bytes -= 2;
+			bytes -= USER_DATA_SZ;
 		}
 
 		/*

-- 
2.53.0

[PATCH 3/3] mtd: rawnand: sunxi: use a stack buffer for BBM randomization

[James Hilliard]

Variable protected user-data lengths replaced the original four-byte
stack buffer with a heap allocation before compensating the bad block
marker. The allocation is unchecked, and a zero length returns
ZERO_SIZE_PTR which sunxi_nfc_randomize_bbm() dereferences.

The controller supports at most 32 protected user-data bytes per ECC
step, so use a zero-initialized buffer of that bounded size. This
removes the allocation failure path and keeps zero-sized invalid layouts
from dereferencing ZERO_SIZE_PTR.

Fixes: 54dcd6aa69db ("mtd: rawnand: sunxi: introduce maximize variable user data length")
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 drivers/mtd/nand/raw/sunxi_nand.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/mtd/nand/raw/sunxi_nand.c b/drivers/mtd/nand/raw/sunxi_nand.c
index 01a0d0fa7b62..83666dd6cb2a 100644
--- a/drivers/mtd/nand/raw/sunxi_nand.c
+++ b/drivers/mtd/nand/raw/sunxi_nand.c
@@ -222,6 +222,7 @@
  * USER_DATA_LEN registers.
  */
 #define USER_DATA_SZ 4
+#define SUNXI_NFC_MAX_USER_DATA_SZ 32
 
 /**
  * struct sunxi_nand_chip_sel - stores information related to NAND Chip Select
@@ -1003,11 +1004,10 @@ static void sunxi_nfc_hw_ecc_set_prot_oob_bytes(struct nand_chip *nand,
 	struct sunxi_nfc *nfc = to_sunxi_nfc(nand->controller);
 	struct sunxi_nand_chip *sunxi_nand = to_sunxi_nand(nand);
 	unsigned int user_data_sz = sunxi_nfc_user_data_sz(sunxi_nand, step);
-	u8 *user_data = NULL;
+	u8 user_data[SUNXI_NFC_MAX_USER_DATA_SZ] = {};
 
 	/* Randomize the Bad Block Marker. */
 	if (bbm && (nand->options & NAND_NEED_SCRAMBLING)) {
-		user_data = kmalloc(user_data_sz, GFP_KERNEL);
 		memcpy(user_data, oob, user_data_sz);
 		sunxi_nfc_randomize_bbm(nand, page, user_data);
 		oob = user_data;
@@ -1040,7 +1040,6 @@ static void sunxi_nfc_hw_ecc_set_prot_oob_bytes(struct nand_chip *nand,
 		}
 	}
 
-	kfree(user_data);
 }
 
 static void sunxi_nfc_hw_ecc_update_stats(struct nand_chip *nand,

-- 
2.53.0

Re: [PATCH 0/3] mtd: rawnand: sunxi: harden protected OOB handling

[Miquel Raynal]

On Tue, 11 Aug 2026 00:01:58 -0600, James Hilliard wrote:
> The H6/H616 variable protected-OOB implementation has three related
> failure cases around its per-step user-data lengths.
> 
> First, propagate failures while allocating the per-step length array.
> Then reserve an encodable four-byte first section for the bad block
> marker and reject layouts which cannot provide it. Finally, replace the
> per-write heap allocation used for BBM compensation with the
> controller's bounded 32-byte stack buffer.
> 
> [...]

Applied to nand/next, thanks!

[1/3] mtd: rawnand: sunxi: propagate user-data allocation errors
      commit: 8696502b0aa96bcbb1f1e78ae117726378823ae8
[2/3] mtd: rawnand: sunxi: reserve a full user-data word for BBM
      commit: bde8fd880d84627072c224729c818306c4b30d58
[3/3] mtd: rawnand: sunxi: use a stack buffer for BBM randomization
      commit: a8596bb7148136b9e83980900d508d0ef86507ef

Patche(s) should be available on mtd/linux.git and will be
part of the next PR (provided that no robot complains by then).

Kind regards,
Miquèl