[U-Boot] [PATCHv3 3/4] spl: Lightweight UBI and UBI fastmap support

[Heiko Schocher <hs@denx.de>]

Hello Ladislav,

Am 11.01.2016 um 00:17 schrieb Ladislav Michl:
> From: Thomas Gleixner <tglx@linutronix.de>
>
> Booting a payload out of NAND FLASH from the SPL is a crux today, as
> it requires hard partioned FLASH. Not a brilliant idea with the
> reliability of todays NAND FLASH chips.
>
> The upstream UBI + UBI fastmap implementation which is about to
> brought to u-boot is too heavy weight for SPLs as it provides way more
> functionality than needed for a SPL and does not even fit into the
> restricted SPL areas which are loaded from the SoC boot ROM.
>
> So this provides a fast and lightweight implementation of UBI scanning
> and UBI fastmap attach. The scan and logical to physical block mapping
> code is developed from scratch, while the fastmap implementation is
> lifted from the linux kernel source and stripped down to fit the SPL
> needs.
>
> The text foot print on the board which I used for development is:
>
> 6854	0	0	6854	1abd
> drivers/mtd/ubispl/built-in.o
>
> Attaching a NAND chip with 4096 physical eraseblocks (4 blocks are
> reserved for the SPL) takes:
>
> In full scan mode:      1172ms
> In fastmap mode:          95ms
>
> The code requires quite some storage. The largest and unknown part of
> it is the number of fastmap blocks to read. Therefor the data
> structure is not put into the BSS. The code requires a pointer to free
> memory handed in which is initialized by the UBI attach code itself.
>
> See doc/README.ubispl for further information on how to use it.
>
> This shares the ubi-media.h and crc32 implementation of drivers/mtd/ubi
> There is no way to share the fastmap code, as UBISPL only utilizes the
> slightly modified functions ubi_attach_fastmap() and ubi_scan_fastmap()
> from the original kernel ubi fastmap implementation.
>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Signed-off-by: Ladislav Michl <ladis@linux-mips.org>
> ---
>
> Changes in v2:
>    - fixes ubi_calc_fm_size to include also sizeof(struct ubi_fm_sb)
>    - dropped private copy of ubi-media.h
>    - ubi-wrapper.h now contains only needed definitions from ubi.h
>      and ubi-user.h
>    - used return values from errno.h
>
> Changes in v3:
>    - move vol_id check to ubi_scan_vid_hdr to verify it has meaningfull value
>      before testing ubi->toload
>    - fixed checkpatch errors except those present also in linux code

Thanks! Patch applies now ;-)

>
>   README                           |   4 +
>   doc/README.ubispl                | 143 ++++++
>   drivers/Makefile                 |   1 +
>   drivers/mtd/ubispl/Makefile      |   1 +
>   drivers/mtd/ubispl/ubi-wrapper.h | 106 +++++
>   drivers/mtd/ubispl/ubispl.c      | 926 +++++++++++++++++++++++++++++++++++++++
>   drivers/mtd/ubispl/ubispl.h      | 136 ++++++
>   include/ubispl.h                 |  92 ++++
>   8 files changed, 1409 insertions(+)
>   create mode 100644 doc/README.ubispl
>   create mode 100644 drivers/mtd/ubispl/Makefile
>   create mode 100644 drivers/mtd/ubispl/ubi-wrapper.h
>   create mode 100644 drivers/mtd/ubispl/ubispl.c
>   create mode 100644 drivers/mtd/ubispl/ubispl.h
>   create mode 100644 include/ubispl.h

Only 2 nitpicks
[...]
> diff --git a/drivers/mtd/ubispl/ubispl.c b/drivers/mtd/ubispl/ubispl.c
> new file mode 100644
> index 0000000..e2d5aff
> --- /dev/null
> +++ b/drivers/mtd/ubispl/ubispl.c
> @@ -0,0 +1,926 @@
[...]
> +static int ubi_attach_fastmap(struct ubi_scan_info *ubi,
> +			      struct ubi_attach_info *ai,
> +			      struct ubi_fastmap_layout *fm)
> +{
> +	struct ubi_fm_hdr *fmhdr;
> +	struct ubi_fm_scan_pool *fmpl1, *fmpl2;
> +	struct ubi_fm_ec *fmec;
> +	struct ubi_fm_volhdr *fmvhdr;
> +	struct ubi_fm_eba *fm_eba;
> +	int ret, i, j, pool_size, wl_pool_size;
> +	size_t fm_pos = 0, fm_size = ubi->fm_size;
> +	void *fm_raw = ubi->fm_buf;
> +
> +	memset(ubi->fm_used, 0, sizeof(ubi->fm_used));
> +
> +	fm_pos += sizeof(struct ubi_fm_sb);
> +	if (fm_pos >= fm_size)
> +		goto fail_bad;
> +
> +	fmhdr = (struct ubi_fm_hdr *)(fm_raw + fm_pos);
> +	fm_pos += sizeof(*fmhdr);
> +	if (fm_pos >= fm_size)
> +		goto fail_bad;
> +
> +	if (be32_to_cpu(fmhdr->magic) != UBI_FM_HDR_MAGIC) {
> +		ubi_err("bad fastmap header magic: 0x%x, expected: 0x%x",
> +			be32_to_cpu(fmhdr->magic), UBI_FM_HDR_MAGIC);
> +		goto fail_bad;
> +	}
> +
> +	fmpl1 = (struct ubi_fm_scan_pool *)(fm_raw + fm_pos);
> +	fm_pos += sizeof(*fmpl1);
> +	if (fm_pos >= fm_size)
> +		goto fail_bad;
> +	if (be32_to_cpu(fmpl1->magic) != UBI_FM_POOL_MAGIC) {
> +		ubi_err("bad fastmap pool magic: 0x%x, expected: 0x%x",
> +			be32_to_cpu(fmpl1->magic), UBI_FM_POOL_MAGIC);
> +		goto fail_bad;
> +	}
> +
> +	fmpl2 = (struct ubi_fm_scan_pool *)(fm_raw + fm_pos);
> +	fm_pos += sizeof(*fmpl2);
> +	if (fm_pos >= fm_size)
> +		goto fail_bad;
> +	if (be32_to_cpu(fmpl2->magic) != UBI_FM_POOL_MAGIC) {
> +		ubi_err("bad fastmap pool magic: 0x%x, expected: 0x%x",
> +			be32_to_cpu(fmpl2->magic), UBI_FM_POOL_MAGIC);
> +		goto fail_bad;
> +	}
> +
> +	pool_size = be16_to_cpu(fmpl1->size);
> +	wl_pool_size = be16_to_cpu(fmpl2->size);
> +	fm->max_pool_size = be16_to_cpu(fmpl1->max_size);
> +	fm->max_wl_pool_size = be16_to_cpu(fmpl2->max_size);
> +
> +	if (pool_size > UBI_FM_MAX_POOL_SIZE || pool_size < 0) {
> +		ubi_err("bad pool size: %i", pool_size);
> +		goto fail_bad;
> +	}
> +
> +	if (wl_pool_size > UBI_FM_MAX_POOL_SIZE || wl_pool_size < 0) {
> +		ubi_err("bad WL pool size: %i", wl_pool_size);
> +		goto fail_bad;
> +	}
> +
> +	if (fm->max_pool_size > UBI_FM_MAX_POOL_SIZE ||
> +	    fm->max_pool_size < 0) {
> +		ubi_err("bad maximal pool size: %i", fm->max_pool_size);
> +		goto fail_bad;
> +	}
> +
> +	if (fm->max_wl_pool_size > UBI_FM_MAX_POOL_SIZE ||
> +	    fm->max_wl_pool_size < 0) {
> +		ubi_err("bad maximal WL pool size: %i", fm->max_wl_pool_size);
> +		goto fail_bad;
> +	}
> +
> +	/* read EC values from free list */
> +	for (i = 0; i < be32_to_cpu(fmhdr->free_peb_count); i++) {
> +		fmec = (struct ubi_fm_ec *)(fm_raw + fm_pos);
> +		fm_pos += sizeof(*fmec);
> +		if (fm_pos >= fm_size)
> +			goto fail_bad;
> +	}
> +
> +	/* read EC values from used list */
> +	for (i = 0; i < be32_to_cpu(fmhdr->used_peb_count); i++) {
> +		fmec = (struct ubi_fm_ec *)(fm_raw + fm_pos);
> +		fm_pos += sizeof(*fmec);
> +		if (fm_pos >= fm_size)
> +			goto fail_bad;
> +
> +		generic_set_bit(be32_to_cpu(fmec->pnum), ubi->fm_used);
> +	}
> +
> +	/* read EC values from scrub list */
> +	for (i = 0; i < be32_to_cpu(fmhdr->scrub_peb_count); i++) {
> +		fmec = (struct ubi_fm_ec *)(fm_raw + fm_pos);
> +		fm_pos += sizeof(*fmec);
> +		if (fm_pos >= fm_size)
> +			goto fail_bad;
> +	}
> +
> +	/* read EC values from erase list */
> +	for (i = 0; i < be32_to_cpu(fmhdr->erase_peb_count); i++) {
> +		fmec = (struct ubi_fm_ec *)(fm_raw + fm_pos);
> +		fm_pos += sizeof(*fmec);
> +		if (fm_pos >= fm_size)
> +			goto fail_bad;
> +	}
> +
> +	/* Iterate over all volumes and read their EBA table */
> +	for (i = 0; i < be32_to_cpu(fmhdr->vol_count); i++) {
> +		u32 vol_id, vol_type, used, reserved;
> +
> +		fmvhdr = (struct ubi_fm_volhdr *)(fm_raw + fm_pos);
> +		fm_pos += sizeof(*fmvhdr);
> +		if (fm_pos >= fm_size)
> +			goto fail_bad;
> +
> +		if (be32_to_cpu(fmvhdr->magic) != UBI_FM_VHDR_MAGIC) {
> +			ubi_err("bad fastmap vol header magic: 0x%x, " \
> +				"expected: 0x%x",
> +				be32_to_cpu(fmvhdr->magic), UBI_FM_VHDR_MAGIC);
> +			goto fail_bad;
> +		}
> +
> +		vol_id = be32_to_cpu(fmvhdr->vol_id);
> +		vol_type = fmvhdr->vol_type;
> +		used = be32_to_cpu(fmvhdr->used_ebs);
> +
> +		fm_eba = (struct ubi_fm_eba *)(fm_raw + fm_pos);
> +		fm_pos += sizeof(*fm_eba);
> +		fm_pos += (sizeof(__be32) * be32_to_cpu(fm_eba->reserved_pebs));
> +		if (fm_pos >= fm_size)
> +			goto fail_bad;
> +
> +		if (be32_to_cpu(fm_eba->magic) != UBI_FM_EBA_MAGIC) {
> +			ubi_err("bad fastmap EBA header magic: 0x%x, " \
> +				"expected: 0x%x",
> +				be32_to_cpu(fm_eba->magic), UBI_FM_EBA_MAGIC);
> +			goto fail_bad;
> +		}
> +
> +		reserved = be32_to_cpu(fm_eba->reserved_pebs);
> +		ubi_dbg("FA: vol %u used %u res: %u", vol_id, used, reserved);
> +		for (j = 0; j < reserved; j++) {
> +			int pnum = be32_to_cpu(fm_eba->pnum[j]);
> +
> +			if ((int)be32_to_cpu(fm_eba->pnum[j]) < 0)
> +				continue;
> +
> +			if (!__test_and_clear_bit(pnum, ubi->fm_used))
> +				continue;
> +
> +			/*
> +			 * We only handle static volumes so used_ebs
> +			 * needs to be handed in. And we do not assign
> +			 * the reserved blocks
> +			 */
> +			if (j >= used)
> +				continue;
> +
> +			ret = assign_aeb_to_av(ubi, pnum, j, vol_id,
> +					       vol_type, used);
> +			if (!ret)
> +				continue;
> +
> +			/*
> +			 * Nasty: The fastmap claims that the volume
> +			 * has one block more than it, but that block
> +			 * is always empty and the other blocks have
> +			 * the correct number of total LEBs in the
> +			 * headers. Deal with it.
> +			 */
> +			if (ret != UBI_IO_FF && j != used - 1)
> +				goto fail_bad;
> +			ubi_dbg("FA: Vol: %u Ignoring empty LEB %d of %d",
> +				vol_id, j, used);
> +		}
> +	}
> +
> +	ret = scan_pool(ubi, fmpl1->pebs, pool_size);
> +	if (ret)
> +		goto fail;
> +
> +	ret = scan_pool(ubi, fmpl2->pebs, wl_pool_size);
> +	if (ret)
> +		goto fail;
> +
> +#ifdef CHECKME

What means this?

> +	/*
> +	 * If fastmap is leaking PEBs (must not happen), raise a
> +	 * fat warning and fall back to scanning mode.
> +	 * We do this here because in ubi_wl_init() it's too late
> +	 * and we cannot fall back to scanning.
> +	 */
> +	if (WARN_ON(count_fastmap_pebs(ai) != ubi->peb_count -
> +		    ai->bad_peb_count - fm->used_blocks))
> +		goto fail_bad;
> +#endif
> +
> +	return 0;
> +
> +fail_bad:
> +	ret = UBI_BAD_FASTMAP;
> +fail:
> +	return ret;
> +}
> +
> +static int ubi_scan_fastmap(struct ubi_scan_info *ubi,
> +			    struct ubi_attach_info *ai,
> +			    int fm_anchor)
> +{
> +	struct ubi_fm_sb *fmsb, *fmsb2;
> +	struct ubi_vid_hdr *vh;
> +	struct ubi_fastmap_layout *fm;
> +	int i, used_blocks, pnum, ret = 0;
> +	size_t fm_size;
> +	__be32 crc, tmp_crc;
> +	unsigned long long sqnum = 0;
> +
> +	fmsb = &ubi->fm_sb;
> +	fm = &ubi->fm_layout;
> +
> +	ret = ubi_io_read(ubi, fmsb, fm_anchor, ubi->leb_start, sizeof(*fmsb));
> +	if (ret && ret != UBI_IO_BITFLIPS)
> +		goto free_fm_sb;
> +	else if (ret == UBI_IO_BITFLIPS)
> +		fm->to_be_tortured[0] = 1;
> +
> +	if (be32_to_cpu(fmsb->magic) != UBI_FM_SB_MAGIC) {
> +		ubi_err("bad super block magic: 0x%x, expected: 0x%x",
> +			be32_to_cpu(fmsb->magic), UBI_FM_SB_MAGIC);
> +		ret = UBI_BAD_FASTMAP;
> +		goto free_fm_sb;
> +	}
> +
> +	if (fmsb->version != UBI_FM_FMT_VERSION) {
> +		ubi_err("bad fastmap version: %i, expected: %i",
> +			fmsb->version, UBI_FM_FMT_VERSION);
> +		ret = UBI_BAD_FASTMAP;
> +		goto free_fm_sb;
> +	}
> +
> +	used_blocks = be32_to_cpu(fmsb->used_blocks);
> +	if (used_blocks > UBI_FM_MAX_BLOCKS || used_blocks < 1) {
> +		ubi_err("number of fastmap blocks is invalid: %i", used_blocks);
> +		ret = UBI_BAD_FASTMAP;
> +		goto free_fm_sb;
> +	}
> +
> +	fm_size = ubi->leb_size * used_blocks;
> +	if (fm_size != ubi->fm_size) {
> +		ubi_err("bad fastmap size: %zi, expected: %zi", fm_size,
> +			ubi->fm_size);
> +		ret = UBI_BAD_FASTMAP;
> +		goto free_fm_sb;
> +	}
> +
> +	vh = &ubi->fm_vh;
> +
> +	for (i = 0; i < used_blocks; i++) {
> +		pnum = be32_to_cpu(fmsb->block_loc[i]);
> +
> +		if (ubi_io_is_bad(ubi, pnum)) {
> +			ret = UBI_BAD_FASTMAP;
> +			goto free_hdr;
> +		}
> +
> +#ifdef LATER

Here to ... what means this?

> +		int image_seq;
> +		ret = ubi_io_read_ec_hdr(ubi, pnum, ech, 0);
> +		if (ret && ret != UBI_IO_BITFLIPS) {
> +			ubi_err("unable to read fastmap block# %i EC (PEB: %i)",
> +				i, pnum);
> +			if (ret > 0)
> +				ret = UBI_BAD_FASTMAP;
> +			goto free_hdr;
> +		} else if (ret == UBI_IO_BITFLIPS)
> +			fm->to_be_tortured[i] = 1;
> +
> +		image_seq = be32_to_cpu(ech->image_seq);
> +		if (!ubi->image_seq)
> +			ubi->image_seq = image_seq;
> +		/*
> +		 * Older UBI implementations have image_seq set to zero, so
> +		 * we shouldn't fail if image_seq == 0.
> +		 */
> +		if (image_seq && (image_seq != ubi->image_seq)) {
> +			ubi_err("wrong image seq:%d instead of %d",
> +				be32_to_cpu(ech->image_seq), ubi->image_seq);
> +			ret = UBI_BAD_FASTMAP;
> +			goto free_hdr;
> +		}
> +#endif

Beside of this:

Acked-by: Heiko Schocher <hs@denx.de>

bye,
Heiko
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany