Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Peter Korsgaard <peter@korsgaard.com>]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

Hi,

 >> NIT: No, it is the sparseness (E.G. the holes). Compression in fact
 >> breaks the holes (replaces them by zeros):

 > You didn't understand my point, and Arnout explained it better on the
 > chat.

 > My point is that it is *very* easy to lose the sparseness. You copy the
 > file around, you share it in Google Drive, or download it over HTTP or
 > what not, and bang the sparseness is lost.

Sure, but that is all about distribution, where as mentioned before you
may want to compress it with gz/xz/zstd/.. and/or stick it in a sensible
archival format like tar, ..

The specifics of that is very use case specific, so IMHO not something
where we should enforce silly old .gz - Especially as bmap supports a
number of different formats (by shelling out to the extractors). From
bmaptool/TransRead.py:

This module allows opening and reading local and remote files and decompress
them on-the-fly if needed. Remote files are read using urllib (except of
"ssh://" URLs, which are handled differently). Supported file extentions are:
'bz2', 'gz', 'xz', 'lzo', 'zst' and a "tar" version of them: 'tar.bz2', 'tbz2',
'tbz', 'tb2', 'tar.gz', 'tgz', 'tar.xz', 'txz', 'tar.lzo', 'tzo', 'tar.lz4',
'tlz4', '.tar.zst', 'tzst'.
This module uses the following system programs for decompressing: pbzip2, bzip2,
gzip, pigz, xz, lzop, lz4, zstd, tar and unzip.


 > Instead, with a compressed file, you get a file that has pretty much
 > the same size as the sparse file, but you can move it around, transfer
 > it, it won't consume its real size. And the bmap metadata file allows
 > to use/flash that compressed file with the same efficiency as the
 > original sparse file.

 > In fact, bmap is quite useless is the file sparseness is retained:
 > instead of using bmap metadata, you could just ask what are the holes
 > of the file and skip them when flashing. So really the whole point of
 > bmap metadata IMO is that they allow to do fast flashing (skipping
 > holes) even if the sparseness of the file has been lost.

Does such a tool exist? I am not aware of one. If so, maybe we should
recommend that instead of dd for writing sdcard.img from the host?

Anyway, I find bmap a bit half-baked, E.G. rather than wasting time (and
space) compressing and decompressing all the holes that are then just
ignored it would make more sense to have a file format encoding the
holes (E.G. tar --sparse or something custom), then optionally compress
that for transmission and have something small and simple (E.G. not in
python) on the host/target to write it to a device.

But oh well, it is what it is. I don't need such a tool often enough to
care to write it ;)

I'll send a patch to drop the gzip step from genimage.sh.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot