[Buildroot] [RFC PATCH 0/2] Add support to genimage.sh for creating a bmap image

[Buildroot] [RFC PATCH 0/2] Add support to genimage.sh for creating a bmap image

[Dario Binacchi]

The series adds to the `genimage.sh` script an option for creating a
bmap image, which significantly reduces the time required to write an
image to an SD card. It also includes a use case applied to the
`ti_am62x_sk_defconfig` configuration to demonstrate how to use the
new option.

Dario Binacchi (2):
  support/scripts/genimage.sh: support creating a bmap image
  configs/ti_am62x_sk_defconfig: create the bmap image

 configs/ti_am62x_sk_defconfig |  3 ++-
 support/scripts/genimage.sh   | 21 +++++++++++++++++++--
 2 files changed, 21 insertions(+), 3 deletions(-)

-- 
2.43.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Dario Binacchi]

The patch adds an option to create, in addition to the usual image, an
image of type bmap that drastically reduces the amount of data that
needs to be written to an SD card, resulting in time savings.

This makes it possible to activate this option easily and maintain
backward compatibility for all configurations already using the genimage
tool for creating the image to be written to the SD card:

BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
+BR2_ROOTFS_POST_IMAGE_SCRIPT_ARGS="-c board/<manufacturer>/<boardname>/genimage.cfg -b"
+BR2_PACKAGE_HOST_BMAP_TOOLS=y

It follows that the script now assumes a broader functionality that
extends beyond just being an interface for the genimage tool.

An alternative implementation could have been to create another script,
such as support/scripts/bmap.sh, capable of creating the bmap image
using the same parameters passed to support/scripts/genimage.sh (i. e.
-c board/<manufacturer>/<boardname>/genimage.cfg). However, in this case,
the script would also need to know that the image is located in the
${BINARIES} directory. This could be achieved by adding an additional
parameter, but it might cause genimage.sh to fail due to an unrecognized
parameter.

Signed-off-by: Dario Binacchi <dario.binacchi@amarulasolutions.com>
---
 support/scripts/genimage.sh | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/support/scripts/genimage.sh b/support/scripts/genimage.sh
index 2796e19eb778..72592ebb710d 100755
--- a/support/scripts/genimage.sh
+++ b/support/scripts/genimage.sh
@@ -4,19 +4,26 @@ die() {
   cat <<EOF >&2
 Error: $@
 
-Usage: ${0} -c GENIMAGE_CONFIG_FILE
+Usage: ${0} -c GENIMAGE_CONFIG_FILE [-b]
+  -b    create bmap image
+  -c    configuration file
+
 EOF
   exit 1
 }
 
 # Parse arguments and put into argument list of the script
-opts="$(getopt -n "${0##*/}" -o c: -- "$@")" || exit $?
+opts="$(getopt -n "${0##*/}" -o bc: -- "$@")" || exit $?
 eval set -- "$opts"
 
 GENIMAGE_TMP="${BUILD_DIR}/genimage.tmp"
+GENIMAGE_CREATE_BMAP="no"
 
 while true ; do
 	case "$1" in
+	-b)
+	  GENIMAGE_CREATE_BMAP="yes"
+	  shift 1 ;;
 	-c)
 	  GENIMAGE_CFG="${2}";
 	  shift 2 ;;
@@ -46,3 +53,13 @@ genimage \
 	--inputpath "${BINARIES_DIR}"  \
 	--outputpath "${BINARIES_DIR}" \
 	--config "${GENIMAGE_CFG}"
+
+if [ "${GENIMAGE_CREATE_BMAP}" = "yes" ]; then
+  while IFS= read -r image; do
+    cnt=$(grep -c "${image}" "${GENIMAGE_CFG}")
+    [ "${cnt}" -gt 1 ] && continue;
+    image_path="${BINARIES_DIR}/${image}"
+    bmaptool create "${image_path}" -o "${image_path}.bmap"
+    gzip -c "${image_path}" > "${image_path}.gz"
+  done < <(grep '^image ' "${GENIMAGE_CFG}" | cut -d ' ' -f 2)
+fi
-- 
2.43.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [RFC PATCH 2/2] configs/ti_am62x_sk_defconfig: create the bmap image

[Dario Binacchi]

The use of a bmap image allows for speeding up the writing of the
image to the SD card, as can be seen from the following results:

$time sudo dd if=output/images/sdcard.img of=/dev/sda
[sudo] password for dario:
655361+0 records in
655361+0 records out
335544832 bytes (336 MB, 320 MiB) copied, 49,1122 s, 6,8 MB/s

real	0m52,177s
user	0m0,022s
sys	0m0,017s

$time sudo bmaptool copy output/images/sdcard.img.gz /dev/sda
bmaptool: info: discovered bmap file 'sdcard.img.bmap'
bmaptool: info: block map format version 2.0
bmaptool: info: 81921 blocks of size 4096 (320.0 MiB), mapped 29339 blocks (114.6 MiB or 35.8%)
bmaptool: info: copying image 'sdcard.img.gz' to block device '/dev/sda' using bmap file 'sdcard.img.bmap'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sda'
bmaptool: info: copying time: 11.8s, copying speed 9.7 MiB/sec

real	0m11,885s
user	0m0,004s
sys	0m0,011s

Signed-off-by: Dario Binacchi <dario.binacchi@amarulasolutions.com>
---
 configs/ti_am62x_sk_defconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/configs/ti_am62x_sk_defconfig b/configs/ti_am62x_sk_defconfig
index 5c7b97b3b0b1..708cb98f22d5 100644
--- a/configs/ti_am62x_sk_defconfig
+++ b/configs/ti_am62x_sk_defconfig
@@ -5,7 +5,7 @@ BR2_DOWNLOAD_FORCE_CHECK_HASHES=y
 BR2_ROOTFS_POST_BUILD_SCRIPT="board/ti/common/am6xx/post-build.sh"
 BR2_ROOTFS_POST_BUILD_SCRIPT_ARGS="-c ttyS2,115200n8 -d k3-am625-sk.dtb -l am62x-sk-buildroot -r /dev/mmcblk1p2 -x earlycon=ns16550a,mmio32,0x02800000"
 BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
-BR2_ROOTFS_POST_IMAGE_SCRIPT_ARGS="-c board/ti/am62x-sk/genimage.cfg"
+BR2_ROOTFS_POST_IMAGE_SCRIPT_ARGS="-c board/ti/am62x-sk/genimage.cfg -b"
 BR2_LINUX_KERNEL=y
 BR2_LINUX_KERNEL_CUSTOM_VERSION=y
 BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="6.6.18"
@@ -44,6 +44,7 @@ BR2_TARGET_UBOOT_FORMAT_IMG=y
 BR2_TARGET_UBOOT_SPL=y
 BR2_TARGET_UBOOT_SPL_NAME="tispl.bin"
 BR2_TARGET_UBOOT_CUSTOM_MAKEOPTS="TEE=$(BINARIES_DIR)/tee-pager_v2.bin"
+BR2_PACKAGE_HOST_BMAP_TOOLS=y
 BR2_PACKAGE_HOST_DOSFSTOOLS=y
 BR2_PACKAGE_HOST_GENIMAGE=y
 BR2_PACKAGE_HOST_MTOOLS=y
-- 
2.43.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Gero Schwäricke via buildroot]

Hi Dario,

I tried your patch, but it didn't work for me. bmaptool failed with an
error. The issue was that my home directory is encrypted using ecryptfs.
This does not implement all ioctl options that are needed for bmaptool
to read the block allocation of the image file. There also seem to be
issues with ZFS, so I would recommend to not enable this by default for
defconfigs.

A workaround for the issue described above would be to sparse copy the
image to a mktemp file and run bmaptool on that. However that will only
work if /tmp uses tmpfs (iiuc). And will potentially eat a lot of memory
for this step.

I generally like the approach of bmaptool and think including this in
buildroot seems like a good idea. Maybe the documentation should also
include a section on that in "14.1 Boot the generated images". Currently
it only covers NFS boot and Live CD. Different optimizations for sdcard
flashing seem to be missing.

Best,
Gero

-- 



---


grandcentrix GmbH | A Vodafone Company · Holzmarkt 1 · 50676 
*Cologne* | Phoenixseestrasse 18 · 44263 *Dortmund* · Germany
| in 
<https://www.linkedin.com/company/grandcentrix> | phone: +49-221-677-860-0 
| email: hello@grandcentrix.net <mailto:hello@grandcentrix.net>


grandcentrix cares about privacy 
<https://grandcentrix.net/de/pages/privacymail/>

Amtsgericht Köln | HRB  
70119 | Geschäftsführer: M. Krömer,  R. Hänel | USt.-IdNr.: DE266333969

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Quentin Schulz via buildroot]

Hi Gero,

On 4/23/24 13:33, Gero Schwäricke via buildroot wrote:
> Hi Dario,
> 
> I tried your patch, but it didn't work for me. bmaptool failed with an
> error. The issue was that my home directory is encrypted using ecryptfs.
> This does not implement all ioctl options that are needed for bmaptool
> to read the block allocation of the image file. There also seem to be
> issues with ZFS, so I would recommend to not enable this by default for
> defconfigs.
> 

Is this something you could report to the upstream project?

https://github.com/yoctoproject/bmaptool/

The ZFS issue seems to be documented, c.f. 
https://github.com/yoctoproject/bmaptool/?tab=readme-ov-file#known-issues

Cheers,
Quentin
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Gero Schwäricke via buildroot]

Hi Quentin,

On Tue, Apr 23, 2024 at 02:55:57PM +0200, Quentin Schulz via buildroot wrote:
> Hi Gero,
> 
> On 4/23/24 13:33, Gero Schwäricke via buildroot wrote:
> > Hi Dario,
> > 
> > I tried your patch, but it didn't work for me. bmaptool failed with an
> > error. The issue was that my home directory is encrypted using ecryptfs.
> > This does not implement all ioctl options that are needed for bmaptool
> > to read the block allocation of the image file. There also seem to be
> > issues with ZFS, so I would recommend to not enable this by default for
> > defconfigs.
> > 
> 
> Is this something you could report to the upstream project?
> 
> https://github.com/yoctoproject/bmaptool/

I reported my finding upstream in

https://github.com/yoctoproject/bmaptool/issues/22

though I don't expect a fix. Iiuc the issue is that ecryptfs does not
support the ioctl FIEMAP, which bmaptool needs to find the holes in
a file. So it would require implementing this for the ecryptfs kernel
module (if even possible) and then it'll take a while for that to hit
the mayor Linux distributions. But maybe a new command to bmaptool
could help in the midterm, like `bmaptool verify --check-fs`, so this
case could be detected and handled gracefully in scripts.

> 
> The ZFS issue seems to be documented, c.f.
> https://github.com/yoctoproject/bmaptool/?tab=readme-ov-file#known-issues

Yes, and it's fixable for ZFS. But I expect mailtraffic might increase
if bmaptool would be used by default in buildroot. That's why I would
recommend not to apply this to buildroot provided defconfigs
(PATCH 2/2). But I'm much in favor of supporting bmaptool in general
(PATCH 1/2).

Best,
Gero

> 
> Cheers,
> Quentin
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 



–––


grandcentrix GmbH | A Vodafone Company · Holzmarkt 1 · 50676 
*Cologne* | Phoenixseestrasse 18 · 44263 *Dortmund* · Germany
| in 
<https://www.linkedin.com/company/grandcentrix> | phone: +49-221-677-860-0 
| email: hello@grandcentrix.net <mailto:hello@grandcentrix.net>


grandcentrix cares about privacy 
<https://grandcentrix.net/de/pages/privacymail/>

Amtsgericht Köln | HRB  
70119 | Geschäftsführer: M. Krömer,  R. Hänel | USt.-IdNr.: DE266333969

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Gero Schwäricke via buildroot]

Hi all,

I have to correct myself:

On Tue, Apr 23, 2024 at 01:33:13PM +0200, Gero Schwäricke wrote:
> Hi Dario,
> 
> I tried your patch, but it didn't work for me. bmaptool failed with an
> error. The issue was that my home directory is encrypted using ecryptfs.
> This does not implement all ioctl options that are needed for bmaptool
> to read the block allocation of the image file. There also seem to be
> issues with ZFS, so I would recommend to not enable this by default for
> defconfigs.
> 
> A workaround for the issue described above would be to sparse copy the
> image to a mktemp file and run bmaptool on that. However that will only
> work if /tmp uses tmpfs (iiuc). And will potentially eat a lot of memory
> for this step.

This won't work. On second thought I wondered how cp can determine a
sparse file while bmaptool can't. So I checked coreutils, and iiuc,
`cp --sparse=always` will detect holes by reading the file and
generating holes for all blocks that contain only zeros. Copying a file
like this and then using bmaptool on the copy would destroy the benefit
of bmaptool. Because bmaptool can differentiate between blocks
containing zeroes and blocks that are empty. Empty blocks are holes (no
need to copy those), blocks containing zeroes are relevant (will be
copied).

Best,
Gero

> 
> I generally like the approach of bmaptool and think including this in
> buildroot seems like a good idea. Maybe the documentation should also
> include a section on that in "14.1 Boot the generated images". Currently
> it only covers NFS boot and Live CD. Different optimizations for sdcard
> flashing seem to be missing.
> 
> Best,
> Gero

-- 



–––


grandcentrix GmbH | A Vodafone Company · Holzmarkt 1 · 50676 
*Cologne* | Phoenixseestrasse 18 · 44263 *Dortmund* · Germany
| in 
<https://www.linkedin.com/company/grandcentrix> | phone: +49-221-677-860-0 
| email: hello@grandcentrix.net <mailto:hello@grandcentrix.net>


grandcentrix cares about privacy 
<https://grandcentrix.net/de/pages/privacymail/>

Amtsgericht Köln | HRB  
70119 | Geschäftsführer: M. Krömer,  R. Hänel | USt.-IdNr.: DE266333969

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Thomas Petazzoni via buildroot]

Hello Dario,

On Sun, 21 Apr 2024 11:53:52 +0200
Dario Binacchi <dario.binacchi@amarulasolutions.com> wrote:

> The patch adds an option to create, in addition to the usual image, an
> image of type bmap that drastically reduces the amount of data that
> needs to be written to an SD card, resulting in time savings.
> 
> This makes it possible to activate this option easily and maintain
> backward compatibility for all configurations already using the genimage
> tool for creating the image to be written to the SD card:
> 
> BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
> +BR2_ROOTFS_POST_IMAGE_SCRIPT_ARGS="-c board/<manufacturer>/<boardname>/genimage.cfg -b"
> +BR2_PACKAGE_HOST_BMAP_TOOLS=y
> 
> It follows that the script now assumes a broader functionality that
> extends beyond just being an interface for the genimage tool.
> 
> An alternative implementation could have been to create another script,
> such as support/scripts/bmap.sh, capable of creating the bmap image
> using the same parameters passed to support/scripts/genimage.sh (i. e.
> -c board/<manufacturer>/<boardname>/genimage.cfg). However, in this case,
> the script would also need to know that the image is located in the
> ${BINARIES} directory. This could be achieved by adding an additional
> parameter, but it might cause genimage.sh to fail due to an unrecognized
> parameter.
> 
> Signed-off-by: Dario Binacchi <dario.binacchi@amarulasolutions.com>

Thanks for this proposal! Actually, I very much like the idea of
extending the usage of bmap-tools in Buildroot. I think it's a great
tool, and it makes a lot of sense to encourage more of its use.

Now the question is how to do this. To me your proposal looks
reasonable, though some people had the feedback that bmap-tools will
not work in all cases (should we then gracefully handle this failure
case?).

Arnout, Yann, Peter, Romain, what do you think? Do you see other
approaches to easily allow generating bmap-compatible images?

Another question, perhaps for Dario, or other people familiar with
genimage: if the rootfs.ext4 generated by Buildroot is sparse, is
genimage able to preserve the "holes" when generating the sdcard.img
that includes the rootfs.ext4 ?

Thanks!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Michael Nazzareno Trimarchi]

Hi Thomas and all

On Fri, May 10, 2024 at 9:54 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Dario,
>
> On Sun, 21 Apr 2024 11:53:52 +0200
> Dario Binacchi <dario.binacchi@amarulasolutions.com> wrote:
>
> > The patch adds an option to create, in addition to the usual image, an
> > image of type bmap that drastically reduces the amount of data that
> > needs to be written to an SD card, resulting in time savings.
> >
> > This makes it possible to activate this option easily and maintain
> > backward compatibility for all configurations already using the genimage
> > tool for creating the image to be written to the SD card:
> >
> > BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
> > +BR2_ROOTFS_POST_IMAGE_SCRIPT_ARGS="-c board/<manufacturer>/<boardname>/genimage.cfg -b"
> > +BR2_PACKAGE_HOST_BMAP_TOOLS=y
> >
> > It follows that the script now assumes a broader functionality that
> > extends beyond just being an interface for the genimage tool.
> >
> > An alternative implementation could have been to create another script,
> > such as support/scripts/bmap.sh, capable of creating the bmap image
> > using the same parameters passed to support/scripts/genimage.sh (i. e.
> > -c board/<manufacturer>/<boardname>/genimage.cfg). However, in this case,
> > the script would also need to know that the image is located in the
> > ${BINARIES} directory. This could be achieved by adding an additional
> > parameter, but it might cause genimage.sh to fail due to an unrecognized
> > parameter.
> >
> > Signed-off-by: Dario Binacchi <dario.binacchi@amarulasolutions.com>
>
> Thanks for this proposal! Actually, I very much like the idea of
> extending the usage of bmap-tools in Buildroot. I think it's a great
> tool, and it makes a lot of sense to encourage more of its use.
>
> Now the question is how to do this. To me your proposal looks
> reasonable, though some people had the feedback that bmap-tools will
> not work in all cases (should we then gracefully handle this failure
> case?).
>
> Arnout, Yann, Peter, Romain, what do you think? Do you see other
> approaches to easily allow generating bmap-compatible images?
>
> Another question, perhaps for Dario, or other people familiar with
> genimage: if the rootfs.ext4 generated by Buildroot is sparse, is
> genimage able to preserve the "holes" when generating the sdcard.img
> that includes the rootfs.ext4 ?

If you consider sparse the way that fastboot handle it, so I have
quick tested and seems
genimage can create sdcard.img and sdcard-sparse.img using a specific genconfig

I have tested with this

image sdcard.img {
    hdimage {
    }

    partition u-boot-tpl-spl-dtb {
        in-partition-table = "no"
        image = "idbloader.img"
        offset = 32K
    }

    partition u-boot-dtb {
        in-partition-table = "no"
        image = "u-boot.itb"
        offset = 8M
        size = 30M
    }

    partition rootfs {
        partition-type = 0x83
        image = "rootfs.ext4"
    }
}

image sdcard-sparse.img {
    android-sparse {
        image = sdcard.img
    }
}

output/images/sdcard.img:        DOS/MBR boot sector; partition 1 :
ID=0x83, start-CHS (0x4,215,20), end-CHS (0x51,85,4), startsector
77824, 1228800 sectors, extended partition table (last)
output/images/sdcard-sparse.img: Android sparse image, version: 1.0,
Total of 163328 4096-byte output blocks in 29 input chunks.

Michael

>
> Thanks!
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com



-- 
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Dario Binacchi]

Hi Thomas, Gero and All

On Fri, May 10, 2024 at 9:54 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Dario,
>
> On Sun, 21 Apr 2024 11:53:52 +0200
> Dario Binacchi <dario.binacchi@amarulasolutions.com> wrote:
>
> > The patch adds an option to create, in addition to the usual image, an
> > image of type bmap that drastically reduces the amount of data that
> > needs to be written to an SD card, resulting in time savings.
> >
> > This makes it possible to activate this option easily and maintain
> > backward compatibility for all configurations already using the genimage
> > tool for creating the image to be written to the SD card:
> >
> > BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
> > +BR2_ROOTFS_POST_IMAGE_SCRIPT_ARGS="-c board/<manufacturer>/<boardname>/genimage.cfg -b"
> > +BR2_PACKAGE_HOST_BMAP_TOOLS=y
> >
> > It follows that the script now assumes a broader functionality that
> > extends beyond just being an interface for the genimage tool.
> >
> > An alternative implementation could have been to create another script,
> > such as support/scripts/bmap.sh, capable of creating the bmap image
> > using the same parameters passed to support/scripts/genimage.sh (i. e.
> > -c board/<manufacturer>/<boardname>/genimage.cfg). However, in this case,
> > the script would also need to know that the image is located in the
> > ${BINARIES} directory. This could be achieved by adding an additional
> > parameter, but it might cause genimage.sh to fail due to an unrecognized
> > parameter.
> >
> > Signed-off-by: Dario Binacchi <dario.binacchi@amarulasolutions.com>
>
> Thanks for this proposal! Actually, I very much like the idea of
> extending the usage of bmap-tools in Buildroot. I think it's a great
> tool, and it makes a lot of sense to encourage more of its use.
>
> Now the question is how to do this. To me your proposal looks
> reasonable, though some people had the feedback that bmap-tools will
> not work in all cases (should we then gracefully handle this failure
> case?).

If I understand correctly, the first patch in the series seems
reasonable to Gero as well.
Therefore, both of you are in favor of merging this patch. The issue
arises for Gero
regarding the application of the second patch, which I added only as
an example use case.
I think merging the first patch would give the opportunity to use
bmaptool and realize its
usefulness. After all, the patch is backward compatible. Its use could
then lead to subsequent
improvements.

Regarding the second patch, I then ask you: Is it correct not to
proceed with merging this
patch for the reason mentioned by Gero (i.e., encrypted directory)? It
seems to me that
bmaptool is used in Yocto, so is it correct to say that you would have
the same problem
with Yocto? I hope, Gero, you do not misunderstand me.

Thanks and regards,
Dario

>
> Arnout, Yann, Peter, Romain, what do you think? Do you see other
> approaches to easily allow generating bmap-compatible images?
>
> Another question, perhaps for Dario, or other people familiar with
> genimage: if the rootfs.ext4 generated by Buildroot is sparse, is
> genimage able to preserve the "holes" when generating the sdcard.img
> that includes the rootfs.ext4 ?
>
> Thanks!
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com



-- 

Dario Binacchi

Senior Embedded Linux Developer

dario.binacchi@amarulasolutions.com

__________________________________


Amarula Solutions SRL

Via Le Canevare 30, 31100 Treviso, Veneto, IT

T. +39 042 243 5310
info@amarulasolutions.com

www.amarulasolutions.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Thomas Petazzoni via buildroot]

Hello Dario,

[ Your e-mail client configuration seems somewhat broken, your lines
  are badly wrapped ]

On Mon, 20 May 2024 10:05:34 +0200
Dario Binacchi <dario.binacchi@amarulasolutions.com> wrote:

> > Now the question is how to do this. To me your proposal looks
> > reasonable, though some people had the feedback that bmap-tools will
> > not work in all cases (should we then gracefully handle this failure
> > case?).  
> 
> If I understand correctly, the first patch in the series seems
> reasonable to Gero as well.
> Therefore, both of you are in favor of merging this patch. The issue
> arises for Gero
> regarding the application of the second patch, which I added only as
> an example use case.
> I think merging the first patch would give the opportunity to use
> bmaptool and realize its
> usefulness. After all, the patch is backward compatible. Its use could
> then lead to subsequent
> improvements.
> 
> Regarding the second patch, I then ask you: Is it correct not to
> proceed with merging this
> patch for the reason mentioned by Gero (i.e., encrypted directory)? It
> seems to me that
> bmaptool is used in Yocto, so is it correct to say that you would have
> the same problem
> with Yocto? I hope, Gero, you do not misunderstand me.

Yes, the first patch is reasonable, but if we can't apply something
like the second patch, then it might mean that the approach taken in
the first patch isn't correct. Indeed, if we can't enable by default
the generation of bmap-capable images in defconfigs, it kind of limits
the usefulness of this bmap-tool integration.

I was hoping for the other BR maintainers to chime in with some opinion.

Do we have a way to detect that the bmap-tool generation will fail, and
in this case gracefully skip the bmap-tool logic?

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Yann E. MORIN]

Michael, All,

On 2024-05-10 22:17 +0200, Michael Nazzareno Trimarchi spake thusly:
> On Fri, May 10, 2024 at 9:54 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> > On Sun, 21 Apr 2024 11:53:52 +0200
> > Dario Binacchi <dario.binacchi@amarulasolutions.com> wrote:
> > > The patch adds an option to create, in addition to the usual image, an
> > > image of type bmap that drastically reduces the amount of data that
> > > needs to be written to an SD card, resulting in time savings.
[--SNIP--]
> > Another question, perhaps for Dario, or other people familiar with
> > genimage: if the rootfs.ext4 generated by Buildroot is sparse, is
> > genimage able to preserve the "holes" when generating the sdcard.img
> > that includes the rootfs.ext4 ?

> Arnout, Yann, Peter, Romain, what do you think? Do you see other
> approaches to easily allow generating bmap-compatible images?

I wonder if we should really do that, in fact.

I believe it is in-scope that we do generate disk images, which we do
with genimage. However, transferring, distributing, writing, etc.. the
generated images I think is out of scope. We may provide tools to help
with that, though, and we do by providing {host-,}bmap-tools.

> If you consider sparse the way that fastboot handle it, so I have
> quick tested and seems
> genimage can create sdcard.img and sdcard-sparse.img using a specific genconfig

I think the question by Thomas was whether, given an input sparse file,
genimage will keep the generated image sparse in the same places the
input file was (only shifted by the offset that input file was copied
at in the image).

For example, mkfs.ext can create sparse files, and such a file can be
fed to genimage, to create a disk image. We want the generated disk
image to be sparsed in the same positions the ext2 image was, e.g. with
'X' as no-hole blocks, '.' as holes, and 'B' as the bootloader:

  rootfs.ext2        XXXXX........XXX...XXXX....XXXXX........X....
  disk.img       BBBBXXXXX........XXX...XXXX....XXXXX........X....

Note that some X block can be all-zeroes, because they are the bitmap of
the filesystem; those must not be skipped when writing to the final
storage, otherwise the filesystem would be corrupted. If genimage
decides that all-zeroes block are sparsed, then this is broken.

I think that was the question raised by Thomas, and I don't understand
how your reply answers that, so maybe I missed something...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Michael Nazzareno Trimarchi]

Yann, all

On Tue, May 21, 2024 at 7:25 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Michael, All,
>
> On 2024-05-10 22:17 +0200, Michael Nazzareno Trimarchi spake thusly:
> > On Fri, May 10, 2024 at 9:54 PM Thomas Petazzoni
> > <thomas.petazzoni@bootlin.com> wrote:
> > > On Sun, 21 Apr 2024 11:53:52 +0200
> > > Dario Binacchi <dario.binacchi@amarulasolutions.com> wrote:
> > > > The patch adds an option to create, in addition to the usual image, an
> > > > image of type bmap that drastically reduces the amount of data that
> > > > needs to be written to an SD card, resulting in time savings.
> [--SNIP--]
> > > Another question, perhaps for Dario, or other people familiar with
> > > genimage: if the rootfs.ext4 generated by Buildroot is sparse, is
> > > genimage able to preserve the "holes" when generating the sdcard.img
> > > that includes the rootfs.ext4 ?
>
> > Arnout, Yann, Peter, Romain, what do you think? Do you see other
> > approaches to easily allow generating bmap-compatible images?
>
> I wonder if we should really do that, in fact.
>
> I believe it is in-scope that we do generate disk images, which we do
> with genimage. However, transferring, distributing, writing, etc.. the
> generated images I think is out of scope. We may provide tools to help
> with that, though, and we do by providing {host-,}bmap-tools.
>
> > If you consider sparse the way that fastboot handle it, so I have
> > quick tested and seems
> > genimage can create sdcard.img and sdcard-sparse.img using a specific genconfig
>
> I think the question by Thomas was whether, given an input sparse file,
> genimage will keep the generated image sparse in the same places the
> input file was (only shifted by the offset that input file was copied
> at in the image).
>
> For example, mkfs.ext can create sparse files, and such a file can be
> fed to genimage, to create a disk image. We want the generated disk
> image to be sparsed in the same positions the ext2 image was, e.g. with
> 'X' as no-hole blocks, '.' as holes, and 'B' as the bootloader:
>
>   rootfs.ext2        XXXXX........XXX...XXXX....XXXXX........X....
>   disk.img       BBBBXXXXX........XXX...XXXX....XXXXX........X....
>
> Note that some X block can be all-zeroes, because they are the bitmap of
> the filesystem; those must not be skipped when writing to the final
> storage, otherwise the filesystem would be corrupted. If genimage
> decides that all-zeroes block are sparsed, then this is broken.
>
> I think that was the question raised by Thomas, and I don't understand
> how your reply answers that, so maybe I missed something...
>

Seems I did not understand the question. My idea on the topic is how
we can generate
images that we can easily flash. Now I understand the question of
Thomas and thank you.

Yann can I ask how we can use an image generated as you describe before?

Michael

> Regards,
> Yann E. MORIN.
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'



-- 
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Dario Binacchi]

Hello Thomas, Gero and All,

On Mon, May 20, 2024 at 11:23 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Dario,
>
> [ Your e-mail client configuration seems somewhat broken, your lines
>   are badly wrapped ]
>
> On Mon, 20 May 2024 10:05:34 +0200
> Dario Binacchi <dario.binacchi@amarulasolutions.com> wrote:
>
> > > Now the question is how to do this. To me your proposal looks
> > > reasonable, though some people had the feedback that bmap-tools will
> > > not work in all cases (should we then gracefully handle this failure
> > > case?).
> >
> > If I understand correctly, the first patch in the series seems
> > reasonable to Gero as well.
> > Therefore, both of you are in favor of merging this patch. The issue
> > arises for Gero
> > regarding the application of the second patch, which I added only as
> > an example use case.
> > I think merging the first patch would give the opportunity to use
> > bmaptool and realize its
> > usefulness. After all, the patch is backward compatible. Its use could
> > then lead to subsequent
> > improvements.
> >
> > Regarding the second patch, I then ask you: Is it correct not to
> > proceed with merging this
> > patch for the reason mentioned by Gero (i.e., encrypted directory)? It
> > seems to me that
> > bmaptool is used in Yocto, so is it correct to say that you would have
> > the same problem
> > with Yocto? I hope, Gero, you do not misunderstand me.
>
> Yes, the first patch is reasonable, but if we can't apply something
> like the second patch, then it might mean that the approach taken in
> the first patch isn't correct. Indeed, if we can't enable by default
> the generation of bmap-capable images in defconfigs, it kind of limits
> the usefulness of this bmap-tool integration.
>
> I was hoping for the other BR maintainers to chime in with some opinion.
>
> Do we have a way to detect that the bmap-tool generation will fail, and
> in this case gracefully skip the bmap-tool logic?

I ran some tests and tried to reproduce Gero's issue.
With these changes, in case of an error, the make command would not fail.

diff --git a/support/scripts/genimage.sh b/support/scripts/genimage.sh
index 72592ebb710d..122aa0e0dc81 100755
--- a/support/scripts/genimage.sh
+++ b/support/scripts/genimage.sh
@@ -59,7 +59,12 @@ if [ "${GENIMAGE_CREATE_BMAP}" = "yes" ]; then
     cnt=$(grep -c "${image}" "${GENIMAGE_CFG}")
     [ "${cnt}" -gt 1 ] && continue;
     image_path="${BINARIES_DIR}/${image}"
-    bmaptool create "${image_path}" -o "${image_path}.bmap"
-    gzip -c "${image_path}" > "${image_path}.gz"
+    TMP_STDERR=$(mktemp -t .tmp.genimage.XXXXXXXXXX)
+    bmaptool create "${image_path}" -o "${image_path}.bmap" 2>${TMP_STDERR}
+    if test $? -eq 0 ; then
+      gzip -c "${image_path}" > "${image_path}.gz"
+    else
+      echo "Cannot generate bmap for file ${image_path}. Please read
log error ${TMP_STDERR} for details"
+    fi
   done < <(grep '^image ' "${GENIMAGE_CFG}" | cut -d ' ' -f 2)
 fi

The change generally handles any type of error, but in the case
highlighted by Gero, it seems to me
that not too much time is wasted encountering the error, since
bmaptool initially tries to understand if
the system supports the FIEMAP ioctl or the SEEK_HOLE/SEEK_DATA
functionality of the file seek
syscall.

As can be seen in the Filemap.py file:

def filemap(image):
    """
    Create and return an instance of a Filemap class - 'FilemapFiemap' or
    'FilemapSeek', depending on what the system we run on supports. If the
    FIEMAP ioctl is supported, an instance of the 'FilemapFiemap' class is
    returned. Otherwise, if 'SEEK_HOLE' is supported an instance of the
    'FilemapSeek' class is returned. If none of these are supported, the
    function generates an 'Error' type exception.
    """

    try:
        return FilemapFiemap(image)
    except ErrorNotSupp:
        return FilemapSeek(image)

Can these changes make the patches acceptable?

Thanks and regards,
Dario

>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com



-- 

Dario Binacchi

Senior Embedded Linux Developer

dario.binacchi@amarulasolutions.com

__________________________________


Amarula Solutions SRL

Via Le Canevare 30, 31100 Treviso, Veneto, IT

T. +39 042 243 5310
info@amarulasolutions.com

www.amarulasolutions.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Yann E. MORIN]

Michael, All,

On 2024-05-21 11:28 +0200, Michael Nazzareno Trimarchi spake thusly:
> On Tue, May 21, 2024 at 7:25 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > I think the question by Thomas was whether, given an input sparse file,
> > genimage will keep the generated image sparse in the same places the
> > input file was (only shifted by the offset that input file was copied
> > at in the image).
[--SNIP--]
> Seems I did not understand the question. My idea on the topic is how
> we can generate
> images that we can easily flash. Now I understand the question of
> Thomas and thank you.
> 
> Yann can I ask how we can use an image generated as you describe before?

    $ make pc_x86_64_efi_defconfig
    $ make
    $ ls -sl output/images/{rootfs.ext2,disk.img}

I tried a bit earlier, and the genimage file is indeed sparse, but I
could not find a tool that would report the list of holes/non-holes in a
file, and their offset, e.g. something like:

                size    offset
    non-hole    2MiB    @0
    hole        1MiB    @2MiB
    non-hole    16MiB   @3MiB
 and so on...

So I could not validate whether the hole/non-hole structure is the same
in the generated disk.img and the rootfs.ext2. If you have any idea how
to do that, I'm all eyes.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Yann E. MORIN]

Thomas, Dario, All,

On 2024-05-20 11:23 +0200, Thomas Petazzoni via buildroot spake thusly:
[--SNIP--]
> Yes, the first patch is reasonable, but if we can't apply something
> like the second patch, then it might mean that the approach taken in
> the first patch isn't correct. Indeed, if we can't enable by default
> the generation of bmap-capable images in defconfigs, it kind of limits
> the usefulness of this bmap-tool integration.
> 
> I was hoping for the other BR maintainers to chime in with some opinion.

Given the issues on existing setups, I am of the opinion that we should
not enable the creation of bmap images by default, in any of our
defconfig.

Now, do we want to include that support in our somewhat-generic genimage
helper script? Technically, I don't have a strong opinion against,
except it would not be exercised very often. We could have our CI jobs
turn it on explicitly, though, so that's not too much of an issue.

However, users will not have an easy way to turn it on, except by adding
somewhat-arcane command line options in the post-image script args.

Also, it is relatively easy to add a site-local post-image script, which
can be added to the list after the bundled one, and which will call
bmap-tool, e.g. something like (kinda pseudo shell in places):

    #!/usr/bin/env bash

    getopts for -c configfile -z -j -J blablalba

    # Only the last image is interesting, as it presumably contains all
    # the others
    last_image="$(
        sed -r -e '/^image ([^[:space:]]+).*/!d; s//\1/' "${cfg_file}" \
        |tail -n 1
    )"
    image_path="${BINARIES_DIR}/${last_image}"

    bmaptool create "${image_path}" -o "${image_path}.bmap"

    if -z; then
        gzip "${image_path}"
    elif -j; then
        bzip2 "${image_path}"
    elif -J; then
        xz "${image_path}"
    fi

Ah one point: I think our generic geninage helper should ignore unknown
options, rather than error out. This would I believe help with users
adding site-local scripts like the above (notice the -z -j -J options
for example).

Still, my opinion is that our defconfigs, and the associated scripts,
are a starting point for booting known boards and somehow show-casing
Buildroot, and set the user on tracks for them to further customise the
configuration and the scripts to match their needs.

In this case, I don't think bmap-tools provides any special interest for
that goal, given the integration in Buildroot is really easy, and very
use-case specific (i.e. I know projects for which bmap-tools would be a
drawback, given the generated images have zero hole).

> Do we have a way to detect that the bmap-tool generation will fail, and
> in this case gracefully skip the bmap-tool logic?

I kind of disagree there. Given we won't enable the creation of bmap
images by default, then it means the user will have to explicitly
request the creation of a bmap image and thus, if we can't generate it,
then we must fail explicitly.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Gero Schwäricke via buildroot]

On Tue, May 21, 2024 at 07:44:24PM +0200, Yann E. MORIN wrote:
> Michael, All,
> 
> On 2024-05-21 11:28 +0200, Michael Nazzareno Trimarchi spake thusly:
> > On Tue, May 21, 2024 at 7:25 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > > I think the question by Thomas was whether, given an input sparse file,
> > > genimage will keep the generated image sparse in the same places the
> > > input file was (only shifted by the offset that input file was copied
> > > at in the image).
> [--SNIP--]
> > Seems I did not understand the question. My idea on the topic is how
> > we can generate
> > images that we can easily flash. Now I understand the question of
> > Thomas and thank you.
> > 
> > Yann can I ask how we can use an image generated as you describe before?
> 
>     $ make pc_x86_64_efi_defconfig
>     $ make
>     $ ls -sl output/images/{rootfs.ext2,disk.img}
> 
> I tried a bit earlier, and the genimage file is indeed sparse, but I
> could not find a tool that would report the list of holes/non-holes in a
> file, and their offset, e.g. something like:
> 
>                 size    offset
>     non-hole    2MiB    @0
>     hole        1MiB    @2MiB
>     non-hole    16MiB   @3MiB
>  and so on...
> 
> So I could not validate whether the hole/non-hole structure is the same
> in the generated disk.img and the rootfs.ext2. If you have any idea how
> to do that, I'm all eyes.

Hi Yann,

iiuc, the bmaptool mainly uses the FIEMAP ioctl. I found this tool [1], 
which prints the FIEMAP values.

Here's a quick test I did:

$ truncate -s 1G /tmp/image.raw
$ ./fiemap /tmp/image.raw
File /tmp/image.raw has 0 extents:
#       Logical          Physical         Length           Flags

$ stat /tmp/image.raw 
  File: /tmp/image.raw
  Size: 1073741824      Blocks: 0          IO Block: 4096   regular file
Device: 10305h/66309d   Inode: 25692108    Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/gschwaer)   Gid: ( 1000/gschwaer)
Access: 2024-05-22 17:09:41.463441946 +0200
Modify: 2024-05-22 17:09:41.463441946 +0200
Change: 2024-05-22 17:09:41.463441946 +0200
 Birth: -
$ echo test >> /tmp/image.raw 
$ ./fiemap /tmp/image.raw
File /tmp/image.raw has 1 extents:
#       Logical          Physical         Length           Flags
0:      0000000040000000 0000006208224000 0000000000001000 0001
$ stat /tmp/image.raw 
  File: /tmp/image.raw
  Size: 1073741829      Blocks: 8          IO Block: 4096   regular file
Device: 10305h/66309d   Inode: 25692114    Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/gschwaer)   Gid: ( 1000/gschwaer)
Access: 2024-05-22 16:54:48.729576604 +0200
Modify: 2024-05-22 16:59:24.905638708 +0200
Change: 2024-05-22 16:59:24.905638708 +0200
 Birth: -

You can see the allocated section starting at logical 0x40000000 (1G).
Without the `echo` line, the extents list is empty, suggesting the file
has zero allocated blocks. `stat` also hints that the file is sparse,
by the small number of blocks (why 8? idk). I think this can be used to
verify the behavior of genimage.

[1] https://github.com/ColinIanKing/fiemap

Best,
Gero

> 
> Regards,
> Yann E. MORIN.
> 
> -- 
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 



–––


grandcentrix GmbH | A Vodafone Company · Holzmarkt 1 · 50676 
*Cologne* | Phoenixseestrasse 18 · 44263 *Dortmund* · Germany
| in 
<https://www.linkedin.com/company/grandcentrix> | phone: +49-221-677-860-0 
| email: hello@grandcentrix.net <mailto:hello@grandcentrix.net>


grandcentrix cares about privacy 
<https://grandcentrix.net/de/pages/privacymail/>

Amtsgericht Köln | HRB  
70119 | Geschäftsführer: M. Krömer,  R. Hänel | USt.-IdNr.: DE266333969

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Gero Schwäricke via buildroot]

Hi all,

On Tue, May 21, 2024 at 10:36:39PM +0200, Yann E. MORIN wrote:
> Thomas, Dario, All,
> 
> On 2024-05-20 11:23 +0200, Thomas Petazzoni via buildroot spake thusly:
> [--SNIP--]
> > Yes, the first patch is reasonable, but if we can't apply something
> > like the second patch, then it might mean that the approach taken in
> > the first patch isn't correct. Indeed, if we can't enable by default
> > the generation of bmap-capable images in defconfigs, it kind of limits
> > the usefulness of this bmap-tool integration.

I disagree, I think not having an option for speeding up image flashing
enabled by default in defconfigs is fair. Defconfigs are intentionally
minimal, and for the sole purpose of getting people started with a
working config.

> > 
> > I was hoping for the other BR maintainers to chime in with some opinion.
> 
> Given the issues on existing setups, I am of the opinion that we should
> not enable the creation of bmap images by default, in any of our
> defconfig.
> 
> Now, do we want to include that support in our somewhat-generic genimage
> helper script? Technically, I don't have a strong opinion against,
> except it would not be exercised very often. We could have our CI jobs
> turn it on explicitly, though, so that's not too much of an issue.
> 
> However, users will not have an easy way to turn it on, except by adding
> somewhat-arcane command line options in the post-image script args.

True, it's not a beautiful solution, but a flexible one. We could also
convert the often used genimage post install step into a kconfig-urable
build step, but I don't think that's much better.

> 
> Also, it is relatively easy to add a site-local post-image script, which
> can be added to the list after the bundled one, and which will call
> bmap-tool, e.g. something like (kinda pseudo shell in places):
> 
>     #!/usr/bin/env bash
> 
>     getopts for -c configfile -z -j -J blablalba
> 
>     # Only the last image is interesting, as it presumably contains all
>     # the others
>     last_image="$(
>         sed -r -e '/^image ([^[:space:]]+).*/!d; s//\1/' "${cfg_file}" \
>         |tail -n 1
>     )"
>     image_path="${BINARIES_DIR}/${last_image}"
> 
>     bmaptool create "${image_path}" -o "${image_path}.bmap"
> 
>     if -z; then
>         gzip "${image_path}"
>     elif -j; then
>         bzip2 "${image_path}"
>     elif -J; then
>         xz "${image_path}"
>     fi

Not sure I understand correctly, but you are suggesting that each user
that wants to use bmap-tools should write their own post install script
and call genimage.sh in it? I think it won't be long until this very
generic wrapper script is requested to be added to support/scripts/. For
me including this in genimage.sh is preferable.

> 
> Ah one point: I think our generic geninage helper should ignore unknown
> options, rather than error out. This would I believe help with users
> adding site-local scripts like the above (notice the -z -j -J options
> for example).
> 
> Still, my opinion is that our defconfigs, and the associated scripts,
> are a starting point for booting known boards and somehow show-casing
> Buildroot, and set the user on tracks for them to further customise the
> configuration and the scripts to match their needs.
> 
> In this case, I don't think bmap-tools provides any special interest for
> that goal, given the integration in Buildroot is really easy, and very
> use-case specific (i.e. I know projects for which bmap-tools would be a
> drawback, given the generated images have zero hole).
> 
> > Do we have a way to detect that the bmap-tool generation will fail, and
> > in this case gracefully skip the bmap-tool logic?
> 
> I kind of disagree there. Given we won't enable the creation of bmap
> images by default, then it means the user will have to explicitly
> request the creation of a bmap image and thus, if we can't generate it,
> then we must fail explicitly.
> 

I think there are three options:

1. don't implement this and let each user figure bmap out on their own
2. implement it opt-in and generate an error if it fails
3. implement it opt-out and generate a warning if it fails

Personally I would prefer the second option: This would mean accepting
the original patch 1/2, but not 2/2. Instead of 2/2, it might be a good
idea to describe this in the manual so this option is easier to
discover. Currently only the genimage.cfg format is briefly described,
but not how this file is usually used in buildroot with genimage.sh. I'd
volunteer to update the documentation if this is accepted.

Best,
Gero

> Regards,
> Yann E. MORIN.
> 
> -- 
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'

-- 



---


grandcentrix GmbH | A Vodafone Company · Holzmarkt 1 · 50676 
*Cologne* | Phoenixseestrasse 18 · 44263 *Dortmund* · Germany
| in 
<https://www.linkedin.com/company/grandcentrix> | phone: +49-221-677-860-0 
| email: hello@grandcentrix.net <mailto:hello@grandcentrix.net>


grandcentrix cares about privacy 
<https://grandcentrix.net/de/pages/privacymail/>

Amtsgericht Köln | HRB  
70119 | Geschäftsführer: M. Krömer,  R. Hänel | USt.-IdNr.: DE266333969

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Thomas Petazzoni via buildroot]

On Tue, 21 May 2024 07:25:11 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> > Arnout, Yann, Peter, Romain, what do you think? Do you see other
> > approaches to easily allow generating bmap-compatible images?  
> 
> I wonder if we should really do that, in fact.
> 
> I believe it is in-scope that we do generate disk images, which we do
> with genimage. However, transferring, distributing, writing, etc.. the
> generated images I think is out of scope. We may provide tools to help
> with that, though, and we do by providing {host-,}bmap-tools.

I think it is in the scope of Buildroot to have defconfigs that
generate images that are fast to flash.

> I think the question by Thomas was whether, given an input sparse file,
> genimage will keep the generated image sparse in the same places the
> input file was (only shifted by the offset that input file was copied
> at in the image).
> 
> For example, mkfs.ext can create sparse files, and such a file can be
> fed to genimage, to create a disk image. We want the generated disk
> image to be sparsed in the same positions the ext2 image was, e.g. with
> 'X' as no-hole blocks, '.' as holes, and 'B' as the bootloader:
> 
>   rootfs.ext2        XXXXX........XXX...XXXX....XXXXX........X....
>   disk.img       BBBBXXXXX........XXX...XXXX....XXXXX........X....
> 
> Note that some X block can be all-zeroes, because they are the bitmap of
> the filesystem; those must not be skipped when writing to the final
> storage, otherwise the filesystem would be corrupted. If genimage
> decides that all-zeroes block are sparsed, then this is broken.
> 
> I think that was the question raised by Thomas, and I don't understand
> how your reply answers that, so maybe I missed something...

This was exactly my question, indeed.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Thomas Petazzoni via buildroot]

On Tue, 21 May 2024 19:44:24 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> I tried a bit earlier, and the genimage file is indeed sparse, but I
> could not find a tool that would report the list of holes/non-holes in a
> file, and their offset, e.g. something like:
> 
>                 size    offset
>     non-hole    2MiB    @0
>     hole        1MiB    @2MiB
>     non-hole    16MiB   @3MiB
>  and so on...

$ pip3 install sparseutils
$ truncate -s 512M file.img
$ sparsemap file.img 
HOLE 536870912

Seems like it's what we're looking for :-)

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Thomas Petazzoni via buildroot]

Hello Dario,

On Sun, 21 Apr 2024 11:53:52 +0200
Dario Binacchi <dario.binacchi@amarulasolutions.com> wrote:

> The patch adds an option to create, in addition to the usual image, an
> image of type bmap that drastically reduces the amount of data that
> needs to be written to an SD card, resulting in time savings.
> 
> This makes it possible to activate this option easily and maintain
> backward compatibility for all configurations already using the genimage
> tool for creating the image to be written to the SD card:
> 
> BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
> +BR2_ROOTFS_POST_IMAGE_SCRIPT_ARGS="-c board/<manufacturer>/<boardname>/genimage.cfg -b"
> +BR2_PACKAGE_HOST_BMAP_TOOLS=y
> 
> It follows that the script now assumes a broader functionality that
> extends beyond just being an interface for the genimage tool.
> 
> An alternative implementation could have been to create another script,
> such as support/scripts/bmap.sh, capable of creating the bmap image
> using the same parameters passed to support/scripts/genimage.sh (i. e.
> -c board/<manufacturer>/<boardname>/genimage.cfg). However, in this case,
> the script would also need to know that the image is located in the
> ${BINARIES} directory. This could be achieved by adding an additional
> parameter, but it might cause genimage.sh to fail due to an unrecognized
> parameter.
> 
> Signed-off-by: Dario Binacchi <dario.binacchi@amarulasolutions.com>

We took advantage of the on-going Buildroot hackathon to discuss live,
and we pushed your patch, but with some changes:

(1) We don't add an option to script, but rather we look in the
Buildroot .config file is host-bmap-tools has been enabled. If it is
enabled, we assume the user wants bmap images

(2) We generate bmap images for all images, as long as they exist in
$(BINARIES_DIR). Indeed your trick of counting how many times the name
of the image was referenced in the genimage.cfg to try to guess what is
the "final" image (like sdcard.img) would fail badly if the final image
was:

image a {

}

because the image name "a" is probably found more than once in the
genimage.cfg.

However, with that, we are not going to apply your PATCH 2/2, because
we don't want to force everyone to build host-bmap-tools. Now everyone
only has to enable BR2_PACKAGE_HOST_BMAP_TOOLS=y in their configuration
to automagically get bmap-capable images generated. Perhaps what should
be done now is to document this somewhere in the manual. The question
is of course where to document it, and on that I'm not sure.

Note: it might be relevant to also adjust the filesystem generation
code in fs/common.mk to also generate a bmap-capable image of the
rootfs image itself, also based on the availability of host-bmap-tools.

Thanks for your work!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Arnout Vandecappelle via buildroot]

On 15/07/2024 16:05, Thomas Petazzoni via buildroot wrote:
> Hello Dario,
> 
> On Sun, 21 Apr 2024 11:53:52 +0200
> Dario Binacchi <dario.binacchi@amarulasolutions.com> wrote:
> 
>> The patch adds an option to create, in addition to the usual image, an
>> image of type bmap that drastically reduces the amount of data that
>> needs to be written to an SD card, resulting in time savings.
>>
>> This makes it possible to activate this option easily and maintain
>> backward compatibility for all configurations already using the genimage
>> tool for creating the image to be written to the SD card:
>>
>> BR2_ROOTFS_POST_IMAGE_SCRIPT="support/scripts/genimage.sh"
>> +BR2_ROOTFS_POST_IMAGE_SCRIPT_ARGS="-c board/<manufacturer>/<boardname>/genimage.cfg -b"
>> +BR2_PACKAGE_HOST_BMAP_TOOLS=y
>>
>> It follows that the script now assumes a broader functionality that
>> extends beyond just being an interface for the genimage tool.
>>
>> An alternative implementation could have been to create another script,
>> such as support/scripts/bmap.sh, capable of creating the bmap image
>> using the same parameters passed to support/scripts/genimage.sh (i. e.
>> -c board/<manufacturer>/<boardname>/genimage.cfg). However, in this case,
>> the script would also need to know that the image is located in the
>> ${BINARIES} directory. This could be achieved by adding an additional
>> parameter, but it might cause genimage.sh to fail due to an unrecognized
>> parameter.
>>
>> Signed-off-by: Dario Binacchi <dario.binacchi@amarulasolutions.com>
> 
> We took advantage of the on-going Buildroot hackathon to discuss live,
> and we pushed your patch, but with some changes:
> 
> (1) We don't add an option to script, but rather we look in the
> Buildroot .config file is host-bmap-tools has been enabled. If it is
> enabled, we assume the user wants bmap images

  Instead of looking in the config, we could also look for it in PATH (with "if 
which bmaptool >/dev/null 2>&1").

  Regards,
  Arnout

> 
> (2) We generate bmap images for all images, as long as they exist in
> $(BINARIES_DIR). Indeed your trick of counting how many times the name
> of the image was referenced in the genimage.cfg to try to guess what is
> the "final" image (like sdcard.img) would fail badly if the final image
> was:
> 
> image a {
> 
> }
> 
> because the image name "a" is probably found more than once in the
> genimage.cfg.
> 
> However, with that, we are not going to apply your PATCH 2/2, because
> we don't want to force everyone to build host-bmap-tools. Now everyone
> only has to enable BR2_PACKAGE_HOST_BMAP_TOOLS=y in their configuration
> to automagically get bmap-capable images generated. Perhaps what should
> be done now is to document this somewhere in the manual. The question
> is of course where to document it, and on that I'm not sure.
> 
> Note: it might be relevant to also adjust the filesystem generation
> code in fs/common.mk to also generate a bmap-capable image of the
> rootfs image itself, also based on the availability of host-bmap-tools.
> 
> Thanks for your work!
> 
> Thomas
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Thomas Petazzoni via buildroot]

On Mon, 15 Jul 2024 16:23:53 +0200
Arnout Vandecappelle <arnout@mind.be> wrote:

> > (1) We don't add an option to script, but rather we look in the
> > Buildroot .config file is host-bmap-tools has been enabled. If it is
> > enabled, we assume the user wants bmap images  
> 
>   Instead of looking in the config, we could also look for it in PATH (with "if 
> which bmaptool >/dev/null 2>&1").

Yes, that was another option, but then for the same Buildroot
configuration and version, the build results would be different from
machine A (which has bmap-tools installed system-wide) and machine B
(which doesn't have bmap-tools installed system-wide). From my
perspective, this breaks the principle of least surprise.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Dario Binacchi]

Hello Thomas and Arnount,

Thank you for applying the patch. More than once, during the patch
tests, I thought to myself:
If only I could speed up the flashing of the board using bmaptool.
And finally, it is possible.

And thanks also for the suggestions on further developments that could
be carried out.

Thanks and regards,
Dario

On Mon, Jul 15, 2024 at 4:27 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> On Mon, 15 Jul 2024 16:23:53 +0200
> Arnout Vandecappelle <arnout@mind.be> wrote:
>
> > > (1) We don't add an option to script, but rather we look in the
> > > Buildroot .config file is host-bmap-tools has been enabled. If it is
> > > enabled, we assume the user wants bmap images
> >
> >   Instead of looking in the config, we could also look for it in PATH (with "if
> > which bmaptool >/dev/null 2>&1").
>
> Yes, that was another option, but then for the same Buildroot
> configuration and version, the build results would be different from
> machine A (which has bmap-tools installed system-wide) and machine B
> (which doesn't have bmap-tools installed system-wide). From my
> perspective, this breaks the principle of least surprise.
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com



-- 

Dario Binacchi

Senior Embedded Linux Developer

dario.binacchi@amarulasolutions.com

__________________________________


Amarula Solutions SRL

Via Le Canevare 30, 31100 Treviso, Veneto, IT

T. +39 042 243 5310
info@amarulasolutions.com

www.amarulasolutions.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni via buildroot <buildroot@buildroot.org> writes:

Hi,

 >> An alternative implementation could have been to create another script,
 >> such as support/scripts/bmap.sh, capable of creating the bmap image
 >> using the same parameters passed to support/scripts/genimage.sh (i. e.
 >> -c board/<manufacturer>/<boardname>/genimage.cfg). However, in this case,
 >> the script would also need to know that the image is located in the
 >> ${BINARIES} directory. This could be achieved by adding an additional
 >> parameter, but it might cause genimage.sh to fail due to an unrecognized
 >> parameter.
 >> 
 >> Signed-off-by: Dario Binacchi <dario.binacchi@amarulasolutions.com>

 > We took advantage of the on-going Buildroot hackathon to discuss live,
 > and we pushed your patch, but with some changes:

 > (1) We don't add an option to script, but rather we look in the
 > Buildroot .config file is host-bmap-tools has been enabled. If it is
 > enabled, we assume the user wants bmap images

Sorry for the slow response, I only got to this now while backporting
for LTS, but I don't like the fact that it hardcodes gzip compression
(with a specific compression level), especially as it is
hidden/optionless now, E.G. it seems quite random that if you have a
genimage.cfg creating a sdcard.img (or whatever) and you then enable
host-bmaptools you all of a sudden don't have sdcard.img anymore, but
sdcard.img.gz (and sdcard.img.bmap) and "waste" time doing the
compression you may or may not want.

I would suggest to drop the gzip step and only create the .bmap files
(which presumably is quite fast?)

What do you say?

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Thomas Petazzoni via buildroot]

On Tue, 27 Aug 2024 20:55:00 +0200
Peter Korsgaard <peter@korsgaard.com> wrote:

> Sorry for the slow response, I only got to this now while backporting
> for LTS, but I don't like the fact that it hardcodes gzip compression
> (with a specific compression level), especially as it is
> hidden/optionless now, E.G. it seems quite random that if you have a
> genimage.cfg creating a sdcard.img (or whatever) and you then enable
> host-bmaptools you all of a sudden don't have sdcard.img anymore, but
> sdcard.img.gz (and sdcard.img.bmap) and "waste" time doing the
> compression you may or may not want.
> 
> I would suggest to drop the gzip step and only create the .bmap files
> (which presumably is quite fast?)
> 
> What do you say?

I understand your concern, but compression is also what helps in making
those sparse images small. Of course, as long as you keep them where
Buildroot generated them, they are sparse, so they don't take much
space. But as soon as you "distribute" them, they usually loose their
sparse-ness, and this is where compressing the image + the bmap
metadata file helps in lot in reducing the amount of data.

But fair enough, I assume this compression can be done as a custom
post-processing step by whoever needs it.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

Hi,

 >> I would suggest to drop the gzip step and only create the .bmap files
 >> (which presumably is quite fast?)
 >> 
 >> What do you say?

 > I understand your concern, but compression is also what helps in making
 > those sparse images small.

NIT: No, it is the sparseness (E.G. the holes). Compression in fact
breaks the holes (replaces them by zeros):

truncate -s 1G disk.img
du -hs disk.img
0       disk.img

gzip disk.img
du -hs disk.img.gz
1020K   disk.img.gz

gunzip disk.img.gz
du -hs disk.img
1.0G    disk.img

To transfer a file with holes without losing them you need an archival
format that understands sparse files, E.G. tar -S:

truncate -s 1G disk.img
du -hs disk.img
0       disk.img

tar cfSz disk.tar.gz disk.img
du -hs disk.tar.gz
4.0K    disk.tar.gz

rm disk.img
tar zxf disk.tar.gz
du -hs disk.img
0       disk.img

Notice also that the .tar.gz is a lot smaller than the img.gz


 > Of course, as long as you keep them where Buildroot generated them,
 > they are sparse, so they don't take much space. But as soon as you
 > "distribute" them, they usually loose their sparse-ness, and this is
 > where compressing the image + the bmap metadata file helps in lot in
 > reducing the amount of data.

Correct, unless an archival format supporting sparse files is used, so
if anything we should make a tarball of the images - But that is also
not so simple, E.G.:

- Busybox tar cannot extract sparse tar files
- One tarball for everything or a tarball for each image?
- What compression algorithm and parameters to use?

So it very fast ends up in a very use case specific setup, hence my
suggestion to NOT do it and leave such customizations to a post-image
script.

 > But fair enough, I assume this compression can be done as a custom
 > post-processing step by whoever needs it.

Indeed.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Thomas Petazzoni via buildroot]

Hello Peter,

On Wed, 28 Aug 2024 15:58:21 +0200
Peter Korsgaard <peter@korsgaard.com> wrote:

>  > I understand your concern, but compression is also what helps in making
>  > those sparse images small.  
> 
> NIT: No, it is the sparseness (E.G. the holes). Compression in fact
> breaks the holes (replaces them by zeros):

You didn't understand my point, and Arnout explained it better on the
chat.

My point is that it is *very* easy to lose the sparseness. You copy the
file around, you share it in Google Drive, or download it over HTTP or
what not, and bang the sparseness is lost.

Instead, with a compressed file, you get a file that has pretty much
the same size as the sparse file, but you can move it around, transfer
it, it won't consume its real size. And the bmap metadata file allows
to use/flash that compressed file with the same efficiency as the
original sparse file.

In fact, bmap is quite useless is the file sparseness is retained:
instead of using bmap metadata, you could just ask what are the holes
of the file and skip them when flashing. So really the whole point of
bmap metadata IMO is that they allow to do fast flashing (skipping
holes) even if the sparseness of the file has been lost.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

Hi,

 >> NIT: No, it is the sparseness (E.G. the holes). Compression in fact
 >> breaks the holes (replaces them by zeros):

 > You didn't understand my point, and Arnout explained it better on the
 > chat.

 > My point is that it is *very* easy to lose the sparseness. You copy the
 > file around, you share it in Google Drive, or download it over HTTP or
 > what not, and bang the sparseness is lost.

Sure, but that is all about distribution, where as mentioned before you
may want to compress it with gz/xz/zstd/.. and/or stick it in a sensible
archival format like tar, ..

The specifics of that is very use case specific, so IMHO not something
where we should enforce silly old .gz - Especially as bmap supports a
number of different formats (by shelling out to the extractors). From
bmaptool/TransRead.py:

This module allows opening and reading local and remote files and decompress
them on-the-fly if needed. Remote files are read using urllib (except of
"ssh://" URLs, which are handled differently). Supported file extentions are:
'bz2', 'gz', 'xz', 'lzo', 'zst' and a "tar" version of them: 'tar.bz2', 'tbz2',
'tbz', 'tb2', 'tar.gz', 'tgz', 'tar.xz', 'txz', 'tar.lzo', 'tzo', 'tar.lz4',
'tlz4', '.tar.zst', 'tzst'.
This module uses the following system programs for decompressing: pbzip2, bzip2,
gzip, pigz, xz, lzop, lz4, zstd, tar and unzip.


 > Instead, with a compressed file, you get a file that has pretty much
 > the same size as the sparse file, but you can move it around, transfer
 > it, it won't consume its real size. And the bmap metadata file allows
 > to use/flash that compressed file with the same efficiency as the
 > original sparse file.

 > In fact, bmap is quite useless is the file sparseness is retained:
 > instead of using bmap metadata, you could just ask what are the holes
 > of the file and skip them when flashing. So really the whole point of
 > bmap metadata IMO is that they allow to do fast flashing (skipping
 > holes) even if the sparseness of the file has been lost.

Does such a tool exist? I am not aware of one. If so, maybe we should
recommend that instead of dd for writing sdcard.img from the host?

Anyway, I find bmap a bit half-baked, E.G. rather than wasting time (and
space) compressing and decompressing all the holes that are then just
ignored it would make more sense to have a file format encoding the
holes (E.G. tar --sparse or something custom), then optionally compress
that for transmission and have something small and simple (E.G. not in
python) on the host/target to write it to a device.

But oh well, it is what it is. I don't need such a tool often enough to
care to write it ;)

I'll send a patch to drop the gzip step from genimage.sh.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Thomas Petazzoni via buildroot]

Hello Peter,

On Fri, 30 Aug 2024 09:14:37 +0200
Peter Korsgaard <peter@korsgaard.com> wrote:

> Does such a tool exist? I am not aware of one. If so, maybe we should
> recommend that instead of dd for writing sdcard.img from the host?

I'm not aware of any tool like this.

> 
> Anyway, I find bmap a bit half-baked, E.G. rather than wasting time (and
> space) compressing and decompressing all the holes that are then just
> ignored it would make more sense to have a file format encoding the
> holes (E.G. tar --sparse or something custom), then optionally compress
> that for transmission and have something small and simple (E.G. not in
> python) on the host/target to write it to a device.

The Android sparse image format looks like to be what you want.
See https://2net.co.uk/tutorial/android-sparse-image-format. There are some
C/C++ tools at https://android.googlesource.com/platform/system/core/+/master/libsparse/
to generate/manipulate Android sparse images.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Michael Nazzareno Trimarchi]

Hi Thomas

On Fri, Aug 30, 2024 at 5:29 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Peter,
>
> On Fri, 30 Aug 2024 09:14:37 +0200
> Peter Korsgaard <peter@korsgaard.com> wrote:
>
> > Does such a tool exist? I am not aware of one. If so, maybe we should
> > recommend that instead of dd for writing sdcard.img from the host?
>
> I'm not aware of any tool like this.
>
> >
> > Anyway, I find bmap a bit half-baked, E.G. rather than wasting time (and
> > space) compressing and decompressing all the holes that are then just
> > ignored it would make more sense to have a file format encoding the
> > holes (E.G. tar --sparse or something custom), then optionally compress
> > that for transmission and have something small and simple (E.G. not in
> > python) on the host/target to write it to a device.
>
> The Android sparse image format looks like to be what you want.
> See https://2net.co.uk/tutorial/android-sparse-image-format. There are some
> C/C++ tools at https://android.googlesource.com/platform/system/core/+/master/libsparse/
> to generate/manipulate Android sparse images.
>

As I mention some time ago android sparse format can be generate with
genimage.cfg

image sdcard-sparse.img {
    android-sparse {
        image = sdcard.img
    }
}

Nothing needs to be changed. Fastboot should able to handle it

Michael

> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com



-- 
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

Hi,

>> Anyway, I find bmap a bit half-baked, E.G. rather than wasting time (and
 >> space) compressing and decompressing all the holes that are then just
 >> ignored it would make more sense to have a file format encoding the
 >> holes (E.G. tar --sparse or something custom), then optionally compress
 >> that for transmission and have something small and simple (E.G. not in
 >> python) on the host/target to write it to a device.

 > The Android sparse image format looks like to be what you want.
 > See https://2net.co.uk/tutorial/android-sparse-image-format. There are some
 > C/C++ tools at https://android.googlesource.com/platform/system/core/+/master/libsparse/
 > to generate/manipulate Android sparse images.

Ahh yes, that looks more sensible than the bmap stuff - Thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC PATCH 1/2] support/scripts/genimage.sh: support creating a bmap image

[Peter Korsgaard]

>>>>> "Michael" == Michael Nazzareno Trimarchi <michael@amarulasolutions.com> writes:

Hi,

 >> The Android sparse image format looks like to be what you want.
 >> See https://2net.co.uk/tutorial/android-sparse-image-format. There are some
 >> C/C++ tools at https://android.googlesource.com/platform/system/core/+/master/libsparse/
 >> to generate/manipulate Android sparse images.
 >> 

 > As I mention some time ago android sparse format can be generate with
 > genimage.cfg

 > image sdcard-sparse.img {
 >     android-sparse {
 >         image = sdcard.img
 >     }
 > }

 > Nothing needs to be changed. Fastboot should able to handle it

Ahh nice, I wasn't aware of that!

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot