[PATCH] doc/wic,README.hardware: add dd option conv=sparse

[PATCH] doc/wic,README.hardware: add dd option conv=sparse

[Jörg Sommer]

From: Jörg Sommer <joerg.sommer@navimatix.de>

The GNU implementation of dd supports the skipping of sparse blocks which
can speed-up the write, esp. of wic images with big holes:

    $ sudo dd if=image.wic of=/dev/sdb bs=2M
    2683+1 Datensätze ein
    2683+1 Datensätze aus
    5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 84,2773 s, 66,8 MB/s

    $ sudo dd if=image.wic of=/dev/sdb bs=2M conv=sparse
    2683+1 Datensätze ein
    2683+1 Datensätze aus
    5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 11,3314 s, 497 MB/s

Signed-off-by: Jörg Sommer <joerg.sommer@navimatix.de>
---
 documentation/dev-manual/wic.rst  |  8 +++++++-
 meta-yocto-bsp/README.hardware.md | 20 ++++++++++++++------
 2 files changed, 21 insertions(+), 7 deletions(-)


I don't know if I should amend the "dd if=directdisksdb-gpt-201710090938-sdb.direct" in line
595, because this wouldn't match the output of dd.


diff --git a/documentation/dev-manual/wic.rst b/documentation/dev-manual/wic.rst
index fced0e170c..792fa35fa7 100644
--- a/documentation/dev-manual/wic.rst
+++ b/documentation/dev-manual/wic.rst
@@ -507,7 +507,13 @@ built your image, and boot from the media. You can write the image by using
 
 or ::
 
-   $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
+   $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX bs=2M conv=sparse
+
+.. note::
+
+   The option ``conv=sparse`` is a GNU extension to skip writing of blocks
+   with purely zeros. If your ``dd`` (e.g. BusyBox) does not support it, you
+   can safely remove this option.
 
 .. note::
 
diff --git a/meta-yocto-bsp/README.hardware.md b/meta-yocto-bsp/README.hardware.md
index d2cc05f909..a824963e4e 100644
--- a/meta-yocto-bsp/README.hardware.md
+++ b/meta-yocto-bsp/README.hardware.md
@@ -91,7 +91,7 @@ Depending on the device, it can boot from a traditional hard-disk, a USB device,
 or over the network. Writing generated images to physical media is
 straightforward with a caveat for USB devices. The following examples assume the
 target boot device is /dev/sdb, be sure to verify this and use the correct
-device as the following commands are run as root and are not reversable.
+device as the following commands are run as root and are not reversible.
 
 USB Device:
 
@@ -105,7 +105,11 @@ USB Device:
   2. Use the `dd` utility to write the image to the raw block device. For
      example:
 
-         # dd if=core-image-minimal-genericx86.hddimg of=/dev/sdb
+         # dd if=core-image-minimal-genericx86.hddimg of=/dev/sdb bs=2M conv=sparse
+
+      The option `conv=sparse` is a GNU extension to skip writing of blocks with
+      purely zeros. If your `dd` doesn't support it, you can safely drop this
+      option.
 
   If the device fails to boot with "Boot error" displayed, or apparently
   stops just after the SYSLINUX version banner, it is likely the BIOS cannot
@@ -119,9 +123,11 @@ USB Device:
 
   2. Use a ".wic" image with an EFI partition
 
-     1. With a default grub-efi bootloader:
+     1. With a default grub-efi bootloader (`conv=sparse` is a GNU extension to
+        skip empty blocks for speed up; remove it, if your `dd` doesn't support
+        it):
 
-            # dd if=core-image-minimal-genericx86-64.wic of=/dev/sdb
+            # dd if=core-image-minimal-genericx86-64.wic of=/dev/sdb bs=2M conv=sparse
 
      2. Use systemd-boot instead. Build an image with `EFI_PROVIDER="systemd-boot"` then use the above
        `dd` command to write the image to a USB stick.
@@ -179,8 +185,10 @@ From a Linux system with access to the image files perform the following steps:
 
          $ bitbake core-image-minimal
 
-  2. Use the "dd" utility to write the image to the SD card. For example:
+  2. Use the `dd` utility to write the image to the SD card. For example
+     (`conv=sparse` is a GNU extension to skip empty blocks for speed up; remove
+     it, if your `dd` doesn't support it):
 
-         # dd if=core-image-minimal-beaglebone-yocto.wic of=/dev/sdb
+         # dd if=core-image-minimal-beaglebone-yocto.wic of=/dev/sdb bs=2M conv=sparse
 
   3. Insert the SD card into the Beaglebone and boot the board.
-- 
2.47.1

Re: [docs] [PATCH] doc/wic,README.hardware: add dd option conv=sparse

[Richard Purdie]

On Sun, 2025-01-26 at 13:37 +0100, Jörg Sommer via lists.yoctoproject.org wrote:
> From: Jörg Sommer <joerg.sommer@navimatix.de>
> 
> The GNU implementation of dd supports the skipping of sparse blocks which
> can speed-up the write, esp. of wic images with big holes:
> 
>     $ sudo dd if=image.wic of=/dev/sdb bs=2M
>     2683+1 Datensätze ein
>     2683+1 Datensätze aus
>     5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 84,2773 s, 66,8 MB/s
> 
>     $ sudo dd if=image.wic of=/dev/sdb bs=2M conv=sparse
>     2683+1 Datensätze ein
>     2683+1 Datensätze aus
>     5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 11,3314 s, 497 MB/s
> 
> Signed-off-by: Jörg Sommer <joerg.sommer@navimatix.de>
> ---
>  documentation/dev-manual/wic.rst  |  8 +++++++-
>  meta-yocto-bsp/README.hardware.md | 20 ++++++++++++++------
>  2 files changed, 21 insertions(+), 7 deletions(-)

I'm torn on this as it is potentially dangerous. It depends whether the
blocks really need to be zero or their contents doesn't matter. You
could leave existing data behind if you use the sparse option.

Cheers,

Richard

Re: [docs] [PATCH] doc/wic,README.hardware: add dd option conv=sparse

[Jörg Sommer]

Richard Purdie schrieb am So 26. Jan, 12:53 (+0000):
> On Sun, 2025-01-26 at 13:37 +0100, Jörg Sommer via lists.yoctoproject.org wrote:
> > From: Jörg Sommer <joerg.sommer@navimatix.de>
> > 
> > The GNU implementation of dd supports the skipping of sparse blocks which
> > can speed-up the write, esp. of wic images with big holes:
> > 
> >     $ sudo dd if=image.wic of=/dev/sdb bs=2M
> >     2683+1 Datensätze ein
> >     2683+1 Datensätze aus
> >     5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 84,2773 s, 66,8 MB/s
> > 
> >     $ sudo dd if=image.wic of=/dev/sdb bs=2M conv=sparse
> >     2683+1 Datensätze ein
> >     2683+1 Datensätze aus
> >     5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 11,3314 s, 497 MB/s
> > 
> > Signed-off-by: Jörg Sommer <joerg.sommer@navimatix.de>
> > ---
> >  documentation/dev-manual/wic.rst  |  8 +++++++-
> >  meta-yocto-bsp/README.hardware.md | 20 ++++++++++++++------
> >  2 files changed, 21 insertions(+), 7 deletions(-)
> 
> I'm torn on this as it is potentially dangerous. It depends whether the
> blocks really need to be zero or their contents doesn't matter. You
> could leave existing data behind if you use the sparse option.

Isn't this the same what bmaptool does?
https://github.com/yoctoproject/bmaptool?tab=readme-ov-file#concept


Bye, Jörg

-- 
Es ist außerdem ein weit verbreiteter Irrtum das USENET ‚helfen‘ soll.
Tatsächlich wurde USENET nachweislich zur persönlichen Belustigung
seiner Erfinder geschaffen.
Jörg Klemenz <joerg@gmx.net>, <b4ai4o$1u8vmt$2@ID-21915.news.dfncis.de>

Navimatix GmbH           T: 03641 - 327 99 0
Tatzendpromenade 2       F: 03641 - 526 306
07745 Jena               www.navimatix.de

Geschäftsführer: Steffen Späthe, Jan Rommeley
Registergericht: Amtsgericht Jena, HRB 501480

Re: [docs] [PATCH] doc/wic,README.hardware: add dd option conv=sparse

[Yoann Congal]

[-- Attachment #1: Type: text/plain, Size: 2373 bytes --]

Le dim. 26 janv. 2025 à 16:25, Jörg Sommer via lists.yoctoproject.org
<joerg.sommer=navimatix.de@lists.yoctoproject.org> a écrit :

> Richard Purdie schrieb am So 26. Jan, 12:53 (+0000):
> > On Sun, 2025-01-26 at 13:37 +0100, Jörg Sommer via
> lists.yoctoproject.org wrote:
> > > From: Jörg Sommer <joerg.sommer@navimatix.de>
> > >
> > > The GNU implementation of dd supports the skipping of sparse blocks
> which
> > > can speed-up the write, esp. of wic images with big holes:
> > >
> > >     $ sudo dd if=image.wic of=/dev/sdb bs=2M
> > >     2683+1 Datensätze ein
> > >     2683+1 Datensätze aus
> > >     5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 84,2773 s, 66,8 MB/s
> > >
> > >     $ sudo dd if=image.wic of=/dev/sdb bs=2M conv=sparse
> > >     2683+1 Datensätze ein
> > >     2683+1 Datensätze aus
> > >     5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 11,3314 s, 497 MB/s
> > >
> > > Signed-off-by: Jörg Sommer <joerg.sommer@navimatix.de>
> > > ---
> > >  documentation/dev-manual/wic.rst  |  8 +++++++-
> > >  meta-yocto-bsp/README.hardware.md | 20 ++++++++++++++------
> > >  2 files changed, 21 insertions(+), 7 deletions(-)
> >
> > I'm torn on this as it is potentially dangerous. It depends whether the
> > blocks really need to be zero or their contents doesn't matter. You
> > could leave existing data behind if you use the sparse option.
>
> Isn't this the same what bmaptool does?
> https://github.com/yoctoproject/bmaptool?tab=readme-ov-file#concept
>

It look it does not. bmaptool readme is very particular on what a "unmapped
block" is. And is not a zero block.
TL;DR: (a gross reinterpretation) that is a block not referenced anywhere
in your filesystem.

Imagine a file filled with zeroes on your filesystem. Those zeros are
referenced so must be flashed. bmaptool will flash them.
"dd conv=sparse" will not and this is only right in a corner case : if you
already know that your disk/memory is zero filled.
But, in the general case, you'll get a "zero file" filled with garbage.

Sorry, but I'm against this change since it will be a source of weird/hard
bugs.
Maybe instead of changing the dd command to speedup we can push users to
"bmaptool copy" (we will get speedup without compromising on the validity
of the filesystem)?

Regards,
-- 
Yoann Congal
Smile ECS - Tech expert

[-- Attachment #2: Type: text/html, Size: 3755 bytes --]

Re: [docs] [PATCH] doc/wic,README.hardware: add dd option conv=sparse

[Antonin Godard]

Hi all,

On Sun Jan 26, 2025 at 5:56 PM CET, Yoann Congal via lists.yoctoproject.org wrote:
> Le dim. 26 janv. 2025 à 16:25, Jörg Sommer via lists.yoctoproject.org
> <joerg.sommer=navimatix.de@lists.yoctoproject.org> a écrit :
>
>> Richard Purdie schrieb am So 26. Jan, 12:53 (+0000):
>> > On Sun, 2025-01-26 at 13:37 +0100, Jörg Sommer via
>> lists.yoctoproject.org wrote:
>> > > From: Jörg Sommer <joerg.sommer@navimatix.de>
>> > >
>> > > The GNU implementation of dd supports the skipping of sparse blocks
>> which
>> > > can speed-up the write, esp. of wic images with big holes:
>> > >
>> > >     $ sudo dd if=image.wic of=/dev/sdb bs=2M
>> > >     2683+1 Datensätze ein
>> > >     2683+1 Datensätze aus
>> > >     5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 84,2773 s, 66,8 MB/s
>> > >
>> > >     $ sudo dd if=image.wic of=/dev/sdb bs=2M conv=sparse
>> > >     2683+1 Datensätze ein
>> > >     2683+1 Datensätze aus
>> > >     5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 11,3314 s, 497 MB/s
>> > >
>> > > Signed-off-by: Jörg Sommer <joerg.sommer@navimatix.de>
>> > > ---
>> > >  documentation/dev-manual/wic.rst  |  8 +++++++-
>> > >  meta-yocto-bsp/README.hardware.md | 20 ++++++++++++++------
>> > >  2 files changed, 21 insertions(+), 7 deletions(-)
>> >
>> > I'm torn on this as it is potentially dangerous. It depends whether the
>> > blocks really need to be zero or their contents doesn't matter. You
>> > could leave existing data behind if you use the sparse option.
>>
>> Isn't this the same what bmaptool does?
>> https://github.com/yoctoproject/bmaptool?tab=readme-ov-file#concept
>>
>
> It look it does not. bmaptool readme is very particular on what a "unmapped
> block" is. And is not a zero block.
> TL;DR: (a gross reinterpretation) that is a block not referenced anywhere
> in your filesystem.
>
> Imagine a file filled with zeroes on your filesystem. Those zeros are
> referenced so must be flashed. bmaptool will flash them.
> "dd conv=sparse" will not and this is only right in a corner case : if you
> already know that your disk/memory is zero filled.
> But, in the general case, you'll get a "zero file" filled with garbage.
>
> Sorry, but I'm against this change since it will be a source of weird/hard
> bugs.
> Maybe instead of changing the dd command to speedup we can push users to
> "bmaptool copy" (we will get speedup without compromising on the validity
> of the filesystem)?

I agree with Yoann, I'm not for providing conv=sparse as a default option for
dd.

I also vote for redirecting to bmaptool. This can be done with:

  :ref:`bmaptool <dev-manual/bmaptool:flashing images using \`bmaptool\`>`

I think in a `..note ::` like you did Jörg, would be good.


Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com