Re: [OE-core] [PATCH] wic partition.py: add --apparent-size to du calls

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Tue, 2024-02-27 at 13:55 +0200, Mikko Rapeli wrote:
> On Tue, Feb 27, 2024 at 11:48:27AM +0000, Richard Purdie wrote:
> > On Tue, 2024-02-27 at 13:23 +0200, Mikko Rapeli wrote:
> > > If build happens on zfs filesystem with compression enabled,
> > > then image size calculations in do_image_wic task can fail:
> > > 
> > > output: mke2fs 1.47.0 (5-Feb-2023)
> > > Discarding device blocks: done
> > > Creating filesystem with 351999 4k blocks and 176000 inodes
> > > Filesystem UUID: 6091b3a4-ce08-3020-93a6-f755a22ef03b
> > > Superblock backups stored on blocks:
> > >         32768, 98304, 163840, 229376, 294912
> > > 
> > > Allocating group tables: done
> > > Writing inode tables: done
> > > Creating journal (8192 blocks): done
> > > Copying files into the device: __populate_fs: Could not allocate
> > > block in ext2 filesystem while writing file "service-2.json"
> > > mkfs.ext4: Could not allocate block in ext2 filesystem while
> > > populating file system
> > > 
> > > du --help says:
> > > 
> > >       --apparent-size   print apparent sizes, rather than disk usage;
> > > although
> > >                           the apparent size is usually smaller, it
> > > may be
> > >                           larger due to holes in ('sparse') files,
> > > internal
> > >                           fragmentation, indirect blocks, and the
> > > like
> > > 
> > > du -b already includes --apparent-size.
> > > 
> > > Same issue reported also in
> > > https://lists.yoctoproject.org/g/poky/message/12389
> > > 
> > > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> > 
> > 
> > From memory we've gone around in circles on this. In some cases we do
> > really want the size in bytes but in others we want it rounded up to
> > block size for each file. I think this was why what bug was never fixed
> > as you can address one set of issues but create a new set.
> > 
> > So for that reason I'm extremely cautious about this, particularly this
> > late in the release cycle. I don't really know what to do with this
> > change.
> 
> I have a build machine with zfs and this issue pops up ever now and then.
> Right now I can't build there without this change. I don't see a way to
> fix this issue in my build/host specific configuration.

Should we break it for others so it works for you?

If I merge this patch and then I get other reports in of issues, I'm
going to be left to somehow fix those other problems. Since your case
will work you will object if I revert this but you likely won't have
time to work on the resulting other issues, or be able to reproduce
them since you're building on zfs.

So you can see my dilemma? :/

Most people don't build on zfs which is probably what I need to balance
this on.

Cheers,

Richard