Re: [PATCH] iotests: Improve mirror-sparse on various filesystems

[Fiona Ebner <f.ebner@proxmox.com>]

Am 22.05.25 um 23:14 schrieb Eric Blake:
> Fiona reported that an ext4 filesystem on top of LVM can sometimes
> report over-allocation to du (based on the hueristics the filesystem

Typo: heuristics

> is making while observing the contents being mirrored); even though
> the contents and actual size matched, about 50% of the time the size
> reported by disk_usage was too large by 4k, failing the test.
> 
> Similarly, on ZFS where a file is created with preallocation=full, du
> does not see the full allocation until things have had time to settle;
> adding a sync call reduces the chance of catching that async window:
> 
> | [I] febner@enia ~/qemu/build/tests/qemu-iotests (master)> qemu-img create my.raw 20M -f
> |w -o preallocation=full
> | Formatting 'my.raw', fmt=raw size=20971520 preallocation=full
> | [I] febner@enia ~/qemu/build/tests/qemu-iotests (master)> du --block-size=1 my.raw
> | 512 my.raw
> | [I] febner@enia ~/qemu/build/tests/qemu-iotests (master)> du --block-size=1 my.raw
> | 20980224    my.raw
> 
> Fiona also reported that on a compressed ZFS, the filesystem can end
> up reporting smaller disk_usage if it re-compresses a file, despite a
> fully-allocating mirror - but since I don't have a compressed ZFS
> handy for reproducing that test, that may remain a sporadic problem
> for another day.
> 
> Reported-by: Fiona Ebner <f.ebner@proxmox.com>
> Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
> Signed-off-by: Eric Blake <eblake@redhat.com>

If you drop the hunk with the sync:
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Fiona Ebner <f.ebner@proxmox.com>

> ---
>  tests/qemu-iotests/common.rc           | 2 ++
>  tests/qemu-iotests/tests/mirror-sparse | 4 +++-
>  2 files changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
> index 237f746af88..c3fc0bcf02a 100644
> --- a/tests/qemu-iotests/common.rc
> +++ b/tests/qemu-iotests/common.rc
> @@ -143,6 +143,8 @@ _optstr_add()
>  # report real disk usage for sparse files
>  disk_usage()
>  {
> +    # ZFS has lazy allocation; sync the file first for best results
> +    sync "$1"

Unfortunately, just syncing the file seems to be not enough. It seems to
be necessary to sync the whole filesystem, but that of course is too
expensive for the helper here:

[I] febner@enia ~> rm my.raw; qemu-img create my.raw 20M -f raw -o
preallocation=full; sync my.raw; du --block-size=1 my.raw
Formatting 'my.raw', fmt=raw size=20971520 preallocation=full
512	my.raw
[I] febner@enia ~> rm my.raw; qemu-img create my.raw 20M -f raw -o
preallocation=full; sync -f my.raw; du --block-size=1 my.raw
Formatting 'my.raw', fmt=raw size=20971520 preallocation=full
20980224	my.raw

There's already quite a few other test failures on ZFS, so I guess it's
not worth it right now if there's no easy fix (mirror-sparse also still
fails, because the file is sparse again after mirroring, ZFS seems very
aggressive trying to reduce allocation):

For -raw:
Failures: 106 109 150 175 221 240 253 308 mirror-sparse write-zeroes-unmap
Failed 10 of 86 iotests


>      du --block-size=1 "$1" | awk '{print $1}'
>  }
> 

Best Regards,
Fiona