Re: xfs/076 takes a long long time testing with a realtime volume

Darrick J. Wong
2021-11-06T02:08:06Z

On Fri, Nov 05, 2021 at 09:58:14PM -0400, Theodore Ts'o wrote:
> After committing some exclusions into my test runner framework (see
> below), I tested a potential fix to xfs/076 which disables the
> real-time volume when creating the scratch volume. Should I send it
> as a formal patch to fstests?

Does adding:

_xfs_force_bdev data $SCRATCH_MNT

right after _scratch_mount make the performance problem go away? Sparse
inodes and realtime are a supported configuration.

--D

> diff --git a/tests/xfs/076 b/tests/xfs/076
> index eac7410e..5628c08f 100755
> --- a/tests/xfs/076
> +++ b/tests/xfs/076
> @@ -60,6 +60,7 @@ _require_xfs_io_command "falloc"
> _require_xfs_io_command "fpunch"
> _require_xfs_sparse_inodes
>
> +unset SCRATCH_RTDEV
> _scratch_mkfs "-d size=50m -m crc=1 -i sparse" |
> _filter_mkfs > /dev/null 2> $tmp.mkfs
> . $tmp.mkfs # for isize
>
> - Ted
>
> For why this is needed, see the commit description below:
>
> commit c41ae1cc0b21eafd2858541c0bc195f951c0726c
> Author: Theodore Ts'o <tytso@mit.edu>
> Date: Fri Nov 5 20:46:19 2021 -0400
>
> test-appliance: exclude xfs/076 from the realtime configs
>
> The xfs/076 test takes two minutes on a normal xfs file system (e.g.,
> a normal 4k block size file system). However, when there is a
> real-time volume attached, this test takes over 80 minutes. The
> reason for this seems to be because the test is spending a lot more
> time failing to create files due to missing directories. Compare:
>
> root@xfstests-2:~# ls -sh /results/xfs/results-4k/xfs/076.full
> 48K /results/xfs/results-4k/xfs/076.full
> root@xfstests-2:~# ls -sh /tmp/realtime-076.full
> 25M /tmp/realtime-076.full
>
> and:
>
> root@xfstests-2:~# grep "cannot touch" /results/xfs/results-4k/xfs/076.full | wc -l
> 656
> root@xfstests-2:~# grep "cannot touch" /tmp/realtime-076.full | wc -l
> 327664
>
> The failures from 076.full look like this:
>
> touch: cannot touch '/xt-vdc/offset.21473722368/25659': No space left on device
> touch: cannot touch '/xt-vdc/offset.21473656832/0': No such file or directory
> touch: cannot touch '/xt-vdc/offset.21473591296/0': No such file or directory
> ...
> touch: cannot touch '/xt-vdc/offset.196608/0': No such file or directory
> touch: cannot touch '/xt-vdc/offset.131072/0': No such file or directory
> touch: cannot touch '/xt-vdc/offset.65536/0': No such file or directory
> touch: cannot touch '/xt-vdc/offset.0/0': No such file or directory
>
> What seems to be going on is that xfs/076 tries to create a small
> scratch file system --- but when we attach a real-time volume this
> balloons the available size of the file system. Of course, that space
> can't be used for normal files. As a result, xfs/076 is incorrectly
> estimating how many files it needs to create to fill the file system.
>
> I'm not sure what's the best way to fix this in the test; perhaps the
> test should forcibly unset SCRATCH_RTDEV environment variable before
> running _scratch_mkfs? Anyway, for now, we'll just skip running
> xfs/076 for the xfs/realtime* configs.
>
> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
>
> diff --git a/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime.exclude b/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime.exclude
> index a9acba9c..bafce552 100644
> --- a/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime.exclude
> +++ b/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime.exclude
> @@ -1,2 +1,7 @@
> # Normal configurations don't support dax
> -g dax
> +
> +# The xfs/076 test takes well over an hour (80 minutes using 100GB GCE
> +# PD/SSD) when run with an external realtime device, which triggers
> +# the ltm "test is stalled" failsafe which aborts the VM.
> +xfs/076
> diff --git a/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime_28k_logdev.exclude b/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime_28k_logdev.exclude
> index a9acba9c..bafce552 100644
> --- a/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime_28k_logdev.exclude
> +++ b/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime_28k_logdev.exclude
> @@ -1,2 +1,7 @@
> # Normal configurations don't support dax
> -g dax
> +
> +# The xfs/076 test takes well over an hour (80 minutes using 100GB GCE
> +# PD/SSD) when run with an external realtime device, which triggers
> +# the ltm "test is stalled" failsafe which aborts the VM.
> +xfs/076
> diff --git a/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime_logdev.exclude b/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime_logdev.exclude
> index a9acba9c..bafce552 100644
> --- a/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime_logdev.exclude
> +++ b/kvm-xfstests/test-appliance/files/root/fs/xfs/cfg/realtime_logdev.exclude
> @@ -1,2 +1,7 @@
> # Normal configurations don't support dax
> -g dax
> +
> +# The xfs/076 test takes well over an hour (80 minutes using 100GB GCE
> +# PD/SSD) when run with an external realtime device, which triggers
> +# the ltm "test is stalled" failsafe which aborts the VM.
> +xfs/076