[PATCH] btrfs: test fallocate against a file range with a mix of holes and extents

[PATCH] btrfs: test fallocate against a file range with a mix of holes and extents

[fdmanana]

From: Filipe Manana <fdmanana@suse.com>

Test that if we call fallocate against a file range that has a mix of holes
and written extents, the fallocate succeeds if the filesystem has enough free
space to allocate extents for the holes.

This currently fails on btrfs and is fixed by a patch that has the subject:

  "btrfs: only reserve the needed data space amount during fallocate"

There's nothing that is really specific to btrfs in the test case, and the
same test passes on ext4 and f2fs for example. But it fails on xfs, and
after some discussion with Darrick, it seems it's due to technical reasons
that would require a significant effort to xfs's implementation, and at
the moment there isn't enough motivation to do such change. The relevent
thread is at:

   https://lore.kernel.org/linux-btrfs/20220315164011.GF8241@magnolia/

So for the time being, make the test btrfs specific.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
 tests/btrfs/261     | 58 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/261.out | 20 ++++++++++++++++
 2 files changed, 78 insertions(+)
 create mode 100755 tests/btrfs/261
 create mode 100644 tests/btrfs/261.out

diff --git a/tests/btrfs/261 b/tests/btrfs/261
new file mode 100755
index 00000000..e7e9f15e
--- /dev/null
+++ b/tests/btrfs/261
@@ -0,0 +1,58 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022 SUSE Linux Products GmbH.  All Rights Reserved.
+#
+# FS QA Test 261
+#
+# Test that if we call fallocate against a large file range that is nearly full
+# with written extents, the fallocate succeeds and allocates unwritten extents
+# for the holes in the range.
+#
+. ./common/preamble
+_begin_fstest auto quick prealloc
+
+. ./common/rc
+. ./common/filter
+. ./common/punch
+
+# real QA test starts here
+
+_supported_fs btrfs
+_require_scratch
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fiemap"
+
+rm -f $seqres.full
+
+# Create a 1G filesystem.
+_scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1
+_scratch_mount
+
+# Create a file with a size of 600M and two holes, each with a size of 1M and
+# at file ranges [200, 201M[ and [401M, 402M[.
+$XFS_IO_PROG -f -c "pwrite -S 0xab 0 200M" \
+                -c "pwrite -S 0xcd 201M 200M" \
+                -c "pwrite -S 0xef 402M 198M" \
+		$SCRATCH_MNT/foobar | _filter_xfs_io
+
+# Now call fallocate against the whole file range.
+# It should succeed, because only 2M of data space needs to be allocated,
+# and not 600M (which isn't available since our fs has a size of 1G).
+$XFS_IO_PROG -c "falloc 0 600M" $SCRATCH_MNT/foobar
+
+# Unmount and mount again the filesystem. We want to veriy that the fallocate
+# results were persisted and that all the file metadata and data on disk are
+# also correct.
+_scratch_cycle_mount
+
+echo -n "Number of unwritten extents in the file: "
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foobar | _filter_fiemap | \
+    grep 'unwritten' | wc -l
+
+# Verify we don't have any corruption caused by the fallocate.
+echo "File content after fallocate:"
+od -A d -t x1 $SCRATCH_MNT/foobar
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/261.out b/tests/btrfs/261.out
new file mode 100644
index 00000000..db7f0d6d
--- /dev/null
+++ b/tests/btrfs/261.out
@@ -0,0 +1,20 @@
+QA output created by 261
+wrote 209715200/209715200 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 209715200/209715200 bytes at offset 210763776
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 207618048/207618048 bytes at offset 421527552
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Number of unwritten extents in the file: 2
+File content after fallocate:
+0000000 ab ab ab ab ab ab ab ab ab ab ab ab ab ab ab ab
+*
+209715200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
+*
+210763776 cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd
+*
+420478976 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
+*
+421527552 ef ef ef ef ef ef ef ef ef ef ef ef ef ef ef ef
+*
+629145600
-- 
2.33.0

Re: [PATCH] btrfs: test fallocate against a file range with a mix of holes and extents

[Darrick J. Wong]

On Thu, Mar 17, 2022 at 03:03:40PM +0000, fdmanana@kernel.org wrote:
> From: Filipe Manana <fdmanana@suse.com>
> 
> Test that if we call fallocate against a file range that has a mix of holes
> and written extents, the fallocate succeeds if the filesystem has enough free
> space to allocate extents for the holes.
> 
> This currently fails on btrfs and is fixed by a patch that has the subject:
> 
>   "btrfs: only reserve the needed data space amount during fallocate"
> 
> There's nothing that is really specific to btrfs in the test case, and the
> same test passes on ext4 and f2fs for example. But it fails on xfs, and
> after some discussion with Darrick, it seems it's due to technical reasons
> that would require a significant effort to xfs's implementation, and at
> the moment there isn't enough motivation to do such change. The relevent
> thread is at:
> 
>    https://lore.kernel.org/linux-btrfs/20220315164011.GF8241@magnolia/
> 
> So for the time being, make the test btrfs specific.
> 
> Signed-off-by: Filipe Manana <fdmanana@suse.com>
> ---
>  tests/btrfs/261     | 58 +++++++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/261.out | 20 ++++++++++++++++
>  2 files changed, 78 insertions(+)
>  create mode 100755 tests/btrfs/261
>  create mode 100644 tests/btrfs/261.out
> 
> diff --git a/tests/btrfs/261 b/tests/btrfs/261
> new file mode 100755
> index 00000000..e7e9f15e
> --- /dev/null
> +++ b/tests/btrfs/261
> @@ -0,0 +1,58 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2022 SUSE Linux Products GmbH.  All Rights Reserved.
> +#
> +# FS QA Test 261
> +#
> +# Test that if we call fallocate against a large file range that is nearly full
> +# with written extents, the fallocate succeeds and allocates unwritten extents
> +# for the holes in the range.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick prealloc
> +
> +. ./common/rc
> +. ./common/filter
> +. ./common/punch
> +
> +# real QA test starts here
> +
> +_supported_fs btrfs

Or make this a generic test and add:

test $FSTYP == "xfs"  && _notrun "xfs doesn't work that way lol"

> +_require_scratch
> +_require_xfs_io_command "falloc"
> +_require_xfs_io_command "fiemap"
> +
> +rm -f $seqres.full
> +
> +# Create a 1G filesystem.
> +_scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1
> +_scratch_mount
> +
> +# Create a file with a size of 600M and two holes, each with a size of 1M and
> +# at file ranges [200, 201M[ and [401M, 402M[.
> +$XFS_IO_PROG -f -c "pwrite -S 0xab 0 200M" \
> +                -c "pwrite -S 0xcd 201M 200M" \
> +                -c "pwrite -S 0xef 402M 198M" \
> +		$SCRATCH_MNT/foobar | _filter_xfs_io
> +
> +# Now call fallocate against the whole file range.
> +# It should succeed, because only 2M of data space needs to be allocated,
> +# and not 600M (which isn't available since our fs has a size of 1G).
> +$XFS_IO_PROG -c "falloc 0 600M" $SCRATCH_MNT/foobar
> +
> +# Unmount and mount again the filesystem. We want to veriy that the fallocate

'verify'

Other than those two things, this test looks sound to me.

--D

> +# results were persisted and that all the file metadata and data on disk are
> +# also correct.
> +_scratch_cycle_mount
> +
> +echo -n "Number of unwritten extents in the file: "
> +$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foobar | _filter_fiemap | \
> +    grep 'unwritten' | wc -l
> +
> +# Verify we don't have any corruption caused by the fallocate.
> +echo "File content after fallocate:"
> +od -A d -t x1 $SCRATCH_MNT/foobar
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/261.out b/tests/btrfs/261.out
> new file mode 100644
> index 00000000..db7f0d6d
> --- /dev/null
> +++ b/tests/btrfs/261.out
> @@ -0,0 +1,20 @@
> +QA output created by 261
> +wrote 209715200/209715200 bytes at offset 0
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +wrote 209715200/209715200 bytes at offset 210763776
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +wrote 207618048/207618048 bytes at offset 421527552
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +Number of unwritten extents in the file: 2
> +File content after fallocate:
> +0000000 ab ab ab ab ab ab ab ab ab ab ab ab ab ab ab ab
> +*
> +209715200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> +*
> +210763776 cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd
> +*
> +420478976 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> +*
> +421527552 ef ef ef ef ef ef ef ef ef ef ef ef ef ef ef ef
> +*
> +629145600
> -- 
> 2.33.0
> 

[PATCH v2] generic: test fallocate against a file range with a mix of holes and extents

[fdmanana]

From: Filipe Manana <fdmanana@suse.com>

Test that if we call fallocate against a file range that has a mix of
holes and written extents, the fallocate succeeds if the filesystem has
enough free space to allocate extents for the holes.

This test currently fails on btrfs and is fixed by a patch that has the
following subject:

    "btrfs: only reserve the needed data space amount during fallocate"

The test also fails on xfs, and after some discussion with Darrick, it
seems it's due to technical reasons that would require a significant
effort to xfs's implementation, and at the moment there isn't enough
motivation to do such change. The relevent thread is at:

    https://lore.kernel.org/linux-btrfs/20220315164011.GF8241@magnolia/

Therefore the test is intentionally skipped on xfs only. Ext4 and f2fs
pass this test.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
---

V2: Fixed a typo, moved the test into the generic group while also
    making it skip xfs as per Darrick's suggestion.

 tests/generic/678     | 63 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/678.out | 20 ++++++++++++++
 2 files changed, 83 insertions(+)
 create mode 100755 tests/generic/678
 create mode 100644 tests/generic/678.out

diff --git a/tests/generic/678 b/tests/generic/678
new file mode 100755
index 00000000..9fe1bdcc
--- /dev/null
+++ b/tests/generic/678
@@ -0,0 +1,63 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022 SUSE Linux Products GmbH.  All Rights Reserved.
+#
+# FS QA Test 678
+#
+# Test that if we call fallocate against a file range that has a mix of holes
+# and written extents, the fallocate succeeds if the filesystem has enough free
+# space to allocate extents for the holes.
+#
+. ./common/preamble
+_begin_fstest auto quick prealloc
+
+. ./common/rc
+. ./common/filter
+. ./common/punch
+
+# real QA test starts here
+
+_supported_fs generic
+_require_scratch
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fiemap"
+
+# This test is currently not valid for xfs, see the following thread for details:
+#
+#   https://lore.kernel.org/linux-btrfs/20220315164011.GF8241@magnolia/
+#
+[ $FSTYP == "xfs" ] && _notrun "test not valid for xfs"
+
+rm -f $seqres.full
+
+# Create a 1G filesystem.
+_scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1
+_scratch_mount
+
+# Create a file with a size of 600M and two holes, each with a size of 1M and
+# at file ranges [200, 201M[ and [401M, 402M[.
+$XFS_IO_PROG -f -c "pwrite -S 0xab 0 200M" \
+                -c "pwrite -S 0xcd 201M 200M" \
+                -c "pwrite -S 0xef 402M 198M" \
+		$SCRATCH_MNT/foobar | _filter_xfs_io
+
+# Now call fallocate against the whole file range.
+# It should succeed, because only 2M of data space needs to be allocated,
+# and not 600M (which isn't available since our fs has a size of 1G).
+$XFS_IO_PROG -c "falloc 0 600M" $SCRATCH_MNT/foobar
+
+# Unmount and mount again the filesystem. We want to verify that the fallocate
+# results were persisted and that all the file data on disk are also correct.
+_scratch_cycle_mount
+
+echo -n "Number of unwritten extents in the file: "
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foobar | _filter_fiemap | \
+    grep "unwritten" | wc -l
+
+# Verify we don't have any corruption caused by the fallocate.
+echo "File content after fallocate:"
+od -A d -t x1 $SCRATCH_MNT/foobar
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/678.out b/tests/generic/678.out
new file mode 100644
index 00000000..61f800c1
--- /dev/null
+++ b/tests/generic/678.out
@@ -0,0 +1,20 @@
+QA output created by 678
+wrote 209715200/209715200 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 209715200/209715200 bytes at offset 210763776
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 207618048/207618048 bytes at offset 421527552
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Number of unwritten extents in the file: 2
+File content after fallocate:
+0000000 ab ab ab ab ab ab ab ab ab ab ab ab ab ab ab ab
+*
+209715200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
+*
+210763776 cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd
+*
+420478976 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
+*
+421527552 ef ef ef ef ef ef ef ef ef ef ef ef ef ef ef ef
+*
+629145600
-- 
2.33.0