[PATCH V3 0/4] Large extent counters tests

[PATCH V3 0/4] Large extent counters tests

[Chandan Babu R]

Hi all,

This patchset adds two tests for verifying the behaviour of Large
extent counters feature. It also fixes xfs/270 test failure when
executed on a filesystem with Large extent counters enabled.

Changelog:
V2 -> V3:
   1. xfs/270: Fix regular expresssion used in inline awk script to
      match hex number.

V1 -> V2:
   1. xfs/270: Replace bashisms with inline awk scripts.
   2. Use _scratch_mkfs_xfs_supported helper in _require_xfs_nrext64().
   3. Remove invocation of $XFS_INFO_PROG from _require_xfs_nrext64().
   4. Use xfs_db's 'path' command instead of saving test file's inode
      number in the two new tests introduced by the patchset.

Chandan Babu R (4):
  xfs/270: Fix ro mount failure when nrext64 option is enabled
  common/xfs: Add helper to check if nrext64 option is supported
  xfs: Verify that the correct inode extent counters are updated
    with/without nrext64
  xfs: Verify correctness of upgrading an fs to support large extent
    counters

 common/xfs        |  12 +++++
 tests/xfs/270     |  26 ++++++++++-
 tests/xfs/270.out |   1 -
 tests/xfs/547     |  92 +++++++++++++++++++++++++++++++++++++
 tests/xfs/547.out |  13 ++++++
 tests/xfs/548     | 112 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/548.out |  12 +++++
 7 files changed, 265 insertions(+), 3 deletions(-)
 create mode 100755 tests/xfs/547
 create mode 100644 tests/xfs/547.out
 create mode 100755 tests/xfs/548
 create mode 100644 tests/xfs/548.out

-- 
2.35.1

[PATCH V3 1/4] xfs/270: Fix ro mount failure when nrext64 option is enabled

[Chandan Babu R]

With nrext64 option enabled at run time, the read-only mount performed by the
test fails because,
1. mkfs.xfs would have calculated log size based on reflink being enabled.
2. Clearing the reflink ro compat bit causes log size calculations to yield a
   different value.
3. In the case where nrext64 is enabled, this causes attr reservation to be
   the largest among all the transaction reservations.
4. This ends up causing XFS to require a larger ondisk log size than that
   which is available.

This commit fixes the problem by setting features_ro_compat to the value
obtained by the bitwise-OR of features_ro_compat field with 2^31.

This commit includes changes suggested by Dave Chinner to replace bashisms
with invocations to inline awk scripts.

Signed-off-by: Chandan Babu R <chandan.babu@oracle.com>
---
 tests/xfs/270     | 26 ++++++++++++++++++++++++--
 tests/xfs/270.out |  1 -
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/tests/xfs/270 b/tests/xfs/270
index 0ab0c7d8..b740c379 100755
--- a/tests/xfs/270
+++ b/tests/xfs/270
@@ -27,8 +27,30 @@ _scratch_mkfs_xfs >>$seqres.full 2>&1
 # set the highest bit of features_ro_compat, use it as an unknown
 # feature bit. If one day this bit become known feature, please
 # change this case.
-_scratch_xfs_set_metadata_field "features_ro_compat" "$((2**31))" "sb 0" | \
-	grep 'features_ro_compat'
+
+ro_compat=$(_scratch_xfs_get_metadata_field "features_ro_compat" "sb 0")
+echo $ro_compat | grep -q -E '^0x[[:xdigit:]]$'
+if [[ $? != 0  ]]; then
+	echo "features_ro_compat has an invalid value."
+fi
+
+ro_compat=$(echo $ro_compat | \
+		    awk '/^0x[[:xdigit:]]+/ {
+				printf("0x%x\n", or(strtonum($1), 0x80000000))
+			}')
+
+# write the new ro compat field to the superblock
+_scratch_xfs_set_metadata_field "features_ro_compat" "$ro_compat" "sb 0" \
+				> $seqres.full 2>&1
+
+# read the newly set ro compat filed for verification
+new_ro_compat=$(_scratch_xfs_get_metadata_field "features_ro_compat" "sb 0" \
+						2>/dev/null)
+
+# verify the new ro_compat field is correct.
+if [ $new_ro_compat != $ro_compat ]; then
+	echo "Unable to set new features_ro_compat. Wanted $ro_compat, got $new_ro_compat"
+fi
 
 # rw mount with unknown ro-compat feature should fail
 echo "rw mount test"
diff --git a/tests/xfs/270.out b/tests/xfs/270.out
index 0a8b3851..edf4c254 100644
--- a/tests/xfs/270.out
+++ b/tests/xfs/270.out
@@ -1,5 +1,4 @@
 QA output created by 270
-features_ro_compat = 0x80000000
 rw mount test
 ro mount test
 rw remount test
-- 
2.35.1

Re: [PATCH V3 1/4] xfs/270: Fix ro mount failure when nrext64 option is enabled

[Darrick J. Wong]

On Sat, Jun 11, 2022 at 04:40:34PM +0530, Chandan Babu R wrote:
> With nrext64 option enabled at run time, the read-only mount performed by the
> test fails because,
> 1. mkfs.xfs would have calculated log size based on reflink being enabled.
> 2. Clearing the reflink ro compat bit causes log size calculations to yield a
>    different value.
> 3. In the case where nrext64 is enabled, this causes attr reservation to be
>    the largest among all the transaction reservations.
> 4. This ends up causing XFS to require a larger ondisk log size than that
>    which is available.
> 
> This commit fixes the problem by setting features_ro_compat to the value
> obtained by the bitwise-OR of features_ro_compat field with 2^31.
> 
> This commit includes changes suggested by Dave Chinner to replace bashisms
> with invocations to inline awk scripts.
> 
> Signed-off-by: Chandan Babu R <chandan.babu@oracle.com>
> ---
>  tests/xfs/270     | 26 ++++++++++++++++++++++++--
>  tests/xfs/270.out |  1 -
>  2 files changed, 24 insertions(+), 3 deletions(-)
> 
> diff --git a/tests/xfs/270 b/tests/xfs/270
> index 0ab0c7d8..b740c379 100755
> --- a/tests/xfs/270
> +++ b/tests/xfs/270
> @@ -27,8 +27,30 @@ _scratch_mkfs_xfs >>$seqres.full 2>&1
>  # set the highest bit of features_ro_compat, use it as an unknown
>  # feature bit. If one day this bit become known feature, please
>  # change this case.
> -_scratch_xfs_set_metadata_field "features_ro_compat" "$((2**31))" "sb 0" | \
> -	grep 'features_ro_compat'
> +
> +ro_compat=$(_scratch_xfs_get_metadata_field "features_ro_compat" "sb 0")
> +echo $ro_compat | grep -q -E '^0x[[:xdigit:]]$'
> +if [[ $? != 0  ]]; then
> +	echo "features_ro_compat has an invalid value."
> +fi
> +
> +ro_compat=$(echo $ro_compat | \
> +		    awk '/^0x[[:xdigit:]]+/ {

I think the double-braces around :xdigit: were the only changes between
v2 and v3, right?

If so,
Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D


> +				printf("0x%x\n", or(strtonum($1), 0x80000000))
> +			}')
> +
> +# write the new ro compat field to the superblock
> +_scratch_xfs_set_metadata_field "features_ro_compat" "$ro_compat" "sb 0" \
> +				> $seqres.full 2>&1
> +
> +# read the newly set ro compat filed for verification
> +new_ro_compat=$(_scratch_xfs_get_metadata_field "features_ro_compat" "sb 0" \
> +						2>/dev/null)
> +
> +# verify the new ro_compat field is correct.
> +if [ $new_ro_compat != $ro_compat ]; then
> +	echo "Unable to set new features_ro_compat. Wanted $ro_compat, got $new_ro_compat"
> +fi
>  
>  # rw mount with unknown ro-compat feature should fail
>  echo "rw mount test"
> diff --git a/tests/xfs/270.out b/tests/xfs/270.out
> index 0a8b3851..edf4c254 100644
> --- a/tests/xfs/270.out
> +++ b/tests/xfs/270.out
> @@ -1,5 +1,4 @@
>  QA output created by 270
> -features_ro_compat = 0x80000000
>  rw mount test
>  ro mount test
>  rw remount test
> -- 
> 2.35.1
> 

[PATCH V3 2/4] common/xfs: Add helper to check if nrext64 option is supported

[Chandan Babu R]

This commit adds a new helper to allow tests to check if xfsprogs and xfs
kernel module support nrext64 option.

Signed-off-by: Chandan Babu R <chandan.babu@oracle.com>
---
 common/xfs | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/common/xfs b/common/xfs
index 2123a4ab..9f84dffb 100644
--- a/common/xfs
+++ b/common/xfs
@@ -444,6 +444,18 @@ _require_xfs_sparse_inodes()
 	_scratch_unmount
 }
 
+# this test requires the xfs large extent counter feature
+#
+_require_xfs_nrext64()
+{
+	_scratch_mkfs_xfs_supported -m crc=1 -i nrext64 > /dev/null 2>&1 \
+		|| _notrun "mkfs.xfs does not support nrext64"
+	_scratch_mkfs_xfs -m crc=1 -i nrext64 > /dev/null 2>&1
+	_try_scratch_mount >/dev/null 2>&1 \
+		|| _notrun "kernel does not support nrext64"
+	_scratch_unmount
+}
+
 # check that xfs_db supports a specific command
 _require_xfs_db_command()
 {
-- 
2.35.1

[PATCH V3 3/4] xfs: Verify that the correct inode extent counters are updated with/without nrext64

[Chandan Babu R]

This commit adds a new test to verify if the correct inode extent counter
fields are updated with/without nrext64 mkfs option.

Signed-off-by: Chandan Babu R <chandan.babu@oracle.com>
---
 tests/xfs/547     | 92 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/547.out | 13 +++++++
 2 files changed, 105 insertions(+)
 create mode 100755 tests/xfs/547
 create mode 100644 tests/xfs/547.out

diff --git a/tests/xfs/547 b/tests/xfs/547
new file mode 100755
index 00000000..9d4216ca
--- /dev/null
+++ b/tests/xfs/547
@@ -0,0 +1,92 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022 Oracle.  All Rights Reserved.
+#
+# FS QA Test 547
+#
+# Verify that correct inode extent count fields are populated with and without
+# nrext64 feature.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata
+
+# Import common functions.
+. ./common/filter
+. ./common/attr
+. ./common/inject
+. ./common/populate
+
+# real QA test starts here
+_supported_fs xfs
+_require_scratch
+_require_xfs_nrext64
+_require_attrs
+_require_xfs_debug
+_require_xfs_db_command path
+_require_test_program "punch-alternating"
+_require_xfs_io_error_injection "bmap_alloc_minlen_extent"
+
+for nrext64 in 0 1; do
+	echo "* Verify extent counter fields with nrext64=${nrext64} option"
+
+	_scratch_mkfs -i nrext64=${nrext64} -d size=$((512 * 1024 * 1024)) \
+		      >> $seqres.full
+	_scratch_mount >> $seqres.full
+
+	bsize=$(_get_file_block_size $SCRATCH_MNT)
+
+	testfile=$SCRATCH_MNT/testfile
+
+	nr_blks=20
+
+	echo "Add blocks to test file's data fork"
+	$XFS_IO_PROG -f -c "pwrite 0 $((nr_blks * bsize))" $testfile \
+		     >> $seqres.full
+	$here/src/punch-alternating $testfile
+
+	echo "Consume free space"
+	fillerdir=$SCRATCH_MNT/fillerdir
+	nr_free_blks=$(stat -f -c '%f' $SCRATCH_MNT)
+	nr_free_blks=$((nr_free_blks * 90 / 100))
+
+	_fill_fs $((bsize * nr_free_blks)) $fillerdir $bsize 0 \
+		 >> $seqres.full 2>&1
+
+	echo "Create fragmented filesystem"
+	for dentry in $(ls -1 $fillerdir/); do
+		$here/src/punch-alternating $fillerdir/$dentry >> $seqres.full
+	done
+
+	echo "Inject bmap_alloc_minlen_extent error tag"
+	_scratch_inject_error bmap_alloc_minlen_extent 1
+
+	echo "Add blocks to test file's attr fork"
+	attr_len=255
+	nr_attrs=$((nr_blks * bsize / attr_len))
+	for i in $(seq 1 $nr_attrs); do
+		attr="$(printf "trusted.%0247d" $i)"
+		$SETFATTR_PROG -n "$attr" $testfile >> $seqres.full 2>&1
+		[[ $? != 0 ]] && break
+	done
+
+	_scratch_unmount >> $seqres.full
+
+	dcnt=$(_scratch_xfs_get_metadata_field core.nextents \
+					       "path /$(basename $testfile)")
+	acnt=$(_scratch_xfs_get_metadata_field core.naextents \
+					       "path /$(basename $testfile)")
+
+	if (( $dcnt != 10 )); then
+		echo "Invalid data fork extent count: $dextcnt"
+		exit 1
+	fi
+
+	if (( $acnt < 10 )); then
+		echo "Invalid attr fork extent count: $aextcnt"
+		exit 1
+	fi
+done
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/547.out b/tests/xfs/547.out
new file mode 100644
index 00000000..49fcc3c2
--- /dev/null
+++ b/tests/xfs/547.out
@@ -0,0 +1,13 @@
+QA output created by 547
+* Verify extent counter fields with nrext64=0 option
+Add blocks to test file's data fork
+Consume free space
+Create fragmented filesystem
+Inject bmap_alloc_minlen_extent error tag
+Add blocks to test file's attr fork
+* Verify extent counter fields with nrext64=1 option
+Add blocks to test file's data fork
+Consume free space
+Create fragmented filesystem
+Inject bmap_alloc_minlen_extent error tag
+Add blocks to test file's attr fork
-- 
2.35.1

[PATCH V3 4/4] xfs: Verify correctness of upgrading an fs to support large extent counters

[Chandan Babu R]

This commit adds a test to verify upgrade of an existing V5 filesystem to
support large extent counters.

Signed-off-by: Chandan Babu R <chandan.babu@oracle.com>
---
 tests/xfs/548     | 112 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/548.out |  12 +++++
 2 files changed, 124 insertions(+)
 create mode 100755 tests/xfs/548
 create mode 100644 tests/xfs/548.out

diff --git a/tests/xfs/548 b/tests/xfs/548
new file mode 100755
index 00000000..560c90fd
--- /dev/null
+++ b/tests/xfs/548
@@ -0,0 +1,112 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022 Oracle.  All Rights Reserved.
+#
+# FS QA Test 548
+#
+# Test to verify upgrade of an existing V5 filesystem to support large extent
+# counters.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata
+
+# Import common functions.
+. ./common/filter
+. ./common/attr
+. ./common/inject
+. ./common/populate
+
+# real QA test starts here
+_supported_fs xfs
+_require_scratch
+_require_xfs_nrext64
+_require_attrs
+_require_xfs_debug
+_require_xfs_db_command path
+_require_test_program "punch-alternating"
+_require_xfs_io_error_injection "bmap_alloc_minlen_extent"
+
+_scratch_mkfs -d size=$((512 * 1024 * 1024)) >> $seqres.full
+_scratch_mount >> $seqres.full
+
+bsize=$(_get_file_block_size $SCRATCH_MNT)
+
+testfile=$SCRATCH_MNT/testfile
+
+nr_blks=20
+
+echo "Add blocks to file's data fork"
+$XFS_IO_PROG -f -c "pwrite 0 $((nr_blks * bsize))" $testfile \
+	     >> $seqres.full
+$here/src/punch-alternating $testfile
+
+echo "Consume free space"
+fillerdir=$SCRATCH_MNT/fillerdir
+nr_free_blks=$(stat -f -c '%f' $SCRATCH_MNT)
+nr_free_blks=$((nr_free_blks * 90 / 100))
+
+_fill_fs $((bsize * nr_free_blks)) $fillerdir $bsize 0 \
+	 >> $seqres.full 2>&1
+
+echo "Create fragmented filesystem"
+for dentry in $(ls -1 $fillerdir/); do
+	$here/src/punch-alternating $fillerdir/$dentry >> $seqres.full
+done
+
+echo "Inject bmap_alloc_minlen_extent error tag"
+_scratch_inject_error bmap_alloc_minlen_extent 1
+
+echo "Add blocks to file's attr fork"
+nr_blks=10
+attr_len=255
+nr_attrs=$((nr_blks * bsize / attr_len))
+for i in $(seq 1 $nr_attrs); do
+	attr="$(printf "trusted.%0247d" $i)"
+	$SETFATTR_PROG -n "$attr" $testfile >> $seqres.full 2>&1
+	[[ $? != 0 ]] && break
+done
+
+echo "Unmount filesystem"
+_scratch_unmount >> $seqres.full
+
+orig_dcnt=$(_scratch_xfs_get_metadata_field core.nextents \
+					    "path /$(basename $testfile)")
+orig_acnt=$(_scratch_xfs_get_metadata_field core.naextents \
+					    "path /$(basename $testfile)")
+
+echo "Upgrade filesystem to support large extent counters"
+_scratch_xfs_admin -O nrext64=1 >> $seqres.full 2>&1
+if [[ $? != 0 ]]; then
+	_notrun "Filesystem geometry is not suitable for upgrading"
+fi
+
+
+echo "Mount filesystem"
+_scratch_mount >> $seqres.full
+
+echo "Modify inode core"
+touch $testfile
+
+echo "Unmount filesystem"
+_scratch_unmount >> $seqres.full
+
+dcnt=$(_scratch_xfs_get_metadata_field core.nextents \
+				       "path /$(basename $testfile)")
+acnt=$(_scratch_xfs_get_metadata_field core.naextents \
+				       "path /$(basename $testfile)")
+
+echo "Verify inode extent counter values after fs upgrade"
+
+if [[ $orig_dcnt != $dcnt ]]; then
+	echo "Corrupt data extent counter"
+	exit 1
+fi
+
+if [[ $orig_acnt != $acnt ]]; then
+	echo "Corrupt attr extent counter"
+	exit 1
+fi
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/548.out b/tests/xfs/548.out
new file mode 100644
index 00000000..19a7f907
--- /dev/null
+++ b/tests/xfs/548.out
@@ -0,0 +1,12 @@
+QA output created by 548
+Add blocks to file's data fork
+Consume free space
+Create fragmented filesystem
+Inject bmap_alloc_minlen_extent error tag
+Add blocks to file's attr fork
+Unmount filesystem
+Upgrade filesystem to support large extent counters
+Mount filesystem
+Modify inode core
+Unmount filesystem
+Verify inode extent counter values after fs upgrade
-- 
2.35.1

Re: [PATCH V3 0/4] Large extent counters tests

[Darrick J. Wong]

On Sat, Jun 11, 2022 at 04:40:33PM +0530, Chandan Babu R wrote:
> Hi all,
> 
> This patchset adds two tests for verifying the behaviour of Large
> extent counters feature. It also fixes xfs/270 test failure when
> executed on a filesystem with Large extent counters enabled.
> 
> Changelog:
> V2 -> V3:
>    1. xfs/270: Fix regular expresssion used in inline awk script to
>       match hex number.

Why did all the patches lose their RVB tags?

--D

> V1 -> V2:
>    1. xfs/270: Replace bashisms with inline awk scripts.
>    2. Use _scratch_mkfs_xfs_supported helper in _require_xfs_nrext64().
>    3. Remove invocation of $XFS_INFO_PROG from _require_xfs_nrext64().
>    4. Use xfs_db's 'path' command instead of saving test file's inode
>       number in the two new tests introduced by the patchset.
> 
> Chandan Babu R (4):
>   xfs/270: Fix ro mount failure when nrext64 option is enabled
>   common/xfs: Add helper to check if nrext64 option is supported
>   xfs: Verify that the correct inode extent counters are updated
>     with/without nrext64
>   xfs: Verify correctness of upgrading an fs to support large extent
>     counters
> 
>  common/xfs        |  12 +++++
>  tests/xfs/270     |  26 ++++++++++-
>  tests/xfs/270.out |   1 -
>  tests/xfs/547     |  92 +++++++++++++++++++++++++++++++++++++
>  tests/xfs/547.out |  13 ++++++
>  tests/xfs/548     | 112 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/xfs/548.out |  12 +++++
>  7 files changed, 265 insertions(+), 3 deletions(-)
>  create mode 100755 tests/xfs/547
>  create mode 100644 tests/xfs/547.out
>  create mode 100755 tests/xfs/548
>  create mode 100644 tests/xfs/548.out
> 
> -- 
> 2.35.1
>