[PATCHSET 0/3] fstests: random fixes for v2023.05.28

[PATCHSET 0/3] fstests: random fixes for v2023.05.28

[Darrick J. Wong]

Hi all,

Here's the usual odd fixes for fstests.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=random-fixes

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=random-fixes

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes
---
 tests/xfs/108 |   14 +++++++++++---
 tests/xfs/155 |    7 ++++---
 2 files changed, 15 insertions(+), 6 deletions(-)

[PATCH 1/3] xfs/108: allow slightly higher block usage

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

With pmem and fsdax enabled, I occasionally see this test fail on XFS:

   Mode: (0600/-rw-------)         Uid: (1)  Gid: (2)
 Disk quotas for User #1 (1)
 Filesystem Blocks Quota Limit Warn/Time Mounted on
-SCRATCH_DEV 48M 0 0 00 [------] SCRATCH_MNT
+SCRATCH_DEV 48.0M 0 0 00 [------] SCRATCH_MNT
 Disk quotas for User #1 (1)
 Filesystem Files Quota Limit Warn/Time Mounted on
 SCRATCH_DEV 3 0 0 00 [------] SCRATCH_MNT

The cause of this failure is fragmentation in the file mappings that
results in a block mapping structure that no longer fits in the inode.
Hence the block usage is 49160K instead of the 49152K that was written.
Use some fugly sed duct tape to make this test accomodate this
possiblity.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/xfs/108 |   14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)


diff --git a/tests/xfs/108 b/tests/xfs/108
index 4607000544..8593edbdd2 100755
--- a/tests/xfs/108
+++ b/tests/xfs/108
@@ -32,6 +32,14 @@ test_files()
 	done
 }
 
+# Some filesystem configurations fragment the file mapping more than others,
+# which leads to the quota block counts being slightly higher than the 48MB
+# written.
+filter_quota()
+{
+	sed -e 's/48\.[01]M/48M/g' | _filter_quota
+}
+
 test_accounting()
 {
 	echo "### some controlled buffered, direct and mmapd IO (type=$type)"
@@ -49,9 +57,9 @@ test_accounting()
 		$here/src/lstat64 $file | head -3 | _filter_scratch
 	done
 	$XFS_IO_PROG -c syncfs $SCRATCH_MNT
-	$XFS_QUOTA_PROG -c "quota -hnb -$type $id" $QARGS | _filter_quota
-	$XFS_QUOTA_PROG -c "quota -hni -$type $id" $QARGS | _filter_quota
-	$XFS_QUOTA_PROG -c "quota -hnr -$type $id" $QARGS | _filter_quota
+	$XFS_QUOTA_PROG -c "quota -hnb -$type $id" $QARGS | filter_quota
+	$XFS_QUOTA_PROG -c "quota -hni -$type $id" $QARGS | filter_quota
+	$XFS_QUOTA_PROG -c "quota -hnr -$type $id" $QARGS | filter_quota
 }
 
 export MOUNT_OPTIONS="-opquota"

Re: [PATCH 1/3] xfs/108: allow slightly higher block usage

[Bill O'Donnell]

On Tue, Jun 06, 2023 at 03:29:08PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> With pmem and fsdax enabled, I occasionally see this test fail on XFS:
> 
>    Mode: (0600/-rw-------)         Uid: (1)  Gid: (2)
>  Disk quotas for User #1 (1)
>  Filesystem Blocks Quota Limit Warn/Time Mounted on
> -SCRATCH_DEV 48M 0 0 00 [------] SCRATCH_MNT
> +SCRATCH_DEV 48.0M 0 0 00 [------] SCRATCH_MNT
>  Disk quotas for User #1 (1)
>  Filesystem Files Quota Limit Warn/Time Mounted on
>  SCRATCH_DEV 3 0 0 00 [------] SCRATCH_MNT
> 
> The cause of this failure is fragmentation in the file mappings that
> results in a block mapping structure that no longer fits in the inode.
> Hence the block usage is 49160K instead of the 49152K that was written.
> Use some fugly sed duct tape to make this test accomodate this
> possiblity.
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>

LGTM.
Reviewed-by: Bill O'Donnell <bodonnel@redhat.com>

> ---
>  tests/xfs/108 |   14 +++++++++++---
>  1 file changed, 11 insertions(+), 3 deletions(-)
> 
> 
> diff --git a/tests/xfs/108 b/tests/xfs/108
> index 4607000544..8593edbdd2 100755
> --- a/tests/xfs/108
> +++ b/tests/xfs/108
> @@ -32,6 +32,14 @@ test_files()
>  	done
>  }
>  
> +# Some filesystem configurations fragment the file mapping more than others,
> +# which leads to the quota block counts being slightly higher than the 48MB
> +# written.
> +filter_quota()
> +{
> +	sed -e 's/48\.[01]M/48M/g' | _filter_quota
> +}
> +
>  test_accounting()
>  {
>  	echo "### some controlled buffered, direct and mmapd IO (type=$type)"
> @@ -49,9 +57,9 @@ test_accounting()
>  		$here/src/lstat64 $file | head -3 | _filter_scratch
>  	done
>  	$XFS_IO_PROG -c syncfs $SCRATCH_MNT
> -	$XFS_QUOTA_PROG -c "quota -hnb -$type $id" $QARGS | _filter_quota
> -	$XFS_QUOTA_PROG -c "quota -hni -$type $id" $QARGS | _filter_quota
> -	$XFS_QUOTA_PROG -c "quota -hnr -$type $id" $QARGS | _filter_quota
> +	$XFS_QUOTA_PROG -c "quota -hnb -$type $id" $QARGS | filter_quota
> +	$XFS_QUOTA_PROG -c "quota -hni -$type $id" $QARGS | filter_quota
> +	$XFS_QUOTA_PROG -c "quota -hnr -$type $id" $QARGS | filter_quota
>  }
>  
>  export MOUNT_OPTIONS="-opquota"
> 

[PATCH 2/3] xfs/155: discard stderr when checking for NEEDSREPAIR

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

This test deliberate crashes xfs_repair midway through writing metadata
to check that NEEDSREPAIR is always triggered by filesystem writes.
However, the subsequent scan for the NEEDSREPAIR feature bit prints
verifier errors to stderr.

On a filesystem with metadata directories, this leads to the test
failing with this recorded in the golden output:

+Metadata CRC error detected at 0x55c0a2dd0d38, xfs_dir3_block block 0xc0/0x1000
+dir block owner 0x82 doesnt match block 0xbb8cd37e44eb3623

This isn't specific to metadata directories -- any repair crash could
leave a metadata structure in a weird state such that starting xfs_db
will spray verifier errors.  For _check_scratch_xfs_features here, we
don't care if the filesystem is corrupt; we /only/ care that the
superblock feature bit is set.  Route all that noise to devnull.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/xfs/155 |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


diff --git a/tests/xfs/155 b/tests/xfs/155
index c4ee8e20ef..25cc84069c 100755
--- a/tests/xfs/155
+++ b/tests/xfs/155
@@ -55,7 +55,7 @@ for ((nr_writes = 1; nr_writes < max_writes; nr_writes += nr_incr)); do
 	# but repair -n says the fs is clean, then it's possible that the
 	# injected error caused it to abort immediately after the write that
 	# cleared NEEDSREPAIR.
-	if ! _check_scratch_xfs_features NEEDSREPAIR > /dev/null &&
+	if ! _check_scratch_xfs_features NEEDSREPAIR &> /dev/null &&
 	   ! _scratch_xfs_repair -n &>> $seqres.full; then
 		echo "NEEDSREPAIR should be set on corrupt fs"
 	fi
@@ -63,7 +63,7 @@ done
 
 # If NEEDSREPAIR is still set on the filesystem, ensure that a full run
 # cleans everything up.
-if _check_scratch_xfs_features NEEDSREPAIR > /dev/null; then
+if _check_scratch_xfs_features NEEDSREPAIR &> /dev/null; then
 	echo "Clearing NEEDSREPAIR" >> $seqres.full
 	_scratch_xfs_repair 2>> $seqres.full
 	_check_scratch_xfs_features NEEDSREPAIR > /dev/null && \

Re: [PATCH 2/3] xfs/155: discard stderr when checking for NEEDSREPAIR

[Andrey Albershteyn]

On 2023-06-06 15:29:14, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> This test deliberate crashes xfs_repair midway through writing metadata
> to check that NEEDSREPAIR is always triggered by filesystem writes.
> However, the subsequent scan for the NEEDSREPAIR feature bit prints
> verifier errors to stderr.
> 
> On a filesystem with metadata directories, this leads to the test
> failing with this recorded in the golden output:
> 
> +Metadata CRC error detected at 0x55c0a2dd0d38, xfs_dir3_block block 0xc0/0x1000
> +dir block owner 0x82 doesnt match block 0xbb8cd37e44eb3623
> 
> This isn't specific to metadata directories -- any repair crash could
> leave a metadata structure in a weird state such that starting xfs_db
> will spray verifier errors.  For _check_scratch_xfs_features here, we
> don't care if the filesystem is corrupt; we /only/ care that the
> superblock feature bit is set.  Route all that noise to devnull.
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---

Looks good to me.
Reviewed-by: Andrey Albershteyn <aalbersh@redhat.com>

[PATCH 3/3] xfs/155: improve logging in this test

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

If this test fails after a certain number of writes, we should state
the exact number of writes so that we can coordinate with 155.full.
Instead, we state the pre-randomization number, which isn't all that
helpful.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/xfs/155 |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


diff --git a/tests/xfs/155 b/tests/xfs/155
index 25cc84069c..302607b510 100755
--- a/tests/xfs/155
+++ b/tests/xfs/155
@@ -63,11 +63,12 @@ done
 
 # If NEEDSREPAIR is still set on the filesystem, ensure that a full run
 # cleans everything up.
+echo "Checking filesystem one last time after $allowed_writes writes." >> $seqres.full
 if _check_scratch_xfs_features NEEDSREPAIR &> /dev/null; then
 	echo "Clearing NEEDSREPAIR" >> $seqres.full
 	_scratch_xfs_repair 2>> $seqres.full
 	_check_scratch_xfs_features NEEDSREPAIR > /dev/null && \
-		echo "Repair failed to clear NEEDSREPAIR on the $nr_writes writes test"
+		echo "Repair failed to clear NEEDSREPAIR on the $allowed_writes writes test"
 fi
 
 # success, all done

Re: [PATCH 3/3] xfs/155: improve logging in this test

[Bill O'Donnell]

On Tue, Jun 06, 2023 at 03:29:19PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> If this test fails after a certain number of writes, we should state
> the exact number of writes so that we can coordinate with 155.full.
> Instead, we state the pre-randomization number, which isn't all that
> helpful.
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>

Makes sense.
Reviewed-by: Bill O'Donnell <bodonnel@redhat.com>


> ---
>  tests/xfs/155 |    3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> 
> diff --git a/tests/xfs/155 b/tests/xfs/155
> index 25cc84069c..302607b510 100755
> --- a/tests/xfs/155
> +++ b/tests/xfs/155
> @@ -63,11 +63,12 @@ done
>  
>  # If NEEDSREPAIR is still set on the filesystem, ensure that a full run
>  # cleans everything up.
> +echo "Checking filesystem one last time after $allowed_writes writes." >> $seqres.full
>  if _check_scratch_xfs_features NEEDSREPAIR &> /dev/null; then
>  	echo "Clearing NEEDSREPAIR" >> $seqres.full
>  	_scratch_xfs_repair 2>> $seqres.full
>  	_check_scratch_xfs_features NEEDSREPAIR > /dev/null && \
> -		echo "Repair failed to clear NEEDSREPAIR on the $nr_writes writes test"
> +		echo "Repair failed to clear NEEDSREPAIR on the $allowed_writes writes test"
>  fi
>  
>  # success, all done
>