[PATCH 0/2] xfs/438: test the dquot retry again

[PATCH 0/2] xfs/438: test the dquot retry again

[Daejun Park via B4 Relay]

Because fstests turns off metadata write retries for XFS scratch
mounts, xfs/438 passes on a kernel that has the bug. Its dm-flakey
table also relies on dm dropping a flush error, and the dm patch [1]
fixes that. Patch 1 turns the retries back on. Patch 2 moves the test
to the buf_ioerror error tag. Tested with an internal and an external
log on v7.4-rc2, with and without [1], and on a kernel with the bug
brought back.

[1] https://lore.kernel.org/r/20261112-dm-flush-data-error-v1-1-c83d3f8f3ac4@samsung.com

---
Daejun Park (2):
      xfs/438: retry failed metadata writes again
      xfs/438: fail metadata writes with the buf_ioerror error tag

 tests/xfs/438 | 89 ++++++++++++++---------------------------------------------
 1 file changed, 20 insertions(+), 69 deletions(-)
---
base-commit: 71641219a506ac62bfa231592394265ca812670e
change-id: 20261111-xfs-438-dquot-retry-bdff828ab6fa

Best regards,
-- 
Daejun Park <daejun7.park@samsung.com>

[PATCH 1/2] xfs/438: retry failed metadata writes again

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

xfs/438 checks that a dquot buffer write that keeps failing does not
hang the unmount (kernel commit 373b0589dc8d). Commit eb2ad950ea90
moved the test to _scratch_mount. Since commit 1a49022fab9b,
_scratch_mount sets the metadata EIO retries to 0. XFS then shuts
down on the second failure, before the AIL can retry the write. The
test passes even with the bug.

Turn the retries back on after the mount. Fail the test if XFS has
shut down before the unmount.

Fixes: eb2ad950ea90 ("dmflakey: override SCRATCH_DEV in _init_flakey")
Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
Tested on kernels with and without XFS_DEBUG. Patch 1 gives the same
results on both. With an internal log, v7.4-rc2 passes 3 of 3. A TEST
ONLY kernel that skips the resubmission of failed AIL items hangs at
unmount (1 of 1). Without patch 2, the new check fails in three cases:

- with the dm patch and a write cache (3 of 3)
- with an external log (1 of 1)
- with an rt device that has a write cache (from the code, not run)

In each, XFS shuts down on a log error before the unmount. Patch 2
passes in the first two cases.
---
 tests/xfs/438 | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/tests/xfs/438 b/tests/xfs/438
index 4cd0b26..e4850ff 100755
--- a/tests/xfs/438
+++ b/tests/xfs/438
@@ -95,6 +95,9 @@ _require_flakey_with_error_writes
 _require_user
 _require_xfs_quota
 _require_freeze
+_require_test_sysfs_dev_attr error/fail_at_unmount
+_require_test_sysfs_dev_attr error/metadata/EIO/max_retries
+_require_test_sysfs_dev_attr error/metadata/EIO/retry_timeout_seconds
 
 echo "Silence is golden"
 
@@ -117,6 +120,12 @@ sysctl -w fs.xfs.xfssyncd_centisecs=100 >> $seqres.full 2>&1
 _qmount_option "usrquota"
 _scratch_mount
 
+# _scratch_mount turns off the retries of failed metadata writes.
+# Go back to the kernel defaults, so that the failed dquot buffer
+# write is retried from the AIL until unmount. fail_at_unmount
+# stays on, so the unmount ends the retries on a fixed kernel.
+_reset_xfs_sysfs_error_handling $SCRATCH_DEV >> $seqres.full
+
 # We need to set the quota limitation twice, and inject the write error
 # after the second setting. If we try to inject the write error after
 # the first setting, the initialization of the dquota buffer will get
@@ -145,6 +154,10 @@ _scratch_sync
 # the completion of the retried write of dquota buffer
 sleep 2
 
+# The dquot buffer write must still be failing and retried here. A shut
+# down filesystem means that the test missed the hang it checks.
+stat $SCRATCH_MNT > /dev/null
+
 _scratch_unmount
 
 _cleanup_flakey

-- 
2.43.0

Re: [PATCH 1/2] xfs/438: retry failed metadata writes again

[Christoph Hellwig]

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 1/2] xfs/438: retry failed metadata writes again

[Brian Foster]

On Thu, Nov 12, 2026 at 10:37:17AM +0900, Daejun Park via B4 Relay wrote:
> From: Daejun Park <daejun7.park@samsung.com>
> 
> xfs/438 checks that a dquot buffer write that keeps failing does not
> hang the unmount (kernel commit 373b0589dc8d). Commit eb2ad950ea90
> moved the test to _scratch_mount. Since commit 1a49022fab9b,
> _scratch_mount sets the metadata EIO retries to 0. XFS then shuts
> down on the second failure, before the AIL can retry the write. The
> test passes even with the bug.
> 
> Turn the retries back on after the mount. Fail the test if XFS has
> shut down before the unmount.
> 
> Fixes: eb2ad950ea90 ("dmflakey: override SCRATCH_DEV in _init_flakey")
> Signed-off-by: Daejun Park <daejun7.park@samsung.com>
> ---
> Tested on kernels with and without XFS_DEBUG. Patch 1 gives the same
> results on both. With an internal log, v7.4-rc2 passes 3 of 3. A TEST
> ONLY kernel that skips the resubmission of failed AIL items hangs at
> unmount (1 of 1). Without patch 2, the new check fails in three cases:
> 
> - with the dm patch and a write cache (3 of 3)
> - with an external log (1 of 1)
> - with an rt device that has a write cache (from the code, not run)
> 
> In each, XFS shuts down on a log error before the unmount. Patch 2
> passes in the first two cases.
> ---
>  tests/xfs/438 | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/tests/xfs/438 b/tests/xfs/438
> index 4cd0b26..e4850ff 100755
> --- a/tests/xfs/438
> +++ b/tests/xfs/438
...
> @@ -117,6 +120,12 @@ sysctl -w fs.xfs.xfssyncd_centisecs=100 >> $seqres.full 2>&1
>  _qmount_option "usrquota"
>  _scratch_mount
>  
> +# _scratch_mount turns off the retries of failed metadata writes.
> +# Go back to the kernel defaults, so that the failed dquot buffer
> +# write is retried from the AIL until unmount. fail_at_unmount
> +# stays on, so the unmount ends the retries on a fixed kernel.
> +_reset_xfs_sysfs_error_handling $SCRATCH_DEV >> $seqres.full
> +
>  # We need to set the quota limitation twice, and inject the write error
>  # after the second setting. If we try to inject the write error after
>  # the first setting, the initialization of the dquota buffer will get
> @@ -145,6 +154,10 @@ _scratch_sync
>  # the completion of the retried write of dquota buffer
>  sleep 2
>  
> +# The dquot buffer write must still be failing and retried here. A shut
> +# down filesystem means that the test missed the hang it checks.

Nit on the comment.. something like this is a little more clear to me:

# Run a stat to confirm we haven't shutdown prematurely. We expect the
# fs to be running metadata write retries. A shutdown fs at this point
# means that we haven't properly tested for the hang.

But this LGTM regardless. Maybe Zorro can fix that up if you agree:

Reviewed-by: Brian Foster <bfoster@redhat.com>

> +stat $SCRATCH_MNT > /dev/null
> +
>  _scratch_unmount
>  
>  _cleanup_flakey
> 
> -- 
> 2.43.0
> 
> 

[PATCH 2/2] xfs/438: fail metadata writes with the buf_ioerror error tag

[Daejun Park via B4 Relay]

From: Daejun Park <daejun7.park@samsung.com>

xfs/438 needs metadata writes to fail while the log keeps working.
With a write cache, dm-flakey also fails the flushes that dm sends it
with REQ_PREFLUSH log writes. The internal log survives only because
dm drops that error. The dm patch "dm: fail a REQ_PREFLUSH bio with
data when its preflush fails" fixes that. With an external log, the
flakey table fails the log device too.

Use the buf_ioerror error tag. It fails async metadata buffer writes
and leaves log writes alone. The test now needs XFS_DEBUG.

Signed-off-by: Daejun Park <daejun7.park@samsung.com>
---
On v7.4-rc2 and on v7.4-rc2 with the dm patch, the test passes with
an internal log (3 of 3) and an external log (1 of 1). The failed
dquot write is retried until the unmount. On the TEST ONLY kernel of
patch 1 the unmount hangs with both logs (1 of 1 each).
---
 tests/xfs/438 | 76 ++++++-----------------------------------------------------
 1 file changed, 7 insertions(+), 69 deletions(-)

diff --git a/tests/xfs/438 b/tests/xfs/438
index e4850ff..5c228e4 100755
--- a/tests/xfs/438
+++ b/tests/xfs/438
@@ -14,9 +14,9 @@
 # get the same IO error, XFS will let xfsaild to restart the write and
 # xfs_buf_do_callbacks() will not be inovked.
 #
-# This test emulates the write error by using dm-flakey. The log
-# area of the XFS filesystem is excluded from the range covered by
-# dm-flakey, so the XFS will not be shutdown prematurely.
+# This test emulates the write error with the buf_ioerror error tag,
+# which fails async metadata buffer writes. Log writes are not buffer
+# writes, so the XFS will not be shutdown prematurely.
 #
 # Fixed by upstream commit 373b058 ("xfs: Properly retry failed dquot
 # items in case of error during buffer writeback")
@@ -33,85 +33,27 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_scratch_unmount >/dev/null 2>&1
-	_cleanup_flakey > /dev/null 2>&1
-}
-
-# inject IO write error for the XFS filesystem except its log section
-make_xfs_scratch_flakey_table()
-{
-	local tgt=flakey
-	local opt="0 1 1 error_writes"
-	local dev=${SCRATCH_DEV}
-	local dev_sz=$(blockdev --getsz $dev)
-
-	# If using an external log device, just making the writing of
-	# entire data/metadata area fail forever.
-	if [ "${USE_EXTERNAL}" = "yes" -a ! -z "$SCRATCH_LOGDEV" ]; then
-		echo "0 ${dev_sz} $tgt $dev 0 $opt"
-		return
-	fi
-
-	local blk_sz=$(_scratch_xfs_get_sb_field blocksize)
-	local log_ofs=$(_scratch_xfs_get_sb_field logstart)
-	local log_sz=$(_scratch_xfs_get_sb_field logblocks)
-	local table=""
-	local ofs=0
-	local sz
-
-	log_ofs=$(_scratch_xfs_db -r -c "convert fsb ${log_ofs} bb" | \
-			  $AWK_PROG '{gsub("[()]", "", $2); print $2}')
-	let "log_sz *= blk_sz / 512"
-
-	# Add a flakey target for the area before the log section
-	# to make the data/metadata write fail forever
-	if [ "$ofs" -lt "${log_ofs}" ]; then
-		let "sz = log_ofs - ofs"
-		table="$ofs $sz $tgt $dev $ofs $opt"
-	fi
-
-	# Add a linear target for the log section, so the log write
-	# will work normally
-	table="$table\n${log_ofs} ${log_sz} linear $dev ${log_ofs}"
-
-	# Add a flakey target for the area after the log section
-	# to make the data/metadata write fail forever
-	let "ofs = log_ofs + log_sz"
-	if [ "$ofs" -lt "${dev_sz}" ]; then
-		let "sz = dev_sz - ofs"
-		table="$table\n$ofs $sz $tgt $dev $ofs $opt"
-	fi
-
-	echo -e $table
 }
 
 # Import common functions.
-. ./common/dmflakey
 . ./common/quota
+. ./common/inject
 
 
 # due to the injection of write IO error, the fs will be inconsistent
 _require_scratch_nocheck
-_require_flakey_with_error_writes
 _require_user
 _require_xfs_quota
 _require_freeze
 _require_test_sysfs_dev_attr error/fail_at_unmount
 _require_test_sysfs_dev_attr error/metadata/EIO/max_retries
 _require_test_sysfs_dev_attr error/metadata/EIO/retry_timeout_seconds
+_require_xfs_io_error_injection buf_ioerror
 
 echo "Silence is golden"
 
 _scratch_mkfs > $seqres.full 2>&1
 
-# this needs to happen after mkfs, but before _init_flakey overrides SCRATCH_DEV
-FLAKEY_TABLE_NON_LOG_ERROR=$(make_xfs_scratch_flakey_table)
-
-# no error will be injected
-_init_flakey
-FLAKEY_TABLE_ERROR=${FLAKEY_TABLE_NON_LOG_ERROR}
-$DMSETUP_PROG info >> $seqres.full
-$DMSETUP_PROG table >> $seqres.full
-
 # save the old value for _cleanup()
 interval=$(sysctl -n fs.xfs.xfssyncd_centisecs 2>/dev/null)
 # shorten the time waiting for the push of ail items
@@ -139,10 +81,8 @@ $XFS_QUOTA_PROG -x -c "report -ih" $SCRATCH_MNT >> $seqres.full
 xfs_freeze -f $SCRATCH_MNT
 xfs_freeze -u $SCRATCH_MNT
 
-# inject write IO error
-_load_flakey_table ${FLAKEY_ERROR_WRITES}
-$DMSETUP_PROG info >> $seqres.full
-$DMSETUP_PROG table >> $seqres.full
+# fail every async metadata buffer write from now on
+_scratch_inject_error buf_ioerror 1
 
 # update the dquota buffer
 $XFS_QUOTA_PROG -x -c "limit -u isoft=400 $qa_user" $SCRATCH_MNT
@@ -160,8 +100,6 @@ stat $SCRATCH_MNT > /dev/null
 
 _scratch_unmount
 
-_cleanup_flakey
-
 # success, all done
 status=0
 exit

-- 
2.43.0

Re: [PATCH 2/2] xfs/438: fail metadata writes with the buf_ioerror error tag

[Christoph Hellwig]

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH 2/2] xfs/438: fail metadata writes with the buf_ioerror error tag

[Brian Foster]

On Thu, Nov 12, 2026 at 10:37:18AM +0900, Daejun Park via B4 Relay wrote:
> From: Daejun Park <daejun7.park@samsung.com>
> 
> xfs/438 needs metadata writes to fail while the log keeps working.
> With a write cache, dm-flakey also fails the flushes that dm sends it
> with REQ_PREFLUSH log writes. The internal log survives only because
> dm drops that error. The dm patch "dm: fail a REQ_PREFLUSH bio with
> data when its preflush fails" fixes that. With an external log, the
> flakey table fails the log device too.
> 
> Use the buf_ioerror error tag. It fails async metadata buffer writes
> and leaves log writes alone. The test now needs XFS_DEBUG.
> 
> Signed-off-by: Daejun Park <daejun7.park@samsung.com>
> ---
> On v7.4-rc2 and on v7.4-rc2 with the dm patch, the test passes with
> an internal log (3 of 3) and an external log (1 of 1). The failed
> dquot write is retried until the unmount. On the TEST ONLY kernel of
> patch 1 the unmount hangs with both logs (1 of 1 each).
> ---

Nice simplification.

Reviewed-by: Brian Foster <bfoster@redhat.com>

>  tests/xfs/438 | 76 ++++++-----------------------------------------------------
>  1 file changed, 7 insertions(+), 69 deletions(-)
> 
> diff --git a/tests/xfs/438 b/tests/xfs/438
> index e4850ff..5c228e4 100755
> --- a/tests/xfs/438
> +++ b/tests/xfs/438
> @@ -14,9 +14,9 @@
>  # get the same IO error, XFS will let xfsaild to restart the write and
>  # xfs_buf_do_callbacks() will not be inovked.
>  #
> -# This test emulates the write error by using dm-flakey. The log
> -# area of the XFS filesystem is excluded from the range covered by
> -# dm-flakey, so the XFS will not be shutdown prematurely.
> +# This test emulates the write error with the buf_ioerror error tag,
> +# which fails async metadata buffer writes. Log writes are not buffer
> +# writes, so the XFS will not be shutdown prematurely.
>  #
>  # Fixed by upstream commit 373b058 ("xfs: Properly retry failed dquot
>  # items in case of error during buffer writeback")
> @@ -33,85 +33,27 @@ _cleanup()
>  	cd /
>  	rm -f $tmp.*
>  	_scratch_unmount >/dev/null 2>&1
> -	_cleanup_flakey > /dev/null 2>&1
> -}
> -
> -# inject IO write error for the XFS filesystem except its log section
> -make_xfs_scratch_flakey_table()
> -{
> -	local tgt=flakey
> -	local opt="0 1 1 error_writes"
> -	local dev=${SCRATCH_DEV}
> -	local dev_sz=$(blockdev --getsz $dev)
> -
> -	# If using an external log device, just making the writing of
> -	# entire data/metadata area fail forever.
> -	if [ "${USE_EXTERNAL}" = "yes" -a ! -z "$SCRATCH_LOGDEV" ]; then
> -		echo "0 ${dev_sz} $tgt $dev 0 $opt"
> -		return
> -	fi
> -
> -	local blk_sz=$(_scratch_xfs_get_sb_field blocksize)
> -	local log_ofs=$(_scratch_xfs_get_sb_field logstart)
> -	local log_sz=$(_scratch_xfs_get_sb_field logblocks)
> -	local table=""
> -	local ofs=0
> -	local sz
> -
> -	log_ofs=$(_scratch_xfs_db -r -c "convert fsb ${log_ofs} bb" | \
> -			  $AWK_PROG '{gsub("[()]", "", $2); print $2}')
> -	let "log_sz *= blk_sz / 512"
> -
> -	# Add a flakey target for the area before the log section
> -	# to make the data/metadata write fail forever
> -	if [ "$ofs" -lt "${log_ofs}" ]; then
> -		let "sz = log_ofs - ofs"
> -		table="$ofs $sz $tgt $dev $ofs $opt"
> -	fi
> -
> -	# Add a linear target for the log section, so the log write
> -	# will work normally
> -	table="$table\n${log_ofs} ${log_sz} linear $dev ${log_ofs}"
> -
> -	# Add a flakey target for the area after the log section
> -	# to make the data/metadata write fail forever
> -	let "ofs = log_ofs + log_sz"
> -	if [ "$ofs" -lt "${dev_sz}" ]; then
> -		let "sz = dev_sz - ofs"
> -		table="$table\n$ofs $sz $tgt $dev $ofs $opt"
> -	fi
> -
> -	echo -e $table
>  }
>  
>  # Import common functions.
> -. ./common/dmflakey
>  . ./common/quota
> +. ./common/inject
>  
>  
>  # due to the injection of write IO error, the fs will be inconsistent
>  _require_scratch_nocheck
> -_require_flakey_with_error_writes
>  _require_user
>  _require_xfs_quota
>  _require_freeze
>  _require_test_sysfs_dev_attr error/fail_at_unmount
>  _require_test_sysfs_dev_attr error/metadata/EIO/max_retries
>  _require_test_sysfs_dev_attr error/metadata/EIO/retry_timeout_seconds
> +_require_xfs_io_error_injection buf_ioerror
>  
>  echo "Silence is golden"
>  
>  _scratch_mkfs > $seqres.full 2>&1
>  
> -# this needs to happen after mkfs, but before _init_flakey overrides SCRATCH_DEV
> -FLAKEY_TABLE_NON_LOG_ERROR=$(make_xfs_scratch_flakey_table)
> -
> -# no error will be injected
> -_init_flakey
> -FLAKEY_TABLE_ERROR=${FLAKEY_TABLE_NON_LOG_ERROR}
> -$DMSETUP_PROG info >> $seqres.full
> -$DMSETUP_PROG table >> $seqres.full
> -
>  # save the old value for _cleanup()
>  interval=$(sysctl -n fs.xfs.xfssyncd_centisecs 2>/dev/null)
>  # shorten the time waiting for the push of ail items
> @@ -139,10 +81,8 @@ $XFS_QUOTA_PROG -x -c "report -ih" $SCRATCH_MNT >> $seqres.full
>  xfs_freeze -f $SCRATCH_MNT
>  xfs_freeze -u $SCRATCH_MNT
>  
> -# inject write IO error
> -_load_flakey_table ${FLAKEY_ERROR_WRITES}
> -$DMSETUP_PROG info >> $seqres.full
> -$DMSETUP_PROG table >> $seqres.full
> +# fail every async metadata buffer write from now on
> +_scratch_inject_error buf_ioerror 1
>  
>  # update the dquota buffer
>  $XFS_QUOTA_PROG -x -c "limit -u isoft=400 $qa_user" $SCRATCH_MNT
> @@ -160,8 +100,6 @@ stat $SCRATCH_MNT > /dev/null
>  
>  _scratch_unmount
>  
> -_cleanup_flakey
> -
>  # success, all done
>  status=0
>  exit
> 
> -- 
> 2.43.0
> 
>