public inbox for fstests@vger.kernel.org
 help / color / mirror / Atom feed
From: David Disseldorp <ddiss@suse.de>
To: fstests@vger.kernel.org
Cc: Qu Wenruo <wqu@suse.com>, David Disseldorp <ddiss@suse.de>
Subject: [PATCH 3/5] btrfs/282: dmdelay scrub I/O to fix intermittent failures
Date: Thu,  7 Dec 2023 18:20:54 +1100	[thread overview]
Message-ID: <20231207072056.14588-4-ddiss@suse.de> (raw)
In-Reply-To: <20231207072056.14588-1-ddiss@suse.de>

btrfs/282 can fail intermittently if scrub completes quickly (e.g. atop
  a RAM backed scratch device):
- if under one second, the reported scrub duration metric may be rounded
  down to zero, which results in a "Rate: 0/s" metric
  + I have a btrfs-progs patch to address this confusing Rate reporting
- I've also observed one second scrub durations fail intermittently,
  presumably because scheduling jitter doesn't allow throttling to take
  effect

Avoid these intermittent failures by injecting a 100ms delay to device
read I/Os using dm-delay.

Signed-off-by: David Disseldorp <ddiss@suse.de>
---
 tests/btrfs/282 | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/tests/btrfs/282 b/tests/btrfs/282
index 66d14f82..f9e22e12 100755
--- a/tests/btrfs/282
+++ b/tests/btrfs/282
@@ -10,6 +10,15 @@
 _begin_fstest auto scrub
 
 . ./common/filter
+. ./common/dmdelay
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	_scratch_unmount > /dev/null 2>&1
+	_cleanup_delay > /dev/null 2>&1
+}
 
 # real QA test starts here
 _supported_fs btrfs
@@ -18,6 +27,7 @@ _wants_kernel_commit eb3b50536642 \
 
 # We want at least 5G for the scratch device.
 _require_scratch_size $(( 5 * 1024 * 1024))
+_require_dm_target delay
 
 # Make sure we can create scrub progress data file
 if [ -e /var/lib/btrfs ]; then
@@ -27,7 +37,8 @@ else
 fi
 
 _scratch_mkfs >> $seqres.full 2>&1
-_scratch_mount
+_init_delay
+_mount_delay
 
 uuid=$(findmnt -n -o UUID $SCRATCH_MNT)
 
@@ -45,6 +56,10 @@ $XFS_IO_PROG -f -c "pwrite -i /dev/urandom 0 2G" $SCRATCH_MNT/file | _filter_xfs
 # Writeback above data, as scrub only verify the committed data.
 sync
 
+# 100ms read I/O delay, so that the scrub doesn't complete too quickly
+bs=$(blockdev --getsz $SCRATCH_DEV)
+_load_delay_table "0 $bs delay $SCRATCH_DEV 0 100 $SCRATCH_DEV 0 0"
+
 # The first scrub, mostly to grab the speed of the scrub.
 $BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >> $seqres.full
 
@@ -70,7 +85,8 @@ if [ -z "$init_speed" ]; then
 fi
 
 # Cycle mount to drop any possible cache.
-_scratch_cycle_mount
+_unmount_delay
+_mount_delay
 
 target_speed=$(( $init_speed / 2 ))
 echo "$target_speed" > "${devinfo_dir}/scrub_speed_max"
-- 
2.35.3


  parent reply	other threads:[~2023-12-07  7:21 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-12-07  7:20 [PATCH 0/5] btrfs/282: resolve intermittent failures David Disseldorp
2023-12-07  7:20 ` [PATCH 1/5] common/dmdelay: remove DELAY_X enums David Disseldorp
2023-12-07  7:20 ` [PATCH 2/5] btrfs/282: append scrub status output to 282.full David Disseldorp
2023-12-07  9:46   ` Qu Wenruo
2023-12-07  7:20 ` David Disseldorp [this message]
2023-12-07  7:20 ` [PATCH 4/5] btrfs/282: reduce scrub dataset size David Disseldorp
2023-12-07  7:20 ` [PATCH 5/5] btrfs/282: increase throttled scrub rate tolerance David Disseldorp
2023-12-07  9:49 ` [PATCH 0/5] btrfs/282: resolve intermittent failures Qu Wenruo
2023-12-07 13:49   ` David Disseldorp
2023-12-07 20:02     ` Qu Wenruo

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20231207072056.14588-4-ddiss@suse.de \
    --to=ddiss@suse.de \
    --cc=fstests@vger.kernel.org \
    --cc=wqu@suse.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox