From: Anand Jain <anand.jain@oracle.com>
To: Qu Wenruo <quwenruo.btrfs@gmx.com>,
Christoph Hellwig <hch@lst.de>,
fstests@vger.kernel.org
Cc: linux-btrfs@vger.kernel.org
Subject: Re: [PATCH 2/2] btrfs: test repair with corrupted sectors interleaved over multiple mirrors
Date: Sun, 22 May 2022 18:26:26 +0530 [thread overview]
Message-ID: <1fc36e3f-8fa9-2bfb-ede1-d4f852bcb8cf@oracle.com> (raw)
In-Reply-To: <c28607d6-34c5-5c9e-cb55-2a64273c477f@gmx.com>
On 5/22/22 12:01, Qu Wenruo wrote:
>
>
> On 2022/5/22 09:13, Anand Jain wrote:
>> On 5/20/22 22:17, Christoph Hellwig wrote:
>>> Test that repair handles the case where it needs to read from more than
>>> a single mirror on the raid1c3 profile and needs to take turns over the
>>> mirrors to recover data for the whole read.
>>>
>>> Signed-off-by: Christoph Hellwig <hch@lst.de>
>>> ---
>>> tests/btrfs/266 | 145 ++++++++++++++++++++++++++++++++++++++++++++
>>> tests/btrfs/266.out | 109 +++++++++++++++++++++++++++++++++
>>> 2 files changed, 254 insertions(+)
>>> create mode 100755 tests/btrfs/266
>>> create mode 100644 tests/btrfs/266.out
>>>
>>> diff --git a/tests/btrfs/266 b/tests/btrfs/266
>>> new file mode 100755
>>> index 00000000..24c2b5fd
>>> --- /dev/null
>>> +++ b/tests/btrfs/266
>>> @@ -0,0 +1,145 @@
>>> +#! /bin/bash
>>> +# SPDX-License-Identifier: GPL-2.0
>>> +# Copyright (c) 2017 Liu Bo. All Rights Reserved.
>>> +# Copyright (c) 2022 Christoph Hellwig.
>>> +#
>>> +# FS QA Test 266
>>> +#
>>> +# Test that btrfs raid repair on a raid1c3 profile can repair
>>> interleaving
>>> +# errors on all mirrors.
>>> +#
>>> +
>>> +. ./common/preamble
>>> +_begin_fstest auto quick read_repair
>>> +
>>> +# Import common functions.
>>> +. ./common/filter
>>> +
>>> +# real QA test starts here
>>> +
>>> +_supported_fs btrfs
>>> +_require_scratch_dev_pool 3
>>> +
>>> +BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical)
>>> +
>>> +_require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical
>>> +_require_command "$FILEFRAG_PROG" filefrag
>>> +_require_odirect
>>> +# Overwriting data is forbidden on a zoned block device
>>> +_require_non_zoned_device "${SCRATCH_DEV}"
>>> +
>>> +get_physical()
>>> +{
>>> + local logical=$1
>>> + local stripe=$2
>>> +
>>> + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV >> $seqres.full
>>> 2>&1
>>> + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
>>> + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }"
>>> +}
>>> +
>>> +get_device_path()
>>> +{
>>> + local logical=$1
>>> + local stripe=$2
>>> +
>>> + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
>>> + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }"
>>> +}
>>> +
>>
>> Can wrap into a helper in common/btrfs.
>>
>>> +_scratch_dev_pool_get 3
>>> +# step 1, create a raid1 btrfs which contains one 128k file.
>>> +echo "step 1......mkfs.btrfs"
>>> +
>>> +mkfs_opts="-d raid1c3 -b 1G"
>>> +_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
>>> +
>>> +# make sure data is written to the start position of the data chunk
>>> +_scratch_mount $(_btrfs_no_v1_cache_opt)
>>> +
>>> +$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" \
>>> + "$SCRATCH_MNT/foobar" | \
>>> + _filter_xfs_io_offset
>>> +
>>> +# ensure btrfs-map-logical sees the tree updates
>>> +sync
>>> +
>>> +# step 2, corrupt 4k in each copy
>>> +echo "step 2......corrupt file extent"
>>> +
>>> +${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
>>> +logical=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag |
>>> cut -d '#' -f 1`
>>> +
>>> +physical1=$(get_physical ${logical} 1)
>>> +devpath1=$(get_device_path ${logical} 1)
>>> +
>>> +physical2=$(get_physical ${logical} 2)
>>> +devpath2=$(get_device_path ${logical} 2)
>>> +
>>> +physical3=$(get_physical ${logical} 3)
>>> +devpath3=$(get_device_path ${logical} 3)
>>> +
>>> +_scratch_unmount
>>> +
>>
>>
>>
>>> +$XFS_IO_PROG -d -c "pwrite -S 0xbd -b 4K $physical3 4K" $devpath3 \
>>> + > /dev/null
>>> +
>>> +$XFS_IO_PROG -d -c "pwrite -S 0xba -b 4K $physical1 8K" $devpath1 \
>>> + > /dev/null
>>> +
>>> +$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 4K $((physical2 + 4096)) 8K"
>>> $devpath2 \
>>> + > /dev/null
>>> +
>>> +$XFS_IO_PROG -d -c "pwrite -S 0xbc -b 4K $((physical3 + (2 * 4096)))
>>> 8K" \
>>> + $devpath3 > /dev/null
>>> +
>>
>> Could you please make this test case compatible with the 64K sectorsize
>> configs too?
>>
>>
>>
>>
>>> +_scratch_mount
>>> +
>>> +# step 3, 128k dio read (this read can repair bad copy)
>>> +echo "step 3......repair the bad copy"
>>> +
>>> +# since raid1c3 consists of three copies, and the bad copy was put on
>>> stripe #1
>>> +# while the good copy lies the other stripes, the bad copy only gets
>>> accessed
>>> +# when the reader's pid % 3 is 1
>>> +while true; do
>>> + echo 3 > /proc/sys/vm/drop_caches
>>> + $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" >
>>> /dev/null &
>>> + pid=$!
>>> + wait
>>> + if [ $((pid % 3)) == 1 ]; then
>>> + break
>>> + fi
>>> +done
>>> +while true; do
>>> + echo 3 > /proc/sys/vm/drop_caches
>>> + $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" >
>>> /dev/null &
>>> + pid=$!
>>> + wait
>>> + if [ $((pid % 3)) == 2 ]; then
>>> + break
>>> + fi
>>> +done
>>> +while true; do
>>> + echo 3 > /proc/sys/vm/drop_caches
>>> + $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" >
>>> /dev/null &
>>> + pid=$!
>>> + wait
>>> + if [ $((pid % 3)) == 0 ]; then
>>> + break
>>> + fi
>>> +done
>>> +
>>
>> Same here. They can wrap into a helper.
>
> In fact, we should recheck the way we handle the read on specific mirror
> behavior.
>
> For a lot of runs, btrfs/14[01] can take super long time (over 100s) to
> really trigger the data read on expected mirror.
>
> For a lot of runs, the read just got the same pid again and again.
>
> Can't we at least add some noise at background to make the pid more random?
>
Agreed this method is unreliable. But there is no other choice.
Unless we integrated [1] patch in the ML.
[1] btrfs: introduce new read_policy device
[1] is more reliable. You can set which mirrored device to read.
Thanks, Anand
> Thanks,
> Qu
>>
>> Thanks, Anand
>>
>>> +_scratch_unmount
>>> +
>>> +echo "step 4......check if the repair works"
>>> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical1 512" $devpath1 |\
>>> + _filter_xfs_io_offset
>>> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical2 512" $devpath2 |\
>>> + _filter_xfs_io_offset
>>> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical3 512" $devpath3 |\
>>> + _filter_xfs_io_offset
>>> +
>>> +_scratch_dev_pool_put
>>> +# success, all done
>>> +status=0
>>> +exit
>>> diff --git a/tests/btrfs/266.out b/tests/btrfs/266.out
>>> new file mode 100644
>>> index 00000000..243d1e1d
>>> --- /dev/null
>>> +++ b/tests/btrfs/266.out
>>> @@ -0,0 +1,109 @@
>>> +QA output created by 266
>>> +step 1......mkfs.btrfs
>>> +wrote 131072/131072 bytes
>>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>> +step 2......corrupt file extent
>>> +step 3......repair the bad copy
>>> +step 4......check if the repair works
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +read 512/512 bytes
>>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +read 512/512 bytes
>>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +XXXXXXXX: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa
>>> ................
>>> +read 512/512 bytes
>>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>
next prev parent reply other threads:[~2022-05-22 12:56 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-20 16:47 two more btrfs read repair tests Christoph Hellwig
2022-05-20 16:47 ` [PATCH 1/2] btrfs: test repair with sectors corrupted in multiple mirrors Christoph Hellwig
2022-05-21 2:02 ` Zorro Lang
2022-05-22 1:13 ` Anand Jain
2022-05-20 16:47 ` [PATCH 2/2] btrfs: test repair with corrupted sectors interleaved over " Christoph Hellwig
2022-05-22 1:13 ` Anand Jain
2022-05-22 6:31 ` Qu Wenruo
2022-05-22 12:56 ` Anand Jain [this message]
2022-05-22 13:00 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1fc36e3f-8fa9-2bfb-ede1-d4f852bcb8cf@oracle.com \
--to=anand.jain@oracle.com \
--cc=fstests@vger.kernel.org \
--cc=hch@lst.de \
--cc=linux-btrfs@vger.kernel.org \
--cc=quwenruo.btrfs@gmx.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox