Re: misc-next and for-next: kernel BUG at fs/btrfs/extent_io.c:2350! during raid5 recovery

[Wang Yugui <wangyugui@e16-tech.com>]

[-- Attachment #1: Type: text/plain, Size: 2382 bytes --]

Hi,

If we add 'echo 3 > /proc/sys/vm/drop_caches; free -h' before
'blkdiscard', 'uncorrectable errors'  will happen in 1st loop.

please see the attachment file for the v2 script.

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2022/08/13

> Hi,
> 
> With a modified script based on this reproducer,
> 'uncorrectable errors' is easy to reproduce
> 
> kernel:
> 	5.20.0(2022/08/09) + some patches in misc-next
> 	0*-btrfs-scrub-properly-report-super-block-errors-in-sy.patch
> 	0*-btrfs-scrub-try-to-fix-super-block-errors.patch
> 	0*-btrfs-fix-lost-error-handling-when-looking-up-extend.patch
> 
> script:
> 	source ~/xfstests/local.config
> 
> 	SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
> 	umount $SCRATCH_MNT
> 	set -e
> 	uname -a
> 
> 	mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_ARRAY[@]}
> 	mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o
> 	compress=zstd,noatime
> 	mkdir -p $SCRATCH_MNT/dir1
> 
> 	/bin/cp -a /usr/hpc-bio $SCRATCH_MNT/dir1/
> 	sync
> 	du -sh $SCRATCH_MNT
> 	cd $SCRATCH_MNT
> 	while true; do
> 		find -type f -exec cat {} + > /dev/null
> 	done &
> 
> 	for((i=0;i>=0;++i)); do
> 
> 		sync; sleep 2; sync; sleep 4; sync; sleep 20; # change the
> 	device to discard in every loop
> 		j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} ))
> 		/usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} >/dev/null 2>&1
> 
> 		btrfs scrub start -Bd $SCRATCH_MNT | grep
> 'summary\|Uncorrectable'
> done &
> wait
> 
> Result:
> 	'uncorrectable errors' is reported in 2nd loop.
> 
> Is this a problem of this test script or btrfs kernel?
> 
> Best Regards
> Wang Yugui (wangyugui@e16-tech.com)
> 2022/08/12
> 
> 
> > > Any chance you could share a script for your reproducer?
> > 
> > The simplest reproducer is some variant of:
> > 
> > 	mkfs.btrfs -draid5 -mraid1 /dev/vdb /dev/vdc /dev/vdd
> > 	mount /dev/vdb /mnt -ocompress=zstd,noatime
> > 	cd /mnt
> > 	cp -a /40gb-test-data .
> > 	sync
> > 	while true; do
> > 		find -type f -exec cat {} + > /dev/null
> > 	done &
> > 	while true; do
> > 		cat /dev/zero > /dev/vdb
> > 	done &
> > 	while true; do
> > 		btrfs scrub start -Bd /mnt
> > 	done &
> > 	wait
> > 
> > but it can take a long time to hit a failure with something that gentle.
> > I throw on some extra test workload (e.g. lots of rsyncs) to keep the
> > page cache full and under memory pressure, which seems to speed up the
> > failure rate to once every few hours.
> 
> 
> 


[-- Attachment #2: raid5.disk.fail-v2.sh --]
[-- Type: application/octet-stream, Size: 788 bytes --]

#!/bin/bash
set -ux -o pipefail

source ~/xfstests/local.config

SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
umount $SCRATCH_MNT
set -e
uname -a

mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_ARRAY[@]}
mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
mkdir -p $SCRATCH_MNT/dir1

/bin/cp -a /usr/hpc-bio $SCRATCH_MNT/dir1/
sync
du -sh $SCRATCH_MNT
cd $SCRATCH_MNT
while true; do
	find -type f -exec cat {} + > /dev/null
done &

for((i=1;i>=0;++i)); do

	sync; sleep 2; sync; sleep 4; sync; sleep 20; # change the device to discard in every loop
	echo 3 > /proc/sys/vm/drop_caches; free -h
	j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} ))
	/usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} >/dev/null 2>&1

	btrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'
done &
wait