Re: Uncorrectable error during multiple scrub (raid5 recovery).

[Wang Yugui <wangyugui@e16-tech.com>]

Hi,

> Uncorrectable error during multiple scrub (raid5 recovery).
> 
> This reproducer is based on some reproducer [1],
> but it seems a new problem, so I open a new thread.
> 
> reproducer:
> 
> mkfs.btrfs -f -draid5 -mraid1 ${SCRATCH_DEV_POOL}
> SCRATCH_DEV_ARRAY=($SCRATCH_DEV_POOL)
> mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
> 
> /bin/cp -a /usr/bin $SCRATCH_MNT/
> #(OK)dd if=/dev/urandom bs=1M count=1K of=$SCRATCH_MNT/1G.img
> du -sh $SCRATCH_MNT
> 
> for((i=1;i<=15;++i)); do
> 
> 	#(OK)umount $SCRATCH_MNT; mount ${SCRATCH_DEV_ARRAY[0]} $SCRATCH_MNT # -o compress=zstd,noatime
> 	sync; sleep 5; sync; sleep 5; sync; sleep 25;
> 
> 	# change the device to discard in every loop
> 	j=$(( i % ${#SCRATCH_DEV_ARRAY[@]} ))
> 	/usr/sbin/blkdiscard -f ${SCRATCH_DEV_ARRAY[$j]} # --offset 2M
> 
> 	btrfs scrub start -Bd $SCRATCH_MNT | grep 'summary\|Uncorrectable'
> 
> done


Add some info about the logical of this reproducer.

For btrfs RAID5, if only one disk is corrupted (blkdiscard in this
reproducer) at the same time, we can recovery it by 'btrfs scrub'.

Once this recovery is finished, and then if another disk is corrupted
(blkdiscard in this reproducer) too, we can recovery it by 'btrfs scrub'
too.

We do multiple RAID5 recovery(btrfs scrub) with different disk, as a way
to try to detect some mismatched info inside.

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2022/08/14


> 
> This problem will not happen if we change the test data to simpler one.
> # about 220M data of '/usr/bin' to single 1G file
> 
> This problem will not happen if we clear cache with 'umount; mount'
> between multiple loop.
> # 'sync; sleep 5; ...' to  'umount; mount'
> 
> so it seems that some info in memory is wrong after RAID5 recovery?
> 
> [1]
> Subject: misc-next and for-next: kernel BUG at fs/btrfs/extent_io.c:2350!
> during raid5 recovery
> https://lore.kernel.org/linux-btrfs/9dfb0b60-9178-7bbe-6ba1-10d056a7e84c@gmx.com/T/#t
> 
> Best Regards
> Wang Yugui (wangyugui@e16-tech.com)
> 2022/08/14
>