Re: btrfs: raid1C3 and raid1C4 fails to go ro when all but 1 drive removed.

[Nikolay Borisov <nborisov@suse.com>]

On 25.03.22 г. 6:44 ч., peter brown wrote:
> Hi,
> 
>   If I set up raid1C3 or raid1C4 and pull drives to simulate a drive 
> failure the fs does not go readonly.
> 
>   If I perform the same test on a raid1 setup the fs goes readonly when 
> the second last drive is removed. Ie when the fs can no longer maintain 
> a two copy mirror.
> 
> 
> 
> kernel 5.16.17-gentoo
> btrfs-progs v5.16.2
> 
> short version of the logs...
> 
> test1 ~ # mkfs.btrfs -f -L test -d raid1C3 -m raid1C3  /dev/sdb /dev/sdc 
> /dev/sde /dev/sdf
> test1 ~ # mount -t btrfs -o noatime /dev/sdb /mnt/btrfs/
> 
> 
> 
> RAID1C3
> pull drive 1 (3 left)
> pull drive 2 (2 left)

For raid1c3 we tolerate 2 failures, since you've removed 2 device you 
are within the boundaries so no failure is expected.

<snip>

> 
> 
> 
> pulling drive 3  we no longer can support 2 copies.. just 1

Here you've pulled 3 drives so when we try to writeback stuff to disk 
(like during transaction commit) the writes should fail and the 
filesystem switch to RO.

<snip>

> 
> read from fs
> find .
> no errors and no logs
> 
> The fs should go ro when we write the the drive in this state.

Yes it should, however this likely won't happen the same instant as you 
remove the drive. Instead it will wait until a write has to hit disk and 
the bio mapping routines figures out that the number of missing devices 
is above the threshold.

> 
> write to fs
> touch 3
> no errors ---  dmesg logs.
> [Thu Mar 24 23:31:34 2022] btrfs_dev_stat_print_on_error: 2 callbacks 
> suppressed
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs: 
> wr 8, rd 0, flush 1, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs: 
> wr 1, rd 0, flush 0, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs: 
> wr 9, rd 0, flush 1, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs: 
> wr 2, rd 0, flush 0, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs: 
> wr 10, rd 0, flush 1, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs: 
> wr 3, rd 0, flush 0, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdb errs: 
> wr 6, rd 0, flush 3, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs: 
> wr 10, rd 0, flush 2, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs: 
> wr 3, rd 0, flush 1, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write 
> due to IO error on /dev/sdb (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdb errs: 
> wr 7, rd 0, flush 3, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write 
> due to IO error on /dev/sdb (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write 
> due to IO error on /dev/sdb (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write 
> due to IO error on /dev/sde (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write 
> due to IO error on /dev/sde (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write 
> due to IO error on /dev/sde (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write 
> due to IO error on /dev/sdf (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write 
> due to IO error on /dev/sdf (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write 
> due to IO error on /dev/sdf (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): error writing 
> primary super block to device 1
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): error writing 
> primary super block to device 3
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): error writing 
> primary super block to device 4
> 
> There is no going ro log and the write should fail.
> 
> raid1C4 is the same thing. As 4 copies are required it should do ro when
> the first drive is pulled.

No, it shouldn't , raid1c4 tolerates up to 3 missing devices.

> 
> 
> If I configure as raid1

<snip>

>   Am I missing something here?

Raid1 tolerates 1 drive failure.

> 
> 
> 
> 
> 
> 
> 
>