* btrfs: raid1C3 and raid1C4 fails to go ro when all but 1 drive removed.
@ 2022-03-25 4:44 peter brown
2022-03-29 9:54 ` Nikolay Borisov
0 siblings, 1 reply; 2+ messages in thread
From: peter brown @ 2022-03-25 4:44 UTC (permalink / raw)
To: linux-btrfs
Hi,
If I set up raid1C3 or raid1C4 and pull drives to simulate a drive failure the
fs does not go readonly.
If I perform the same test on a raid1 setup the fs goes readonly when the
second last drive is removed. Ie when the fs can no longer maintain a two copy
mirror.
kernel 5.16.17-gentoo
btrfs-progs v5.16.2
short version of the logs...
test1 ~ # mkfs.btrfs -f -L test -d raid1C3 -m raid1C3 /dev/sdb /dev/sdc
/dev/sde /dev/sdf
test1 ~ # mount -t btrfs -o noatime /dev/sdb /mnt/btrfs/
RAID1C3
pull drive 1 (3 left)
pull drive 2 (2 left)
Pulling drive 2 should trigger a ro fs. Ie we no longer support 3 copies.
[ 1098.411396] BTRFS error (device sdb): bdev /dev/sdb errs: wr 0, rd 0, flush
1, corrupt 0, gen 0
[ 1098.430918] BTRFS warning (device sdb): lost page write due to IO error on
/dev/sdb (-5)
[ 1098.430923] BTRFS error (device sdb): bdev /dev/sdb errs: wr 1, rd 0, flush
1, corrupt 0, gen 0
[ 1098.430936] BTRFS warning (device sdb): lost page write due to IO error on
/dev/sdb (-5)
[ 1098.430939] BTRFS error (device sdb): bdev /dev/sdb errs: wr 2, rd 0, flush
1, corrupt 0, gen 0
[ 1098.430949] BTRFS warning (device sdb): lost page write due to IO error on
/dev/sdb (-5)
[ 1098.430952] BTRFS error (device sdb): bdev /dev/sdb errs: wr 3, rd 0, flush
1, corrupt 0, gen 0
[ 1098.431101] BTRFS error (device sdb): error writing primary super block to
device 1
[ 1111.722182] ata6: SATA link down (SStatus 0 SControl 300)
[ 1117.150299] ata6: SATA link down (SStatus 0 SControl 300)
[ 1122.782299] ata6: SATA link down (SStatus 0 SControl 300)
[ 1122.782308] ata6.00: disabled
[ 1122.782324] ata6.00: detaching (SCSI 5:0:0:0)
[ 1122.792186] sd 5:0:0:0: [sdf] Synchronizing SCSI cache
[ 1122.792218] sd 5:0:0:0: [sdf] Synchronize Cache(10) failed: Result:
hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[ 1122.792222] sd 5:0:0:0: [sdf] Stopping disk
[ 1122.792231] sd 5:0:0:0: [sdf] Start/Stop Unit failed: Result:
hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
btrfs device usage /mnt/btrfs/
/dev/sdb, ID: 1
Device size: 0.00B <---------
Device slack: 0.00B
Data,RAID1C3: 3.00GiB
Unallocated: 462.76GiB
/dev/sdc, ID: 2
Device size: 1.82TiB
Device slack: 0.00B
Data,RAID1C3: 4.00GiB
Metadata,RAID1C3: 1.00GiB
System,RAID1C3: 8.00MiB
Unallocated: 1.81TiB
/dev/sde, ID: 3
Device size: 465.76GiB
Device slack: 0.00B
Data,RAID1C3: 3.00GiB
Metadata,RAID1C3: 1.00GiB
System,RAID1C3: 8.00MiB
Unallocated: 461.75GiB
/dev/sdf, ID: 4
Device size: 0.00B <--------
Device slack: 0.00B
Data,RAID1C3: 2.00GiB
Metadata,RAID1C3: 1.00GiB
System,RAID1C3: 8.00MiB
Unallocated: 462.75GiB
I can read the fs.
Writing should trigger a ro fs.
touch k
[ 1381.545803] BTRFS error (device sdb): bdev /dev/sdf errs: wr 1, rd 0, flush
0, corrupt 0, gen 0
[ 1381.546043] BTRFS error (device sdb): bdev /dev/sdf errs: wr 2, rd 0, flush
0, corrupt 0, gen 0
[ 1381.546186] BTRFS error (device sdb): bdev /dev/sdf errs: wr 3, rd 0, flush
0, corrupt 0, gen 0
[ 1381.547239] BTRFS error (device sdb): bdev /dev/sdb errs: wr 3, rd 0, flush
2, corrupt 0, gen 0
[ 1381.572225] BTRFS error (device sdb): bdev /dev/sdf errs: wr 3, rd 0, flush
1, corrupt 0, gen 0
[ 1381.572244] BTRFS warning (device sdb): lost page write due to IO error on
/dev/sdb (-5)
[ 1381.572247] BTRFS error (device sdb): bdev /dev/sdb errs: wr 4, rd 0, flush
2, corrupt 0, gen 0
[ 1381.572257] BTRFS warning (device sdb): lost page write due to IO error on
/dev/sdb (-5)
[ 1381.572260] BTRFS error (device sdb): bdev /dev/sdb errs: wr 5, rd 0, flush
2, corrupt 0, gen 0
[ 1381.572270] BTRFS warning (device sdb): lost page write due to IO error on
/dev/sdb (-5)
[ 1381.572272] BTRFS error (device sdb): bdev /dev/sdb errs: wr 6, rd 0, flush
2, corrupt 0, gen 0
[ 1381.572376] BTRFS warning (device sdb): lost page write due to IO error on
/dev/sdf (-5)
[ 1381.572380] BTRFS error (device sdb): bdev /dev/sdf errs: wr 4, rd 0, flush
1, corrupt 0, gen 0
[ 1381.572392] BTRFS warning (device sdb): lost page write due to IO error on
/dev/sdf (-5)
[ 1381.572410] BTRFS error (device sdb): bdev /dev/sdf errs: wr 5, rd 0, flush
1, corrupt 0, gen 0
[ 1381.572423] BTRFS warning (device sdb): lost page write due to IO error on
/dev/sdf (-5)
[ 1381.572427] BTRFS error (device sdb): error writing primary super block to
device 1
[ 1381.613884] BTRFS error (device sdb): error writing primary super block to
device 4
write was sucessful
pulling drive 3 we no longer can support 2 copies.. just 1
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 1,
rd 0, flush 0, corrupt 0, gen 0
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 2,
rd 0, flush 0, corrupt 0, gen 0
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 3,
rd 0, flush 0, corrupt 0, gen 0
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 4,
rd 0, flush 0, corrupt 0, gen 0
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): bdev /dev/sdb errs: wr 3,
rd 0, flush 2, corrupt 0, gen 0
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 4,
rd 0, flush 1, corrupt 0, gen 0
[Thu Mar 24 23:27:28 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sdb (-5)
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): bdev /dev/sdb errs: wr 4,
rd 0, flush 2, corrupt 0, gen 0
[Thu Mar 24 23:27:28 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sdb (-5)
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): bdev /dev/sdb errs: wr 5,
rd 0, flush 2, corrupt 0, gen 0
[Thu Mar 24 23:27:28 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sdb (-5)
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): bdev /dev/sdb errs: wr 6,
rd 0, flush 2, corrupt 0, gen 0
[Thu Mar 24 23:27:28 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sde (-5)
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 5,
rd 0, flush 1, corrupt 0, gen 0
[Thu Mar 24 23:27:28 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sde (-5)
[Thu Mar 24 23:27:28 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sde (-5)
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): error writing primary super
block to device 1
[Thu Mar 24 23:27:28 2022] BTRFS error (device sdb): error writing primary super
block to device 3
[Thu Mar 24 23:28:28 2022] ata6: SATA link down (SStatus 0 SControl 300)
[Thu Mar 24 23:28:33 2022] ata6: SATA link down (SStatus 0 SControl 300)
[Thu Mar 24 23:28:39 2022] ata6: SATA link down (SStatus 0 SControl 300)
[Thu Mar 24 23:28:39 2022] ata6.00: disabled
[Thu Mar 24 23:28:39 2022] ata6.00: detaching (SCSI 5:0:0:0)
[Thu Mar 24 23:28:39 2022] sd 5:0:0:0: [sdf] Synchronizing SCSI cache
[Thu Mar 24 23:28:39 2022] sd 5:0:0:0: [sdf] Synchronize Cache(10) failed:
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[Thu Mar 24 23:28:39 2022] sd 5:0:0:0: [sdf] Stopping disk
[Thu Mar 24 23:28:39 2022] sd 5:0:0:0: [sdf] Start/Stop Unit failed: Result:
hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
btrfs device usage /mnt/btrfs/
/dev/sdb, ID: 1
Device size: 0.00B <-----------
Device slack: 0.00B
Data,RAID1C3: 3.00GiB
Unallocated: 462.76GiB
/dev/sdc, ID: 2
Device size: 1.82TiB
Device slack: 0.00B
Data,RAID1C3: 4.00GiB
Metadata,RAID1C3: 1.00GiB
System,RAID1C3: 8.00MiB
Unallocated: 1.81TiB
/dev/sde, ID: 3
Device size: 0.00B <-----------
Device slack: 0.00B
Data,RAID1C3: 3.00GiB
Metadata,RAID1C3: 1.00GiB
System,RAID1C3: 8.00MiB
Unallocated: 461.75GiB
/dev/sdf, ID: 4
Device size: 0.00B <-----------
Device slack: 0.00B
Data,RAID1C3: 2.00GiB
Metadata,RAID1C3: 1.00GiB
System,RAID1C3: 8.00MiB
Unallocated: 462.75GiB
read from fs
find .
no errors and no logs
The fs should go ro when we write the the drive in this state.
write to fs
touch 3
no errors --- dmesg logs.
[Thu Mar 24 23:31:34 2022] btrfs_dev_stat_print_on_error: 2 callbacks suppressed
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 8,
rd 0, flush 1, corrupt 0, gen 0
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs: wr 1,
rd 0, flush 0, corrupt 0, gen 0
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 9,
rd 0, flush 1, corrupt 0, gen 0
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs: wr 2,
rd 0, flush 0, corrupt 0, gen 0
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 10,
rd 0, flush 1, corrupt 0, gen 0
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs: wr 3,
rd 0, flush 0, corrupt 0, gen 0
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdb errs: wr 6,
rd 0, flush 3, corrupt 0, gen 0
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 10,
rd 0, flush 2, corrupt 0, gen 0
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs: wr 3,
rd 0, flush 1, corrupt 0, gen 0
[Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sdb (-5)
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdb errs: wr 7,
rd 0, flush 3, corrupt 0, gen 0
[Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sdb (-5)
[Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sdb (-5)
[Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sde (-5)
[Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sde (-5)
[Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sde (-5)
[Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sdf (-5)
[Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sdf (-5)
[Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write due to IO
error on /dev/sdf (-5)
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): error writing primary super
block to device 1
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): error writing primary super
block to device 3
[Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): error writing primary super
block to device 4
There is no going ro log and the write should fail.
raid1C4 is the same thing. As 4 copies are required it should do ro when
the first drive is pulled.
If I configure as raid1
mkfs.btrfs -f -L test -d raid1 -m raid1 /dev/sdb /dev/sdc /dev/sde /dev/sdf
mount -t btrfs -o noatime /dev/sdb /mnt/btrfs/
when I get to pulling the 3rd drive the fs goes ro when I try and write to it.
As it should
[Fri Mar 25 00:05:55 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 1,
rd 0, flush 0, corrupt 0, gen 0
[Fri Mar 25 00:05:55 2022] BTRFS error (device sdb): bdev /dev/sdf errs: wr 1,
rd 0, flush 0, corrupt 0, gen 0
[Fri Mar 25 00:05:55 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 2,
rd 0, flush 0, corrupt 0, gen 0
[Fri Mar 25 00:05:55 2022] BTRFS error (device sdb): bdev /dev/sdf errs: wr 2,
rd 0, flush 0, corrupt 0, gen 0
[Fri Mar 25 00:05:55 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 3,
rd 0, flush 0, corrupt 0, gen 0
[Fri Mar 25 00:05:55 2022] BTRFS error (device sdb): bdev /dev/sdf errs: wr 3,
rd 0, flush 0, corrupt 0, gen 0
[Fri Mar 25 00:05:55 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 4,
rd 0, flush 0, corrupt 0, gen 0
[Fri Mar 25 00:05:55 2022] BTRFS error (device sdb): bdev /dev/sdf errs: wr 4,
rd 0, flush 0, corrupt 0, gen 0
[Fri Mar 25 00:05:55 2022] BTRFS error (device sdb): bdev /dev/sde errs: wr 5,
rd 0, flush 0, corrupt 0, gen 0
[Fri Mar 25 00:05:55 2022] BTRFS error (device sdb): bdev /dev/sdf errs: wr 5,
rd 0, flush 0, corrupt 0, gen 0
[Fri Mar 25 00:05:55 2022] BTRFS: error (device sdb) in
btrfs_commit_transaction:2437: errno=-5 IO failure (Error while writing out
transaction)
[Fri Mar 25 00:05:55 2022] BTRFS info (device sdb): forced readonly
[Fri Mar 25 00:05:55 2022] BTRFS warning (device sdb): Skipping commit of
aborted transaction.
[Fri Mar 25 00:05:55 2022] BTRFS: error (device sdb) in
cleanup_transaction:2010: errno=-5 IO failure
[Fri Mar 25 00:06:01 2022] btrfs_dev_stat_print_on_error: 4 callbacks suppressed
[Fri Mar 25 00:06:01 2022] BTRFS error (device sdb): bdev /dev/sdb errs: wr 4,
rd 0, flush 1, corrupt 0, gen 0
Am I missing something here?
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: btrfs: raid1C3 and raid1C4 fails to go ro when all but 1 drive removed.
2022-03-25 4:44 btrfs: raid1C3 and raid1C4 fails to go ro when all but 1 drive removed peter brown
@ 2022-03-29 9:54 ` Nikolay Borisov
0 siblings, 0 replies; 2+ messages in thread
From: Nikolay Borisov @ 2022-03-29 9:54 UTC (permalink / raw)
To: peter brown, linux-btrfs
On 25.03.22 г. 6:44 ч., peter brown wrote:
> Hi,
>
> If I set up raid1C3 or raid1C4 and pull drives to simulate a drive
> failure the fs does not go readonly.
>
> If I perform the same test on a raid1 setup the fs goes readonly when
> the second last drive is removed. Ie when the fs can no longer maintain
> a two copy mirror.
>
>
>
> kernel 5.16.17-gentoo
> btrfs-progs v5.16.2
>
> short version of the logs...
>
> test1 ~ # mkfs.btrfs -f -L test -d raid1C3 -m raid1C3 /dev/sdb /dev/sdc
> /dev/sde /dev/sdf
> test1 ~ # mount -t btrfs -o noatime /dev/sdb /mnt/btrfs/
>
>
>
> RAID1C3
> pull drive 1 (3 left)
> pull drive 2 (2 left)
For raid1c3 we tolerate 2 failures, since you've removed 2 device you
are within the boundaries so no failure is expected.
<snip>
>
>
>
> pulling drive 3 we no longer can support 2 copies.. just 1
Here you've pulled 3 drives so when we try to writeback stuff to disk
(like during transaction commit) the writes should fail and the
filesystem switch to RO.
<snip>
>
> read from fs
> find .
> no errors and no logs
>
> The fs should go ro when we write the the drive in this state.
Yes it should, however this likely won't happen the same instant as you
remove the drive. Instead it will wait until a write has to hit disk and
the bio mapping routines figures out that the number of missing devices
is above the threshold.
>
> write to fs
> touch 3
> no errors --- dmesg logs.
> [Thu Mar 24 23:31:34 2022] btrfs_dev_stat_print_on_error: 2 callbacks
> suppressed
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs:
> wr 8, rd 0, flush 1, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs:
> wr 1, rd 0, flush 0, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs:
> wr 9, rd 0, flush 1, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs:
> wr 2, rd 0, flush 0, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs:
> wr 10, rd 0, flush 1, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs:
> wr 3, rd 0, flush 0, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdb errs:
> wr 6, rd 0, flush 3, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sde errs:
> wr 10, rd 0, flush 2, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdf errs:
> wr 3, rd 0, flush 1, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write
> due to IO error on /dev/sdb (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): bdev /dev/sdb errs:
> wr 7, rd 0, flush 3, corrupt 0, gen 0
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write
> due to IO error on /dev/sdb (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write
> due to IO error on /dev/sdb (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write
> due to IO error on /dev/sde (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write
> due to IO error on /dev/sde (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write
> due to IO error on /dev/sde (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write
> due to IO error on /dev/sdf (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write
> due to IO error on /dev/sdf (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS warning (device sdb): lost page write
> due to IO error on /dev/sdf (-5)
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): error writing
> primary super block to device 1
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): error writing
> primary super block to device 3
> [Thu Mar 24 23:31:34 2022] BTRFS error (device sdb): error writing
> primary super block to device 4
>
> There is no going ro log and the write should fail.
>
> raid1C4 is the same thing. As 4 copies are required it should do ro when
> the first drive is pulled.
No, it shouldn't , raid1c4 tolerates up to 3 missing devices.
>
>
> If I configure as raid1
<snip>
> Am I missing something here?
Raid1 tolerates 1 drive failure.
>
>
>
>
>
>
>
>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2022-03-29 9:55 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-03-25 4:44 btrfs: raid1C3 and raid1C4 fails to go ro when all but 1 drive removed peter brown
2022-03-29 9:54 ` Nikolay Borisov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox