Re: Converting RAID1 to single fails if RAID1 is missing device

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

在 2025/2/3 12:49, Jussi Kansanen 写道:
> On 2/3/25 02:44, Qu Wenruo wrote:
>>
>>
>> 在 2025/2/3 09:13, Jeff Siddall 写道:
>>> After a device failed on RAID1 filesystem, an attempt to convert the
>>> online filesystem from RAID1 to single failed.  This isn't an uncommon
>>> use case if the failed device isn't readily replaceable.
>>>
>>> The command ran was:
>>>
>>> btrfs balance start -f -sconvert=single -mconvert=single -
>>> dconvert=single /mountpoint
>>>
>>> and the kernel logs were:
>>>
>>> kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single -
>>> mconvert=dup -sconvert=dup
>>> kernel: BTRFS info (device nvme0n1p3): relocating block group
>>> 1222049267712 flags data|raid1
>>> kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1
>>> devices, max tolerance is 0 for writable mount
>>
>> This is not the chunk to be relocated. Considering the tolerance is only
>> 0, meaning it's the newly created single chunk.
>>
>> I tried locally, but failed to reproduce the same problem.
>>
>> Mind to provide the following info:
>>
>> - Kernel version
>> - Btrfs fi usage output
>> - Mount option of the fs
>>
>> My major concern is, the failing devices is still considered online, but
>> will fail all read/write/flush commands.
>> (Btrfs only has read time repair, not failing device detection)
>>
>> In that case, converting to single is the worst thing you can do, as it
>> will write metadata chunks into that failing devices, and lost
>> everything.
>>
>> The proper solution is to unmount the fs (if possible), remove the
>> failing device, then mount the fs in degraded mode, add replace the
>> missing device with a newer one.
>>
>> Thanks,
>> Qu
>>
>>> kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370:
>>> errno=-5 IO failure (errors while submitting device barriers.)
>>> kernel: BTRFS info (device nvme0n1p3: state E): forced readonly
>>> kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of
>>> aborted transaction.
>>> kernel: BTRFS: error (device nvme0n1p3: state EA) in
>>> cleanup_transaction:1992: errno=-5 IO failure
>>> kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with
>>> status: -5
>>>
>>> Either it should be made possible to convert a RAID1 device with a
>>> missing device to a single device filesystem without errors, or the
>>> command should return a message stating that it is not supported to
>>> convert RAID1 array with missing devices to a single.  Having the
>>> process fail and then going forced readonly is a significant failure on
>>> an otherwise working system.
>>>
>>>
>>>
>>
>>
>
> Hi, here's a reproducer for similar issue that Jeff had:
>
> debian:/mnt# uname -a
> Linux debian 6.12.11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.11-1
> (2025-01-25) x86_64 GNU/Linux
>
> debian:/mnt# findmnt .
> TARGET SOURCE   FSTYPE OPTIONS
> /mnt   /dev/sdd btrfs  rw,relatime,space_cache=v2,subvolid=5,subvol=/
>
> debian:/mnt# btrfs fi usage .
> Overall:
>      Device size:          16.00GiB
>      Device allocated:           4.52GiB
>      Device unallocated:          11.48GiB
>      Device missing:             0.00B
>      Device slack:             0.00B
>      Used:               2.00GiB
>      Free (estimated):           6.74GiB    (min: 6.74GiB)
>      Free (statfs, df):           6.74GiB
>      Data ratio:                  2.00
>      Metadata ratio:              2.00
>      Global reserve:           5.50MiB    (used: 0.00B)
>      Multiple profiles:                no
>
> Data,RAID1: Size:2.00GiB, Used:1.00GiB (50.05%)
>     /dev/sdd       2.00GiB
>     /dev/sde       2.00GiB
>
> Metadata,RAID1: Size:256.00MiB, Used:1.14MiB (0.45%)
>     /dev/sdd     256.00MiB
>     /dev/sde     256.00MiB
>
> System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
>     /dev/sdd       8.00MiB
>     /dev/sde       8.00MiB
>
> Unallocated:
>     /dev/sdd       5.74GiB
>     /dev/sde       5.74GiB
>
>
> debian:/mnt# echo 1 > /sys/block/sde/device/delete
>
> debian:/mnt# btrfs balance start -mconvert=dup -dconvert=single .
> ERROR: error during balancing '.': Input/output error
> There may be more info in syslog - try dmesg | tail
>
> debian:/mnt# dmesg | tail -35
> [  582.117314] BTRFS info (device sdd): first mount of filesystem
> d4b36ef9-3518-43a6-bc68-a2b4df751896
> [  582.117329] BTRFS info (device sdd): using crc32c (crc32c-intel)
> checksum algorithm
> [  582.117333] BTRFS info (device sdd): using free-space-tree
> [  582.119952] BTRFS info (device sdd): checking UUID tree
> [  979.011795] sd 6:0:0:0: [sde] Synchronizing SCSI cache
> [  979.013096] ata7.00: Entering standby power mode
> [ 1002.726249] btrfs: attempt to access beyond end of device
>                 sde: rw=6145, sector=21696, nr_sectors = 32 limit=0

As you can see, btrfs is still trying to access the removed device, thus
still the worst case scenario.

IIRC the sde deletion using sysfs interface is exactly what test case
generic/730 is doing (but with single devices fs), and unfortunately
btrfs doesn't support proper device shutdown callback, thus using sysfs
delete is just leading to the worst situation.

I can enhance the document related to convert, so that one should not
try to convert if there is any unreliable device.

But I think the ultimate solution is to make btrfs to properly detect
and support device shut down request.
Although that would also introduce new complexity, e.g. what if the
missing devices show up again after missing several writes?

Thanks,
Qu

> [ 1002.726274] btrfs: attempt to access beyond end of device
>                 sde: rw=6145, sector=21728, nr_sectors = 32 limit=0
> [ 1002.726281] btrfs: attempt to access beyond end of device
>                 sde: rw=6145, sector=21760, nr_sectors = 32 limit=0
> [ 1002.726442] BTRFS error (device sdd): bdev /dev/sde errs: wr 1, rd 0,
> flush 0, corrupt 0, gen 0
> [ 1002.726501] BTRFS error (device sdd): bdev /dev/sde errs: wr 2, rd 0,
> flush 0, corrupt 0, gen 0
> [ 1002.726534] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0,
> flush 0, corrupt 0, gen 0
> [ 1002.726659] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0,
> flush 1, corrupt 0, gen 0
> [ 1002.726692] btrfs: attempt to access beyond end of device
>                 sde: rw=145409, sector=128, nr_sectors = 8 limit=0
> [ 1002.726702] BTRFS warning (device sdd): lost super block write due to
> IO error on /dev/sde (-5)
> [ 1002.726704] BTRFS error (device sdd): bdev /dev/sde errs: wr 4, rd 0,
> flush 1, corrupt 0, gen 0
> [ 1002.726728] btrfs: attempt to access beyond end of device
>                 sde: rw=14337, sector=131072, nr_sectors = 8 limit=0
> [ 1002.726733] BTRFS warning (device sdd): lost super block write due to
> IO error on /dev/sde (-5)
> [ 1002.726737] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0,
> flush 1, corrupt 0, gen 0
> [ 1002.726835] BTRFS error (device sdd): error writing primary super
> block to device 2
> [ 1002.726844] BTRFS info (device sdd): balance: start -dconvert=single
> -mconvert=dup -sconvert=dup
> [ 1002.726990] BTRFS info (device sdd): relocating block group
> 1372585984 flags data|raid1
> [ 1002.727811] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0,
> flush 2, corrupt 0, gen 0
> [ 1002.727816] BTRFS warning (device sdd): chunk 2446327808 missing 1
> devices, max tolerance is 0 for writable mount
> [ 1002.727818] BTRFS: error (device sdd) in write_all_supers:4044:
> errno=-5 IO failure (errors while submitting device barriers.)
> [ 1002.727821] BTRFS info (device sdd state E): forced readonly
> [ 1002.727823] BTRFS warning (device sdd state E): Skipping commit of
> aborted transaction.
> [ 1002.727824] BTRFS error (device sdd state EA): Transaction aborted
> (error -5)
> [ 1002.727826] BTRFS: error (device sdd state EA) in
> cleanup_transaction:2017: errno=-5 IO failure
> [ 1002.727838] BTRFS info (device sdd state EA): balance: ended with
> status: -5
>
> debian:~# umount /mnt
>
> debian:~# btrfs fi show
> warning, device 2 is missing
> Label: none  uuid: d4b36ef9-3518-43a6-bc68-a2b4df751896
>      Total devices 2 FS bytes used 1.00GiB
>      devid    1 size 8.00GiB used 2.26GiB path /dev/sdd
>      *** Some devices missing
>
> mount -odegraded /dev/sdd /mnt
>
> debian:~# dmesg | tail -15
> [ 1332.628959] BTRFS info (device sdd): first mount of filesystem
> d4b36ef9-3518-43a6-bc68-a2b4df751896
> [ 1332.628974] BTRFS info (device sdd): using crc32c (crc32c-intel)
> checksum algorithm
> [ 1332.628978] BTRFS info (device sdd): using free-space-tree
> [ 1332.630281] BTRFS warning (device sdd): devid 2 uuid
> 0afd8d45-96d6-4393-b7ce-e55abd4b668e is missing
> [ 1332.632964] BTRFS info (device sdd): balance: resume -
> dconvert=single,soft -mconvert=dup,soft -sconvert=dup,soft
> [ 1332.633041] BTRFS info (device sdd): relocating block group
> 1372585984 flags data|raid1
> [ 1332.638220] BTRFS info (device sdd): found 1 extents, stage: move
> data extents
> [ 1332.640073] BTRFS info (device sdd): found 1 extents, stage: update
> data pointers
> [ 1332.641350] BTRFS info (device sdd): relocating block group 298844160
> flags data|raid1
> [ 1333.991088] BTRFS info (device sdd): found 8 extents, stage: move
> data extents
> [ 1333.994336] BTRFS info (device sdd): found 8 extents, stage: update
> data pointers
> [ 1333.997220] BTRFS info (device sdd): relocating block group 30408704
> flags metadata|raid1
> [ 1333.998489] BTRFS info (device sdd): found 1 extents, stage: move
> data extents
> [ 1333.999617] BTRFS info (device sdd): relocating block group 22020096
> flags system|raid1
> [ 1334.000838] BTRFS info (device sdd): balance: ended with status: 0
>
> debian:/mnt# btrfs fi usage .
> Overall:
>      Device size:          16.00GiB
>      Device allocated:           3.00GiB
>      Device unallocated:          13.00GiB
>      Device missing:           8.00GiB
>      Device slack:             0.00B
>      Used:               1.00GiB
>      Free (estimated):          14.44GiB    (min: 7.94GiB)
>      Free (statfs, df):           6.44GiB
>      Data ratio:                  1.00
>      Metadata ratio:              2.00
>      Global reserve:           5.50MiB    (used: 0.00B)
>      Multiple profiles:                no
>
> Data,single: Size:2.44GiB, Used:1.00GiB (41.07%)
>     /dev/sdd       2.44GiB
>
> Metadata,DUP: Size:256.00MiB, Used:1.17MiB (0.46%)
>     /dev/sdd     512.00MiB
>
> System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%)
>     /dev/sdd      64.00MiB
>
> Unallocated:
>     /dev/sdd       5.00GiB
>     <missing disk>       8.00GiB
>
> debian:/mnt# btrfs device remove missing .
>
> debian:/mnt# dmesg | tail -1
> [ 1714.116147] BTRFS info (device sdd): device deleted: missing
>
>
>