Re: Converting RAID1 to single fails if RAID1 is missing device

[Jussi Kansanen <jussi.kansanen@gmail.com>]

On 2/3/25 02:44, Qu Wenruo wrote:
> 
> 
> 在 2025/2/3 09:13, Jeff Siddall 写道:
>> After a device failed on RAID1 filesystem, an attempt to convert the
>> online filesystem from RAID1 to single failed.  This isn't an uncommon
>> use case if the failed device isn't readily replaceable.
>>
>> The command ran was:
>>
>> btrfs balance start -f -sconvert=single -mconvert=single -
>> dconvert=single /mountpoint
>>
>> and the kernel logs were:
>>
>> kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single -
>> mconvert=dup -sconvert=dup
>> kernel: BTRFS info (device nvme0n1p3): relocating block group
>> 1222049267712 flags data|raid1
>> kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1
>> devices, max tolerance is 0 for writable mount
> 
> This is not the chunk to be relocated. Considering the tolerance is only
> 0, meaning it's the newly created single chunk.
> 
> I tried locally, but failed to reproduce the same problem.
> 
> Mind to provide the following info:
> 
> - Kernel version
> - Btrfs fi usage output
> - Mount option of the fs
> 
> My major concern is, the failing devices is still considered online, but
> will fail all read/write/flush commands.
> (Btrfs only has read time repair, not failing device detection)
> 
> In that case, converting to single is the worst thing you can do, as it
> will write metadata chunks into that failing devices, and lost everything.
> 
> The proper solution is to unmount the fs (if possible), remove the
> failing device, then mount the fs in degraded mode, add replace the
> missing device with a newer one.
> 
> Thanks,
> Qu
> 
>> kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370:
>> errno=-5 IO failure (errors while submitting device barriers.)
>> kernel: BTRFS info (device nvme0n1p3: state E): forced readonly
>> kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of
>> aborted transaction.
>> kernel: BTRFS: error (device nvme0n1p3: state EA) in
>> cleanup_transaction:1992: errno=-5 IO failure
>> kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with
>> status: -5
>>
>> Either it should be made possible to convert a RAID1 device with a
>> missing device to a single device filesystem without errors, or the
>> command should return a message stating that it is not supported to
>> convert RAID1 array with missing devices to a single.  Having the
>> process fail and then going forced readonly is a significant failure on
>> an otherwise working system.
>>
>>
>>
> 
> 

Hi, here's a reproducer for similar issue that Jeff had:

debian:/mnt# uname -a
Linux debian 6.12.11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.11-1 (2025-01-25) 
x86_64 GNU/Linux

debian:/mnt# findmnt .
TARGET SOURCE   FSTYPE OPTIONS
/mnt   /dev/sdd btrfs  rw,relatime,space_cache=v2,subvolid=5,subvol=/

debian:/mnt# btrfs fi usage .
Overall:
     Device size:		  16.00GiB
     Device allocated:		   4.52GiB
     Device unallocated:		  11.48GiB
     Device missing:		     0.00B
     Device slack:		     0.00B
     Used:			   2.00GiB
     Free (estimated):		   6.74GiB	(min: 6.74GiB)
     Free (statfs, df):		   6.74GiB
     Data ratio:			      2.00
     Metadata ratio:		      2.00
     Global reserve:		   5.50MiB	(used: 0.00B)
     Multiple profiles:		        no

Data,RAID1: Size:2.00GiB, Used:1.00GiB (50.05%)
    /dev/sdd	   2.00GiB
    /dev/sde	   2.00GiB

Metadata,RAID1: Size:256.00MiB, Used:1.14MiB (0.45%)
    /dev/sdd	 256.00MiB
    /dev/sde	 256.00MiB

System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
    /dev/sdd	   8.00MiB
    /dev/sde	   8.00MiB

Unallocated:
    /dev/sdd	   5.74GiB
    /dev/sde	   5.74GiB


debian:/mnt# echo 1 > /sys/block/sde/device/delete

debian:/mnt# btrfs balance start -mconvert=dup -dconvert=single .
ERROR: error during balancing '.': Input/output error
There may be more info in syslog - try dmesg | tail

debian:/mnt# dmesg | tail -35
[  582.117314] BTRFS info (device sdd): first mount of filesystem 
d4b36ef9-3518-43a6-bc68-a2b4df751896
[  582.117329] BTRFS info (device sdd): using crc32c (crc32c-intel) checksum 
algorithm
[  582.117333] BTRFS info (device sdd): using free-space-tree
[  582.119952] BTRFS info (device sdd): checking UUID tree
[  979.011795] sd 6:0:0:0: [sde] Synchronizing SCSI cache
[  979.013096] ata7.00: Entering standby power mode
[ 1002.726249] btrfs: attempt to access beyond end of device
                sde: rw=6145, sector=21696, nr_sectors = 32 limit=0
[ 1002.726274] btrfs: attempt to access beyond end of device
                sde: rw=6145, sector=21728, nr_sectors = 32 limit=0
[ 1002.726281] btrfs: attempt to access beyond end of device
                sde: rw=6145, sector=21760, nr_sectors = 32 limit=0
[ 1002.726442] BTRFS error (device sdd): bdev /dev/sde errs: wr 1, rd 0, flush 
0, corrupt 0, gen 0
[ 1002.726501] BTRFS error (device sdd): bdev /dev/sde errs: wr 2, rd 0, flush 
0, corrupt 0, gen 0
[ 1002.726534] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0, flush 
0, corrupt 0, gen 0
[ 1002.726659] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0, flush 
1, corrupt 0, gen 0
[ 1002.726692] btrfs: attempt to access beyond end of device
                sde: rw=145409, sector=128, nr_sectors = 8 limit=0
[ 1002.726702] BTRFS warning (device sdd): lost super block write due to IO 
error on /dev/sde (-5)
[ 1002.726704] BTRFS error (device sdd): bdev /dev/sde errs: wr 4, rd 0, flush 
1, corrupt 0, gen 0
[ 1002.726728] btrfs: attempt to access beyond end of device
                sde: rw=14337, sector=131072, nr_sectors = 8 limit=0
[ 1002.726733] BTRFS warning (device sdd): lost super block write due to IO 
error on /dev/sde (-5)
[ 1002.726737] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0, flush 
1, corrupt 0, gen 0
[ 1002.726835] BTRFS error (device sdd): error writing primary super block to 
device 2
[ 1002.726844] BTRFS info (device sdd): balance: start -dconvert=single 
-mconvert=dup -sconvert=dup
[ 1002.726990] BTRFS info (device sdd): relocating block group 1372585984 flags 
data|raid1
[ 1002.727811] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0, flush 
2, corrupt 0, gen 0
[ 1002.727816] BTRFS warning (device sdd): chunk 2446327808 missing 1 devices, 
max tolerance is 0 for writable mount
[ 1002.727818] BTRFS: error (device sdd) in write_all_supers:4044: errno=-5 IO 
failure (errors while submitting device barriers.)
[ 1002.727821] BTRFS info (device sdd state E): forced readonly
[ 1002.727823] BTRFS warning (device sdd state E): Skipping commit of aborted 
transaction.
[ 1002.727824] BTRFS error (device sdd state EA): Transaction aborted (error -5)
[ 1002.727826] BTRFS: error (device sdd state EA) in cleanup_transaction:2017: 
errno=-5 IO failure
[ 1002.727838] BTRFS info (device sdd state EA): balance: ended with status: -5

debian:~# umount /mnt

debian:~# btrfs fi show
warning, device 2 is missing
Label: none  uuid: d4b36ef9-3518-43a6-bc68-a2b4df751896
	Total devices 2 FS bytes used 1.00GiB
	devid    1 size 8.00GiB used 2.26GiB path /dev/sdd
	*** Some devices missing

mount -odegraded /dev/sdd /mnt

debian:~# dmesg | tail -15
[ 1332.628959] BTRFS info (device sdd): first mount of filesystem 
d4b36ef9-3518-43a6-bc68-a2b4df751896
[ 1332.628974] BTRFS info (device sdd): using crc32c (crc32c-intel) checksum 
algorithm
[ 1332.628978] BTRFS info (device sdd): using free-space-tree
[ 1332.630281] BTRFS warning (device sdd): devid 2 uuid 
0afd8d45-96d6-4393-b7ce-e55abd4b668e is missing
[ 1332.632964] BTRFS info (device sdd): balance: resume -dconvert=single,soft 
-mconvert=dup,soft -sconvert=dup,soft
[ 1332.633041] BTRFS info (device sdd): relocating block group 1372585984 flags 
data|raid1
[ 1332.638220] BTRFS info (device sdd): found 1 extents, stage: move data extents
[ 1332.640073] BTRFS info (device sdd): found 1 extents, stage: update data pointers
[ 1332.641350] BTRFS info (device sdd): relocating block group 298844160 flags 
data|raid1
[ 1333.991088] BTRFS info (device sdd): found 8 extents, stage: move data extents
[ 1333.994336] BTRFS info (device sdd): found 8 extents, stage: update data pointers
[ 1333.997220] BTRFS info (device sdd): relocating block group 30408704 flags 
metadata|raid1
[ 1333.998489] BTRFS info (device sdd): found 1 extents, stage: move data extents
[ 1333.999617] BTRFS info (device sdd): relocating block group 22020096 flags 
system|raid1
[ 1334.000838] BTRFS info (device sdd): balance: ended with status: 0

debian:/mnt# btrfs fi usage .
Overall:
     Device size:		  16.00GiB
     Device allocated:		   3.00GiB
     Device unallocated:		  13.00GiB
     Device missing:		   8.00GiB
     Device slack:		     0.00B
     Used:			   1.00GiB
     Free (estimated):		  14.44GiB	(min: 7.94GiB)
     Free (statfs, df):		   6.44GiB
     Data ratio:			      1.00
     Metadata ratio:		      2.00
     Global reserve:		   5.50MiB	(used: 0.00B)
     Multiple profiles:		        no

Data,single: Size:2.44GiB, Used:1.00GiB (41.07%)
    /dev/sdd	   2.44GiB

Metadata,DUP: Size:256.00MiB, Used:1.17MiB (0.46%)
    /dev/sdd	 512.00MiB

System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%)
    /dev/sdd	  64.00MiB

Unallocated:
    /dev/sdd	   5.00GiB
    <missing disk>	   8.00GiB

debian:/mnt# btrfs device remove missing .

debian:/mnt# dmesg | tail -1
[ 1714.116147] BTRFS info (device sdd): device deleted: missing