Converting RAID1 to single fails if RAID1 is missing device

Converting RAID1 to single fails if RAID1 is missing device

[Jeff Siddall]

After a device failed on RAID1 filesystem, an attempt to convert the 
online filesystem from RAID1 to single failed.  This isn't an uncommon 
use case if the failed device isn't readily replaceable.

The command ran was:

btrfs balance start -f -sconvert=single -mconvert=single 
-dconvert=single /mountpoint

and the kernel logs were:

kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single 
-mconvert=dup -sconvert=dup
kernel: BTRFS info (device nvme0n1p3): relocating block group 
1222049267712 flags data|raid1
kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1 
devices, max tolerance is 0 for writable mount
kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370: 
errno=-5 IO failure (errors while submitting device barriers.)
kernel: BTRFS info (device nvme0n1p3: state E): forced readonly
kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of 
aborted transaction.
kernel: BTRFS: error (device nvme0n1p3: state EA) in 
cleanup_transaction:1992: errno=-5 IO failure
kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with 
status: -5

Either it should be made possible to convert a RAID1 device with a 
missing device to a single device filesystem without errors, or the 
command should return a message stating that it is not supported to 
convert RAID1 array with missing devices to a single.  Having the 
process fail and then going forced readonly is a significant failure on 
an otherwise working system.

Re: Converting RAID1 to single fails if RAID1 is missing device

[Qu Wenruo]

在 2025/2/3 09:13, Jeff Siddall 写道:
> After a device failed on RAID1 filesystem, an attempt to convert the
> online filesystem from RAID1 to single failed.  This isn't an uncommon
> use case if the failed device isn't readily replaceable.
>
> The command ran was:
>
> btrfs balance start -f -sconvert=single -mconvert=single -
> dconvert=single /mountpoint
>
> and the kernel logs were:
>
> kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single -
> mconvert=dup -sconvert=dup
> kernel: BTRFS info (device nvme0n1p3): relocating block group
> 1222049267712 flags data|raid1
> kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1
> devices, max tolerance is 0 for writable mount

This is not the chunk to be relocated. Considering the tolerance is only
0, meaning it's the newly created single chunk.

I tried locally, but failed to reproduce the same problem.

Mind to provide the following info:

- Kernel version
- Btrfs fi usage output
- Mount option of the fs

My major concern is, the failing devices is still considered online, but
will fail all read/write/flush commands.
(Btrfs only has read time repair, not failing device detection)

In that case, converting to single is the worst thing you can do, as it
will write metadata chunks into that failing devices, and lost everything.

The proper solution is to unmount the fs (if possible), remove the
failing device, then mount the fs in degraded mode, add replace the
missing device with a newer one.

Thanks,
Qu

> kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370:
> errno=-5 IO failure (errors while submitting device barriers.)
> kernel: BTRFS info (device nvme0n1p3: state E): forced readonly
> kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of
> aborted transaction.
> kernel: BTRFS: error (device nvme0n1p3: state EA) in
> cleanup_transaction:1992: errno=-5 IO failure
> kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with
> status: -5
>
> Either it should be made possible to convert a RAID1 device with a
> missing device to a single device filesystem without errors, or the
> command should return a message stating that it is not supported to
> convert RAID1 array with missing devices to a single.  Having the
> process fail and then going forced readonly is a significant failure on
> an otherwise working system.
>
>
>

Re: Converting RAID1 to single fails if RAID1 is missing device

[Jeff Siddall]

To be clear, when I say "device failed" I mean it disappeared entirely 
so from btrfs perspective it was missing.

Kernel string is: 6.1.0-1025-rockchip #25-Ubuntu SMP Mon Aug 26 23:01:14 
UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

I have since rebuilt the array so fi usage doesn't contain anything 
relevant to this issue any more.

Mount options are defaults.

I was chatting with @multicore over on libera.chat/#btrfs who said they 
were able to reproduce.  I can see if I can find out how they did if you 
are still unable to see this for yourself.

Thanks,

Jeff


On 2025-02-02 19:44, Qu Wenruo wrote:
>
>
> 在 2025/2/3 09:13, Jeff Siddall 写道:
>> After a device failed on RAID1 filesystem, an attempt to convert the
>> online filesystem from RAID1 to single failed.  This isn't an uncommon
>> use case if the failed device isn't readily replaceable.
>>
>> The command ran was:
>>
>> btrfs balance start -f -sconvert=single -mconvert=single -
>> dconvert=single /mountpoint
>>
>> and the kernel logs were:
>>
>> kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single -
>> mconvert=dup -sconvert=dup
>> kernel: BTRFS info (device nvme0n1p3): relocating block group
>> 1222049267712 flags data|raid1
>> kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1
>> devices, max tolerance is 0 for writable mount
>
> This is not the chunk to be relocated. Considering the tolerance is only
> 0, meaning it's the newly created single chunk.
>
> I tried locally, but failed to reproduce the same problem.
>
> Mind to provide the following info:
>
> - Kernel version
> - Btrfs fi usage output
> - Mount option of the fs
>
> My major concern is, the failing devices is still considered online, but
> will fail all read/write/flush commands.
> (Btrfs only has read time repair, not failing device detection)
>
> In that case, converting to single is the worst thing you can do, as it
> will write metadata chunks into that failing devices, and lost 
> everything.
>
> The proper solution is to unmount the fs (if possible), remove the
> failing device, then mount the fs in degraded mode, add replace the
> missing device with a newer one.
>
> Thanks,
> Qu
>
>> kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370:
>> errno=-5 IO failure (errors while submitting device barriers.)
>> kernel: BTRFS info (device nvme0n1p3: state E): forced readonly
>> kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of
>> aborted transaction.
>> kernel: BTRFS: error (device nvme0n1p3: state EA) in
>> cleanup_transaction:1992: errno=-5 IO failure
>> kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with
>> status: -5
>>
>> Either it should be made possible to convert a RAID1 device with a
>> missing device to a single device filesystem without errors, or the
>> command should return a message stating that it is not supported to
>> convert RAID1 array with missing devices to a single.  Having the
>> process fail and then going forced readonly is a significant failure on
>> an otherwise working system.
>>
>>
>>
>

Re: Converting RAID1 to single fails if RAID1 is missing device

[Jussi Kansanen]

On 2/3/25 02:44, Qu Wenruo wrote:
> 
> 
> 在 2025/2/3 09:13, Jeff Siddall 写道:
>> After a device failed on RAID1 filesystem, an attempt to convert the
>> online filesystem from RAID1 to single failed.  This isn't an uncommon
>> use case if the failed device isn't readily replaceable.
>>
>> The command ran was:
>>
>> btrfs balance start -f -sconvert=single -mconvert=single -
>> dconvert=single /mountpoint
>>
>> and the kernel logs were:
>>
>> kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single -
>> mconvert=dup -sconvert=dup
>> kernel: BTRFS info (device nvme0n1p3): relocating block group
>> 1222049267712 flags data|raid1
>> kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1
>> devices, max tolerance is 0 for writable mount
> 
> This is not the chunk to be relocated. Considering the tolerance is only
> 0, meaning it's the newly created single chunk.
> 
> I tried locally, but failed to reproduce the same problem.
> 
> Mind to provide the following info:
> 
> - Kernel version
> - Btrfs fi usage output
> - Mount option of the fs
> 
> My major concern is, the failing devices is still considered online, but
> will fail all read/write/flush commands.
> (Btrfs only has read time repair, not failing device detection)
> 
> In that case, converting to single is the worst thing you can do, as it
> will write metadata chunks into that failing devices, and lost everything.
> 
> The proper solution is to unmount the fs (if possible), remove the
> failing device, then mount the fs in degraded mode, add replace the
> missing device with a newer one.
> 
> Thanks,
> Qu
> 
>> kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370:
>> errno=-5 IO failure (errors while submitting device barriers.)
>> kernel: BTRFS info (device nvme0n1p3: state E): forced readonly
>> kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of
>> aborted transaction.
>> kernel: BTRFS: error (device nvme0n1p3: state EA) in
>> cleanup_transaction:1992: errno=-5 IO failure
>> kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with
>> status: -5
>>
>> Either it should be made possible to convert a RAID1 device with a
>> missing device to a single device filesystem without errors, or the
>> command should return a message stating that it is not supported to
>> convert RAID1 array with missing devices to a single.  Having the
>> process fail and then going forced readonly is a significant failure on
>> an otherwise working system.
>>
>>
>>
> 
> 

Hi, here's a reproducer for similar issue that Jeff had:

debian:/mnt# uname -a
Linux debian 6.12.11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.11-1 (2025-01-25) 
x86_64 GNU/Linux

debian:/mnt# findmnt .
TARGET SOURCE   FSTYPE OPTIONS
/mnt   /dev/sdd btrfs  rw,relatime,space_cache=v2,subvolid=5,subvol=/

debian:/mnt# btrfs fi usage .
Overall:
     Device size:		  16.00GiB
     Device allocated:		   4.52GiB
     Device unallocated:		  11.48GiB
     Device missing:		     0.00B
     Device slack:		     0.00B
     Used:			   2.00GiB
     Free (estimated):		   6.74GiB	(min: 6.74GiB)
     Free (statfs, df):		   6.74GiB
     Data ratio:			      2.00
     Metadata ratio:		      2.00
     Global reserve:		   5.50MiB	(used: 0.00B)
     Multiple profiles:		        no

Data,RAID1: Size:2.00GiB, Used:1.00GiB (50.05%)
    /dev/sdd	   2.00GiB
    /dev/sde	   2.00GiB

Metadata,RAID1: Size:256.00MiB, Used:1.14MiB (0.45%)
    /dev/sdd	 256.00MiB
    /dev/sde	 256.00MiB

System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
    /dev/sdd	   8.00MiB
    /dev/sde	   8.00MiB

Unallocated:
    /dev/sdd	   5.74GiB
    /dev/sde	   5.74GiB


debian:/mnt# echo 1 > /sys/block/sde/device/delete

debian:/mnt# btrfs balance start -mconvert=dup -dconvert=single .
ERROR: error during balancing '.': Input/output error
There may be more info in syslog - try dmesg | tail

debian:/mnt# dmesg | tail -35
[  582.117314] BTRFS info (device sdd): first mount of filesystem 
d4b36ef9-3518-43a6-bc68-a2b4df751896
[  582.117329] BTRFS info (device sdd): using crc32c (crc32c-intel) checksum 
algorithm
[  582.117333] BTRFS info (device sdd): using free-space-tree
[  582.119952] BTRFS info (device sdd): checking UUID tree
[  979.011795] sd 6:0:0:0: [sde] Synchronizing SCSI cache
[  979.013096] ata7.00: Entering standby power mode
[ 1002.726249] btrfs: attempt to access beyond end of device
                sde: rw=6145, sector=21696, nr_sectors = 32 limit=0
[ 1002.726274] btrfs: attempt to access beyond end of device
                sde: rw=6145, sector=21728, nr_sectors = 32 limit=0
[ 1002.726281] btrfs: attempt to access beyond end of device
                sde: rw=6145, sector=21760, nr_sectors = 32 limit=0
[ 1002.726442] BTRFS error (device sdd): bdev /dev/sde errs: wr 1, rd 0, flush 
0, corrupt 0, gen 0
[ 1002.726501] BTRFS error (device sdd): bdev /dev/sde errs: wr 2, rd 0, flush 
0, corrupt 0, gen 0
[ 1002.726534] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0, flush 
0, corrupt 0, gen 0
[ 1002.726659] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0, flush 
1, corrupt 0, gen 0
[ 1002.726692] btrfs: attempt to access beyond end of device
                sde: rw=145409, sector=128, nr_sectors = 8 limit=0
[ 1002.726702] BTRFS warning (device sdd): lost super block write due to IO 
error on /dev/sde (-5)
[ 1002.726704] BTRFS error (device sdd): bdev /dev/sde errs: wr 4, rd 0, flush 
1, corrupt 0, gen 0
[ 1002.726728] btrfs: attempt to access beyond end of device
                sde: rw=14337, sector=131072, nr_sectors = 8 limit=0
[ 1002.726733] BTRFS warning (device sdd): lost super block write due to IO 
error on /dev/sde (-5)
[ 1002.726737] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0, flush 
1, corrupt 0, gen 0
[ 1002.726835] BTRFS error (device sdd): error writing primary super block to 
device 2
[ 1002.726844] BTRFS info (device sdd): balance: start -dconvert=single 
-mconvert=dup -sconvert=dup
[ 1002.726990] BTRFS info (device sdd): relocating block group 1372585984 flags 
data|raid1
[ 1002.727811] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0, flush 
2, corrupt 0, gen 0
[ 1002.727816] BTRFS warning (device sdd): chunk 2446327808 missing 1 devices, 
max tolerance is 0 for writable mount
[ 1002.727818] BTRFS: error (device sdd) in write_all_supers:4044: errno=-5 IO 
failure (errors while submitting device barriers.)
[ 1002.727821] BTRFS info (device sdd state E): forced readonly
[ 1002.727823] BTRFS warning (device sdd state E): Skipping commit of aborted 
transaction.
[ 1002.727824] BTRFS error (device sdd state EA): Transaction aborted (error -5)
[ 1002.727826] BTRFS: error (device sdd state EA) in cleanup_transaction:2017: 
errno=-5 IO failure
[ 1002.727838] BTRFS info (device sdd state EA): balance: ended with status: -5

debian:~# umount /mnt

debian:~# btrfs fi show
warning, device 2 is missing
Label: none  uuid: d4b36ef9-3518-43a6-bc68-a2b4df751896
	Total devices 2 FS bytes used 1.00GiB
	devid    1 size 8.00GiB used 2.26GiB path /dev/sdd
	*** Some devices missing

mount -odegraded /dev/sdd /mnt

debian:~# dmesg | tail -15
[ 1332.628959] BTRFS info (device sdd): first mount of filesystem 
d4b36ef9-3518-43a6-bc68-a2b4df751896
[ 1332.628974] BTRFS info (device sdd): using crc32c (crc32c-intel) checksum 
algorithm
[ 1332.628978] BTRFS info (device sdd): using free-space-tree
[ 1332.630281] BTRFS warning (device sdd): devid 2 uuid 
0afd8d45-96d6-4393-b7ce-e55abd4b668e is missing
[ 1332.632964] BTRFS info (device sdd): balance: resume -dconvert=single,soft 
-mconvert=dup,soft -sconvert=dup,soft
[ 1332.633041] BTRFS info (device sdd): relocating block group 1372585984 flags 
data|raid1
[ 1332.638220] BTRFS info (device sdd): found 1 extents, stage: move data extents
[ 1332.640073] BTRFS info (device sdd): found 1 extents, stage: update data pointers
[ 1332.641350] BTRFS info (device sdd): relocating block group 298844160 flags 
data|raid1
[ 1333.991088] BTRFS info (device sdd): found 8 extents, stage: move data extents
[ 1333.994336] BTRFS info (device sdd): found 8 extents, stage: update data pointers
[ 1333.997220] BTRFS info (device sdd): relocating block group 30408704 flags 
metadata|raid1
[ 1333.998489] BTRFS info (device sdd): found 1 extents, stage: move data extents
[ 1333.999617] BTRFS info (device sdd): relocating block group 22020096 flags 
system|raid1
[ 1334.000838] BTRFS info (device sdd): balance: ended with status: 0

debian:/mnt# btrfs fi usage .
Overall:
     Device size:		  16.00GiB
     Device allocated:		   3.00GiB
     Device unallocated:		  13.00GiB
     Device missing:		   8.00GiB
     Device slack:		     0.00B
     Used:			   1.00GiB
     Free (estimated):		  14.44GiB	(min: 7.94GiB)
     Free (statfs, df):		   6.44GiB
     Data ratio:			      1.00
     Metadata ratio:		      2.00
     Global reserve:		   5.50MiB	(used: 0.00B)
     Multiple profiles:		        no

Data,single: Size:2.44GiB, Used:1.00GiB (41.07%)
    /dev/sdd	   2.44GiB

Metadata,DUP: Size:256.00MiB, Used:1.17MiB (0.46%)
    /dev/sdd	 512.00MiB

System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%)
    /dev/sdd	  64.00MiB

Unallocated:
    /dev/sdd	   5.00GiB
    <missing disk>	   8.00GiB

debian:/mnt# btrfs device remove missing .

debian:/mnt# dmesg | tail -1
[ 1714.116147] BTRFS info (device sdd): device deleted: missing

Re: Converting RAID1 to single fails if RAID1 is missing device

[Qu Wenruo]

在 2025/2/3 12:49, Jussi Kansanen 写道:
> On 2/3/25 02:44, Qu Wenruo wrote:
>>
>>
>> 在 2025/2/3 09:13, Jeff Siddall 写道:
>>> After a device failed on RAID1 filesystem, an attempt to convert the
>>> online filesystem from RAID1 to single failed.  This isn't an uncommon
>>> use case if the failed device isn't readily replaceable.
>>>
>>> The command ran was:
>>>
>>> btrfs balance start -f -sconvert=single -mconvert=single -
>>> dconvert=single /mountpoint
>>>
>>> and the kernel logs were:
>>>
>>> kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single -
>>> mconvert=dup -sconvert=dup
>>> kernel: BTRFS info (device nvme0n1p3): relocating block group
>>> 1222049267712 flags data|raid1
>>> kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1
>>> devices, max tolerance is 0 for writable mount
>>
>> This is not the chunk to be relocated. Considering the tolerance is only
>> 0, meaning it's the newly created single chunk.
>>
>> I tried locally, but failed to reproduce the same problem.
>>
>> Mind to provide the following info:
>>
>> - Kernel version
>> - Btrfs fi usage output
>> - Mount option of the fs
>>
>> My major concern is, the failing devices is still considered online, but
>> will fail all read/write/flush commands.
>> (Btrfs only has read time repair, not failing device detection)
>>
>> In that case, converting to single is the worst thing you can do, as it
>> will write metadata chunks into that failing devices, and lost
>> everything.
>>
>> The proper solution is to unmount the fs (if possible), remove the
>> failing device, then mount the fs in degraded mode, add replace the
>> missing device with a newer one.
>>
>> Thanks,
>> Qu
>>
>>> kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370:
>>> errno=-5 IO failure (errors while submitting device barriers.)
>>> kernel: BTRFS info (device nvme0n1p3: state E): forced readonly
>>> kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of
>>> aborted transaction.
>>> kernel: BTRFS: error (device nvme0n1p3: state EA) in
>>> cleanup_transaction:1992: errno=-5 IO failure
>>> kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with
>>> status: -5
>>>
>>> Either it should be made possible to convert a RAID1 device with a
>>> missing device to a single device filesystem without errors, or the
>>> command should return a message stating that it is not supported to
>>> convert RAID1 array with missing devices to a single.  Having the
>>> process fail and then going forced readonly is a significant failure on
>>> an otherwise working system.
>>>
>>>
>>>
>>
>>
>
> Hi, here's a reproducer for similar issue that Jeff had:
>
> debian:/mnt# uname -a
> Linux debian 6.12.11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.11-1
> (2025-01-25) x86_64 GNU/Linux
>
> debian:/mnt# findmnt .
> TARGET SOURCE   FSTYPE OPTIONS
> /mnt   /dev/sdd btrfs  rw,relatime,space_cache=v2,subvolid=5,subvol=/
>
> debian:/mnt# btrfs fi usage .
> Overall:
>      Device size:          16.00GiB
>      Device allocated:           4.52GiB
>      Device unallocated:          11.48GiB
>      Device missing:             0.00B
>      Device slack:             0.00B
>      Used:               2.00GiB
>      Free (estimated):           6.74GiB    (min: 6.74GiB)
>      Free (statfs, df):           6.74GiB
>      Data ratio:                  2.00
>      Metadata ratio:              2.00
>      Global reserve:           5.50MiB    (used: 0.00B)
>      Multiple profiles:                no
>
> Data,RAID1: Size:2.00GiB, Used:1.00GiB (50.05%)
>     /dev/sdd       2.00GiB
>     /dev/sde       2.00GiB
>
> Metadata,RAID1: Size:256.00MiB, Used:1.14MiB (0.45%)
>     /dev/sdd     256.00MiB
>     /dev/sde     256.00MiB
>
> System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
>     /dev/sdd       8.00MiB
>     /dev/sde       8.00MiB
>
> Unallocated:
>     /dev/sdd       5.74GiB
>     /dev/sde       5.74GiB
>
>
> debian:/mnt# echo 1 > /sys/block/sde/device/delete
>
> debian:/mnt# btrfs balance start -mconvert=dup -dconvert=single .
> ERROR: error during balancing '.': Input/output error
> There may be more info in syslog - try dmesg | tail
>
> debian:/mnt# dmesg | tail -35
> [  582.117314] BTRFS info (device sdd): first mount of filesystem
> d4b36ef9-3518-43a6-bc68-a2b4df751896
> [  582.117329] BTRFS info (device sdd): using crc32c (crc32c-intel)
> checksum algorithm
> [  582.117333] BTRFS info (device sdd): using free-space-tree
> [  582.119952] BTRFS info (device sdd): checking UUID tree
> [  979.011795] sd 6:0:0:0: [sde] Synchronizing SCSI cache
> [  979.013096] ata7.00: Entering standby power mode
> [ 1002.726249] btrfs: attempt to access beyond end of device
>                 sde: rw=6145, sector=21696, nr_sectors = 32 limit=0

As you can see, btrfs is still trying to access the removed device, thus
still the worst case scenario.

IIRC the sde deletion using sysfs interface is exactly what test case
generic/730 is doing (but with single devices fs), and unfortunately
btrfs doesn't support proper device shutdown callback, thus using sysfs
delete is just leading to the worst situation.

I can enhance the document related to convert, so that one should not
try to convert if there is any unreliable device.

But I think the ultimate solution is to make btrfs to properly detect
and support device shut down request.
Although that would also introduce new complexity, e.g. what if the
missing devices show up again after missing several writes?

Thanks,
Qu

> [ 1002.726274] btrfs: attempt to access beyond end of device
>                 sde: rw=6145, sector=21728, nr_sectors = 32 limit=0
> [ 1002.726281] btrfs: attempt to access beyond end of device
>                 sde: rw=6145, sector=21760, nr_sectors = 32 limit=0
> [ 1002.726442] BTRFS error (device sdd): bdev /dev/sde errs: wr 1, rd 0,
> flush 0, corrupt 0, gen 0
> [ 1002.726501] BTRFS error (device sdd): bdev /dev/sde errs: wr 2, rd 0,
> flush 0, corrupt 0, gen 0
> [ 1002.726534] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0,
> flush 0, corrupt 0, gen 0
> [ 1002.726659] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0,
> flush 1, corrupt 0, gen 0
> [ 1002.726692] btrfs: attempt to access beyond end of device
>                 sde: rw=145409, sector=128, nr_sectors = 8 limit=0
> [ 1002.726702] BTRFS warning (device sdd): lost super block write due to
> IO error on /dev/sde (-5)
> [ 1002.726704] BTRFS error (device sdd): bdev /dev/sde errs: wr 4, rd 0,
> flush 1, corrupt 0, gen 0
> [ 1002.726728] btrfs: attempt to access beyond end of device
>                 sde: rw=14337, sector=131072, nr_sectors = 8 limit=0
> [ 1002.726733] BTRFS warning (device sdd): lost super block write due to
> IO error on /dev/sde (-5)
> [ 1002.726737] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0,
> flush 1, corrupt 0, gen 0
> [ 1002.726835] BTRFS error (device sdd): error writing primary super
> block to device 2
> [ 1002.726844] BTRFS info (device sdd): balance: start -dconvert=single
> -mconvert=dup -sconvert=dup
> [ 1002.726990] BTRFS info (device sdd): relocating block group
> 1372585984 flags data|raid1
> [ 1002.727811] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0,
> flush 2, corrupt 0, gen 0
> [ 1002.727816] BTRFS warning (device sdd): chunk 2446327808 missing 1
> devices, max tolerance is 0 for writable mount
> [ 1002.727818] BTRFS: error (device sdd) in write_all_supers:4044:
> errno=-5 IO failure (errors while submitting device barriers.)
> [ 1002.727821] BTRFS info (device sdd state E): forced readonly
> [ 1002.727823] BTRFS warning (device sdd state E): Skipping commit of
> aborted transaction.
> [ 1002.727824] BTRFS error (device sdd state EA): Transaction aborted
> (error -5)
> [ 1002.727826] BTRFS: error (device sdd state EA) in
> cleanup_transaction:2017: errno=-5 IO failure
> [ 1002.727838] BTRFS info (device sdd state EA): balance: ended with
> status: -5
>
> debian:~# umount /mnt
>
> debian:~# btrfs fi show
> warning, device 2 is missing
> Label: none  uuid: d4b36ef9-3518-43a6-bc68-a2b4df751896
>      Total devices 2 FS bytes used 1.00GiB
>      devid    1 size 8.00GiB used 2.26GiB path /dev/sdd
>      *** Some devices missing
>
> mount -odegraded /dev/sdd /mnt
>
> debian:~# dmesg | tail -15
> [ 1332.628959] BTRFS info (device sdd): first mount of filesystem
> d4b36ef9-3518-43a6-bc68-a2b4df751896
> [ 1332.628974] BTRFS info (device sdd): using crc32c (crc32c-intel)
> checksum algorithm
> [ 1332.628978] BTRFS info (device sdd): using free-space-tree
> [ 1332.630281] BTRFS warning (device sdd): devid 2 uuid
> 0afd8d45-96d6-4393-b7ce-e55abd4b668e is missing
> [ 1332.632964] BTRFS info (device sdd): balance: resume -
> dconvert=single,soft -mconvert=dup,soft -sconvert=dup,soft
> [ 1332.633041] BTRFS info (device sdd): relocating block group
> 1372585984 flags data|raid1
> [ 1332.638220] BTRFS info (device sdd): found 1 extents, stage: move
> data extents
> [ 1332.640073] BTRFS info (device sdd): found 1 extents, stage: update
> data pointers
> [ 1332.641350] BTRFS info (device sdd): relocating block group 298844160
> flags data|raid1
> [ 1333.991088] BTRFS info (device sdd): found 8 extents, stage: move
> data extents
> [ 1333.994336] BTRFS info (device sdd): found 8 extents, stage: update
> data pointers
> [ 1333.997220] BTRFS info (device sdd): relocating block group 30408704
> flags metadata|raid1
> [ 1333.998489] BTRFS info (device sdd): found 1 extents, stage: move
> data extents
> [ 1333.999617] BTRFS info (device sdd): relocating block group 22020096
> flags system|raid1
> [ 1334.000838] BTRFS info (device sdd): balance: ended with status: 0
>
> debian:/mnt# btrfs fi usage .
> Overall:
>      Device size:          16.00GiB
>      Device allocated:           3.00GiB
>      Device unallocated:          13.00GiB
>      Device missing:           8.00GiB
>      Device slack:             0.00B
>      Used:               1.00GiB
>      Free (estimated):          14.44GiB    (min: 7.94GiB)
>      Free (statfs, df):           6.44GiB
>      Data ratio:                  1.00
>      Metadata ratio:              2.00
>      Global reserve:           5.50MiB    (used: 0.00B)
>      Multiple profiles:                no
>
> Data,single: Size:2.44GiB, Used:1.00GiB (41.07%)
>     /dev/sdd       2.44GiB
>
> Metadata,DUP: Size:256.00MiB, Used:1.17MiB (0.46%)
>     /dev/sdd     512.00MiB
>
> System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%)
>     /dev/sdd      64.00MiB
>
> Unallocated:
>     /dev/sdd       5.00GiB
>     <missing disk>       8.00GiB
>
> debian:/mnt# btrfs device remove missing .
>
> debian:/mnt# dmesg | tail -1
> [ 1714.116147] BTRFS info (device sdd): device deleted: missing
>
>
>

Re: Converting RAID1 to single fails if RAID1 is missing device

[Colin S]

On 02/02/2025 20:32, Qu Wenruo wrote:
>
> But I think the ultimate solution is to make btrfs to properly detect
> and support device shut down request.

Yes. How many years and how many more users need to have this problem 
before it’s given some priority?

> Although that would also introduce new complexity, e.g. what if the
> missing devices show up again after missing several writes? 

Since I see you wrote a patch in 2022 to add write-intent bitmap for 
raid5/6, don’t you already understand the answer is a write-intent 
bitmap? Further, did you not see any of the several messages I have sent 
to the mailing list talking about exactly this in the last year? I am 
genuinely confused.

Re: Converting RAID1 to single fails if RAID1 is missing device

[Qu Wenruo]

在 2025/2/3 15:52, Colin S 写道:
> On 02/02/2025 20:32, Qu Wenruo wrote:
>>
>> But I think the ultimate solution is to make btrfs to properly detect
>> and support device shut down request.
>
> Yes. How many years and how many more users need to have this problem
> before it’s given some priority?

Complaining is so easy that some one doesn't even know what's going
wrong can do.

>
>> Although that would also introduce new complexity, e.g. what if the
>> missing devices show up again after missing several writes?
>
> Since I see you wrote a patch in 2022 to add write-intent bitmap for
> raid5/6, don’t you already understand the answer is a write-intent
> bitmap? Further, did you not see any of the several messages I have sent
> to the mailing list talking about exactly this in the last year? I am
> genuinely confused.
>


It's completely a different bug.

It doesn't even have RAID56 involved.

And write-intent bitmap is not the solution at all. It doesn't support
things like zoned device support (which is now part of the core btrfs
functionality).

Knowing what you don't know is important.

Re: Converting RAID1 to single fails if RAID1 is missing device

[Colin S]

On 03/02/2025 00:13, Qu Wenruo wrote:
>
>
> 在 2025/2/3 15:52, Colin S 写道:
>> On 02/02/2025 20:32, Qu Wenruo wrote:
>>>
>>> But I think the ultimate solution is to make btrfs to properly detect
>>> and support device shut down request.
>>
>> Yes. How many years and how many more users need to have this problem
>> before it’s given some priority?
>
> Complaining is so easy that some one doesn't even know what's going
> wrong can do.

Please understand that I am acting in good faith to try to make btrfs 
better. I do not want to complain, I want to solve this problem. I sent 
ideas to the list that I thought were sound several times and didn’t see 
any btrfs developer respond at all. Did I miss a reply? If so, I 
apologise for the missed connection on my end, and will look to the 
archive for such a message if you tell me so. Otherwise, I’d like it if 
you could answer my question, please, so I can understand what is 
missing to give this priority?

I don’t think this specific case where btrfs doesn’t notice the device 
has gone away because it doesn’t implement a callback can be separated 
from the larger problem because implementing that callback correctly 
(i.e. with no data loss, corruption, or risk of split-brain) depends on 
btrfs tracking writes better than it does now. If this seems confusing 
or wrong, let me know so I can try to understand and say more.

>>
>>> Although that would also introduce new complexity, e.g. what if the
>>> missing devices show up again after missing several writes?
>>
>> Since I see you wrote a patch in 2022 to add write-intent bitmap for
>> raid5/6, don’t you already understand the answer is a write-intent
>> bitmap? Further, did you not see any of the several messages I have sent
>> to the mailing list talking about exactly this in the last year? I am
>> genuinely confused.
>>
>
>
> It's completely a different bug.
>
> It doesn't even have RAID56 involved.

Are you saying this because the abandoned implementation relied on 
raid5/6 stripes? A write-intent bitmap, as a general concept, doesn’t 
require parity as far as I know, so I don’t understand this statement 
right now.

> And write-intent bitmap is not the solution at all. It doesn't support
> things like zoned device support (which is now part of the core btrfs
> functionality).

How so?

> Knowing what you don't know is important.

I agree. Knowing what I don’t know is the explicit reason I gave in the 
past about why I wouldn’t even attempt to implement and send any patch 
on my own. I don’t mind being wrong since it means I get to learn, but I 
can’t learn a thing from a browbeating.

Best,