* Converting RAID1 to single fails if RAID1 is missing device @ 2025-02-02 22:43 Jeff Siddall 2025-02-03 0:44 ` Qu Wenruo 0 siblings, 1 reply; 8+ messages in thread From: Jeff Siddall @ 2025-02-02 22:43 UTC (permalink / raw) To: linux-btrfs After a device failed on RAID1 filesystem, an attempt to convert the online filesystem from RAID1 to single failed. This isn't an uncommon use case if the failed device isn't readily replaceable. The command ran was: btrfs balance start -f -sconvert=single -mconvert=single -dconvert=single /mountpoint and the kernel logs were: kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single -mconvert=dup -sconvert=dup kernel: BTRFS info (device nvme0n1p3): relocating block group 1222049267712 flags data|raid1 kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1 devices, max tolerance is 0 for writable mount kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370: errno=-5 IO failure (errors while submitting device barriers.) kernel: BTRFS info (device nvme0n1p3: state E): forced readonly kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of aborted transaction. kernel: BTRFS: error (device nvme0n1p3: state EA) in cleanup_transaction:1992: errno=-5 IO failure kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with status: -5 Either it should be made possible to convert a RAID1 device with a missing device to a single device filesystem without errors, or the command should return a message stating that it is not supported to convert RAID1 array with missing devices to a single. Having the process fail and then going forced readonly is a significant failure on an otherwise working system. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Converting RAID1 to single fails if RAID1 is missing device 2025-02-02 22:43 Converting RAID1 to single fails if RAID1 is missing device Jeff Siddall @ 2025-02-03 0:44 ` Qu Wenruo 2025-02-03 1:11 ` Jeff Siddall 2025-02-03 2:19 ` Jussi Kansanen 0 siblings, 2 replies; 8+ messages in thread From: Qu Wenruo @ 2025-02-03 0:44 UTC (permalink / raw) To: Jeff Siddall, linux-btrfs 在 2025/2/3 09:13, Jeff Siddall 写道: > After a device failed on RAID1 filesystem, an attempt to convert the > online filesystem from RAID1 to single failed. This isn't an uncommon > use case if the failed device isn't readily replaceable. > > The command ran was: > > btrfs balance start -f -sconvert=single -mconvert=single - > dconvert=single /mountpoint > > and the kernel logs were: > > kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single - > mconvert=dup -sconvert=dup > kernel: BTRFS info (device nvme0n1p3): relocating block group > 1222049267712 flags data|raid1 > kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1 > devices, max tolerance is 0 for writable mount This is not the chunk to be relocated. Considering the tolerance is only 0, meaning it's the newly created single chunk. I tried locally, but failed to reproduce the same problem. Mind to provide the following info: - Kernel version - Btrfs fi usage output - Mount option of the fs My major concern is, the failing devices is still considered online, but will fail all read/write/flush commands. (Btrfs only has read time repair, not failing device detection) In that case, converting to single is the worst thing you can do, as it will write metadata chunks into that failing devices, and lost everything. The proper solution is to unmount the fs (if possible), remove the failing device, then mount the fs in degraded mode, add replace the missing device with a newer one. Thanks, Qu > kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370: > errno=-5 IO failure (errors while submitting device barriers.) > kernel: BTRFS info (device nvme0n1p3: state E): forced readonly > kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of > aborted transaction. > kernel: BTRFS: error (device nvme0n1p3: state EA) in > cleanup_transaction:1992: errno=-5 IO failure > kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with > status: -5 > > Either it should be made possible to convert a RAID1 device with a > missing device to a single device filesystem without errors, or the > command should return a message stating that it is not supported to > convert RAID1 array with missing devices to a single. Having the > process fail and then going forced readonly is a significant failure on > an otherwise working system. > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Converting RAID1 to single fails if RAID1 is missing device 2025-02-03 0:44 ` Qu Wenruo @ 2025-02-03 1:11 ` Jeff Siddall 2025-02-03 2:19 ` Jussi Kansanen 1 sibling, 0 replies; 8+ messages in thread From: Jeff Siddall @ 2025-02-03 1:11 UTC (permalink / raw) To: Qu Wenruo, linux-btrfs To be clear, when I say "device failed" I mean it disappeared entirely so from btrfs perspective it was missing. Kernel string is: 6.1.0-1025-rockchip #25-Ubuntu SMP Mon Aug 26 23:01:14 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux I have since rebuilt the array so fi usage doesn't contain anything relevant to this issue any more. Mount options are defaults. I was chatting with @multicore over on libera.chat/#btrfs who said they were able to reproduce. I can see if I can find out how they did if you are still unable to see this for yourself. Thanks, Jeff On 2025-02-02 19:44, Qu Wenruo wrote: > > > 在 2025/2/3 09:13, Jeff Siddall 写道: >> After a device failed on RAID1 filesystem, an attempt to convert the >> online filesystem from RAID1 to single failed. This isn't an uncommon >> use case if the failed device isn't readily replaceable. >> >> The command ran was: >> >> btrfs balance start -f -sconvert=single -mconvert=single - >> dconvert=single /mountpoint >> >> and the kernel logs were: >> >> kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single - >> mconvert=dup -sconvert=dup >> kernel: BTRFS info (device nvme0n1p3): relocating block group >> 1222049267712 flags data|raid1 >> kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1 >> devices, max tolerance is 0 for writable mount > > This is not the chunk to be relocated. Considering the tolerance is only > 0, meaning it's the newly created single chunk. > > I tried locally, but failed to reproduce the same problem. > > Mind to provide the following info: > > - Kernel version > - Btrfs fi usage output > - Mount option of the fs > > My major concern is, the failing devices is still considered online, but > will fail all read/write/flush commands. > (Btrfs only has read time repair, not failing device detection) > > In that case, converting to single is the worst thing you can do, as it > will write metadata chunks into that failing devices, and lost > everything. > > The proper solution is to unmount the fs (if possible), remove the > failing device, then mount the fs in degraded mode, add replace the > missing device with a newer one. > > Thanks, > Qu > >> kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370: >> errno=-5 IO failure (errors while submitting device barriers.) >> kernel: BTRFS info (device nvme0n1p3: state E): forced readonly >> kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of >> aborted transaction. >> kernel: BTRFS: error (device nvme0n1p3: state EA) in >> cleanup_transaction:1992: errno=-5 IO failure >> kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with >> status: -5 >> >> Either it should be made possible to convert a RAID1 device with a >> missing device to a single device filesystem without errors, or the >> command should return a message stating that it is not supported to >> convert RAID1 array with missing devices to a single. Having the >> process fail and then going forced readonly is a significant failure on >> an otherwise working system. >> >> >> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Converting RAID1 to single fails if RAID1 is missing device 2025-02-03 0:44 ` Qu Wenruo 2025-02-03 1:11 ` Jeff Siddall @ 2025-02-03 2:19 ` Jussi Kansanen 2025-02-03 2:32 ` Qu Wenruo 1 sibling, 1 reply; 8+ messages in thread From: Jussi Kansanen @ 2025-02-03 2:19 UTC (permalink / raw) To: Qu Wenruo, Jeff Siddall, linux-btrfs On 2/3/25 02:44, Qu Wenruo wrote: > > > 在 2025/2/3 09:13, Jeff Siddall 写道: >> After a device failed on RAID1 filesystem, an attempt to convert the >> online filesystem from RAID1 to single failed. This isn't an uncommon >> use case if the failed device isn't readily replaceable. >> >> The command ran was: >> >> btrfs balance start -f -sconvert=single -mconvert=single - >> dconvert=single /mountpoint >> >> and the kernel logs were: >> >> kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single - >> mconvert=dup -sconvert=dup >> kernel: BTRFS info (device nvme0n1p3): relocating block group >> 1222049267712 flags data|raid1 >> kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1 >> devices, max tolerance is 0 for writable mount > > This is not the chunk to be relocated. Considering the tolerance is only > 0, meaning it's the newly created single chunk. > > I tried locally, but failed to reproduce the same problem. > > Mind to provide the following info: > > - Kernel version > - Btrfs fi usage output > - Mount option of the fs > > My major concern is, the failing devices is still considered online, but > will fail all read/write/flush commands. > (Btrfs only has read time repair, not failing device detection) > > In that case, converting to single is the worst thing you can do, as it > will write metadata chunks into that failing devices, and lost everything. > > The proper solution is to unmount the fs (if possible), remove the > failing device, then mount the fs in degraded mode, add replace the > missing device with a newer one. > > Thanks, > Qu > >> kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370: >> errno=-5 IO failure (errors while submitting device barriers.) >> kernel: BTRFS info (device nvme0n1p3: state E): forced readonly >> kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of >> aborted transaction. >> kernel: BTRFS: error (device nvme0n1p3: state EA) in >> cleanup_transaction:1992: errno=-5 IO failure >> kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with >> status: -5 >> >> Either it should be made possible to convert a RAID1 device with a >> missing device to a single device filesystem without errors, or the >> command should return a message stating that it is not supported to >> convert RAID1 array with missing devices to a single. Having the >> process fail and then going forced readonly is a significant failure on >> an otherwise working system. >> >> >> > > Hi, here's a reproducer for similar issue that Jeff had: debian:/mnt# uname -a Linux debian 6.12.11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.11-1 (2025-01-25) x86_64 GNU/Linux debian:/mnt# findmnt . TARGET SOURCE FSTYPE OPTIONS /mnt /dev/sdd btrfs rw,relatime,space_cache=v2,subvolid=5,subvol=/ debian:/mnt# btrfs fi usage . Overall: Device size: 16.00GiB Device allocated: 4.52GiB Device unallocated: 11.48GiB Device missing: 0.00B Device slack: 0.00B Used: 2.00GiB Free (estimated): 6.74GiB (min: 6.74GiB) Free (statfs, df): 6.74GiB Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 5.50MiB (used: 0.00B) Multiple profiles: no Data,RAID1: Size:2.00GiB, Used:1.00GiB (50.05%) /dev/sdd 2.00GiB /dev/sde 2.00GiB Metadata,RAID1: Size:256.00MiB, Used:1.14MiB (0.45%) /dev/sdd 256.00MiB /dev/sde 256.00MiB System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%) /dev/sdd 8.00MiB /dev/sde 8.00MiB Unallocated: /dev/sdd 5.74GiB /dev/sde 5.74GiB debian:/mnt# echo 1 > /sys/block/sde/device/delete debian:/mnt# btrfs balance start -mconvert=dup -dconvert=single . ERROR: error during balancing '.': Input/output error There may be more info in syslog - try dmesg | tail debian:/mnt# dmesg | tail -35 [ 582.117314] BTRFS info (device sdd): first mount of filesystem d4b36ef9-3518-43a6-bc68-a2b4df751896 [ 582.117329] BTRFS info (device sdd): using crc32c (crc32c-intel) checksum algorithm [ 582.117333] BTRFS info (device sdd): using free-space-tree [ 582.119952] BTRFS info (device sdd): checking UUID tree [ 979.011795] sd 6:0:0:0: [sde] Synchronizing SCSI cache [ 979.013096] ata7.00: Entering standby power mode [ 1002.726249] btrfs: attempt to access beyond end of device sde: rw=6145, sector=21696, nr_sectors = 32 limit=0 [ 1002.726274] btrfs: attempt to access beyond end of device sde: rw=6145, sector=21728, nr_sectors = 32 limit=0 [ 1002.726281] btrfs: attempt to access beyond end of device sde: rw=6145, sector=21760, nr_sectors = 32 limit=0 [ 1002.726442] BTRFS error (device sdd): bdev /dev/sde errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 [ 1002.726501] BTRFS error (device sdd): bdev /dev/sde errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 [ 1002.726534] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0, flush 0, corrupt 0, gen 0 [ 1002.726659] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0, flush 1, corrupt 0, gen 0 [ 1002.726692] btrfs: attempt to access beyond end of device sde: rw=145409, sector=128, nr_sectors = 8 limit=0 [ 1002.726702] BTRFS warning (device sdd): lost super block write due to IO error on /dev/sde (-5) [ 1002.726704] BTRFS error (device sdd): bdev /dev/sde errs: wr 4, rd 0, flush 1, corrupt 0, gen 0 [ 1002.726728] btrfs: attempt to access beyond end of device sde: rw=14337, sector=131072, nr_sectors = 8 limit=0 [ 1002.726733] BTRFS warning (device sdd): lost super block write due to IO error on /dev/sde (-5) [ 1002.726737] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0, flush 1, corrupt 0, gen 0 [ 1002.726835] BTRFS error (device sdd): error writing primary super block to device 2 [ 1002.726844] BTRFS info (device sdd): balance: start -dconvert=single -mconvert=dup -sconvert=dup [ 1002.726990] BTRFS info (device sdd): relocating block group 1372585984 flags data|raid1 [ 1002.727811] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0, flush 2, corrupt 0, gen 0 [ 1002.727816] BTRFS warning (device sdd): chunk 2446327808 missing 1 devices, max tolerance is 0 for writable mount [ 1002.727818] BTRFS: error (device sdd) in write_all_supers:4044: errno=-5 IO failure (errors while submitting device barriers.) [ 1002.727821] BTRFS info (device sdd state E): forced readonly [ 1002.727823] BTRFS warning (device sdd state E): Skipping commit of aborted transaction. [ 1002.727824] BTRFS error (device sdd state EA): Transaction aborted (error -5) [ 1002.727826] BTRFS: error (device sdd state EA) in cleanup_transaction:2017: errno=-5 IO failure [ 1002.727838] BTRFS info (device sdd state EA): balance: ended with status: -5 debian:~# umount /mnt debian:~# btrfs fi show warning, device 2 is missing Label: none uuid: d4b36ef9-3518-43a6-bc68-a2b4df751896 Total devices 2 FS bytes used 1.00GiB devid 1 size 8.00GiB used 2.26GiB path /dev/sdd *** Some devices missing mount -odegraded /dev/sdd /mnt debian:~# dmesg | tail -15 [ 1332.628959] BTRFS info (device sdd): first mount of filesystem d4b36ef9-3518-43a6-bc68-a2b4df751896 [ 1332.628974] BTRFS info (device sdd): using crc32c (crc32c-intel) checksum algorithm [ 1332.628978] BTRFS info (device sdd): using free-space-tree [ 1332.630281] BTRFS warning (device sdd): devid 2 uuid 0afd8d45-96d6-4393-b7ce-e55abd4b668e is missing [ 1332.632964] BTRFS info (device sdd): balance: resume -dconvert=single,soft -mconvert=dup,soft -sconvert=dup,soft [ 1332.633041] BTRFS info (device sdd): relocating block group 1372585984 flags data|raid1 [ 1332.638220] BTRFS info (device sdd): found 1 extents, stage: move data extents [ 1332.640073] BTRFS info (device sdd): found 1 extents, stage: update data pointers [ 1332.641350] BTRFS info (device sdd): relocating block group 298844160 flags data|raid1 [ 1333.991088] BTRFS info (device sdd): found 8 extents, stage: move data extents [ 1333.994336] BTRFS info (device sdd): found 8 extents, stage: update data pointers [ 1333.997220] BTRFS info (device sdd): relocating block group 30408704 flags metadata|raid1 [ 1333.998489] BTRFS info (device sdd): found 1 extents, stage: move data extents [ 1333.999617] BTRFS info (device sdd): relocating block group 22020096 flags system|raid1 [ 1334.000838] BTRFS info (device sdd): balance: ended with status: 0 debian:/mnt# btrfs fi usage . Overall: Device size: 16.00GiB Device allocated: 3.00GiB Device unallocated: 13.00GiB Device missing: 8.00GiB Device slack: 0.00B Used: 1.00GiB Free (estimated): 14.44GiB (min: 7.94GiB) Free (statfs, df): 6.44GiB Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 5.50MiB (used: 0.00B) Multiple profiles: no Data,single: Size:2.44GiB, Used:1.00GiB (41.07%) /dev/sdd 2.44GiB Metadata,DUP: Size:256.00MiB, Used:1.17MiB (0.46%) /dev/sdd 512.00MiB System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%) /dev/sdd 64.00MiB Unallocated: /dev/sdd 5.00GiB <missing disk> 8.00GiB debian:/mnt# btrfs device remove missing . debian:/mnt# dmesg | tail -1 [ 1714.116147] BTRFS info (device sdd): device deleted: missing ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Converting RAID1 to single fails if RAID1 is missing device 2025-02-03 2:19 ` Jussi Kansanen @ 2025-02-03 2:32 ` Qu Wenruo 2025-02-03 5:22 ` Colin S 0 siblings, 1 reply; 8+ messages in thread From: Qu Wenruo @ 2025-02-03 2:32 UTC (permalink / raw) To: Jussi Kansanen, Jeff Siddall, linux-btrfs 在 2025/2/3 12:49, Jussi Kansanen 写道: > On 2/3/25 02:44, Qu Wenruo wrote: >> >> >> 在 2025/2/3 09:13, Jeff Siddall 写道: >>> After a device failed on RAID1 filesystem, an attempt to convert the >>> online filesystem from RAID1 to single failed. This isn't an uncommon >>> use case if the failed device isn't readily replaceable. >>> >>> The command ran was: >>> >>> btrfs balance start -f -sconvert=single -mconvert=single - >>> dconvert=single /mountpoint >>> >>> and the kernel logs were: >>> >>> kernel: BTRFS info (device nvme0n1p3): balance: start -dconvert=single - >>> mconvert=dup -sconvert=dup >>> kernel: BTRFS info (device nvme0n1p3): relocating block group >>> 1222049267712 flags data|raid1 >>> kernel: BTRFS warning (device nvme0n1p3): chunk 1223123009536 missing 1 >>> devices, max tolerance is 0 for writable mount >> >> This is not the chunk to be relocated. Considering the tolerance is only >> 0, meaning it's the newly created single chunk. >> >> I tried locally, but failed to reproduce the same problem. >> >> Mind to provide the following info: >> >> - Kernel version >> - Btrfs fi usage output >> - Mount option of the fs >> >> My major concern is, the failing devices is still considered online, but >> will fail all read/write/flush commands. >> (Btrfs only has read time repair, not failing device detection) >> >> In that case, converting to single is the worst thing you can do, as it >> will write metadata chunks into that failing devices, and lost >> everything. >> >> The proper solution is to unmount the fs (if possible), remove the >> failing device, then mount the fs in degraded mode, add replace the >> missing device with a newer one. >> >> Thanks, >> Qu >> >>> kernel: BTRFS: error (device nvme0n1p3) in write_all_supers:4370: >>> errno=-5 IO failure (errors while submitting device barriers.) >>> kernel: BTRFS info (device nvme0n1p3: state E): forced readonly >>> kernel: BTRFS warning (device nvme0n1p3: state E): Skipping commit of >>> aborted transaction. >>> kernel: BTRFS: error (device nvme0n1p3: state EA) in >>> cleanup_transaction:1992: errno=-5 IO failure >>> kernel: BTRFS info (device nvme0n1p3: state EA): balance: ended with >>> status: -5 >>> >>> Either it should be made possible to convert a RAID1 device with a >>> missing device to a single device filesystem without errors, or the >>> command should return a message stating that it is not supported to >>> convert RAID1 array with missing devices to a single. Having the >>> process fail and then going forced readonly is a significant failure on >>> an otherwise working system. >>> >>> >>> >> >> > > Hi, here's a reproducer for similar issue that Jeff had: > > debian:/mnt# uname -a > Linux debian 6.12.11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.11-1 > (2025-01-25) x86_64 GNU/Linux > > debian:/mnt# findmnt . > TARGET SOURCE FSTYPE OPTIONS > /mnt /dev/sdd btrfs rw,relatime,space_cache=v2,subvolid=5,subvol=/ > > debian:/mnt# btrfs fi usage . > Overall: > Device size: 16.00GiB > Device allocated: 4.52GiB > Device unallocated: 11.48GiB > Device missing: 0.00B > Device slack: 0.00B > Used: 2.00GiB > Free (estimated): 6.74GiB (min: 6.74GiB) > Free (statfs, df): 6.74GiB > Data ratio: 2.00 > Metadata ratio: 2.00 > Global reserve: 5.50MiB (used: 0.00B) > Multiple profiles: no > > Data,RAID1: Size:2.00GiB, Used:1.00GiB (50.05%) > /dev/sdd 2.00GiB > /dev/sde 2.00GiB > > Metadata,RAID1: Size:256.00MiB, Used:1.14MiB (0.45%) > /dev/sdd 256.00MiB > /dev/sde 256.00MiB > > System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%) > /dev/sdd 8.00MiB > /dev/sde 8.00MiB > > Unallocated: > /dev/sdd 5.74GiB > /dev/sde 5.74GiB > > > debian:/mnt# echo 1 > /sys/block/sde/device/delete > > debian:/mnt# btrfs balance start -mconvert=dup -dconvert=single . > ERROR: error during balancing '.': Input/output error > There may be more info in syslog - try dmesg | tail > > debian:/mnt# dmesg | tail -35 > [ 582.117314] BTRFS info (device sdd): first mount of filesystem > d4b36ef9-3518-43a6-bc68-a2b4df751896 > [ 582.117329] BTRFS info (device sdd): using crc32c (crc32c-intel) > checksum algorithm > [ 582.117333] BTRFS info (device sdd): using free-space-tree > [ 582.119952] BTRFS info (device sdd): checking UUID tree > [ 979.011795] sd 6:0:0:0: [sde] Synchronizing SCSI cache > [ 979.013096] ata7.00: Entering standby power mode > [ 1002.726249] btrfs: attempt to access beyond end of device > sde: rw=6145, sector=21696, nr_sectors = 32 limit=0 As you can see, btrfs is still trying to access the removed device, thus still the worst case scenario. IIRC the sde deletion using sysfs interface is exactly what test case generic/730 is doing (but with single devices fs), and unfortunately btrfs doesn't support proper device shutdown callback, thus using sysfs delete is just leading to the worst situation. I can enhance the document related to convert, so that one should not try to convert if there is any unreliable device. But I think the ultimate solution is to make btrfs to properly detect and support device shut down request. Although that would also introduce new complexity, e.g. what if the missing devices show up again after missing several writes? Thanks, Qu > [ 1002.726274] btrfs: attempt to access beyond end of device > sde: rw=6145, sector=21728, nr_sectors = 32 limit=0 > [ 1002.726281] btrfs: attempt to access beyond end of device > sde: rw=6145, sector=21760, nr_sectors = 32 limit=0 > [ 1002.726442] BTRFS error (device sdd): bdev /dev/sde errs: wr 1, rd 0, > flush 0, corrupt 0, gen 0 > [ 1002.726501] BTRFS error (device sdd): bdev /dev/sde errs: wr 2, rd 0, > flush 0, corrupt 0, gen 0 > [ 1002.726534] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0, > flush 0, corrupt 0, gen 0 > [ 1002.726659] BTRFS error (device sdd): bdev /dev/sde errs: wr 3, rd 0, > flush 1, corrupt 0, gen 0 > [ 1002.726692] btrfs: attempt to access beyond end of device > sde: rw=145409, sector=128, nr_sectors = 8 limit=0 > [ 1002.726702] BTRFS warning (device sdd): lost super block write due to > IO error on /dev/sde (-5) > [ 1002.726704] BTRFS error (device sdd): bdev /dev/sde errs: wr 4, rd 0, > flush 1, corrupt 0, gen 0 > [ 1002.726728] btrfs: attempt to access beyond end of device > sde: rw=14337, sector=131072, nr_sectors = 8 limit=0 > [ 1002.726733] BTRFS warning (device sdd): lost super block write due to > IO error on /dev/sde (-5) > [ 1002.726737] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0, > flush 1, corrupt 0, gen 0 > [ 1002.726835] BTRFS error (device sdd): error writing primary super > block to device 2 > [ 1002.726844] BTRFS info (device sdd): balance: start -dconvert=single > -mconvert=dup -sconvert=dup > [ 1002.726990] BTRFS info (device sdd): relocating block group > 1372585984 flags data|raid1 > [ 1002.727811] BTRFS error (device sdd): bdev /dev/sde errs: wr 5, rd 0, > flush 2, corrupt 0, gen 0 > [ 1002.727816] BTRFS warning (device sdd): chunk 2446327808 missing 1 > devices, max tolerance is 0 for writable mount > [ 1002.727818] BTRFS: error (device sdd) in write_all_supers:4044: > errno=-5 IO failure (errors while submitting device barriers.) > [ 1002.727821] BTRFS info (device sdd state E): forced readonly > [ 1002.727823] BTRFS warning (device sdd state E): Skipping commit of > aborted transaction. > [ 1002.727824] BTRFS error (device sdd state EA): Transaction aborted > (error -5) > [ 1002.727826] BTRFS: error (device sdd state EA) in > cleanup_transaction:2017: errno=-5 IO failure > [ 1002.727838] BTRFS info (device sdd state EA): balance: ended with > status: -5 > > debian:~# umount /mnt > > debian:~# btrfs fi show > warning, device 2 is missing > Label: none uuid: d4b36ef9-3518-43a6-bc68-a2b4df751896 > Total devices 2 FS bytes used 1.00GiB > devid 1 size 8.00GiB used 2.26GiB path /dev/sdd > *** Some devices missing > > mount -odegraded /dev/sdd /mnt > > debian:~# dmesg | tail -15 > [ 1332.628959] BTRFS info (device sdd): first mount of filesystem > d4b36ef9-3518-43a6-bc68-a2b4df751896 > [ 1332.628974] BTRFS info (device sdd): using crc32c (crc32c-intel) > checksum algorithm > [ 1332.628978] BTRFS info (device sdd): using free-space-tree > [ 1332.630281] BTRFS warning (device sdd): devid 2 uuid > 0afd8d45-96d6-4393-b7ce-e55abd4b668e is missing > [ 1332.632964] BTRFS info (device sdd): balance: resume - > dconvert=single,soft -mconvert=dup,soft -sconvert=dup,soft > [ 1332.633041] BTRFS info (device sdd): relocating block group > 1372585984 flags data|raid1 > [ 1332.638220] BTRFS info (device sdd): found 1 extents, stage: move > data extents > [ 1332.640073] BTRFS info (device sdd): found 1 extents, stage: update > data pointers > [ 1332.641350] BTRFS info (device sdd): relocating block group 298844160 > flags data|raid1 > [ 1333.991088] BTRFS info (device sdd): found 8 extents, stage: move > data extents > [ 1333.994336] BTRFS info (device sdd): found 8 extents, stage: update > data pointers > [ 1333.997220] BTRFS info (device sdd): relocating block group 30408704 > flags metadata|raid1 > [ 1333.998489] BTRFS info (device sdd): found 1 extents, stage: move > data extents > [ 1333.999617] BTRFS info (device sdd): relocating block group 22020096 > flags system|raid1 > [ 1334.000838] BTRFS info (device sdd): balance: ended with status: 0 > > debian:/mnt# btrfs fi usage . > Overall: > Device size: 16.00GiB > Device allocated: 3.00GiB > Device unallocated: 13.00GiB > Device missing: 8.00GiB > Device slack: 0.00B > Used: 1.00GiB > Free (estimated): 14.44GiB (min: 7.94GiB) > Free (statfs, df): 6.44GiB > Data ratio: 1.00 > Metadata ratio: 2.00 > Global reserve: 5.50MiB (used: 0.00B) > Multiple profiles: no > > Data,single: Size:2.44GiB, Used:1.00GiB (41.07%) > /dev/sdd 2.44GiB > > Metadata,DUP: Size:256.00MiB, Used:1.17MiB (0.46%) > /dev/sdd 512.00MiB > > System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%) > /dev/sdd 64.00MiB > > Unallocated: > /dev/sdd 5.00GiB > <missing disk> 8.00GiB > > debian:/mnt# btrfs device remove missing . > > debian:/mnt# dmesg | tail -1 > [ 1714.116147] BTRFS info (device sdd): device deleted: missing > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Converting RAID1 to single fails if RAID1 is missing device 2025-02-03 2:32 ` Qu Wenruo @ 2025-02-03 5:22 ` Colin S 2025-02-03 6:13 ` Qu Wenruo 0 siblings, 1 reply; 8+ messages in thread From: Colin S @ 2025-02-03 5:22 UTC (permalink / raw) To: linux-btrfs On 02/02/2025 20:32, Qu Wenruo wrote: > > But I think the ultimate solution is to make btrfs to properly detect > and support device shut down request. Yes. How many years and how many more users need to have this problem before it’s given some priority? > Although that would also introduce new complexity, e.g. what if the > missing devices show up again after missing several writes? Since I see you wrote a patch in 2022 to add write-intent bitmap for raid5/6, don’t you already understand the answer is a write-intent bitmap? Further, did you not see any of the several messages I have sent to the mailing list talking about exactly this in the last year? I am genuinely confused. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Converting RAID1 to single fails if RAID1 is missing device 2025-02-03 5:22 ` Colin S @ 2025-02-03 6:13 ` Qu Wenruo 2025-02-04 7:04 ` Colin S 0 siblings, 1 reply; 8+ messages in thread From: Qu Wenruo @ 2025-02-03 6:13 UTC (permalink / raw) To: Colin S, linux-btrfs 在 2025/2/3 15:52, Colin S 写道: > On 02/02/2025 20:32, Qu Wenruo wrote: >> >> But I think the ultimate solution is to make btrfs to properly detect >> and support device shut down request. > > Yes. How many years and how many more users need to have this problem > before it’s given some priority? Complaining is so easy that some one doesn't even know what's going wrong can do. > >> Although that would also introduce new complexity, e.g. what if the >> missing devices show up again after missing several writes? > > Since I see you wrote a patch in 2022 to add write-intent bitmap for > raid5/6, don’t you already understand the answer is a write-intent > bitmap? Further, did you not see any of the several messages I have sent > to the mailing list talking about exactly this in the last year? I am > genuinely confused. > It's completely a different bug. It doesn't even have RAID56 involved. And write-intent bitmap is not the solution at all. It doesn't support things like zoned device support (which is now part of the core btrfs functionality). Knowing what you don't know is important. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Converting RAID1 to single fails if RAID1 is missing device 2025-02-03 6:13 ` Qu Wenruo @ 2025-02-04 7:04 ` Colin S 0 siblings, 0 replies; 8+ messages in thread From: Colin S @ 2025-02-04 7:04 UTC (permalink / raw) To: linux-btrfs On 03/02/2025 00:13, Qu Wenruo wrote: > > > 在 2025/2/3 15:52, Colin S 写道: >> On 02/02/2025 20:32, Qu Wenruo wrote: >>> >>> But I think the ultimate solution is to make btrfs to properly detect >>> and support device shut down request. >> >> Yes. How many years and how many more users need to have this problem >> before it’s given some priority? > > Complaining is so easy that some one doesn't even know what's going > wrong can do. Please understand that I am acting in good faith to try to make btrfs better. I do not want to complain, I want to solve this problem. I sent ideas to the list that I thought were sound several times and didn’t see any btrfs developer respond at all. Did I miss a reply? If so, I apologise for the missed connection on my end, and will look to the archive for such a message if you tell me so. Otherwise, I’d like it if you could answer my question, please, so I can understand what is missing to give this priority? I don’t think this specific case where btrfs doesn’t notice the device has gone away because it doesn’t implement a callback can be separated from the larger problem because implementing that callback correctly (i.e. with no data loss, corruption, or risk of split-brain) depends on btrfs tracking writes better than it does now. If this seems confusing or wrong, let me know so I can try to understand and say more. >> >>> Although that would also introduce new complexity, e.g. what if the >>> missing devices show up again after missing several writes? >> >> Since I see you wrote a patch in 2022 to add write-intent bitmap for >> raid5/6, don’t you already understand the answer is a write-intent >> bitmap? Further, did you not see any of the several messages I have sent >> to the mailing list talking about exactly this in the last year? I am >> genuinely confused. >> > > > It's completely a different bug. > > It doesn't even have RAID56 involved. Are you saying this because the abandoned implementation relied on raid5/6 stripes? A write-intent bitmap, as a general concept, doesn’t require parity as far as I know, so I don’t understand this statement right now. > And write-intent bitmap is not the solution at all. It doesn't support > things like zoned device support (which is now part of the core btrfs > functionality). How so? > Knowing what you don't know is important. I agree. Knowing what I don’t know is the explicit reason I gave in the past about why I wouldn’t even attempt to implement and send any patch on my own. I don’t mind being wrong since it means I get to learn, but I can’t learn a thing from a browbeating. Best, ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2025-02-04 7:04 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-02-02 22:43 Converting RAID1 to single fails if RAID1 is missing device Jeff Siddall 2025-02-03 0:44 ` Qu Wenruo 2025-02-03 1:11 ` Jeff Siddall 2025-02-03 2:19 ` Jussi Kansanen 2025-02-03 2:32 ` Qu Wenruo 2025-02-03 5:22 ` Colin S 2025-02-03 6:13 ` Qu Wenruo 2025-02-04 7:04 ` Colin S
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.