Error: Failed to read chunk root, fs destroyed

Error: Failed to read chunk root, fs destroyed

[Davide Palma]

Hello,
What happened:

I had a Gentoo linux install on a standard, ssd, btrfs partition.
After I removed lots of files, I wanted to shrink the partition, so i
shrunk it as much as possible and started a balance operation to
squeeze out more unallocated space. However, the balance operation was
not progressing. (the behaviour was identical to the one described
here https://www.opensuse-forum.de/thread/43376-btrfs-balance-caught-in-write-loop/
). So I cancelled the balance and, following a piece of advice found
on the net, created a ramdisk that worked as a second device for a
raid1 setup.

dd if=/dev/zero of=/var/tmp/btrfs bs=1G count=5

losetup -v -f /var/tmp/btrfs
btrfs device add /dev/loop0 .

I tried running the balance again, but, as before, it wasn't
progressing. So I decided to remove the ramdisk and reconvert the now
raid1 partition to a single one. This is where the bad stuff began: As
the process of converting raid1 to single is a balance, the balance
looped and didn't progress. I also couldn't cancel it by ^C or SIGTERM
or SIGKILL. Since I didn't know what to do, I just rebooted the PC
(very bad decision!). The filesystem is now broken.
Please note I never tried dangerous options like zero-log or --repair.

I setted up an arch linux install on /dev/sda4 to keep on working and
to try recovery.

My environment:
Gentoo linux: kernel 5.7.8, btrfs-progs 5.7

Arch linux:
uname -a
Linux lenuovo 5.7.9-arch1-1 #1 SMP PREEMPT Thu, 16 Jul 2020 19:34:49
+0000 x86_64 GNU/Linux

btrfs --version
btrfs-progs v5.7

fdisk -l
Disk /dev/sda: 223,58 GiB, 240057409536 bytes, 468862128 sectors
Disk model: KINGSTON SUV400S
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 472C32BA-DDF3-4063-BBF7-629C7A68C195

Dispositivo     Start      Fine   Settori   Size Tipo
/dev/sda1        2048   8194047   8192000   3,9G Linux swap
/dev/sda2     8194048   8521727    327680   160M EFI System
/dev/sda3     8521728 302680063 294158336 140,3G Linux filesystem
/dev/sda4   302680064 345087999  42407936  20,2G Linux filesystem
/dev/sda5   345088000 468860927 123772928    59G Microsoft basic data

The broken FS is /dev/sda3.

Any combination of mount options without degraded fails with:
BTRFS info (device sda3): disk space caching is enabled
BTRFS info (device sda3): has skinny extents
BTRFS error (device sda3): devid 2 uuid
bc7154d2-bf0e-4ff4-b7cb-44e118148976 is missing
BTRFS error (device sda3): failed to read the system array: -2
BTRFS error (device sda3): open_ctree failed

Any combination of mount options with degraded fails with:
BTRFS info (device sda3): allowing degraded mounts
BTRFS info (device sda3): disk space caching is enabled
BTRFS info (device sda3): has skinny extents
BTRFS warning (device sda3): devid 2 uuid
bc7154d2-bf0e-4ff4-b7cb-44e118148976 is missing
BTRFS error (device sda3): failed to read chunk root
BTRFS error (device sda3): open_ctree failed

This is also the error of every btrfs prog that tries to access the volume:
btrfs check /dev/sda3
Opening filesystem to check...
warning, device 2 is missing
bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
ERROR: cannot read chunk root
ERROR: cannot open file system

btrfs restore /dev/sda3 -i -l
warning, device 2 is missing
bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
ERROR: cannot read chunk root
Could not open root, trying backup super
warning, device 2 is missing
bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
ERROR: cannot read chunk root
Could not open root, trying backup super
ERROR: superblock bytenr 274877906944 is larger than device size 150609068032
Could not open root, trying backup super

btrfs rescue super-recover /dev/sda3
All supers are valid, no need to recover

btrfs inspect-internal dump-super -f /dev/sda3
superblock: bytenr=65536, device=/dev/sda3
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x3d057827 [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 8e42af34-3102-4547-b3ff-f3d976eafa9d
metadata_uuid 8e42af34-3102-4547-b3ff-f3d976eafa9d
label gentoo
generation 8872486
root 110264320
sys_array_size 97
chunk_root_generation 8863753
root_level 1
chunk_root 315196702720
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 151926079488
bytes_used 89796988928
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 2
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x171
( MIXED_BACKREF |
 COMPRESS_ZSTD |
 BIG_METADATA |
 EXTENDED_IREF |
 SKINNY_METADATA )
cache_generation 8872486
uuid_tree_generation 283781
dev_item.uuid 877550b0-11cc-4b12-944f-83ac2a08201c
dev_item.fsid 8e42af34-3102-4547-b3ff-f3d976eafa9d [match]
dev_item.type 0
dev_item.total_bytes 148780351488
dev_item.bytes_used 148779302912
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 315196702720)
length 33554432 owner 2 stripe_len 65536 type SYSTEM
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 1
stripe 0 devid 2 offset 38797312
dev_uuid bc7154d2-bf0e-4ff4-b7cb-44e118148976
backup_roots[4]:
backup 0:
backup_tree_root: 110264320 gen: 8872486 level: 1
backup_chunk_root: 315196702720 gen: 8863753 level: 1
backup_extent_root: 110280704 gen: 8872486 level: 2
backup_fs_root: 125812736 gen: 8872478 level: 3
backup_dev_root: 104220278784 gen: 8863753 level: 0
backup_csum_root: 125796352 gen: 8872486 level: 2
backup_total_bytes: 151926079488
backup_bytes_used: 89796988928
backup_num_devices: 2

backup 1:
backup_tree_root: 191561728 gen: 8872483 level: 1
backup_chunk_root: 315196702720 gen: 8863753 level: 1
backup_extent_root: 191578112 gen: 8872483 level: 2
backup_fs_root: 125812736 gen: 8872478 level: 3
backup_dev_root: 104220278784 gen: 8863753 level: 0
backup_csum_root: 297615360 gen: 8872483 level: 2
backup_total_bytes: 151926079488
backup_bytes_used: 89796988928
backup_num_devices: 2

backup 2:
backup_tree_root: 398196736 gen: 8872484 level: 1
backup_chunk_root: 315196702720 gen: 8863753 level: 1
backup_extent_root: 398213120 gen: 8872484 level: 2
backup_fs_root: 125812736 gen: 8872478 level: 3
backup_dev_root: 104220278784 gen: 8863753 level: 0
backup_csum_root: 45662208 gen: 8872484 level: 2
backup_total_bytes: 151926079488
backup_bytes_used: 89796988928
backup_num_devices: 2

backup 3:
backup_tree_root: 70385664 gen: 8872485 level: 1
backup_chunk_root: 315196702720 gen: 8863753 level: 1
backup_extent_root: 70402048 gen: 8872485 level: 2
backup_fs_root: 125812736 gen: 8872478 level: 3
backup_dev_root: 104220278784 gen: 8863753 level: 0
backup_csum_root: 76185600 gen: 8872485 level: 2
backup_total_bytes: 151926079488
backup_bytes_used: 89796988928
backup_num_devices: 2

chunk-recover:
https://pastebin.com/MieHVxbd

So, did I screw up btrfs so badly I don't deserve a recovery, or can
something be done?

Thank you,
David

Re: Error: Failed to read chunk root, fs destroyed

[Lionel Bouton]

Hi,

Le 23/07/2020 à 23:15, Davide Palma a écrit :
> Hello,
> What happened:
>
> I had a Gentoo linux install on a standard, ssd, btrfs partition.
> After I removed lots of files, I wanted to shrink the partition, so i
> shrunk it as much as possible and started a balance operation to
> squeeze out more unallocated space. However, the balance operation was
> not progressing. (the behaviour was identical to the one described
> here https://www.opensuse-forum.de/thread/43376-btrfs-balance-caught-in-write-loop/
> ). So I cancelled the balance and, following a piece of advice found
> on the net, created a ramdisk that worked as a second device for a
> raid1 setup.
>
> dd if=/dev/zero of=/var/tmp/btrfs bs=1G count=5
>
> losetup -v -f /var/tmp/btrfs
> btrfs device add /dev/loop0 .
>
> I tried running the balance again, but, as before, it wasn't
> progressing. So I decided to remove the ramdisk and reconvert the now
> raid1 partition to a single one.

I probably won't be able to help but while this is fresh in your memory
please try to remember the exact sequence.

If I'm not mistaken the sequence of operations above isn't enough to
store data in raid1 block groups.

After device add you should still be using the same allocation policy as
the one chosen during mkfs. By default this should be :
metadata/system : dup,
data : single.

If that's indeed the case, removing the loop device later on will indeed
break the filesystem as several metadata block groups will be entirely
missing (dup will probably have allocated the two duplicates on the loop
block device where space was available).

If you managed to create raid1 block groups the exact sequence of events
will probably be useful for more knowledgeable people.

Best regards,

Lionel

Re: Error: Failed to read chunk root, fs destroyed

[Lionel Bouton]

Hi,

Le 23/07/2020 à 23:51, Davide Palma a écrit :
> Dear Lionel,
>
> > I probably won't be able to help but while this is fresh in your memory
> > please try to remember the exact sequence.
>
> I wish I could give you the .bash_history file,  but the sequence is
> the one I wrote. I tried running many balances before the ramdisk
> stuff by starting with -musage=0 and -dusage=0 and got to about 30
> before it started looping, I don't think this is useful but now you
> have 100% of my memory :)
>
> > If that's indeed the case, removing the loop device later on will indeed
> > break the filesystem as several metadata block groups will be entirely
> > missing (dup will probably have allocated the two duplicates on the loop
> > block device where space was available).
>
> Doesn't this defeat the whole point of raid1?

My point is that I saw nothing that pointed to raid1 being used on your
filesystem in the list of steps you described. I understand it was your
intent to use raid1 (although it isn't clear why as you didn't provide a
link for the advice you were following) but I'm not sure you actually did.

Simply adding a device to a BTRFS filesystem doesn't convert your
filesystem block groups it only adds usable space. You'd need to use a
balance command for converting the block groups to raid1.
Something like :
btrfs balance start -mconvert=raid1 -dconvert=raid1 <yourfs>

Best regards,

Lionel

Re: Error: Failed to read chunk root, fs destroyed

[Davide Palma]

Hi Lionel,
I didn't add a link because I couldn't find it :) here you go:
https://ohthehugemanatee.org/blog/2019/02/11/btrfs-out-of-space-emergency-response/

However, I had the problem written in the first comment: btrfs
automatically decided it was raid1 time and when i tried deleting the
device it said "unable to go below two devices on raid1".

Il giorno ven 24 lug 2020 alle ore 01:46 Lionel Bouton
<lionel-subscription@bouton.name> ha scritto:
>
> Hi,
>
> Le 23/07/2020 à 23:51, Davide Palma a écrit :
>
> Dear Lionel,
>
> > I probably won't be able to help but while this is fresh in your memory
> > please try to remember the exact sequence.
>
> I wish I could give you the .bash_history file,  but the sequence is the one I wrote. I tried running many balances before the ramdisk stuff by starting with -musage=0 and -dusage=0 and got to about 30 before it started looping, I don't think this is useful but now you have 100% of my memory :)
>
> > If that's indeed the case, removing the loop device later on will indeed
> > break the filesystem as several metadata block groups will be entirely
> > missing (dup will probably have allocated the two duplicates on the loop
> > block device where space was available).
>
> Doesn't this defeat the whole point of raid1?
>
>
> My point is that I saw nothing that pointed to raid1 being used on your filesystem in the list of steps you described. I understand it was your intent to use raid1 (although it isn't clear why as you didn't provide a link for the advice you were following) but I'm not sure you actually did.
>
> Simply adding a device to a BTRFS filesystem doesn't convert your filesystem block groups it only adds usable space. You'd need to use a balance command for converting the block groups to raid1.
> Something like :
> btrfs balance start -mconvert=raid1 -dconvert=raid1 <yourfs>
>
> Best regards,
>
> Lionel

Re: Error: Failed to read chunk root, fs destroyed

[Zygo Blaxell]

On Thu, Jul 23, 2020 at 11:15:40PM +0200, Davide Palma wrote:
> Hello,
> What happened:
> 
> I had a Gentoo linux install on a standard, ssd, btrfs partition.
> After I removed lots of files, I wanted to shrink the partition, so i
> shrunk it as much as possible and started a balance operation to
> squeeze out more unallocated space. 

The other order would have been better:  balance first to compact
free space, then shrink.  Balance and shrink are more or less the same
operation, but a balance can use all of the space on the filesystem to
eliminate free space gaps between extents, while a shrink can strictly
only use space below the target device size, considerably limiting its
flexibility.  If you do the shrink first, then balance has no free space
to work with that the shrink didn't already use.  If you do the balance
first, then block groups can be packed until the smallest amount of free
space remains in each one (often zero, but not always), then after the
balance, shrink can move full block groups around.

That said, the amount of IO required increases rapidly for the last few
percent of free space.  You'd have to balance the entire filesystem to
get the last few percent, and you wouldn't be able to get all of it.
If the smallest extent on the filesystem is 112.004MiB then the maximum
space efficiency is 87.5%.

> However, the balance operation was
> not progressing. (the behaviour was identical to the one described
> here https://www.opensuse-forum.de/thread/43376-btrfs-balance-caught-in-write-loop/
> ). 

This is the brtfs balance looping bug.  One of the triggers of the
bug is running low on space.

> So I cancelled the balance and, following a piece of advice found
> on the net, created a ramdisk that worked as a second device for a
> raid1 setup.
> 
> dd if=/dev/zero of=/var/tmp/btrfs bs=1G count=5
> 
> losetup -v -f /var/tmp/btrfs
> btrfs device add /dev/loop0 .
> 
> I tried running the balance again, but, as before, it wasn't
> progressing. So I decided to remove the ramdisk and reconvert the now
> raid1 partition to a single one. This is where the bad stuff began: As
> the process of converting raid1 to single is a balance, the balance
> looped and didn't progress. I also couldn't cancel it by ^C or SIGTERM
> or SIGKILL. Since I didn't know what to do, I just rebooted the PC
> (very bad decision!). The filesystem is now broken.

> Please note I never tried dangerous options like zero-log or --repair.

You didn't need to.  The most dangerous thing you did, by far, was adding
a ramdisk to a single-device btrfs that you intended to keep.  Compared
to that, zero-log is extremely safe.  Even --repair sometimes works.

> I setted up an arch linux install on /dev/sda4 to keep on working and
> to try recovery.
> 
> My environment:
> Gentoo linux: kernel 5.7.8, btrfs-progs 5.7
> 
> Arch linux:
> uname -a
> Linux lenuovo 5.7.9-arch1-1 #1 SMP PREEMPT Thu, 16 Jul 2020 19:34:49
> +0000 x86_64 GNU/Linux
> 
> btrfs --version
> btrfs-progs v5.7
> 
> fdisk -l
> Disk /dev/sda: 223,58 GiB, 240057409536 bytes, 468862128 sectors
> Disk model: KINGSTON SUV400S
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: gpt
> Disk identifier: 472C32BA-DDF3-4063-BBF7-629C7A68C195
> 
> Dispositivo     Start      Fine   Settori   Size Tipo
> /dev/sda1        2048   8194047   8192000   3,9G Linux swap
> /dev/sda2     8194048   8521727    327680   160M EFI System
> /dev/sda3     8521728 302680063 294158336 140,3G Linux filesystem
> /dev/sda4   302680064 345087999  42407936  20,2G Linux filesystem
> /dev/sda5   345088000 468860927 123772928    59G Microsoft basic data
> 
> The broken FS is /dev/sda3.
> 
> Any combination of mount options without degraded fails with:
> BTRFS info (device sda3): disk space caching is enabled
> BTRFS info (device sda3): has skinny extents
> BTRFS error (device sda3): devid 2 uuid
> bc7154d2-bf0e-4ff4-b7cb-44e118148976 is missing
> BTRFS error (device sda3): failed to read the system array: -2
> BTRFS error (device sda3): open_ctree failed
> 
> Any combination of mount options with degraded fails with:
> BTRFS info (device sda3): allowing degraded mounts
> BTRFS info (device sda3): disk space caching is enabled
> BTRFS info (device sda3): has skinny extents
> BTRFS warning (device sda3): devid 2 uuid
> bc7154d2-bf0e-4ff4-b7cb-44e118148976 is missing
> BTRFS error (device sda3): failed to read chunk root
> BTRFS error (device sda3): open_ctree failed
> 
> This is also the error of every btrfs prog that tries to access the volume:
> btrfs check /dev/sda3
> Opening filesystem to check...
> warning, device 2 is missing
> bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
> ERROR: cannot read chunk root
> ERROR: cannot open file system
> 
> btrfs restore /dev/sda3 -i -l
> warning, device 2 is missing
> bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
> ERROR: cannot read chunk root
> Could not open root, trying backup super
> warning, device 2 is missing
> bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
> ERROR: cannot read chunk root
> Could not open root, trying backup super
> ERROR: superblock bytenr 274877906944 is larger than device size 150609068032
> Could not open root, trying backup super
> 
> btrfs rescue super-recover /dev/sda3
> All supers are valid, no need to recover
> 
> btrfs inspect-internal dump-super -f /dev/sda3
> superblock: bytenr=65536, device=/dev/sda3
> ---------------------------------------------------------
> csum_type 0 (crc32c)
> csum_size 4
> csum 0x3d057827 [match]
> bytenr 65536
> flags 0x1
> ( WRITTEN )
> magic _BHRfS_M [match]
> fsid 8e42af34-3102-4547-b3ff-f3d976eafa9d
> metadata_uuid 8e42af34-3102-4547-b3ff-f3d976eafa9d
> label gentoo
> generation 8872486
> root 110264320
> sys_array_size 97
> chunk_root_generation 8863753
> root_level 1
> chunk_root 315196702720
> chunk_root_level 1
> log_root 0
> log_root_transid 0
> log_root_level 0
> total_bytes 151926079488
> bytes_used 89796988928
> sectorsize 4096
> nodesize 16384
> leafsize (deprecated) 16384
> stripesize 4096
> root_dir 6
> num_devices 2
> compat_flags 0x0
> compat_ro_flags 0x0
> incompat_flags 0x171
> ( MIXED_BACKREF |
>  COMPRESS_ZSTD |
>  BIG_METADATA |
>  EXTENDED_IREF |
>  SKINNY_METADATA )
> cache_generation 8872486
> uuid_tree_generation 283781
> dev_item.uuid 877550b0-11cc-4b12-944f-83ac2a08201c
> dev_item.fsid 8e42af34-3102-4547-b3ff-f3d976eafa9d [match]
> dev_item.type 0
> dev_item.total_bytes 148780351488
> dev_item.bytes_used 148779302912
> dev_item.io_align 4096
> dev_item.io_width 4096
> dev_item.sector_size 4096
> dev_item.devid 1
> dev_item.dev_group 0
> dev_item.seek_speed 0
> dev_item.bandwidth 0
> dev_item.generation 0
> sys_chunk_array[2048]:
> item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 315196702720)
> length 33554432 owner 2 stripe_len 65536 type SYSTEM
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 2 offset 38797312
> dev_uuid bc7154d2-bf0e-4ff4-b7cb-44e118148976
> backup_roots[4]:
> backup 0:
> backup_tree_root: 110264320 gen: 8872486 level: 1
> backup_chunk_root: 315196702720 gen: 8863753 level: 1
> backup_extent_root: 110280704 gen: 8872486 level: 2
> backup_fs_root: 125812736 gen: 8872478 level: 3
> backup_dev_root: 104220278784 gen: 8863753 level: 0
> backup_csum_root: 125796352 gen: 8872486 level: 2
> backup_total_bytes: 151926079488
> backup_bytes_used: 89796988928
> backup_num_devices: 2
> 
> backup 1:
> backup_tree_root: 191561728 gen: 8872483 level: 1
> backup_chunk_root: 315196702720 gen: 8863753 level: 1
> backup_extent_root: 191578112 gen: 8872483 level: 2
> backup_fs_root: 125812736 gen: 8872478 level: 3
> backup_dev_root: 104220278784 gen: 8863753 level: 0
> backup_csum_root: 297615360 gen: 8872483 level: 2
> backup_total_bytes: 151926079488
> backup_bytes_used: 89796988928
> backup_num_devices: 2
> 
> backup 2:
> backup_tree_root: 398196736 gen: 8872484 level: 1
> backup_chunk_root: 315196702720 gen: 8863753 level: 1
> backup_extent_root: 398213120 gen: 8872484 level: 2
> backup_fs_root: 125812736 gen: 8872478 level: 3
> backup_dev_root: 104220278784 gen: 8863753 level: 0
> backup_csum_root: 45662208 gen: 8872484 level: 2
> backup_total_bytes: 151926079488
> backup_bytes_used: 89796988928
> backup_num_devices: 2
> 
> backup 3:
> backup_tree_root: 70385664 gen: 8872485 level: 1
> backup_chunk_root: 315196702720 gen: 8863753 level: 1
> backup_extent_root: 70402048 gen: 8872485 level: 2
> backup_fs_root: 125812736 gen: 8872478 level: 3
> backup_dev_root: 104220278784 gen: 8863753 level: 0
> backup_csum_root: 76185600 gen: 8872485 level: 2
> backup_total_bytes: 151926079488
> backup_bytes_used: 89796988928
> backup_num_devices: 2
> 
> chunk-recover:
> https://pastebin.com/MieHVxbd
> 
> So, did I screw up btrfs so badly I don't deserve a recovery, or can
> something be done?

Normally, if we are setting up a RAID0 or JBOD array, we are accepting
the risk that if any device in the array fails, we'll mkfs and start
over with a new filesystem.  Balances and device remove/shrink will move
data aggressively to the most-empty disk.

If the chunk tree roots are now on the RAM device (very likely if the SSD
was full, a device added, then a balance/delete/resize-shrink started)
then they're gone forever.  Even old copies on the partition you were
trying to shrink would have been overwritten by other data by now.

If a metadata block group was moved to the RAM device, the filesystem
is gone.  The data blocks will remain on disk, but there will be nothing
left behind to associate them with files any more.

You might want to try:

	1.  Make a dd copy of the disk, because the following steps
	might destroy the filesystem.

	2.  btrfs rescue chunk-recover /dev/sda3

	3.  btrfs check --repair --init-csum-tree /dev/sda3

but it looks like you already did step 2, and if that doesn't work
there is no step 3.

> Thank you,
> David

Re: Error: Failed to read chunk root, fs destroyed

[Zygo Blaxell]

On Sat, Jul 25, 2020 at 08:01:05PM -0400, Zygo Blaxell wrote:
> On Thu, Jul 23, 2020 at 11:15:40PM +0200, Davide Palma wrote:
> > Hello,
> > What happened:
> > 
> > I had a Gentoo linux install on a standard, ssd, btrfs partition.
> > After I removed lots of files, I wanted to shrink the partition, so i
> > shrunk it as much as possible and started a balance operation to
> > squeeze out more unallocated space. 
> 
> The other order would have been better:  balance first to compact
> free space, then shrink.  Balance and shrink are more or less the same
> operation, but a balance can use all of the space on the filesystem to
> eliminate free space gaps between extents, while a shrink can strictly
> only use space below the target device size, considerably limiting its
> flexibility.  If you do the shrink first, then balance has no free space
> to work with that the shrink didn't already use.  If you do the balance
> first, then block groups can be packed until the smallest amount of free
> space remains in each one (often zero, but not always), then after the
> balance, shrink can move full block groups around.
> 
> That said, the amount of IO required increases rapidly for the last few
> percent of free space.  You'd have to balance the entire filesystem to
> get the last few percent, and you wouldn't be able to get all of it.
> If the smallest extent on the filesystem is 112.004MiB then the maximum
> space efficiency is 87.5%.
> 
> > However, the balance operation was
> > not progressing. (the behaviour was identical to the one described
> > here https://www.opensuse-forum.de/thread/43376-btrfs-balance-caught-in-write-loop/
> > ). 
> 
> This is the brtfs balance looping bug.  One of the triggers of the
> bug is running low on space.
> 
> > So I cancelled the balance and, following a piece of advice found
> > on the net, created a ramdisk that worked as a second device for a
> > raid1 setup.
> > 
> > dd if=/dev/zero of=/var/tmp/btrfs bs=1G count=5
> > 
> > losetup -v -f /var/tmp/btrfs
> > btrfs device add /dev/loop0 .
> > 
> > I tried running the balance again, but, as before, it wasn't
> > progressing. So I decided to remove the ramdisk and reconvert the now
> > raid1 partition to a single one. This is where the bad stuff began: As
> > the process of converting raid1 to single is a balance, the balance
> > looped and didn't progress. I also couldn't cancel it by ^C or SIGTERM
> > or SIGKILL. Since I didn't know what to do, I just rebooted the PC
> > (very bad decision!). The filesystem is now broken.
> 
> > Please note I never tried dangerous options like zero-log or --repair.
> 
> You didn't need to.  The most dangerous thing you did, by far, was adding
> a ramdisk to a single-device btrfs that you intended to keep.  Compared
> to that, zero-log is extremely safe.  Even --repair sometimes works.
> 
> > I setted up an arch linux install on /dev/sda4 to keep on working and
> > to try recovery.
> > 
> > My environment:
> > Gentoo linux: kernel 5.7.8, btrfs-progs 5.7
> > 
> > Arch linux:
> > uname -a
> > Linux lenuovo 5.7.9-arch1-1 #1 SMP PREEMPT Thu, 16 Jul 2020 19:34:49
> > +0000 x86_64 GNU/Linux
> > 
> > btrfs --version
> > btrfs-progs v5.7
> > 
> > fdisk -l
> > Disk /dev/sda: 223,58 GiB, 240057409536 bytes, 468862128 sectors
> > Disk model: KINGSTON SUV400S
> > Units: sectors of 1 * 512 = 512 bytes
> > Sector size (logical/physical): 512 bytes / 512 bytes
> > I/O size (minimum/optimal): 512 bytes / 512 bytes
> > Disklabel type: gpt
> > Disk identifier: 472C32BA-DDF3-4063-BBF7-629C7A68C195
> > 
> > Dispositivo     Start      Fine   Settori   Size Tipo
> > /dev/sda1        2048   8194047   8192000   3,9G Linux swap
> > /dev/sda2     8194048   8521727    327680   160M EFI System
> > /dev/sda3     8521728 302680063 294158336 140,3G Linux filesystem
> > /dev/sda4   302680064 345087999  42407936  20,2G Linux filesystem
> > /dev/sda5   345088000 468860927 123772928    59G Microsoft basic data
> > 
> > The broken FS is /dev/sda3.
> > 
> > Any combination of mount options without degraded fails with:
> > BTRFS info (device sda3): disk space caching is enabled
> > BTRFS info (device sda3): has skinny extents
> > BTRFS error (device sda3): devid 2 uuid
> > bc7154d2-bf0e-4ff4-b7cb-44e118148976 is missing
> > BTRFS error (device sda3): failed to read the system array: -2
> > BTRFS error (device sda3): open_ctree failed
> > 
> > Any combination of mount options with degraded fails with:
> > BTRFS info (device sda3): allowing degraded mounts
> > BTRFS info (device sda3): disk space caching is enabled
> > BTRFS info (device sda3): has skinny extents
> > BTRFS warning (device sda3): devid 2 uuid
> > bc7154d2-bf0e-4ff4-b7cb-44e118148976 is missing
> > BTRFS error (device sda3): failed to read chunk root
> > BTRFS error (device sda3): open_ctree failed
> > 
> > This is also the error of every btrfs prog that tries to access the volume:
> > btrfs check /dev/sda3
> > Opening filesystem to check...
> > warning, device 2 is missing
> > bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
> > ERROR: cannot read chunk root
> > ERROR: cannot open file system
> > 
> > btrfs restore /dev/sda3 -i -l
> > warning, device 2 is missing
> > bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
> > ERROR: cannot read chunk root
> > Could not open root, trying backup super
> > warning, device 2 is missing
> > bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
> > ERROR: cannot read chunk root
> > Could not open root, trying backup super
> > ERROR: superblock bytenr 274877906944 is larger than device size 150609068032
> > Could not open root, trying backup super
> > 
> > btrfs rescue super-recover /dev/sda3
> > All supers are valid, no need to recover
> > 
> > btrfs inspect-internal dump-super -f /dev/sda3
> > superblock: bytenr=65536, device=/dev/sda3
> > ---------------------------------------------------------
> > csum_type 0 (crc32c)
> > csum_size 4
> > csum 0x3d057827 [match]
> > bytenr 65536
> > flags 0x1
> > ( WRITTEN )
> > magic _BHRfS_M [match]
> > fsid 8e42af34-3102-4547-b3ff-f3d976eafa9d
> > metadata_uuid 8e42af34-3102-4547-b3ff-f3d976eafa9d
> > label gentoo
> > generation 8872486
> > root 110264320
> > sys_array_size 97
> > chunk_root_generation 8863753
> > root_level 1
> > chunk_root 315196702720
> > chunk_root_level 1
> > log_root 0
> > log_root_transid 0
> > log_root_level 0
> > total_bytes 151926079488
> > bytes_used 89796988928
> > sectorsize 4096
> > nodesize 16384
> > leafsize (deprecated) 16384
> > stripesize 4096
> > root_dir 6
> > num_devices 2
> > compat_flags 0x0
> > compat_ro_flags 0x0
> > incompat_flags 0x171
> > ( MIXED_BACKREF |
> >  COMPRESS_ZSTD |
> >  BIG_METADATA |
> >  EXTENDED_IREF |
> >  SKINNY_METADATA )
> > cache_generation 8872486
> > uuid_tree_generation 283781
> > dev_item.uuid 877550b0-11cc-4b12-944f-83ac2a08201c
> > dev_item.fsid 8e42af34-3102-4547-b3ff-f3d976eafa9d [match]
> > dev_item.type 0
> > dev_item.total_bytes 148780351488
> > dev_item.bytes_used 148779302912
> > dev_item.io_align 4096
> > dev_item.io_width 4096
> > dev_item.sector_size 4096
> > dev_item.devid 1
> > dev_item.dev_group 0
> > dev_item.seek_speed 0
> > dev_item.bandwidth 0
> > dev_item.generation 0
> > sys_chunk_array[2048]:
> > item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 315196702720)
> > length 33554432 owner 2 stripe_len 65536 type SYSTEM
> > io_align 65536 io_width 65536 sector_size 4096
> > num_stripes 1 sub_stripes 1
> > stripe 0 devid 2 offset 38797312
> > dev_uuid bc7154d2-bf0e-4ff4-b7cb-44e118148976
> > backup_roots[4]:
> > backup 0:
> > backup_tree_root: 110264320 gen: 8872486 level: 1
> > backup_chunk_root: 315196702720 gen: 8863753 level: 1
> > backup_extent_root: 110280704 gen: 8872486 level: 2
> > backup_fs_root: 125812736 gen: 8872478 level: 3
> > backup_dev_root: 104220278784 gen: 8863753 level: 0
> > backup_csum_root: 125796352 gen: 8872486 level: 2
> > backup_total_bytes: 151926079488
> > backup_bytes_used: 89796988928
> > backup_num_devices: 2
> > 
> > backup 1:
> > backup_tree_root: 191561728 gen: 8872483 level: 1
> > backup_chunk_root: 315196702720 gen: 8863753 level: 1
> > backup_extent_root: 191578112 gen: 8872483 level: 2
> > backup_fs_root: 125812736 gen: 8872478 level: 3
> > backup_dev_root: 104220278784 gen: 8863753 level: 0
> > backup_csum_root: 297615360 gen: 8872483 level: 2
> > backup_total_bytes: 151926079488
> > backup_bytes_used: 89796988928
> > backup_num_devices: 2
> > 
> > backup 2:
> > backup_tree_root: 398196736 gen: 8872484 level: 1
> > backup_chunk_root: 315196702720 gen: 8863753 level: 1
> > backup_extent_root: 398213120 gen: 8872484 level: 2
> > backup_fs_root: 125812736 gen: 8872478 level: 3
> > backup_dev_root: 104220278784 gen: 8863753 level: 0
> > backup_csum_root: 45662208 gen: 8872484 level: 2
> > backup_total_bytes: 151926079488
> > backup_bytes_used: 89796988928
> > backup_num_devices: 2
> > 
> > backup 3:
> > backup_tree_root: 70385664 gen: 8872485 level: 1
> > backup_chunk_root: 315196702720 gen: 8863753 level: 1
> > backup_extent_root: 70402048 gen: 8872485 level: 2
> > backup_fs_root: 125812736 gen: 8872478 level: 3
> > backup_dev_root: 104220278784 gen: 8863753 level: 0
> > backup_csum_root: 76185600 gen: 8872485 level: 2
> > backup_total_bytes: 151926079488
> > backup_bytes_used: 89796988928
> > backup_num_devices: 2
> > 
> > chunk-recover:
> > https://pastebin.com/MieHVxbd
> > 
> > So, did I screw up btrfs so badly I don't deserve a recovery, or can
> > something be done?
> 
> Normally, if we are setting up a RAID0 or JBOD array, we are accepting
> the risk that if any device in the array fails, we'll mkfs and start
> over with a new filesystem.  Balances and device remove/shrink will move
> data aggressively to the most-empty disk.
> 
> If the chunk tree roots are now on the RAM device (very likely if the SSD
> was full, a device added, then a balance/delete/resize-shrink started)
> then they're gone forever.  Even old copies on the partition you were
> trying to shrink would have been overwritten by other data by now.
> 
> If a metadata block group was moved to the RAM device, the filesystem
> is gone.  The data blocks will remain on disk, but there will be nothing
> left behind to associate them with files any more.
> 
> You might want to try:
> 
> 	1.  Make a dd copy of the disk, because the following steps
> 	might destroy the filesystem.
> 
> 	2.  btrfs rescue chunk-recover /dev/sda3
> 
> 	3.  btrfs check --repair --init-csum-tree /dev/sda3

Whoops, that should be:

  	3.  btrfs check --repair --init-extent-tree /dev/sda3

Maybe --init-csum-tree will be needed too, but it's not likely that any
of these commands will work at this point.

> but it looks like you already did step 2, and if that doesn't work
> there is no step 3.
> 
> > Thank you,
> > David

Re: Error: Failed to read chunk root, fs destroyed

[Davide Palma]

Thank you Zygo. Any btrfs check --repair will fail to open chunk root.
Looks like my fs is gone. Pro tip: don't use ramdisks in this silly
way :)

Il giorno dom 26 lug 2020 alle ore 02:11 Zygo Blaxell
<ce3g8jdj@umail.furryterror.org> ha scritto:

>
> On Sat, Jul 25, 2020 at 08:01:05PM -0400, Zygo Blaxell wrote:
> > On Thu, Jul 23, 2020 at 11:15:40PM +0200, Davide Palma wrote:
> > > Hello,
> > > What happened:
> > >
> > > I had a Gentoo linux install on a standard, ssd, btrfs partition.
> > > After I removed lots of files, I wanted to shrink the partition, so i
> > > shrunk it as much as possible and started a balance operation to
> > > squeeze out more unallocated space.
> >
> > The other order would have been better:  balance first to compact
> > free space, then shrink.  Balance and shrink are more or less the same
> > operation, but a balance can use all of the space on the filesystem to
> > eliminate free space gaps between extents, while a shrink can strictly
> > only use space below the target device size, considerably limiting its
> > flexibility.  If you do the shrink first, then balance has no free space
> > to work with that the shrink didn't already use.  If you do the balance
> > first, then block groups can be packed until the smallest amount of free
> > space remains in each one (often zero, but not always), then after the
> > balance, shrink can move full block groups around.
> >
> > That said, the amount of IO required increases rapidly for the last few
> > percent of free space.  You'd have to balance the entire filesystem to
> > get the last few percent, and you wouldn't be able to get all of it.
> > If the smallest extent on the filesystem is 112.004MiB then the maximum
> > space efficiency is 87.5%.
> >
> > > However, the balance operation was
> > > not progressing. (the behaviour was identical to the one described
> > > here https://www.opensuse-forum.de/thread/43376-btrfs-balance-caught-in-write-loop/
> > > ).
> >
> > This is the brtfs balance looping bug.  One of the triggers of the
> > bug is running low on space.
> >
> > > So I cancelled the balance and, following a piece of advice found
> > > on the net, created a ramdisk that worked as a second device for a
> > > raid1 setup.
> > >
> > > dd if=/dev/zero of=/var/tmp/btrfs bs=1G count=5
> > >
> > > losetup -v -f /var/tmp/btrfs
> > > btrfs device add /dev/loop0 .
> > >
> > > I tried running the balance again, but, as before, it wasn't
> > > progressing. So I decided to remove the ramdisk and reconvert the now
> > > raid1 partition to a single one. This is where the bad stuff began: As
> > > the process of converting raid1 to single is a balance, the balance
> > > looped and didn't progress. I also couldn't cancel it by ^C or SIGTERM
> > > or SIGKILL. Since I didn't know what to do, I just rebooted the PC
> > > (very bad decision!). The filesystem is now broken.
> >
> > > Please note I never tried dangerous options like zero-log or --repair.
> >
> > You didn't need to.  The most dangerous thing you did, by far, was adding
> > a ramdisk to a single-device btrfs that you intended to keep.  Compared
> > to that, zero-log is extremely safe.  Even --repair sometimes works.
> >
> > > I setted up an arch linux install on /dev/sda4 to keep on working and
> > > to try recovery.
> > >
> > > My environment:
> > > Gentoo linux: kernel 5.7.8, btrfs-progs 5.7
> > >
> > > Arch linux:
> > > uname -a
> > > Linux lenuovo 5.7.9-arch1-1 #1 SMP PREEMPT Thu, 16 Jul 2020 19:34:49
> > > +0000 x86_64 GNU/Linux
> > >
> > > btrfs --version
> > > btrfs-progs v5.7
> > >
> > > fdisk -l
> > > Disk /dev/sda: 223,58 GiB, 240057409536 bytes, 468862128 sectors
> > > Disk model: KINGSTON SUV400S
> > > Units: sectors of 1 * 512 = 512 bytes
> > > Sector size (logical/physical): 512 bytes / 512 bytes
> > > I/O size (minimum/optimal): 512 bytes / 512 bytes
> > > Disklabel type: gpt
> > > Disk identifier: 472C32BA-DDF3-4063-BBF7-629C7A68C195
> > >
> > > Dispositivo     Start      Fine   Settori   Size Tipo
> > > /dev/sda1        2048   8194047   8192000   3,9G Linux swap
> > > /dev/sda2     8194048   8521727    327680   160M EFI System
> > > /dev/sda3     8521728 302680063 294158336 140,3G Linux filesystem
> > > /dev/sda4   302680064 345087999  42407936  20,2G Linux filesystem
> > > /dev/sda5   345088000 468860927 123772928    59G Microsoft basic data
> > >
> > > The broken FS is /dev/sda3.
> > >
> > > Any combination of mount options without degraded fails with:
> > > BTRFS info (device sda3): disk space caching is enabled
> > > BTRFS info (device sda3): has skinny extents
> > > BTRFS error (device sda3): devid 2 uuid
> > > bc7154d2-bf0e-4ff4-b7cb-44e118148976 is missing
> > > BTRFS error (device sda3): failed to read the system array: -2
> > > BTRFS error (device sda3): open_ctree failed
> > >
> > > Any combination of mount options with degraded fails with:
> > > BTRFS info (device sda3): allowing degraded mounts
> > > BTRFS info (device sda3): disk space caching is enabled
> > > BTRFS info (device sda3): has skinny extents
> > > BTRFS warning (device sda3): devid 2 uuid
> > > bc7154d2-bf0e-4ff4-b7cb-44e118148976 is missing
> > > BTRFS error (device sda3): failed to read chunk root
> > > BTRFS error (device sda3): open_ctree failed
> > >
> > > This is also the error of every btrfs prog that tries to access the volume:
> > > btrfs check /dev/sda3
> > > Opening filesystem to check...
> > > warning, device 2 is missing
> > > bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
> > > ERROR: cannot read chunk root
> > > ERROR: cannot open file system
> > >
> > > btrfs restore /dev/sda3 -i -l
> > > warning, device 2 is missing
> > > bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
> > > ERROR: cannot read chunk root
> > > Could not open root, trying backup super
> > > warning, device 2 is missing
> > > bad tree block 315196702720, bytenr mismatch, want=315196702720, have=0
> > > ERROR: cannot read chunk root
> > > Could not open root, trying backup super
> > > ERROR: superblock bytenr 274877906944 is larger than device size 150609068032
> > > Could not open root, trying backup super
> > >
> > > btrfs rescue super-recover /dev/sda3
> > > All supers are valid, no need to recover
> > >
> > > btrfs inspect-internal dump-super -f /dev/sda3
> > > superblock: bytenr=65536, device=/dev/sda3
> > > ---------------------------------------------------------
> > > csum_type 0 (crc32c)
> > > csum_size 4
> > > csum 0x3d057827 [match]
> > > bytenr 65536
> > > flags 0x1
> > > ( WRITTEN )
> > > magic _BHRfS_M [match]
> > > fsid 8e42af34-3102-4547-b3ff-f3d976eafa9d
> > > metadata_uuid 8e42af34-3102-4547-b3ff-f3d976eafa9d
> > > label gentoo
> > > generation 8872486
> > > root 110264320
> > > sys_array_size 97
> > > chunk_root_generation 8863753
> > > root_level 1
> > > chunk_root 315196702720
> > > chunk_root_level 1
> > > log_root 0
> > > log_root_transid 0
> > > log_root_level 0
> > > total_bytes 151926079488
> > > bytes_used 89796988928
> > > sectorsize 4096
> > > nodesize 16384
> > > leafsize (deprecated) 16384
> > > stripesize 4096
> > > root_dir 6
> > > num_devices 2
> > > compat_flags 0x0
> > > compat_ro_flags 0x0
> > > incompat_flags 0x171
> > > ( MIXED_BACKREF |
> > >  COMPRESS_ZSTD |
> > >  BIG_METADATA |
> > >  EXTENDED_IREF |
> > >  SKINNY_METADATA )
> > > cache_generation 8872486
> > > uuid_tree_generation 283781
> > > dev_item.uuid 877550b0-11cc-4b12-944f-83ac2a08201c
> > > dev_item.fsid 8e42af34-3102-4547-b3ff-f3d976eafa9d [match]
> > > dev_item.type 0
> > > dev_item.total_bytes 148780351488
> > > dev_item.bytes_used 148779302912
> > > dev_item.io_align 4096
> > > dev_item.io_width 4096
> > > dev_item.sector_size 4096
> > > dev_item.devid 1
> > > dev_item.dev_group 0
> > > dev_item.seek_speed 0
> > > dev_item.bandwidth 0
> > > dev_item.generation 0
> > > sys_chunk_array[2048]:
> > > item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 315196702720)
> > > length 33554432 owner 2 stripe_len 65536 type SYSTEM
> > > io_align 65536 io_width 65536 sector_size 4096
> > > num_stripes 1 sub_stripes 1
> > > stripe 0 devid 2 offset 38797312
> > > dev_uuid bc7154d2-bf0e-4ff4-b7cb-44e118148976
> > > backup_roots[4]:
> > > backup 0:
> > > backup_tree_root: 110264320 gen: 8872486 level: 1
> > > backup_chunk_root: 315196702720 gen: 8863753 level: 1
> > > backup_extent_root: 110280704 gen: 8872486 level: 2
> > > backup_fs_root: 125812736 gen: 8872478 level: 3
> > > backup_dev_root: 104220278784 gen: 8863753 level: 0
> > > backup_csum_root: 125796352 gen: 8872486 level: 2
> > > backup_total_bytes: 151926079488
> > > backup_bytes_used: 89796988928
> > > backup_num_devices: 2
> > >
> > > backup 1:
> > > backup_tree_root: 191561728 gen: 8872483 level: 1
> > > backup_chunk_root: 315196702720 gen: 8863753 level: 1
> > > backup_extent_root: 191578112 gen: 8872483 level: 2
> > > backup_fs_root: 125812736 gen: 8872478 level: 3
> > > backup_dev_root: 104220278784 gen: 8863753 level: 0
> > > backup_csum_root: 297615360 gen: 8872483 level: 2
> > > backup_total_bytes: 151926079488
> > > backup_bytes_used: 89796988928
> > > backup_num_devices: 2
> > >
> > > backup 2:
> > > backup_tree_root: 398196736 gen: 8872484 level: 1
> > > backup_chunk_root: 315196702720 gen: 8863753 level: 1
> > > backup_extent_root: 398213120 gen: 8872484 level: 2
> > > backup_fs_root: 125812736 gen: 8872478 level: 3
> > > backup_dev_root: 104220278784 gen: 8863753 level: 0
> > > backup_csum_root: 45662208 gen: 8872484 level: 2
> > > backup_total_bytes: 151926079488
> > > backup_bytes_used: 89796988928
> > > backup_num_devices: 2
> > >
> > > backup 3:
> > > backup_tree_root: 70385664 gen: 8872485 level: 1
> > > backup_chunk_root: 315196702720 gen: 8863753 level: 1
> > > backup_extent_root: 70402048 gen: 8872485 level: 2
> > > backup_fs_root: 125812736 gen: 8872478 level: 3
> > > backup_dev_root: 104220278784 gen: 8863753 level: 0
> > > backup_csum_root: 76185600 gen: 8872485 level: 2
> > > backup_total_bytes: 151926079488
> > > backup_bytes_used: 89796988928
> > > backup_num_devices: 2
> > >
> > > chunk-recover:
> > > https://pastebin.com/MieHVxbd
> > >
> > > So, did I screw up btrfs so badly I don't deserve a recovery, or can
> > > something be done?
> >
> > Normally, if we are setting up a RAID0 or JBOD array, we are accepting
> > the risk that if any device in the array fails, we'll mkfs and start
> > over with a new filesystem.  Balances and device remove/shrink will move
> > data aggressively to the most-empty disk.
> >
> > If the chunk tree roots are now on the RAM device (very likely if the SSD
> > was full, a device added, then a balance/delete/resize-shrink started)
> > then they're gone forever.  Even old copies on the partition you were
> > trying to shrink would have been overwritten by other data by now.
> >
> > If a metadata block group was moved to the RAM device, the filesystem
> > is gone.  The data blocks will remain on disk, but there will be nothing
> > left behind to associate them with files any more.
> >
> > You might want to try:
> >
> >       1.  Make a dd copy of the disk, because the following steps
> >       might destroy the filesystem.
> >
> >       2.  btrfs rescue chunk-recover /dev/sda3
> >
> >       3.  btrfs check --repair --init-csum-tree /dev/sda3
>
> Whoops, that should be:
>
>         3.  btrfs check --repair --init-extent-tree /dev/sda3
>
> Maybe --init-csum-tree will be needed too, but it's not likely that any
> of these commands will work at this point.
>
> > but it looks like you already did step 2, and if that doesn't work
> > there is no step 3.
> >
> > > Thank you,
> > > David