Critical error from Tree-checker

Linux Btrfs filesystem development
 help / color / mirror / Atom feed

* Critical error from Tree-checker
@ 2024-09-10 21:05 Archange
  2024-09-10 21:28 ` Qu Wenruo
  0 siblings, 1 reply; 24+ messages in thread
From: Archange @ 2024-09-10 21:05 UTC (permalink / raw)
  To: linux-btrfs

Hi there,

Since today, my system started randomly becoming read-only. At that 
point I can still run dmesg in an open terminal, so I’ve seen it was 
related to a btrfs error, but did not try anything since I could not 
open a web browser anymore. But I’ve seen the error to be “BTRFS 
critical” and related to a “corrupt leaf”.

I’ve tried to run `btrfs scrub` on the device after rebooting, and in 
fact it aborted almost right away triggering the same error in dmesg 
(but not turning the system read-only, so I can copy paste it here):

[  365.268769] BTRFS info (device dm-0): scrub: started on devid 1
[  385.788000] page: refcount:3 mapcount:0 mapping:00000000d0054cae 
index:0x9678888 pfn:0x11ce15
[  385.788015] memcg:ffff9fc94db8f000
[  385.788021] aops:btree_aops [btrfs] ino:1
[  385.788235] flags: 
0x2ffffa000004020(lru|private|node=0|zone=2|lastcpupid=0x1ffff)
[  385.788248] raw: 02ffffa000004020 ffffea9a8574ff88 ffffea9a847385c8 
ffff9fc95b8365b0
[  385.788255] raw: 0000000009678888 ffff9fc9ae554000 00000003ffffffff 
ffff9fc94db8f000
[  385.788259] page dumped because: eb page dump
[  385.788264] BTRFS critical (device dm-0): corrupt leaf: 
block=646267305984 slot=92 extent bytenr=1182031872 len=106496 invalid 
data ref objectid value 257
[  385.788283] BTRFS error (device dm-0): read time tree block 
corruption detected on logical 646267305984 mirror 1
[  385.796803] BTRFS info (device dm-0): scrub: not finished on devid 1 
with status: -5

According to https://btrfs.readthedocs.io/en/latest/Tree-checker.html 
this is not really expected, and the last paragraph says to report 
troubles here. So here I am, in the search for advice about this error 
(web searches returned nothing with this specific error except the 
commit/ml messages that added the code for it) and how to fix my random 
lockups.

Regards.

P.S.: I’m not subscribed to the list, so please keep me in copy when 
answering.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-10 21:05 Critical error from Tree-checker Archange
@ 2024-09-10 21:28 ` Qu Wenruo
  2024-09-10 21:37   ` Qu Wenruo
  0 siblings, 1 reply; 24+ messages in thread
From: Qu Wenruo @ 2024-09-10 21:28 UTC (permalink / raw)
  To: Archange, linux-btrfs



在 2024/9/11 06:35, Archange 写道:
> Hi there,
>
> Since today, my system started randomly becoming read-only. At that
> point I can still run dmesg in an open terminal, so I’ve seen it was
> related to a btrfs error, but did not try anything since I could not
> open a web browser anymore. But I’ve seen the error to be “BTRFS
> critical” and related to a “corrupt leaf”.
>
> I’ve tried to run `btrfs scrub` on the device after rebooting, and in
> fact it aborted almost right away triggering the same error in dmesg
> (but not turning the system read-only, so I can copy paste it here):
>
> [  365.268769] BTRFS info (device dm-0): scrub: started on devid 1
> [  385.788000] page: refcount:3 mapcount:0 mapping:00000000d0054cae
> index:0x9678888 pfn:0x11ce15
> [  385.788015] memcg:ffff9fc94db8f000
> [  385.788021] aops:btree_aops [btrfs] ino:1
> [  385.788235] flags:
> 0x2ffffa000004020(lru|private|node=0|zone=2|lastcpupid=0x1ffff)
> [  385.788248] raw: 02ffffa000004020 ffffea9a8574ff88 ffffea9a847385c8
> ffff9fc95b8365b0
> [  385.788255] raw: 0000000009678888 ffff9fc9ae554000 00000003ffffffff
> ffff9fc94db8f000
> [  385.788259] page dumped because: eb page dump
> [  385.788264] BTRFS critical (device dm-0): corrupt leaf:
> block=646267305984 slot=92 extent bytenr=1182031872 len=106496 invalid
> data ref objectid value 257

Full dmesg please.

Normally it should dump the full content of the tree block, to help
debugging the problem.

Thanks,
Qu
> [  385.788283] BTRFS error (device dm-0): read time tree block
> corruption detected on logical 646267305984 mirror 1
> [  385.796803] BTRFS info (device dm-0): scrub: not finished on devid 1
> with status: -5
>
> According to https://btrfs.readthedocs.io/en/latest/Tree-checker.html
> this is not really expected, and the last paragraph says to report
> troubles here. So here I am, in the search for advice about this error
> (web searches returned nothing with this specific error except the
> commit/ml messages that added the code for it) and how to fix my random
> lockups.
>
> Regards.
>
> P.S.: I’m not subscribed to the list, so please keep me in copy when
> answering.
>
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-10 21:28 ` Qu Wenruo
@ 2024-09-10 21:37   ` Qu Wenruo
  2024-09-11 19:55     ` Archange
  0 siblings, 1 reply; 24+ messages in thread
From: Qu Wenruo @ 2024-09-10 21:37 UTC (permalink / raw)
  To: Archange, linux-btrfs



在 2024/9/11 06:58, Qu Wenruo 写道:
>
>
> 在 2024/9/11 06:35, Archange 写道:
>> Hi there,
>>
>> Since today, my system started randomly becoming read-only. At that
>> point I can still run dmesg in an open terminal, so I’ve seen it was
>> related to a btrfs error, but did not try anything since I could not
>> open a web browser anymore. But I’ve seen the error to be “BTRFS
>> critical” and related to a “corrupt leaf”.
>>
>> I’ve tried to run `btrfs scrub` on the device after rebooting, and in
>> fact it aborted almost right away triggering the same error in dmesg
>> (but not turning the system read-only, so I can copy paste it here):
>>
>> [  365.268769] BTRFS info (device dm-0): scrub: started on devid 1
>> [  385.788000] page: refcount:3 mapcount:0 mapping:00000000d0054cae
>> index:0x9678888 pfn:0x11ce15
>> [  385.788015] memcg:ffff9fc94db8f000
>> [  385.788021] aops:btree_aops [btrfs] ino:1
>> [  385.788235] flags:
>> 0x2ffffa000004020(lru|private|node=0|zone=2|lastcpupid=0x1ffff)
>> [  385.788248] raw: 02ffffa000004020 ffffea9a8574ff88 ffffea9a847385c8
>> ffff9fc95b8365b0
>> [  385.788255] raw: 0000000009678888 ffff9fc9ae554000 00000003ffffffff
>> ffff9fc94db8f000
>> [  385.788259] page dumped because: eb page dump
>> [  385.788264] BTRFS critical (device dm-0): corrupt leaf:
>> block=646267305984 slot=92 extent bytenr=1182031872 len=106496 invalid
>> data ref objectid value 257
>
> Full dmesg please.
>
> Normally it should dump the full content of the tree block, to help
> debugging the problem.

Nevermind, the code doesn't dump the full leaf for debug anyway.

In that case please dump that corrupted leaf by:

  # btrfs ins dump-tree -b 1182031872 /dev/dm-0

Thanks,
Qu
>
> Thanks,
> Qu
>> [  385.788283] BTRFS error (device dm-0): read time tree block
>> corruption detected on logical 646267305984 mirror 1
>> [  385.796803] BTRFS info (device dm-0): scrub: not finished on devid 1
>> with status: -5
>>
>> According to https://btrfs.readthedocs.io/en/latest/Tree-checker.html
>> this is not really expected, and the last paragraph says to report
>> troubles here. So here I am, in the search for advice about this error
>> (web searches returned nothing with this specific error except the
>> commit/ml messages that added the code for it) and how to fix my random
>> lockups.
>>
>> Regards.
>>
>> P.S.: I’m not subscribed to the list, so please keep me in copy when
>> answering.
>>
>>
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-10 21:37   ` Qu Wenruo
@ 2024-09-11 19:55     ` Archange
  2024-09-11 20:54       ` Qu Wenruo
  0 siblings, 1 reply; 24+ messages in thread
From: Archange @ 2024-09-11 19:55 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs

Le 11/09/2024 à 01:37, Qu Wenruo a écrit :
> 在 2024/9/11 06:58, Qu Wenruo 写道:
>> 在 2024/9/11 06:35, Archange 写道:
>>> Hi there,
>>>
>>> Since today, my system started randomly becoming read-only. At that
>>> point I can still run dmesg in an open terminal, so I’ve seen it was
>>> related to a btrfs error, but did not try anything since I could not
>>> open a web browser anymore. But I’ve seen the error to be “BTRFS
>>> critical” and related to a “corrupt leaf”.
>>>
>>> I’ve tried to run `btrfs scrub` on the device after rebooting, and in
>>> fact it aborted almost right away triggering the same error in dmesg
>>> (but not turning the system read-only, so I can copy paste it here):
>>>
>>> [  365.268769] BTRFS info (device dm-0): scrub: started on devid 1
>>> [  385.788000] page: refcount:3 mapcount:0 mapping:00000000d0054cae
>>> index:0x9678888 pfn:0x11ce15
>>> [  385.788015] memcg:ffff9fc94db8f000
>>> [  385.788021] aops:btree_aops [btrfs] ino:1
>>> [  385.788235] flags:
>>> 0x2ffffa000004020(lru|private|node=0|zone=2|lastcpupid=0x1ffff)
>>> [  385.788248] raw: 02ffffa000004020 ffffea9a8574ff88 ffffea9a847385c8
>>> ffff9fc95b8365b0
>>> [  385.788255] raw: 0000000009678888 ffff9fc9ae554000 00000003ffffffff
>>> ffff9fc94db8f000
>>> [  385.788259] page dumped because: eb page dump
>>> [  385.788264] BTRFS critical (device dm-0): corrupt leaf:
>>> block=646267305984 slot=92 extent bytenr=1182031872 len=106496 invalid
>>> data ref objectid value 257
>>
>> Full dmesg please.
>>
>> Normally it should dump the full content of the tree block, to help
>> debugging the problem.
>
> Nevermind, the code doesn't dump the full leaf for debug anyway.
>
> In that case please dump that corrupted leaf by:
>
>  # btrfs ins dump-tree -b 1182031872 /dev/dm-0

Sorry for the delay, in the meantime my computer went unbootable: the 
initramfs went missing, then some systemd files… Last time such things 
happened was when my btrfs went out of free space, but there is plenty 
currently, so I guess this is related to this other kind of btrfs issue 
I’m facing. Now everything seems in order and I’m back to my emails.

Here is the output of the asked command:

# btrfs ins dump-tree -b 1182031872 /dev/dm-0
btrfs-progs v6.10.1
checksum verify failed on 1182031872 wanted 0x00000000 found 0x21b9544e
ERROR: failed to read tree block 1182031872

Some additional informations:

1. I was able to save the log the next time it went read-only, it is a 
bit different:

[ 4588.750188] page: refcount:4 mapcount:0 mapping:00000000d0054cae 
index:0x967c1f0 pfn:0x35077d
[ 4588.750203] memcg:ffff9fc9400ae000
[ 4588.750208] aops:btree_aops [btrfs] ino:1
[ 4588.750407] flags: 
0x2ffff8000004000(private|node=0|zone=2|lastcpupid=0x1ffff)
[ 4588.750419] raw: 02ffff8000004000 0000000000000000 dead000000000122 
ffff9fc95b8365b0
[ 4588.750425] raw: 000000000967c1f0 ffff9fcafdcc2690 00000004ffffffff 
ffff9fc9400ae000
[ 4588.750428] page dumped because: eb page dump
[ 4588.750433] BTRFS critical (device dm-0): corrupt leaf: 
block=646327500800 slot=105 extent bytenr=11287011328 len=114688 invalid 
data ref objectid value 258
[ 4588.750451] BTRFS error (device dm-0): read time tree block 
corruption detected on logical 646327500800 mirror 1
[ 4588.750524] BTRFS error (device dm-0): failed to run delayed ref for 
logical 11285897216 num_bytes 36864 type 178 action 1 ref_mod 1: -5
[ 4588.750542] BTRFS error (device dm-0 state A): Transaction aborted 
(error -5)
[ 4588.750549] BTRFS: error (device dm-0 state A) in 
btrfs_run_delayed_refs:2207: errno=-5 IO failure
[ 4588.750559] BTRFS info (device dm-0 state EA): forced readonly

2. I’ve thus decided to run

# btrfs ins dump-tree -b 11287011328 /dev/dm-0
btrfs-progs v6.10.1
checksum verify failed on 11287011328 wanted 0x00000000 found 0x2c7de3ac
ERROR: failed to read tree block 11287011328

and

# btrfs ins dump-tree -b 11285897216 /dev/dm-0
btrfs-progs v6.10.1
checksum verify failed on 11285897216 wanted 0x28b52ffd found 0xa166b670
ERROR: failed to read tree block 11285897216

3. There is some information on kernel loading

[   12.793025] Btrfs loaded, zoned=yes, fsverity=yes
[   12.886212] BTRFS: device label root devid 1 transid 6065022 
/dev/mapper/root (254:0) scanned by mount (252)
[   12.887845] BTRFS info (device dm-0): first mount of filesystem 
e6614f01-6f56-4776-8b0a-c260089c35e7
[   12.887874] BTRFS info (device dm-0): using crc32c (crc32c-intel) 
checksum algorithm
[   12.887885] BTRFS info (device dm-0): disk space caching is enabled
[   12.907001] BTRFS warning (device dm-0): devid 1 physical 0 len 
4194304 inside the reserved space
[   12.910034] BTRFS info (device dm-0): bdev /dev/mapper/root errs: wr 
0, rd 0, flush 0, corrupt 4, gen 0

(Especially the “corrupt 4” I guess, but the warning above might also be 
relevant?)

4. I’ve run a full scrub while the disk was mounted on another system, 
which returned no error.

5. I’ve also run a check from that system while the fs was not mounted:

# btrfs check /dev/mapper/rootext
Opening filesystem to check...
Checking filesystem on /dev/mapper/rootext
UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 480039342080 bytes used, no error found
total csum bytes: 466668868
total tree bytes: 1618149376
total fs tree bytes: 898007040
total extent tree bytes: 138395648
btree space waste bytes: 331985228
file data blocks allocated: 517562126336
  referenced 492533391360

Waiting for further debugging instructions.

Thanks,
Archange


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-11 19:55     ` Archange
@ 2024-09-11 20:54       ` Qu Wenruo
  2024-09-11 21:20         ` Archange
  0 siblings, 1 reply; 24+ messages in thread
From: Qu Wenruo @ 2024-09-11 20:54 UTC (permalink / raw)
  To: Archange, linux-btrfs



在 2024/9/12 05:25, Archange 写道:
> Le 11/09/2024 à 01:37, Qu Wenruo a écrit :
>> 在 2024/9/11 06:58, Qu Wenruo 写道:
>>> 在 2024/9/11 06:35, Archange 写道:
>>>> Hi there,
>>>>
>>>> Since today, my system started randomly becoming read-only. At that
>>>> point I can still run dmesg in an open terminal, so I’ve seen it was
>>>> related to a btrfs error, but did not try anything since I could not
>>>> open a web browser anymore. But I’ve seen the error to be “BTRFS
>>>> critical” and related to a “corrupt leaf”.
>>>>
>>>> I’ve tried to run `btrfs scrub` on the device after rebooting, and in
>>>> fact it aborted almost right away triggering the same error in dmesg
>>>> (but not turning the system read-only, so I can copy paste it here):
>>>>
>>>> [  365.268769] BTRFS info (device dm-0): scrub: started on devid 1
>>>> [  385.788000] page: refcount:3 mapcount:0 mapping:00000000d0054cae
>>>> index:0x9678888 pfn:0x11ce15
>>>> [  385.788015] memcg:ffff9fc94db8f000
>>>> [  385.788021] aops:btree_aops [btrfs] ino:1
>>>> [  385.788235] flags:
>>>> 0x2ffffa000004020(lru|private|node=0|zone=2|lastcpupid=0x1ffff)
>>>> [  385.788248] raw: 02ffffa000004020 ffffea9a8574ff88 ffffea9a847385c8
>>>> ffff9fc95b8365b0
>>>> [  385.788255] raw: 0000000009678888 ffff9fc9ae554000 00000003ffffffff
>>>> ffff9fc94db8f000
>>>> [  385.788259] page dumped because: eb page dump
>>>> [  385.788264] BTRFS critical (device dm-0): corrupt leaf:
>>>> block=646267305984 slot=92 extent bytenr=1182031872 len=106496 invalid
>>>> data ref objectid value 257
>>>
>>> Full dmesg please.
>>>
>>> Normally it should dump the full content of the tree block, to help
>>> debugging the problem.
>>
>> Nevermind, the code doesn't dump the full leaf for debug anyway.
>>
>> In that case please dump that corrupted leaf by:
>>
>>  # btrfs ins dump-tree -b 1182031872 /dev/dm-0
>
> Sorry for the delay, in the meantime my computer went unbootable: the
> initramfs went missing, then some systemd files… Last time such things
> happened was when my btrfs went out of free space, but there is plenty
> currently, so I guess this is related to this other kind of btrfs issue
> I’m facing. Now everything seems in order and I’m back to my emails.
>
> Here is the output of the asked command:
>
> # btrfs ins dump-tree -b 1182031872 /dev/dm-0
> btrfs-progs v6.10.1
> checksum verify failed on 1182031872 wanted 0x00000000 found 0x21b9544e
> ERROR: failed to read tree block 1182031872
>
> Some additional informations:
>
> 1. I was able to save the log the next time it went read-only, it is a
> bit different:
>
> [ 4588.750188] page: refcount:4 mapcount:0 mapping:00000000d0054cae
> index:0x967c1f0 pfn:0x35077d
> [ 4588.750203] memcg:ffff9fc9400ae000
> [ 4588.750208] aops:btree_aops [btrfs] ino:1
> [ 4588.750407] flags:
> 0x2ffff8000004000(private|node=0|zone=2|lastcpupid=0x1ffff)
> [ 4588.750419] raw: 02ffff8000004000 0000000000000000 dead000000000122
> ffff9fc95b8365b0
> [ 4588.750425] raw: 000000000967c1f0 ffff9fcafdcc2690 00000004ffffffff
> ffff9fc9400ae000
> [ 4588.750428] page dumped because: eb page dump
> [ 4588.750433] BTRFS critical (device dm-0): corrupt leaf:
> block=646327500800 slot=105 extent bytenr=11287011328 len=114688 invalid
> data ref objectid value 258
> [ 4588.750451] BTRFS error (device dm-0): read time tree block
> corruption detected on logical 646327500800 mirror 1
> [ 4588.750524] BTRFS error (device dm-0): failed to run delayed ref for
> logical 11285897216 num_bytes 36864 type 178 action 1 ref_mod 1: -5
> [ 4588.750542] BTRFS error (device dm-0 state A): Transaction aborted
> (error -5)
> [ 4588.750549] BTRFS: error (device dm-0 state A) in
> btrfs_run_delayed_refs:2207: errno=-5 IO failure
> [ 4588.750559] BTRFS info (device dm-0 state EA): forced readonly
>
> 2. I’ve thus decided to run
>
> # btrfs ins dump-tree -b 11287011328 /dev/dm-0
> btrfs-progs v6.10.1
> checksum verify failed on 11287011328 wanted 0x00000000 found 0x2c7de3ac
> ERROR: failed to read tree block 11287011328
>
> and
>
> # btrfs ins dump-tree -b 11285897216 /dev/dm-0
> btrfs-progs v6.10.1
> checksum verify failed on 11285897216 wanted 0x28b52ffd found 0xa166b670
> ERROR: failed to read tree block 11285897216
>
> 3. There is some information on kernel loading
>
> [   12.793025] Btrfs loaded, zoned=yes, fsverity=yes
> [   12.886212] BTRFS: device label root devid 1 transid 6065022
> /dev/mapper/root (254:0) scanned by mount (252)
> [   12.887845] BTRFS info (device dm-0): first mount of filesystem
> e6614f01-6f56-4776-8b0a-c260089c35e7
> [   12.887874] BTRFS info (device dm-0): using crc32c (crc32c-intel)
> checksum algorithm
> [   12.887885] BTRFS info (device dm-0): disk space caching is enabled
> [   12.907001] BTRFS warning (device dm-0): devid 1 physical 0 len
> 4194304 inside the reserved space
> [   12.910034] BTRFS info (device dm-0): bdev /dev/mapper/root errs: wr
> 0, rd 0, flush 0, corrupt 4, gen 0
>
> (Especially the “corrupt 4” I guess, but the warning above might also be
> relevant?)
>
> 4. I’ve run a full scrub while the disk was mounted on another system,
> which returned no error.
>
> 5. I’ve also run a check from that system while the fs was not mounted:
>
> # btrfs check /dev/mapper/rootext
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/rootext
> UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 480039342080 bytes used, no error found
> total csum bytes: 466668868
> total tree bytes: 1618149376
> total fs tree bytes: 898007040
> total extent tree bytes: 138395648
> btree space waste bytes: 331985228
> file data blocks allocated: 517562126336
>   referenced 492533391360
>
> Waiting for further debugging instructions.

This looks exactly like another report that is caused by inode cache.

So in that case, mind to try the following commands?

# btrfs rescue zero-log <device>
# btrfs rescue clear-inode-cache <device>

Then try mounting the fs.

Thanks,
Qu
>
> Thanks,
> Archange
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-11 20:54       ` Qu Wenruo
@ 2024-09-11 21:20         ` Archange
  2024-09-11 21:23           ` Qu Wenruo
  0 siblings, 1 reply; 24+ messages in thread
From: Archange @ 2024-09-11 21:20 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs

Le 12/09/2024 à 00:54, Qu Wenruo a écrit :
> 在 2024/9/12 05:25, Archange 写道:
>> Le 11/09/2024 à 01:37, Qu Wenruo a écrit :
>>> 在 2024/9/11 06:58, Qu Wenruo 写道:
>>>> 在 2024/9/11 06:35, Archange 写道:
>>>> […]
>
> This looks exactly like another report that is caused by inode cache.
>
> So in that case, mind to try the following commands?
>
> # btrfs rescue zero-log <device>
> # btrfs rescue clear-inode-cache <device>

I supposed the second command was meant to be `clear-ino-cache` (I 
remember having to remove `inode_cache` from mount options some time ago 
as it prevented booting, I had then discovered the related feature had 
been deprecated and removed).

Here are the command outputs:

# btrfs rescue zero-log /dev/mapper/rootext
Clearing log on /dev/mapper/rootext, previous log_root 0, level 0

# btrfs rescue clear-ino-cache /dev/mapper/rootext
Successfully cleaned up ino cache for root id: 5
Successfully cleaned up ino cache for root id: 257
Successfully cleaned up ino cache for root id: 258
corrupt node: root=7 block=647369064448 slot=0, invalid level for leaf, 
have 1 expect 0
node 647369064448 level 1 items 252 free space 241 generation 6065173 
owner CSUM_TREE
node 647369064448 flags 0x1(WRITTEN) backref revision 1
fs uuid e6614f01-6f56-4776-8b0a-c260089c35e7
chunk uuid f665f535-4cfd-49e0-8be9-7f94bf59b75d
     key (EXTENT_CSUM EXTENT_CSUM 3714473984) block 677126111232 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 3720785920) block 646338183168 gen 6065030
     key (EXTENT_CSUM EXTENT_CSUM 3728965632) block 677126176768 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 3737710592) block 676941266944 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 3747074048) block 677126209536 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 3754676224) block 676863508480 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 3764305920) block 676863524864 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 3772821504) block 646742212608 gen 6065087
     key (EXTENT_CSUM EXTENT_CSUM 3780558848) block 676863557632 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 3792121856) block 646843056128 gen 6065107
     key (EXTENT_CSUM EXTENT_CSUM 3799425024) block 677126258688 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 3808354304) block 676888068096 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 3822206976) block 647458684928 gen 5744378
     key (EXTENT_CSUM EXTENT_CSUM 3838750720) block 648070070272 gen 1381537
     key (EXTENT_CSUM EXTENT_CSUM 3855294464) block 648070184960 gen 1381537
     key (EXTENT_CSUM EXTENT_CSUM 3871838208) block 647481884672 gen 1381531
     key (EXTENT_CSUM EXTENT_CSUM 3888381952) block 676963188736 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 3899301888) block 676943937536 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 3908460544) block 647367393280 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 3916058624) block 676909989888 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 3922890752) block 676715134976 gen 6064994
     key (EXTENT_CSUM EXTENT_CSUM 3932336128) block 676941283328 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 3943030784) block 676910022656 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 3952046080) block 676854464512 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 3959169024) block 676910071808 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 3966623744) block 676933074944 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 3975307264) block 647369162752 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 3983073280) block 676948852736 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 3990368256) block 676941299712 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 3998121984) block 676948869120 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4009500672) block 676910170112 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4017614848) block 646264160256 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 4025708544) block 676843307008 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 4033413120) block 646515097600 gen 6065053
     key (EXTENT_CSUM EXTENT_CSUM 4041183232) block 677125914624 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4047310848) block 647409041408 gen 6065170
     key (EXTENT_CSUM EXTENT_CSUM 4054364160) block 676723982336 gen 6064994
     key (EXTENT_CSUM EXTENT_CSUM 4062892032) block 677107367936 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4072263680) block 677107417088 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4081119232) block 676844634112 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 4089425920) block 676910350336 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4097601536) block 677107302400 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4105961472) block 676910432256 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4114223104) block 676963237888 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4121116672) block 646473465856 gen 6065044
     key (EXTENT_CSUM EXTENT_CSUM 4130504704) block 677107351552 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4139896832) block 676933156864 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4150054912) block 677107384320 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4159819776) block 677107400704 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4166836224) block 677125980160 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4178882560) block 647369195520 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 4185473024) block 676715347968 gen 6064994
     key (EXTENT_CSUM EXTENT_CSUM 4193738752) block 676715364352 gen 6064994
     key (EXTENT_CSUM EXTENT_CSUM 4203532288) block 676724146176 gen 6064994
     key (EXTENT_CSUM EXTENT_CSUM 4211822592) block 676910612480 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4220416000) block 676754997248 gen 6064994
     key (EXTENT_CSUM EXTENT_CSUM 4234862592) block 676855054336 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 4247011328) block 647369097216 gen 6065173
     key (EXTENT_CSUM EXTENT_CSUM 4257337344) block 647369228288 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 4270247936) block 646992494592 gen 6065114
     key (EXTENT_CSUM EXTENT_CSUM 4284358656) block 676910743552 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4295380992) block 646842253312 gen 6065107
     key (EXTENT_CSUM EXTENT_CSUM 4305231872) block 677126012928 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4316020736) block 676963450880 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4325187584) block 677314379776 gen 6064991
     key (EXTENT_CSUM EXTENT_CSUM 4341161984) block 646746472448 gen 6065087
     key (EXTENT_CSUM EXTENT_CSUM 4345565184) block 676941332480 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4354514944) block 646342459392 gen 6065029
     key (EXTENT_CSUM EXTENT_CSUM 4361768960) block 676963467264 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4369428480) block 676967317504 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4376809472) block 676967350272 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4385017856) block 676963483648 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4392333312) block 676941365248 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4400939008) block 676941381632 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4408795136) block 676941496320 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4416131072) block 646359908352 gen 6065030
     key (EXTENT_CSUM EXTENT_CSUM 4424679424) block 676715528192 gen 6064994
     key (EXTENT_CSUM EXTENT_CSUM 4433436672) block 647351910400 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 4442791936) block 647369261056 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 4451270656) block 646407585792 gen 6065032
     key (EXTENT_CSUM EXTENT_CSUM 4459581440) block 677126569984 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4466970624) block 648044281856 gen 6065171
     key (EXTENT_CSUM EXTENT_CSUM 4473278464) block 647373193216 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 4481544192) block 676967743488 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4489117696) block 676967792640 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4501204992) block 647357153280 gen 6065164
     key (EXTENT_CSUM EXTENT_CSUM 4508569600) block 676967907328 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4513787904) block 676967940096 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4519464960) block 676967972864 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4528398336) block 676968005632 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4537438208) block 677148459008 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 4543373312) block 676968087552 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4553338880) block 676963303424 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4564582400) block 676968169472 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4574752768) block 647373455360 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 4584710144) block 648044314624 gen 6065171
     key (EXTENT_CSUM EXTENT_CSUM 4589998080) block 676932272128 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4596559872) block 676693639168 gen 6064993
     key (EXTENT_CSUM EXTENT_CSUM 4603473920) block 676933222400 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4613885952) block 676948901888 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4624228352) block 676963532800 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4633935872) block 676911284224 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4644610048) block 646338265088 gen 6065030
     key (EXTENT_CSUM EXTENT_CSUM 4653531136) block 646359941120 gen 6065030
     key (EXTENT_CSUM EXTENT_CSUM 4664819712) block 676963565568 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4676636672) block 647386398720 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 4686704640) block 646343639040 gen 6065029
     key (EXTENT_CSUM EXTENT_CSUM 4698435584) block 676963598336 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4713156608) block 647392772096 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 4722995200) block 646264193024 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 4731748352) block 676911611904 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4738572288) block 676933238784 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4748193792) block 676942348288 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4753985536) block 676942364672 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4761124864) block 676911644672 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4770844672) block 646845300736 gen 6065107
     key (EXTENT_CSUM EXTENT_CSUM 4780126208) block 676911677440 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4787306496) block 676942397440 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4798906368) block 676844863488 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 4807143424) block 676963631104 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4817457152) block 676546248704 gen 6064992
     key (EXTENT_CSUM EXTENT_CSUM 4826439680) block 676693966848 gen 6064993
     key (EXTENT_CSUM EXTENT_CSUM 4840206336) block 676911742976 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4851683328) block 647302086656 gen 6065158
     key (EXTENT_CSUM EXTENT_CSUM 4857987072) block 646943653888 gen 6065108
     key (EXTENT_CSUM EXTENT_CSUM 4866301952) block 676933271552 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4875943936) block 676911824896 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4888219648) block 676942413824 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4898308096) block 676911857664 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4912943104) block 647369392128 gen 6065173
     key (EXTENT_CSUM EXTENT_CSUM 4923723776) block 676968497152 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4934074368) block 676632723456 gen 6064993
     key (EXTENT_CSUM EXTENT_CSUM 4948840448) block 676911890432 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 4963790848) block 676968579072 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4975116288) block 676968644608 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 4984283136) block 647080198144 gen 6065133
     key (EXTENT_CSUM EXTENT_CSUM 4994596864) block 646304825344 gen 6065025
     key (EXTENT_CSUM EXTENT_CSUM 5002313728) block 646324338688 gen 6065026
     key (EXTENT_CSUM EXTENT_CSUM 5008089088) block 676963336192 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 5018886144) block 647399227392 gen 6065170
     key (EXTENT_CSUM EXTENT_CSUM 5028757504) block 647385743360 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 5037797376) block 647385858048 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 5049577472) block 647386382336 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 5057458176) block 676949278720 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5072642048) block 676963352576 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 5085605888) block 676911988736 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5093662720) block 676963368960 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 5105045504) block 647178960896 gen 6065146
     key (EXTENT_CSUM EXTENT_CSUM 5116792832) block 647383547904 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 5130141696) block 676912218112 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5139943424) block 676912267264 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5148413952) block 676912300032 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5156237312) block 676942430208 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5165236224) block 676942446592 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5177163776) block 646300827648 gen 6065025
     key (EXTENT_CSUM EXTENT_CSUM 5187325952) block 676749754368 gen 6064994
     key (EXTENT_CSUM EXTENT_CSUM 5201133568) block 676942479360 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5213573120) block 676657332224 gen 6064993
     key (EXTENT_CSUM EXTENT_CSUM 5228421120) block 676845240320 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 5236654080) block 676942495744 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5246169088) block 676942512128 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5254701056) block 646343688192 gen 6065029
     key (EXTENT_CSUM EXTENT_CSUM 5263523840) block 676942528512 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5271793664) block 676932288512 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5284384768) block 676933320704 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5298110464) block 676759650304 gen 6064994
     key (EXTENT_CSUM EXTENT_CSUM 5304373248) block 676942544896 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5314695168) block 646338510848 gen 6065030
     key (EXTENT_CSUM EXTENT_CSUM 5323919360) block 676942561280 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5336117248) block 676912496640 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5345054720) block 676963680256 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 5350932480) block 676863836160 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 5363728384) block 676657709056 gen 6064993
     key (EXTENT_CSUM EXTENT_CSUM 5369249792) block 676912594944 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5379948544) block 646264209408 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5392506880) block 676978917376 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 5402152960) block 646948175872 gen 6065108
     key (EXTENT_CSUM EXTENT_CSUM 5412794368) block 646948208640 gen 6065108
     key (EXTENT_CSUM EXTENT_CSUM 5422727168) block 676694409216 gen 6064993
     key (EXTENT_CSUM EXTENT_CSUM 5436506112) block 646264225792 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5446492160) block 646948241408 gen 6065108
     key (EXTENT_CSUM EXTENT_CSUM 5457653760) block 646264242176 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5469483008) block 646264258560 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5479792640) block 646264143872 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5491097600) block 647262306304 gen 6065152
     key (EXTENT_CSUM EXTENT_CSUM 5504716800) block 646264438784 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5516099584) block 646264504320 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5526601728) block 676942594048 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5538586624) block 646716211200 gen 6065083
     key (EXTENT_CSUM EXTENT_CSUM 5550575616) block 677029806080 gen 6065001
     key (EXTENT_CSUM EXTENT_CSUM 5560479744) block 647350697984 gen 6065163
     key (EXTENT_CSUM EXTENT_CSUM 5570015232) block 647383580672 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 5578293248) block 646264487936 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5590724608) block 676863950848 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 5603069952) block 646263504896 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5615501312) block 676913020928 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5626724352) block 676913102848 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5639708672) block 677029822464 gen 6065001
     key (EXTENT_CSUM EXTENT_CSUM 5653757952) block 647216037888 gen 6065147
     key (EXTENT_CSUM EXTENT_CSUM 5660786688) block 647207895040 gen 6065147
     key (EXTENT_CSUM EXTENT_CSUM 5668933632) block 676715937792 gen 6064994
     key (EXTENT_CSUM EXTENT_CSUM 5677277184) block 647383613440 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 5687111680) block 646264520704 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5694754816) block 647384301568 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 5704658944) block 647417839616 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 5714460672) block 647417872384 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 5725593600) block 647114440704 gen 6065138
     key (EXTENT_CSUM EXTENT_CSUM 5733572608) block 647409156096 gen 6065170
     key (EXTENT_CSUM EXTENT_CSUM 5743837184) block 677138202624 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 5755424768) block 646943686656 gen 6065108
     key (EXTENT_CSUM EXTENT_CSUM 5766553600) block 676963794944 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 5778427904) block 646264651776 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5784666112) block 676913709056 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5792251904) block 647225573376 gen 6065149
     key (EXTENT_CSUM EXTENT_CSUM 5800312832) block 676981161984 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 5811654656) block 676981080064 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 5822210048) block 646336544768 gen 6065030
     key (EXTENT_CSUM EXTENT_CSUM 5831127040) block 677138219008 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 5843570688) block 646264668160 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5854285824) block 646943703040 gen 6065108
     key (EXTENT_CSUM EXTENT_CSUM 5862305792) block 647229030400 gen 6065148
     key (EXTENT_CSUM EXTENT_CSUM 5872865280) block 646834454528 gen 6065104
     key (EXTENT_CSUM EXTENT_CSUM 5883650048) block 646264717312 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5895880704) block 677151473664 gen 6065002
     key (EXTENT_CSUM EXTENT_CSUM 5904617472) block 646827114496 gen 6065100
     key (EXTENT_CSUM EXTENT_CSUM 5915639808) block 677029265408 gen 6065001
     key (EXTENT_CSUM EXTENT_CSUM 5926555648) block 647290617856 gen 6065156
     key (EXTENT_CSUM EXTENT_CSUM 5936168960) block 646264799232 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 5947744256) block 676963827712 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 5956050944) block 676914577408 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 5967777792) block 676963844096 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 5978038272) block 646653607936 gen 6065072
     key (EXTENT_CSUM EXTENT_CSUM 5989384192) block 646578962432 gen 6065059
     key (EXTENT_CSUM EXTENT_CSUM 5997871104) block 647357890560 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 6009282560) block 646263701504 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 6021009408) block 646264930304 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 6032379904) block 646264963072 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 6043267072) block 646265044992 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 6054461440) block 647357300736 gen 6065164
     key (EXTENT_CSUM EXTENT_CSUM 6063206400) block 647417905152 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 6072909824) block 647385645056 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 6084259840) block 647417987072 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 6096568320) block 646263603200 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 6107648000) block 676963958784 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 6116904960) block 646993084416 gen 6065111
     key (EXTENT_CSUM EXTENT_CSUM 6128369664) block 646265094144 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 6138900480) block 647353303040 gen 6065172
     key (EXTENT_CSUM EXTENT_CSUM 6146465792) block 676849172480 gen 6064995
     key (EXTENT_CSUM EXTENT_CSUM 6156619776) block 646265323520 gen 6065024
     key (EXTENT_CSUM EXTENT_CSUM 6168879104) block 676992876544 gen 6064998
     key (EXTENT_CSUM EXTENT_CSUM 6179160064) block 676915036160 gen 6064996
     key (EXTENT_CSUM EXTENT_CSUM 6192357376) block 646396493824 gen 6065032
ERROR: failed to clear ino cache: Input/output error

Not sure what to do with this.

dmesg from subsequent mounting on the rescue system:

[  365.027631] BTRFS: device label root devid 1 transid 6065178 
/dev/mapper/rootext (254:1) scanned by mount (1425)
[  365.028812] BTRFS info (device dm-1): first mount of filesystem 
e6614f01-6f56-4776-8b0a-c260089c35e7
[  365.028823] BTRFS info (device dm-1): using crc32c (crc32c-intel) 
checksum algorithm
[  365.028840] BTRFS info (device dm-1): disk space caching is enabled
[  365.039509] BTRFS warning (device dm-1): devid 1 physical 0 len 
4194304 inside the reserved space
[  365.041044] BTRFS info (device dm-1): bdev /dev/mapper/rootext errs: 
wr 0, rd 0, flush 0, corrupt 4, gen 0
[  365.424564] BTRFS info (device dm-1): checking UUID tree


dmesg from booting on the system:

[   12.697569] Btrfs loaded, zoned=yes, fsverity=yes
[   12.793884] BTRFS: device label root devid 1 transid 6065180 
/dev/mapper/root (254:0) scanned by mount (249)
[   12.795330] BTRFS info (device dm-0): first mount of filesystem 
e6614f01-6f56-4776-8b0a-c260089c35e7
[   12.795358] BTRFS info (device dm-0): using crc32c (crc32c-intel) 
checksum algorithm
[   12.795369] BTRFS info (device dm-0): disk space caching is enabled
[   12.809564] BTRFS warning (device dm-0): devid 1 physical 0 len 
4194304 inside the reserved space
[   12.813203] BTRFS info (device dm-0): bdev /dev/mapper/root errs: wr 
0, rd 0, flush 0, corrupt 4, gen 0
[   16.026687] BTRFS info (device dm-0 state M): use zstd compression, 
level 3
[   16.842214] BTRFS info: devid 1 device path /dev/mapper/root changed 
to /dev/dm-0 scanned by (udev-worker) (364)
[   16.843857] BTRFS info: devid 1 device path /dev/dm-0 changed to 
/dev/mapper/root scanned by (udev-worker) (364)
[   19.888321] BTRFS warning (device dm-0): block group 1094713344 has 
wrong amount of free space
[   19.888325] BTRFS warning (device dm-0): failed to load free space 
cache for block group 1094713344, rebuilding it now
[   19.968700] BTRFS warning (device dm-0): block group 10758389760 has 
wrong amount of free space
[   19.968705] BTRFS warning (device dm-0): failed to load free space 
cache for block group 10758389760, rebuilding it now

That being said no critical error has been encountered since I’ve had to 
repair my boot, but I’ve not tried to scrub from the running system 
again, should I do that (as it used to trigger the error before)?

Regards,
Archange


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-11 21:20         ` Archange
@ 2024-09-11 21:23           ` Qu Wenruo
  2024-09-11 21:42             ` Qu Wenruo
  2024-09-11 22:05             ` Archange
  0 siblings, 2 replies; 24+ messages in thread
From: Qu Wenruo @ 2024-09-11 21:23 UTC (permalink / raw)
  To: Archange, linux-btrfs



在 2024/9/12 06:50, Archange 写道:
> Le 12/09/2024 à 00:54, Qu Wenruo a écrit :
>> 在 2024/9/12 05:25, Archange 写道:
>>> Le 11/09/2024 à 01:37, Qu Wenruo a écrit :
>>>> 在 2024/9/11 06:58, Qu Wenruo 写道:
>>>>> 在 2024/9/11 06:35, Archange 写道:
>>>>> […]
>>
>> This looks exactly like another report that is caused by inode cache.
>>
>> So in that case, mind to try the following commands?
>>
>> # btrfs rescue zero-log <device>
>> # btrfs rescue clear-inode-cache <device>
>
> I supposed the second command was meant to be `clear-ino-cache` (I
> remember having to remove `inode_cache` from mount options some time ago
> as it prevented booting, I had then discovered the related feature had
> been deprecated and removed).
>
> Here are the command outputs:
>
> # btrfs rescue zero-log /dev/mapper/rootext
> Clearing log on /dev/mapper/rootext, previous log_root 0, level 0
>
> # btrfs rescue clear-ino-cache /dev/mapper/rootext
> Successfully cleaned up ino cache for root id: 5
> Successfully cleaned up ino cache for root id: 257
> Successfully cleaned up ino cache for root id: 258
> corrupt node: root=7 block=647369064448 slot=0, invalid level for leaf,
> have 1 expect 0

This is not expected, I guess I have to double check the inode clearing
code to be sure.

> node 647369064448 level 1 items 252 free space 241 generation 6065173
> owner CSUM_TREE
> node 647369064448 flags 0x1(WRITTEN) backref revision 1
> fs uuid e6614f01-6f56-4776-8b0a-c260089c35e7
> chunk uuid f665f535-4cfd-49e0-8be9-7f94bf59b75d
>      key (EXTENT_CSUM EXTENT_CSUM 3714473984) block 677126111232 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 3720785920) block 646338183168 gen
> 6065030
>      key (EXTENT_CSUM EXTENT_CSUM 3728965632) block 677126176768 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 3737710592) block 676941266944 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 3747074048) block 677126209536 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 3754676224) block 676863508480 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 3764305920) block 676863524864 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 3772821504) block 646742212608 gen
> 6065087
>      key (EXTENT_CSUM EXTENT_CSUM 3780558848) block 676863557632 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 3792121856) block 646843056128 gen
> 6065107
>      key (EXTENT_CSUM EXTENT_CSUM 3799425024) block 677126258688 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 3808354304) block 676888068096 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 3822206976) block 647458684928 gen
> 5744378
>      key (EXTENT_CSUM EXTENT_CSUM 3838750720) block 648070070272 gen
> 1381537
>      key (EXTENT_CSUM EXTENT_CSUM 3855294464) block 648070184960 gen
> 1381537
>      key (EXTENT_CSUM EXTENT_CSUM 3871838208) block 647481884672 gen
> 1381531
>      key (EXTENT_CSUM EXTENT_CSUM 3888381952) block 676963188736 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 3899301888) block 676943937536 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 3908460544) block 647367393280 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 3916058624) block 676909989888 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 3922890752) block 676715134976 gen
> 6064994
>      key (EXTENT_CSUM EXTENT_CSUM 3932336128) block 676941283328 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 3943030784) block 676910022656 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 3952046080) block 676854464512 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 3959169024) block 676910071808 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 3966623744) block 676933074944 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 3975307264) block 647369162752 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 3983073280) block 676948852736 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 3990368256) block 676941299712 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 3998121984) block 676948869120 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4009500672) block 676910170112 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4017614848) block 646264160256 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 4025708544) block 676843307008 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 4033413120) block 646515097600 gen
> 6065053
>      key (EXTENT_CSUM EXTENT_CSUM 4041183232) block 677125914624 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4047310848) block 647409041408 gen
> 6065170
>      key (EXTENT_CSUM EXTENT_CSUM 4054364160) block 676723982336 gen
> 6064994
>      key (EXTENT_CSUM EXTENT_CSUM 4062892032) block 677107367936 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4072263680) block 677107417088 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4081119232) block 676844634112 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 4089425920) block 676910350336 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4097601536) block 677107302400 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4105961472) block 676910432256 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4114223104) block 676963237888 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4121116672) block 646473465856 gen
> 6065044
>      key (EXTENT_CSUM EXTENT_CSUM 4130504704) block 677107351552 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4139896832) block 676933156864 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4150054912) block 677107384320 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4159819776) block 677107400704 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4166836224) block 677125980160 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4178882560) block 647369195520 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 4185473024) block 676715347968 gen
> 6064994
>      key (EXTENT_CSUM EXTENT_CSUM 4193738752) block 676715364352 gen
> 6064994
>      key (EXTENT_CSUM EXTENT_CSUM 4203532288) block 676724146176 gen
> 6064994
>      key (EXTENT_CSUM EXTENT_CSUM 4211822592) block 676910612480 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4220416000) block 676754997248 gen
> 6064994
>      key (EXTENT_CSUM EXTENT_CSUM 4234862592) block 676855054336 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 4247011328) block 647369097216 gen
> 6065173
>      key (EXTENT_CSUM EXTENT_CSUM 4257337344) block 647369228288 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 4270247936) block 646992494592 gen
> 6065114
>      key (EXTENT_CSUM EXTENT_CSUM 4284358656) block 676910743552 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4295380992) block 646842253312 gen
> 6065107
>      key (EXTENT_CSUM EXTENT_CSUM 4305231872) block 677126012928 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4316020736) block 676963450880 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4325187584) block 677314379776 gen
> 6064991
>      key (EXTENT_CSUM EXTENT_CSUM 4341161984) block 646746472448 gen
> 6065087
>      key (EXTENT_CSUM EXTENT_CSUM 4345565184) block 676941332480 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4354514944) block 646342459392 gen
> 6065029
>      key (EXTENT_CSUM EXTENT_CSUM 4361768960) block 676963467264 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4369428480) block 676967317504 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4376809472) block 676967350272 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4385017856) block 676963483648 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4392333312) block 676941365248 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4400939008) block 676941381632 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4408795136) block 676941496320 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4416131072) block 646359908352 gen
> 6065030
>      key (EXTENT_CSUM EXTENT_CSUM 4424679424) block 676715528192 gen
> 6064994
>      key (EXTENT_CSUM EXTENT_CSUM 4433436672) block 647351910400 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 4442791936) block 647369261056 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 4451270656) block 646407585792 gen
> 6065032
>      key (EXTENT_CSUM EXTENT_CSUM 4459581440) block 677126569984 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4466970624) block 648044281856 gen
> 6065171
>      key (EXTENT_CSUM EXTENT_CSUM 4473278464) block 647373193216 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 4481544192) block 676967743488 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4489117696) block 676967792640 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4501204992) block 647357153280 gen
> 6065164
>      key (EXTENT_CSUM EXTENT_CSUM 4508569600) block 676967907328 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4513787904) block 676967940096 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4519464960) block 676967972864 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4528398336) block 676968005632 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4537438208) block 677148459008 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 4543373312) block 676968087552 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4553338880) block 676963303424 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4564582400) block 676968169472 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4574752768) block 647373455360 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 4584710144) block 648044314624 gen
> 6065171
>      key (EXTENT_CSUM EXTENT_CSUM 4589998080) block 676932272128 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4596559872) block 676693639168 gen
> 6064993
>      key (EXTENT_CSUM EXTENT_CSUM 4603473920) block 676933222400 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4613885952) block 676948901888 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4624228352) block 676963532800 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4633935872) block 676911284224 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4644610048) block 646338265088 gen
> 6065030
>      key (EXTENT_CSUM EXTENT_CSUM 4653531136) block 646359941120 gen
> 6065030
>      key (EXTENT_CSUM EXTENT_CSUM 4664819712) block 676963565568 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4676636672) block 647386398720 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 4686704640) block 646343639040 gen
> 6065029
>      key (EXTENT_CSUM EXTENT_CSUM 4698435584) block 676963598336 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4713156608) block 647392772096 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 4722995200) block 646264193024 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 4731748352) block 676911611904 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4738572288) block 676933238784 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4748193792) block 676942348288 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4753985536) block 676942364672 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4761124864) block 676911644672 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4770844672) block 646845300736 gen
> 6065107
>      key (EXTENT_CSUM EXTENT_CSUM 4780126208) block 676911677440 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4787306496) block 676942397440 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4798906368) block 676844863488 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 4807143424) block 676963631104 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4817457152) block 676546248704 gen
> 6064992
>      key (EXTENT_CSUM EXTENT_CSUM 4826439680) block 676693966848 gen
> 6064993
>      key (EXTENT_CSUM EXTENT_CSUM 4840206336) block 676911742976 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4851683328) block 647302086656 gen
> 6065158
>      key (EXTENT_CSUM EXTENT_CSUM 4857987072) block 646943653888 gen
> 6065108
>      key (EXTENT_CSUM EXTENT_CSUM 4866301952) block 676933271552 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4875943936) block 676911824896 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4888219648) block 676942413824 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4898308096) block 676911857664 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4912943104) block 647369392128 gen
> 6065173
>      key (EXTENT_CSUM EXTENT_CSUM 4923723776) block 676968497152 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4934074368) block 676632723456 gen
> 6064993
>      key (EXTENT_CSUM EXTENT_CSUM 4948840448) block 676911890432 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 4963790848) block 676968579072 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4975116288) block 676968644608 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 4984283136) block 647080198144 gen
> 6065133
>      key (EXTENT_CSUM EXTENT_CSUM 4994596864) block 646304825344 gen
> 6065025
>      key (EXTENT_CSUM EXTENT_CSUM 5002313728) block 646324338688 gen
> 6065026
>      key (EXTENT_CSUM EXTENT_CSUM 5008089088) block 676963336192 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 5018886144) block 647399227392 gen
> 6065170
>      key (EXTENT_CSUM EXTENT_CSUM 5028757504) block 647385743360 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 5037797376) block 647385858048 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 5049577472) block 647386382336 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 5057458176) block 676949278720 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5072642048) block 676963352576 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 5085605888) block 676911988736 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5093662720) block 676963368960 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 5105045504) block 647178960896 gen
> 6065146
>      key (EXTENT_CSUM EXTENT_CSUM 5116792832) block 647383547904 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 5130141696) block 676912218112 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5139943424) block 676912267264 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5148413952) block 676912300032 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5156237312) block 676942430208 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5165236224) block 676942446592 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5177163776) block 646300827648 gen
> 6065025
>      key (EXTENT_CSUM EXTENT_CSUM 5187325952) block 676749754368 gen
> 6064994
>      key (EXTENT_CSUM EXTENT_CSUM 5201133568) block 676942479360 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5213573120) block 676657332224 gen
> 6064993
>      key (EXTENT_CSUM EXTENT_CSUM 5228421120) block 676845240320 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 5236654080) block 676942495744 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5246169088) block 676942512128 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5254701056) block 646343688192 gen
> 6065029
>      key (EXTENT_CSUM EXTENT_CSUM 5263523840) block 676942528512 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5271793664) block 676932288512 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5284384768) block 676933320704 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5298110464) block 676759650304 gen
> 6064994
>      key (EXTENT_CSUM EXTENT_CSUM 5304373248) block 676942544896 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5314695168) block 646338510848 gen
> 6065030
>      key (EXTENT_CSUM EXTENT_CSUM 5323919360) block 676942561280 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5336117248) block 676912496640 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5345054720) block 676963680256 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 5350932480) block 676863836160 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 5363728384) block 676657709056 gen
> 6064993
>      key (EXTENT_CSUM EXTENT_CSUM 5369249792) block 676912594944 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5379948544) block 646264209408 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5392506880) block 676978917376 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 5402152960) block 646948175872 gen
> 6065108
>      key (EXTENT_CSUM EXTENT_CSUM 5412794368) block 646948208640 gen
> 6065108
>      key (EXTENT_CSUM EXTENT_CSUM 5422727168) block 676694409216 gen
> 6064993
>      key (EXTENT_CSUM EXTENT_CSUM 5436506112) block 646264225792 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5446492160) block 646948241408 gen
> 6065108
>      key (EXTENT_CSUM EXTENT_CSUM 5457653760) block 646264242176 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5469483008) block 646264258560 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5479792640) block 646264143872 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5491097600) block 647262306304 gen
> 6065152
>      key (EXTENT_CSUM EXTENT_CSUM 5504716800) block 646264438784 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5516099584) block 646264504320 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5526601728) block 676942594048 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5538586624) block 646716211200 gen
> 6065083
>      key (EXTENT_CSUM EXTENT_CSUM 5550575616) block 677029806080 gen
> 6065001
>      key (EXTENT_CSUM EXTENT_CSUM 5560479744) block 647350697984 gen
> 6065163
>      key (EXTENT_CSUM EXTENT_CSUM 5570015232) block 647383580672 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 5578293248) block 646264487936 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5590724608) block 676863950848 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 5603069952) block 646263504896 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5615501312) block 676913020928 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5626724352) block 676913102848 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5639708672) block 677029822464 gen
> 6065001
>      key (EXTENT_CSUM EXTENT_CSUM 5653757952) block 647216037888 gen
> 6065147
>      key (EXTENT_CSUM EXTENT_CSUM 5660786688) block 647207895040 gen
> 6065147
>      key (EXTENT_CSUM EXTENT_CSUM 5668933632) block 676715937792 gen
> 6064994
>      key (EXTENT_CSUM EXTENT_CSUM 5677277184) block 647383613440 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 5687111680) block 646264520704 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5694754816) block 647384301568 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 5704658944) block 647417839616 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 5714460672) block 647417872384 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 5725593600) block 647114440704 gen
> 6065138
>      key (EXTENT_CSUM EXTENT_CSUM 5733572608) block 647409156096 gen
> 6065170
>      key (EXTENT_CSUM EXTENT_CSUM 5743837184) block 677138202624 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 5755424768) block 646943686656 gen
> 6065108
>      key (EXTENT_CSUM EXTENT_CSUM 5766553600) block 676963794944 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 5778427904) block 646264651776 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5784666112) block 676913709056 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5792251904) block 647225573376 gen
> 6065149
>      key (EXTENT_CSUM EXTENT_CSUM 5800312832) block 676981161984 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 5811654656) block 676981080064 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 5822210048) block 646336544768 gen
> 6065030
>      key (EXTENT_CSUM EXTENT_CSUM 5831127040) block 677138219008 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 5843570688) block 646264668160 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5854285824) block 646943703040 gen
> 6065108
>      key (EXTENT_CSUM EXTENT_CSUM 5862305792) block 647229030400 gen
> 6065148
>      key (EXTENT_CSUM EXTENT_CSUM 5872865280) block 646834454528 gen
> 6065104
>      key (EXTENT_CSUM EXTENT_CSUM 5883650048) block 646264717312 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5895880704) block 677151473664 gen
> 6065002
>      key (EXTENT_CSUM EXTENT_CSUM 5904617472) block 646827114496 gen
> 6065100
>      key (EXTENT_CSUM EXTENT_CSUM 5915639808) block 677029265408 gen
> 6065001
>      key (EXTENT_CSUM EXTENT_CSUM 5926555648) block 647290617856 gen
> 6065156
>      key (EXTENT_CSUM EXTENT_CSUM 5936168960) block 646264799232 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 5947744256) block 676963827712 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 5956050944) block 676914577408 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 5967777792) block 676963844096 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 5978038272) block 646653607936 gen
> 6065072
>      key (EXTENT_CSUM EXTENT_CSUM 5989384192) block 646578962432 gen
> 6065059
>      key (EXTENT_CSUM EXTENT_CSUM 5997871104) block 647357890560 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 6009282560) block 646263701504 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 6021009408) block 646264930304 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 6032379904) block 646264963072 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 6043267072) block 646265044992 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 6054461440) block 647357300736 gen
> 6065164
>      key (EXTENT_CSUM EXTENT_CSUM 6063206400) block 647417905152 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 6072909824) block 647385645056 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 6084259840) block 647417987072 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 6096568320) block 646263603200 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 6107648000) block 676963958784 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 6116904960) block 646993084416 gen
> 6065111
>      key (EXTENT_CSUM EXTENT_CSUM 6128369664) block 646265094144 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 6138900480) block 647353303040 gen
> 6065172
>      key (EXTENT_CSUM EXTENT_CSUM 6146465792) block 676849172480 gen
> 6064995
>      key (EXTENT_CSUM EXTENT_CSUM 6156619776) block 646265323520 gen
> 6065024
>      key (EXTENT_CSUM EXTENT_CSUM 6168879104) block 676992876544 gen
> 6064998
>      key (EXTENT_CSUM EXTENT_CSUM 6179160064) block 676915036160 gen
> 6064996
>      key (EXTENT_CSUM EXTENT_CSUM 6192357376) block 646396493824 gen
> 6065032
> ERROR: failed to clear ino cache: Input/output error
>
> Not sure what to do with this.
>
> dmesg from subsequent mounting on the rescue system:
>
> [  365.027631] BTRFS: device label root devid 1 transid 6065178
> /dev/mapper/rootext (254:1) scanned by mount (1425)
> [  365.028812] BTRFS info (device dm-1): first mount of filesystem
> e6614f01-6f56-4776-8b0a-c260089c35e7
> [  365.028823] BTRFS info (device dm-1): using crc32c (crc32c-intel)
> checksum algorithm
> [  365.028840] BTRFS info (device dm-1): disk space caching is enabled
> [  365.039509] BTRFS warning (device dm-1): devid 1 physical 0 len
> 4194304 inside the reserved space
> [  365.041044] BTRFS info (device dm-1): bdev /dev/mapper/rootext errs:
> wr 0, rd 0, flush 0, corrupt 4, gen 0
> [  365.424564] BTRFS info (device dm-1): checking UUID tree
>
>
> dmesg from booting on the system:
>
> [   12.697569] Btrfs loaded, zoned=yes, fsverity=yes
> [   12.793884] BTRFS: device label root devid 1 transid 6065180
> /dev/mapper/root (254:0) scanned by mount (249)
> [   12.795330] BTRFS info (device dm-0): first mount of filesystem
> e6614f01-6f56-4776-8b0a-c260089c35e7
> [   12.795358] BTRFS info (device dm-0): using crc32c (crc32c-intel)
> checksum algorithm
> [   12.795369] BTRFS info (device dm-0): disk space caching is enabled
> [   12.809564] BTRFS warning (device dm-0): devid 1 physical 0 len
> 4194304 inside the reserved space
> [   12.813203] BTRFS info (device dm-0): bdev /dev/mapper/root errs: wr
> 0, rd 0, flush 0, corrupt 4, gen 0
> [   16.026687] BTRFS info (device dm-0 state M): use zstd compression,
> level 3
> [   16.842214] BTRFS info: devid 1 device path /dev/mapper/root changed
> to /dev/dm-0 scanned by (udev-worker) (364)
> [   16.843857] BTRFS info: devid 1 device path /dev/dm-0 changed to
> /dev/mapper/root scanned by (udev-worker) (364)
> [   19.888321] BTRFS warning (device dm-0): block group 1094713344 has
> wrong amount of free space
> [   19.888325] BTRFS warning (device dm-0): failed to load free space
> cache for block group 1094713344, rebuilding it now
> [   19.968700] BTRFS warning (device dm-0): block group 10758389760 has
> wrong amount of free space
> [   19.968705] BTRFS warning (device dm-0): failed to load free space
> cache for block group 10758389760, rebuilding it now
>
> That being said no critical error has been encountered since I’ve had to
> repair my boot, but I’ve not tried to scrub from the running system
> again, should I do that (as it used to trigger the error before)?

Please do a `btrfs check` to make sure everything is fine.

For the bad free space cache, I'd recommend to go v2 space cache instead.

Thanks,
Qu
>
> Regards,
> Archange
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-11 21:23           ` Qu Wenruo
@ 2024-09-11 21:42             ` Qu Wenruo
  2024-09-11 22:06               ` Archange
  2024-09-11 22:05             ` Archange
  1 sibling, 1 reply; 24+ messages in thread
From: Qu Wenruo @ 2024-09-11 21:42 UTC (permalink / raw)
  To: Archange, linux-btrfs



在 2024/9/12 06:53, Qu Wenruo 写道:
>
>
> 在 2024/9/12 06:50, Archange 写道:
>> Le 12/09/2024 à 00:54, Qu Wenruo a écrit :
>>> 在 2024/9/12 05:25, Archange 写道:
>>>> Le 11/09/2024 à 01:37, Qu Wenruo a écrit :
>>>>> 在 2024/9/11 06:58, Qu Wenruo 写道:
>>>>>> 在 2024/9/11 06:35, Archange 写道:
>>>>>> […]
>>>
>>> This looks exactly like another report that is caused by inode cache.
>>>
>>> So in that case, mind to try the following commands?
>>>
>>> # btrfs rescue zero-log <device>
>>> # btrfs rescue clear-inode-cache <device>
>>
>> I supposed the second command was meant to be `clear-ino-cache` (I
>> remember having to remove `inode_cache` from mount options some time ago
>> as it prevented booting, I had then discovered the related feature had
>> been deprecated and removed).
>>
>> Here are the command outputs:
>>
>> # btrfs rescue zero-log /dev/mapper/rootext
>> Clearing log on /dev/mapper/rootext, previous log_root 0, level 0
>>
>> # btrfs rescue clear-ino-cache /dev/mapper/rootext
>> Successfully cleaned up ino cache for root id: 5
>> Successfully cleaned up ino cache for root id: 257
>> Successfully cleaned up ino cache for root id: 258
>> corrupt node: root=7 block=647369064448 slot=0, invalid level for leaf,
>> have 1 expect 0
>
> This is not expected, I guess I have to double check the inode clearing
> code to be sure.

It turns out to be a bug in the inode cache clearing code, that it holds
a path meanwhile the fs can be modified halfway.

I'll fix it soon, and keep you updated, as you may have some inode cache
left.

Thanks,
Qu

>
>> node 647369064448 level 1 items 252 free space 241 generation 6065173
>> owner CSUM_TREE
>> node 647369064448 flags 0x1(WRITTEN) backref revision 1
>> fs uuid e6614f01-6f56-4776-8b0a-c260089c35e7
>> chunk uuid f665f535-4cfd-49e0-8be9-7f94bf59b75d
>>      key (EXTENT_CSUM EXTENT_CSUM 3714473984) block 677126111232 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 3720785920) block 646338183168 gen
>> 6065030
>>      key (EXTENT_CSUM EXTENT_CSUM 3728965632) block 677126176768 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 3737710592) block 676941266944 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 3747074048) block 677126209536 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 3754676224) block 676863508480 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 3764305920) block 676863524864 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 3772821504) block 646742212608 gen
>> 6065087
>>      key (EXTENT_CSUM EXTENT_CSUM 3780558848) block 676863557632 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 3792121856) block 646843056128 gen
>> 6065107
>>      key (EXTENT_CSUM EXTENT_CSUM 3799425024) block 677126258688 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 3808354304) block 676888068096 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 3822206976) block 647458684928 gen
>> 5744378
>>      key (EXTENT_CSUM EXTENT_CSUM 3838750720) block 648070070272 gen
>> 1381537
>>      key (EXTENT_CSUM EXTENT_CSUM 3855294464) block 648070184960 gen
>> 1381537
>>      key (EXTENT_CSUM EXTENT_CSUM 3871838208) block 647481884672 gen
>> 1381531
>>      key (EXTENT_CSUM EXTENT_CSUM 3888381952) block 676963188736 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 3899301888) block 676943937536 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 3908460544) block 647367393280 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 3916058624) block 676909989888 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 3922890752) block 676715134976 gen
>> 6064994
>>      key (EXTENT_CSUM EXTENT_CSUM 3932336128) block 676941283328 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 3943030784) block 676910022656 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 3952046080) block 676854464512 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 3959169024) block 676910071808 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 3966623744) block 676933074944 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 3975307264) block 647369162752 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 3983073280) block 676948852736 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 3990368256) block 676941299712 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 3998121984) block 676948869120 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4009500672) block 676910170112 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4017614848) block 646264160256 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 4025708544) block 676843307008 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 4033413120) block 646515097600 gen
>> 6065053
>>      key (EXTENT_CSUM EXTENT_CSUM 4041183232) block 677125914624 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4047310848) block 647409041408 gen
>> 6065170
>>      key (EXTENT_CSUM EXTENT_CSUM 4054364160) block 676723982336 gen
>> 6064994
>>      key (EXTENT_CSUM EXTENT_CSUM 4062892032) block 677107367936 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4072263680) block 677107417088 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4081119232) block 676844634112 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 4089425920) block 676910350336 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4097601536) block 677107302400 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4105961472) block 676910432256 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4114223104) block 676963237888 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4121116672) block 646473465856 gen
>> 6065044
>>      key (EXTENT_CSUM EXTENT_CSUM 4130504704) block 677107351552 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4139896832) block 676933156864 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4150054912) block 677107384320 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4159819776) block 677107400704 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4166836224) block 677125980160 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4178882560) block 647369195520 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 4185473024) block 676715347968 gen
>> 6064994
>>      key (EXTENT_CSUM EXTENT_CSUM 4193738752) block 676715364352 gen
>> 6064994
>>      key (EXTENT_CSUM EXTENT_CSUM 4203532288) block 676724146176 gen
>> 6064994
>>      key (EXTENT_CSUM EXTENT_CSUM 4211822592) block 676910612480 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4220416000) block 676754997248 gen
>> 6064994
>>      key (EXTENT_CSUM EXTENT_CSUM 4234862592) block 676855054336 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 4247011328) block 647369097216 gen
>> 6065173
>>      key (EXTENT_CSUM EXTENT_CSUM 4257337344) block 647369228288 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 4270247936) block 646992494592 gen
>> 6065114
>>      key (EXTENT_CSUM EXTENT_CSUM 4284358656) block 676910743552 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4295380992) block 646842253312 gen
>> 6065107
>>      key (EXTENT_CSUM EXTENT_CSUM 4305231872) block 677126012928 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4316020736) block 676963450880 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4325187584) block 677314379776 gen
>> 6064991
>>      key (EXTENT_CSUM EXTENT_CSUM 4341161984) block 646746472448 gen
>> 6065087
>>      key (EXTENT_CSUM EXTENT_CSUM 4345565184) block 676941332480 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4354514944) block 646342459392 gen
>> 6065029
>>      key (EXTENT_CSUM EXTENT_CSUM 4361768960) block 676963467264 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4369428480) block 676967317504 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4376809472) block 676967350272 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4385017856) block 676963483648 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4392333312) block 676941365248 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4400939008) block 676941381632 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4408795136) block 676941496320 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4416131072) block 646359908352 gen
>> 6065030
>>      key (EXTENT_CSUM EXTENT_CSUM 4424679424) block 676715528192 gen
>> 6064994
>>      key (EXTENT_CSUM EXTENT_CSUM 4433436672) block 647351910400 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 4442791936) block 647369261056 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 4451270656) block 646407585792 gen
>> 6065032
>>      key (EXTENT_CSUM EXTENT_CSUM 4459581440) block 677126569984 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4466970624) block 648044281856 gen
>> 6065171
>>      key (EXTENT_CSUM EXTENT_CSUM 4473278464) block 647373193216 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 4481544192) block 676967743488 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4489117696) block 676967792640 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4501204992) block 647357153280 gen
>> 6065164
>>      key (EXTENT_CSUM EXTENT_CSUM 4508569600) block 676967907328 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4513787904) block 676967940096 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4519464960) block 676967972864 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4528398336) block 676968005632 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4537438208) block 677148459008 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 4543373312) block 676968087552 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4553338880) block 676963303424 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4564582400) block 676968169472 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4574752768) block 647373455360 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 4584710144) block 648044314624 gen
>> 6065171
>>      key (EXTENT_CSUM EXTENT_CSUM 4589998080) block 676932272128 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4596559872) block 676693639168 gen
>> 6064993
>>      key (EXTENT_CSUM EXTENT_CSUM 4603473920) block 676933222400 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4613885952) block 676948901888 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4624228352) block 676963532800 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4633935872) block 676911284224 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4644610048) block 646338265088 gen
>> 6065030
>>      key (EXTENT_CSUM EXTENT_CSUM 4653531136) block 646359941120 gen
>> 6065030
>>      key (EXTENT_CSUM EXTENT_CSUM 4664819712) block 676963565568 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4676636672) block 647386398720 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 4686704640) block 646343639040 gen
>> 6065029
>>      key (EXTENT_CSUM EXTENT_CSUM 4698435584) block 676963598336 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4713156608) block 647392772096 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 4722995200) block 646264193024 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 4731748352) block 676911611904 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4738572288) block 676933238784 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4748193792) block 676942348288 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4753985536) block 676942364672 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4761124864) block 676911644672 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4770844672) block 646845300736 gen
>> 6065107
>>      key (EXTENT_CSUM EXTENT_CSUM 4780126208) block 676911677440 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4787306496) block 676942397440 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4798906368) block 676844863488 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 4807143424) block 676963631104 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4817457152) block 676546248704 gen
>> 6064992
>>      key (EXTENT_CSUM EXTENT_CSUM 4826439680) block 676693966848 gen
>> 6064993
>>      key (EXTENT_CSUM EXTENT_CSUM 4840206336) block 676911742976 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4851683328) block 647302086656 gen
>> 6065158
>>      key (EXTENT_CSUM EXTENT_CSUM 4857987072) block 646943653888 gen
>> 6065108
>>      key (EXTENT_CSUM EXTENT_CSUM 4866301952) block 676933271552 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4875943936) block 676911824896 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4888219648) block 676942413824 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4898308096) block 676911857664 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4912943104) block 647369392128 gen
>> 6065173
>>      key (EXTENT_CSUM EXTENT_CSUM 4923723776) block 676968497152 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4934074368) block 676632723456 gen
>> 6064993
>>      key (EXTENT_CSUM EXTENT_CSUM 4948840448) block 676911890432 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 4963790848) block 676968579072 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4975116288) block 676968644608 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 4984283136) block 647080198144 gen
>> 6065133
>>      key (EXTENT_CSUM EXTENT_CSUM 4994596864) block 646304825344 gen
>> 6065025
>>      key (EXTENT_CSUM EXTENT_CSUM 5002313728) block 646324338688 gen
>> 6065026
>>      key (EXTENT_CSUM EXTENT_CSUM 5008089088) block 676963336192 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 5018886144) block 647399227392 gen
>> 6065170
>>      key (EXTENT_CSUM EXTENT_CSUM 5028757504) block 647385743360 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 5037797376) block 647385858048 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 5049577472) block 647386382336 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 5057458176) block 676949278720 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5072642048) block 676963352576 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 5085605888) block 676911988736 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5093662720) block 676963368960 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 5105045504) block 647178960896 gen
>> 6065146
>>      key (EXTENT_CSUM EXTENT_CSUM 5116792832) block 647383547904 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 5130141696) block 676912218112 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5139943424) block 676912267264 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5148413952) block 676912300032 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5156237312) block 676942430208 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5165236224) block 676942446592 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5177163776) block 646300827648 gen
>> 6065025
>>      key (EXTENT_CSUM EXTENT_CSUM 5187325952) block 676749754368 gen
>> 6064994
>>      key (EXTENT_CSUM EXTENT_CSUM 5201133568) block 676942479360 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5213573120) block 676657332224 gen
>> 6064993
>>      key (EXTENT_CSUM EXTENT_CSUM 5228421120) block 676845240320 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 5236654080) block 676942495744 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5246169088) block 676942512128 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5254701056) block 646343688192 gen
>> 6065029
>>      key (EXTENT_CSUM EXTENT_CSUM 5263523840) block 676942528512 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5271793664) block 676932288512 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5284384768) block 676933320704 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5298110464) block 676759650304 gen
>> 6064994
>>      key (EXTENT_CSUM EXTENT_CSUM 5304373248) block 676942544896 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5314695168) block 646338510848 gen
>> 6065030
>>      key (EXTENT_CSUM EXTENT_CSUM 5323919360) block 676942561280 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5336117248) block 676912496640 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5345054720) block 676963680256 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 5350932480) block 676863836160 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 5363728384) block 676657709056 gen
>> 6064993
>>      key (EXTENT_CSUM EXTENT_CSUM 5369249792) block 676912594944 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5379948544) block 646264209408 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5392506880) block 676978917376 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 5402152960) block 646948175872 gen
>> 6065108
>>      key (EXTENT_CSUM EXTENT_CSUM 5412794368) block 646948208640 gen
>> 6065108
>>      key (EXTENT_CSUM EXTENT_CSUM 5422727168) block 676694409216 gen
>> 6064993
>>      key (EXTENT_CSUM EXTENT_CSUM 5436506112) block 646264225792 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5446492160) block 646948241408 gen
>> 6065108
>>      key (EXTENT_CSUM EXTENT_CSUM 5457653760) block 646264242176 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5469483008) block 646264258560 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5479792640) block 646264143872 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5491097600) block 647262306304 gen
>> 6065152
>>      key (EXTENT_CSUM EXTENT_CSUM 5504716800) block 646264438784 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5516099584) block 646264504320 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5526601728) block 676942594048 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5538586624) block 646716211200 gen
>> 6065083
>>      key (EXTENT_CSUM EXTENT_CSUM 5550575616) block 677029806080 gen
>> 6065001
>>      key (EXTENT_CSUM EXTENT_CSUM 5560479744) block 647350697984 gen
>> 6065163
>>      key (EXTENT_CSUM EXTENT_CSUM 5570015232) block 647383580672 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 5578293248) block 646264487936 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5590724608) block 676863950848 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 5603069952) block 646263504896 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5615501312) block 676913020928 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5626724352) block 676913102848 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5639708672) block 677029822464 gen
>> 6065001
>>      key (EXTENT_CSUM EXTENT_CSUM 5653757952) block 647216037888 gen
>> 6065147
>>      key (EXTENT_CSUM EXTENT_CSUM 5660786688) block 647207895040 gen
>> 6065147
>>      key (EXTENT_CSUM EXTENT_CSUM 5668933632) block 676715937792 gen
>> 6064994
>>      key (EXTENT_CSUM EXTENT_CSUM 5677277184) block 647383613440 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 5687111680) block 646264520704 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5694754816) block 647384301568 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 5704658944) block 647417839616 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 5714460672) block 647417872384 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 5725593600) block 647114440704 gen
>> 6065138
>>      key (EXTENT_CSUM EXTENT_CSUM 5733572608) block 647409156096 gen
>> 6065170
>>      key (EXTENT_CSUM EXTENT_CSUM 5743837184) block 677138202624 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 5755424768) block 646943686656 gen
>> 6065108
>>      key (EXTENT_CSUM EXTENT_CSUM 5766553600) block 676963794944 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 5778427904) block 646264651776 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5784666112) block 676913709056 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5792251904) block 647225573376 gen
>> 6065149
>>      key (EXTENT_CSUM EXTENT_CSUM 5800312832) block 676981161984 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 5811654656) block 676981080064 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 5822210048) block 646336544768 gen
>> 6065030
>>      key (EXTENT_CSUM EXTENT_CSUM 5831127040) block 677138219008 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 5843570688) block 646264668160 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5854285824) block 646943703040 gen
>> 6065108
>>      key (EXTENT_CSUM EXTENT_CSUM 5862305792) block 647229030400 gen
>> 6065148
>>      key (EXTENT_CSUM EXTENT_CSUM 5872865280) block 646834454528 gen
>> 6065104
>>      key (EXTENT_CSUM EXTENT_CSUM 5883650048) block 646264717312 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5895880704) block 677151473664 gen
>> 6065002
>>      key (EXTENT_CSUM EXTENT_CSUM 5904617472) block 646827114496 gen
>> 6065100
>>      key (EXTENT_CSUM EXTENT_CSUM 5915639808) block 677029265408 gen
>> 6065001
>>      key (EXTENT_CSUM EXTENT_CSUM 5926555648) block 647290617856 gen
>> 6065156
>>      key (EXTENT_CSUM EXTENT_CSUM 5936168960) block 646264799232 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 5947744256) block 676963827712 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 5956050944) block 676914577408 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 5967777792) block 676963844096 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 5978038272) block 646653607936 gen
>> 6065072
>>      key (EXTENT_CSUM EXTENT_CSUM 5989384192) block 646578962432 gen
>> 6065059
>>      key (EXTENT_CSUM EXTENT_CSUM 5997871104) block 647357890560 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 6009282560) block 646263701504 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 6021009408) block 646264930304 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 6032379904) block 646264963072 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 6043267072) block 646265044992 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 6054461440) block 647357300736 gen
>> 6065164
>>      key (EXTENT_CSUM EXTENT_CSUM 6063206400) block 647417905152 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 6072909824) block 647385645056 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 6084259840) block 647417987072 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 6096568320) block 646263603200 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 6107648000) block 676963958784 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 6116904960) block 646993084416 gen
>> 6065111
>>      key (EXTENT_CSUM EXTENT_CSUM 6128369664) block 646265094144 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 6138900480) block 647353303040 gen
>> 6065172
>>      key (EXTENT_CSUM EXTENT_CSUM 6146465792) block 676849172480 gen
>> 6064995
>>      key (EXTENT_CSUM EXTENT_CSUM 6156619776) block 646265323520 gen
>> 6065024
>>      key (EXTENT_CSUM EXTENT_CSUM 6168879104) block 676992876544 gen
>> 6064998
>>      key (EXTENT_CSUM EXTENT_CSUM 6179160064) block 676915036160 gen
>> 6064996
>>      key (EXTENT_CSUM EXTENT_CSUM 6192357376) block 646396493824 gen
>> 6065032
>> ERROR: failed to clear ino cache: Input/output error
>>
>> Not sure what to do with this.
>>
>> dmesg from subsequent mounting on the rescue system:
>>
>> [  365.027631] BTRFS: device label root devid 1 transid 6065178
>> /dev/mapper/rootext (254:1) scanned by mount (1425)
>> [  365.028812] BTRFS info (device dm-1): first mount of filesystem
>> e6614f01-6f56-4776-8b0a-c260089c35e7
>> [  365.028823] BTRFS info (device dm-1): using crc32c (crc32c-intel)
>> checksum algorithm
>> [  365.028840] BTRFS info (device dm-1): disk space caching is enabled
>> [  365.039509] BTRFS warning (device dm-1): devid 1 physical 0 len
>> 4194304 inside the reserved space
>> [  365.041044] BTRFS info (device dm-1): bdev /dev/mapper/rootext errs:
>> wr 0, rd 0, flush 0, corrupt 4, gen 0
>> [  365.424564] BTRFS info (device dm-1): checking UUID tree
>>
>>
>> dmesg from booting on the system:
>>
>> [   12.697569] Btrfs loaded, zoned=yes, fsverity=yes
>> [   12.793884] BTRFS: device label root devid 1 transid 6065180
>> /dev/mapper/root (254:0) scanned by mount (249)
>> [   12.795330] BTRFS info (device dm-0): first mount of filesystem
>> e6614f01-6f56-4776-8b0a-c260089c35e7
>> [   12.795358] BTRFS info (device dm-0): using crc32c (crc32c-intel)
>> checksum algorithm
>> [   12.795369] BTRFS info (device dm-0): disk space caching is enabled
>> [   12.809564] BTRFS warning (device dm-0): devid 1 physical 0 len
>> 4194304 inside the reserved space
>> [   12.813203] BTRFS info (device dm-0): bdev /dev/mapper/root errs: wr
>> 0, rd 0, flush 0, corrupt 4, gen 0
>> [   16.026687] BTRFS info (device dm-0 state M): use zstd compression,
>> level 3
>> [   16.842214] BTRFS info: devid 1 device path /dev/mapper/root changed
>> to /dev/dm-0 scanned by (udev-worker) (364)
>> [   16.843857] BTRFS info: devid 1 device path /dev/dm-0 changed to
>> /dev/mapper/root scanned by (udev-worker) (364)
>> [   19.888321] BTRFS warning (device dm-0): block group 1094713344 has
>> wrong amount of free space
>> [   19.888325] BTRFS warning (device dm-0): failed to load free space
>> cache for block group 1094713344, rebuilding it now
>> [   19.968700] BTRFS warning (device dm-0): block group 10758389760 has
>> wrong amount of free space
>> [   19.968705] BTRFS warning (device dm-0): failed to load free space
>> cache for block group 10758389760, rebuilding it now
>>
>> That being said no critical error has been encountered since I’ve had to
>> repair my boot, but I’ve not tried to scrub from the running system
>> again, should I do that (as it used to trigger the error before)?
>
> Please do a `btrfs check` to make sure everything is fine.
>
> For the bad free space cache, I'd recommend to go v2 space cache instead.
>
> Thanks,
> Qu
>>
>> Regards,
>> Archange
>>
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-11 21:42             ` Qu Wenruo
@ 2024-09-11 22:06               ` Archange
  0 siblings, 0 replies; 24+ messages in thread
From: Archange @ 2024-09-11 22:06 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs

Le 12/09/2024 à 01:42, Qu Wenruo a écrit :
> 在 2024/9/12 06:53, Qu Wenruo 写道:
>> 在 2024/9/12 06:50, Archange 写道:
>>> Le 12/09/2024 à 00:54, Qu Wenruo a écrit :
>>>> 在 2024/9/12 05:25, Archange 写道:
>>>>> Le 11/09/2024 à 01:37, Qu Wenruo a écrit :
>>>>>> 在 2024/9/11 06:58, Qu Wenruo 写道:
>>>>>>> 在 2024/9/11 06:35, Archange 写道:
>>>>>>> […]
>>>>
>>>> This looks exactly like another report that is caused by inode cache.
>>>>
>>>> So in that case, mind to try the following commands?
>>>>
>>>> # btrfs rescue zero-log <device>
>>>> # btrfs rescue clear-inode-cache <device>
>>>
>>> I supposed the second command was meant to be `clear-ino-cache` (I
>>> remember having to remove `inode_cache` from mount options some time 
>>> ago
>>> as it prevented booting, I had then discovered the related feature had
>>> been deprecated and removed).
>>>
>>> Here are the command outputs:
>>>
>>> # btrfs rescue zero-log /dev/mapper/rootext
>>> Clearing log on /dev/mapper/rootext, previous log_root 0, level 0
>>>
>>> # btrfs rescue clear-ino-cache /dev/mapper/rootext
>>> Successfully cleaned up ino cache for root id: 5
>>> Successfully cleaned up ino cache for root id: 257
>>> Successfully cleaned up ino cache for root id: 258
>>> corrupt node: root=7 block=647369064448 slot=0, invalid level for leaf,
>>> have 1 expect 0
>>
>> This is not expected, I guess I have to double check the inode clearing
>> code to be sure.
>
> It turns out to be a bug in the inode cache clearing code, that it holds
> a path meanwhile the fs can be modified halfway.
>
> I'll fix it soon, and keep you updated, as you may have some inode cache
> left.

OK, thanks.


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-11 21:23           ` Qu Wenruo
  2024-09-11 21:42             ` Qu Wenruo
@ 2024-09-11 22:05             ` Archange
  2024-09-11 22:34               ` Qu Wenruo
  1 sibling, 1 reply; 24+ messages in thread
From: Archange @ 2024-09-11 22:05 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs

Le 12/09/2024 à 01:23, Qu Wenruo a écrit :
>
>
> 在 2024/9/12 06:50, Archange 写道:
>> Le 12/09/2024 à 00:54, Qu Wenruo a écrit :
>>> 在 2024/9/12 05:25, Archange 写道:
>>>> Le 11/09/2024 à 01:37, Qu Wenruo a écrit :
>>>>> 在 2024/9/11 06:58, Qu Wenruo 写道:
>>>>>> 在 2024/9/11 06:35, Archange 写道:
>>>>>> […]
>>> […]
>>>
>>> That being said no critical error has been encountered since I’ve 
>>> had to
>>> repair my boot, but I’ve not tried to scrub from the running system
>>> again, should I do that (as it used to trigger the error before)?
>
> Please do a `btrfs check` to make sure everything is fine.

While the previous one (see my second message in this thread) had no 
error, there is now one:

# btrfs check /dev/mapper/rootext
Opening filesystem to check...
Checking filesystem on /dev/mapper/rootext
UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
wanted bytes 688128, found 720896 for off 676326604800
cache appears valid but isn't 676326604800
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 478434086912 bytes used, error(s) found
total csum bytes: 465136176
total tree bytes: 1590493184
total fs tree bytes: 870760448
total extent tree bytes: 138231808
btree space waste bytes: 326062620
file data blocks allocated: 516090466304
  referenced 492383629312

> For the bad free space cache, I'd recommend to go v2 space cache instead.

Since the above error seems related to space cache, and that I had 
switching to v2 on my todo list for a long time, I’ve just did so.

dmesg mount messages right after (nothing unexpected I guess, outside 
the warning and “corrupt 4” still present):

[  812.242212] BTRFS: device label root devid 1 transid 6065207 
/dev/mapper/rootext (254:1) scanned by mount (2145)
[  812.243727] BTRFS info (device dm-1): first mount of filesystem 
e6614f01-6f56-4776-8b0a-c260089c35e7
[  812.243770] BTRFS info (device dm-1): using crc32c (crc32c-intel) 
checksum algorithm
[  812.243788] BTRFS info (device dm-1): using free-space-tree
[  812.256356] BTRFS warning (device dm-1): devid 1 physical 0 len 
4194304 inside the reserved space
[  812.258504] BTRFS info (device dm-1): bdev /dev/mapper/rootext errs: 
wr 0, rd 0, flush 0, corrupt 4, gen 0
[  812.810623] BTRFS info (device dm-1): creating free space tree
[  819.778945] BTRFS info (device dm-1): setting compat-ro feature flag 
for FREE_SPACE_TREE (0x1)
[  819.778949] BTRFS info (device dm-1): setting compat-ro feature flag 
for FREE_SPACE_TREE_VALID (0x2)
[  819.877973] BTRFS info (device dm-1): cleaning free space cache v1
[  819.885829] BTRFS info (device dm-1): checking UUID tree
[  866.299565] BTRFS info (device dm-1): last unmount of filesystem 
e6614f01-6f56-4776-8b0a-c260089c35e7

I’ve run check again after that:

# btrfs check /dev/mapper/rootext
Opening filesystem to check...
Checking filesystem on /dev/mapper/rootext
UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
there is no free space entry for 0-65536
cache appears valid but isn't 0
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 478312665088 bytes used, error(s) found
total csum bytes: 465136176
total tree bytes: 1593524224
total fs tree bytes: 870760448
total extent tree bytes: 138231808
btree space waste bytes: 326060271
file data blocks allocated: 515966013440
  referenced 492259176448

So there is still an error, but different this time.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-11 22:05             ` Archange
@ 2024-09-11 22:34               ` Qu Wenruo
  2024-09-12  8:21                 ` Archange
  0 siblings, 1 reply; 24+ messages in thread
From: Qu Wenruo @ 2024-09-11 22:34 UTC (permalink / raw)
  To: Archange, linux-btrfs



在 2024/9/12 07:35, Archange 写道:
>
> Le 12/09/2024 à 01:23, Qu Wenruo a écrit :
[...]
>
> While the previous one (see my second message in this thread) had no
> error, there is now one:
>
> # btrfs check /dev/mapper/rootext
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/rootext
> UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> wanted bytes 688128, found 720896 for off 676326604800
> cache appears valid but isn't 676326604800

Minor problem, still I'd recommend to run
  `btrfs rescue clear-space-cache v1 <dev>` to clear the v1 cache first.

Then you can mount with v2 space cache or keep going with the v1 cache
(not recommended, will be deprecated soon)

And if your fs only have subvolumes 5 (the top level one), 257 and 258,
then you're totally fine to continue.
I guess that's the case?

If you have other subvolumes, then you may need to wait for my fix to
ino-clear code to clear the remaining subvolumes, just in case.

Thanks,
Qu

> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 478434086912 bytes used, error(s) found
> total csum bytes: 465136176
> total tree bytes: 1590493184
> total fs tree bytes: 870760448
> total extent tree bytes: 138231808
> btree space waste bytes: 326062620
> file data blocks allocated: 516090466304
>   referenced 492383629312
>
>> For the bad free space cache, I'd recommend to go v2 space cache instead.
>
> Since the above error seems related to space cache, and that I had
> switching to v2 on my todo list for a long time, I’ve just did so.
>
> dmesg mount messages right after (nothing unexpected I guess, outside
> the warning and “corrupt 4” still present):
>
> [  812.242212] BTRFS: device label root devid 1 transid 6065207
> /dev/mapper/rootext (254:1) scanned by mount (2145)
> [  812.243727] BTRFS info (device dm-1): first mount of filesystem
> e6614f01-6f56-4776-8b0a-c260089c35e7
> [  812.243770] BTRFS info (device dm-1): using crc32c (crc32c-intel)
> checksum algorithm
> [  812.243788] BTRFS info (device dm-1): using free-space-tree
> [  812.256356] BTRFS warning (device dm-1): devid 1 physical 0 len
> 4194304 inside the reserved space
> [  812.258504] BTRFS info (device dm-1): bdev /dev/mapper/rootext errs:
> wr 0, rd 0, flush 0, corrupt 4, gen 0
> [  812.810623] BTRFS info (device dm-1): creating free space tree
> [  819.778945] BTRFS info (device dm-1): setting compat-ro feature flag
> for FREE_SPACE_TREE (0x1)
> [  819.778949] BTRFS info (device dm-1): setting compat-ro feature flag
> for FREE_SPACE_TREE_VALID (0x2)
> [  819.877973] BTRFS info (device dm-1): cleaning free space cache v1
> [  819.885829] BTRFS info (device dm-1): checking UUID tree
> [  866.299565] BTRFS info (device dm-1): last unmount of filesystem
> e6614f01-6f56-4776-8b0a-c260089c35e7
>
> I’ve run check again after that:
>
> # btrfs check /dev/mapper/rootext
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/rootext
> UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space tree
> there is no free space entry for 0-65536
> cache appears valid but isn't 0
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 478312665088 bytes used, error(s) found
> total csum bytes: 465136176
> total tree bytes: 1593524224
> total fs tree bytes: 870760448
> total extent tree bytes: 138231808
> btree space waste bytes: 326060271
> file data blocks allocated: 515966013440
>   referenced 492259176448
>
> So there is still an error, but different this time.
>
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-11 22:34               ` Qu Wenruo
@ 2024-09-12  8:21                 ` Archange
  2024-09-12  8:25                   ` Qu Wenruo
  0 siblings, 1 reply; 24+ messages in thread
From: Archange @ 2024-09-12  8:21 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs

Le 12/09/2024 à 02:34, Qu Wenruo a écrit :
> 在 2024/9/12 07:35, Archange 写道:
>>
>> Le 12/09/2024 à 01:23, Qu Wenruo a écrit :
> [...]
>>
>> While the previous one (see my second message in this thread) had no
>> error, there is now one:
>>
>> # btrfs check /dev/mapper/rootext
>> Opening filesystem to check...
>> Checking filesystem on /dev/mapper/rootext
>> UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
>> [1/7] checking root items
>> [2/7] checking extents
>> [3/7] checking free space cache
>> wanted bytes 688128, found 720896 for off 676326604800
>> cache appears valid but isn't 676326604800
>
> Minor problem, still I'd recommend to run
>  `btrfs rescue clear-space-cache v1 <dev>` to clear the v1 cache first.

I indeed did that as explained in the second part of my message.

> Then you can mount with v2 space cache or keep going with the v1 cache
> (not recommended, will be deprecated soon)

Done too.

> And if your fs only have subvolumes 5 (the top level one), 257 and 258,
> then you're totally fine to continue.
> I guess that's the case?

Indeed!

Thanks a lot for your help,
Archange


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-12  8:21                 ` Archange
@ 2024-09-12  8:25                   ` Qu Wenruo
  2024-09-12  9:57                     ` Archange
  0 siblings, 1 reply; 24+ messages in thread
From: Qu Wenruo @ 2024-09-12  8:25 UTC (permalink / raw)
  To: Archange, linux-btrfs



在 2024/9/12 17:51, Archange 写道:
> Le 12/09/2024 à 02:34, Qu Wenruo a écrit :
>> 在 2024/9/12 07:35, Archange 写道:
>>>
>>> Le 12/09/2024 à 01:23, Qu Wenruo a écrit :
>> [...]
>>>
>>> While the previous one (see my second message in this thread) had no
>>> error, there is now one:
>>>
>>> # btrfs check /dev/mapper/rootext
>>> Opening filesystem to check...
>>> Checking filesystem on /dev/mapper/rootext
>>> UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
>>> [1/7] checking root items
>>> [2/7] checking extents
>>> [3/7] checking free space cache
>>> wanted bytes 688128, found 720896 for off 676326604800
>>> cache appears valid but isn't 676326604800
>>
>> Minor problem, still I'd recommend to run
>>  `btrfs rescue clear-space-cache v1 <dev>` to clear the v1 cache first.
>
> I indeed did that as explained in the second part of my message.
>
>> Then you can mount with v2 space cache or keep going with the v1 cache
>> (not recommended, will be deprecated soon)
>
> Done too.
>
>> And if your fs only have subvolumes 5 (the top level one), 257 and 258,
>> then you're totally fine to continue.
>> I guess that's the case?
>
> Indeed!

Just in case, you can run "btrfs check --mode=lowmem" to check if there
is no more inode cache left.

If there is any left, lowmem mode can detect it with errors like:

ERROR: root 5 INODE[18446744073709551604] nlink(1) not equal to
inode_refs(0)
ERROR: invalid imode mode bits: 00
ERROR: invalid inode generation 18446744073709551604 or transid 1 for
ino 18446744073709551605, expect [0, 72)
ERROR: root 5 INODE[18446744073709551605] is orphan item

And I'm already adding the ability to the original mode check to detect
such problem.

Thanks,
Qu
>
> Thanks a lot for your help,
> Archange
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-12  8:25                   ` Qu Wenruo
@ 2024-09-12  9:57                     ` Archange
  2024-09-12 10:01                       ` Qu Wenruo
  0 siblings, 1 reply; 24+ messages in thread
From: Archange @ 2024-09-12  9:57 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs

Le 12/09/2024 à 12:25, Qu Wenruo a écrit :
> 在 2024/9/12 17:51, Archange 写道:
>> Le 12/09/2024 à 02:34, Qu Wenruo a écrit :
>>> 在 2024/9/12 07:35, Archange 写道:
>>>>
>>>> Le 12/09/2024 à 01:23, Qu Wenruo a écrit :
>>> [...]
>>>>
>>>> While the previous one (see my second message in this thread) had no
>>>> error, there is now one:
>>>>
>>>> # btrfs check /dev/mapper/rootext
>>>> Opening filesystem to check...
>>>> Checking filesystem on /dev/mapper/rootext
>>>> UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
>>>> [1/7] checking root items
>>>> [2/7] checking extents
>>>> [3/7] checking free space cache
>>>> wanted bytes 688128, found 720896 for off 676326604800
>>>> cache appears valid but isn't 676326604800
>>>
>>> Minor problem, still I'd recommend to run
>>>  `btrfs rescue clear-space-cache v1 <dev>` to clear the v1 cache first.
>>
>> I indeed did that as explained in the second part of my message.
>>
>>> Then you can mount with v2 space cache or keep going with the v1 cache
>>> (not recommended, will be deprecated soon)
>>
>> Done too.
>>
>>> And if your fs only have subvolumes 5 (the top level one), 257 and 258,
>>> then you're totally fine to continue.
>>> I guess that's the case?
>>
>> Indeed!
>
> Just in case, you can run "btrfs check --mode=lowmem" to check if there
> is no more inode cache left.
>
> If there is any left, lowmem mode can detect it with errors like:
>
> ERROR: root 5 INODE[18446744073709551604] nlink(1) not equal to
> inode_refs(0)
> ERROR: invalid imode mode bits: 00
> ERROR: invalid inode generation 18446744073709551604 or transid 1 for
> ino 18446744073709551605, expect [0, 72)
> ERROR: root 5 INODE[18446744073709551605] is orphan item
>
> And I'm already adding the ability to the original mode check to detect
> such problem.

No such thing appeared during the lowmem check. Only the

[3/7] checking free space tree
there is no free space entry for 0-65536
cache appears valid but isn't 0

is still there.

Archange


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-12  9:57                     ` Archange
@ 2024-09-12 10:01                       ` Qu Wenruo
  2024-09-12 10:04                         ` Archange
  0 siblings, 1 reply; 24+ messages in thread
From: Qu Wenruo @ 2024-09-12 10:01 UTC (permalink / raw)
  To: Archange, Qu Wenruo, linux-btrfs



在 2024/9/12 19:27, Archange 写道:
> Le 12/09/2024 à 12:25, Qu Wenruo a écrit :
>> 在 2024/9/12 17:51, Archange 写道:
>>> Le 12/09/2024 à 02:34, Qu Wenruo a écrit :
>>>> 在 2024/9/12 07:35, Archange 写道:
>>>>>
>>>>> Le 12/09/2024 à 01:23, Qu Wenruo a écrit :
>>>> [...]
>>>>>
>>>>> While the previous one (see my second message in this thread) had no
>>>>> error, there is now one:
>>>>>
>>>>> # btrfs check /dev/mapper/rootext
>>>>> Opening filesystem to check...
>>>>> Checking filesystem on /dev/mapper/rootext
>>>>> UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
>>>>> [1/7] checking root items
>>>>> [2/7] checking extents
>>>>> [3/7] checking free space cache
>>>>> wanted bytes 688128, found 720896 for off 676326604800
>>>>> cache appears valid but isn't 676326604800
>>>>
>>>> Minor problem, still I'd recommend to run
>>>>  `btrfs rescue clear-space-cache v1 <dev>` to clear the v1 cache first.
>>>
>>> I indeed did that as explained in the second part of my message.
>>>
>>>> Then you can mount with v2 space cache or keep going with the v1 cache
>>>> (not recommended, will be deprecated soon)
>>>
>>> Done too.
>>>
>>>> And if your fs only have subvolumes 5 (the top level one), 257 and 258,
>>>> then you're totally fine to continue.
>>>> I guess that's the case?
>>>
>>> Indeed!
>>
>> Just in case, you can run "btrfs check --mode=lowmem" to check if there
>> is no more inode cache left.
>>
>> If there is any left, lowmem mode can detect it with errors like:
>>
>> ERROR: root 5 INODE[18446744073709551604] nlink(1) not equal to
>> inode_refs(0)
>> ERROR: invalid imode mode bits: 00
>> ERROR: invalid inode generation 18446744073709551604 or transid 1 for
>> ino 18446744073709551605, expect [0, 72)
>> ERROR: root 5 INODE[18446744073709551605] is orphan item
>>
>> And I'm already adding the ability to the original mode check to detect
>> such problem.
> 
> No such thing appeared during the lowmem check. Only the
> 
> [3/7] checking free space tree
> there is no free space entry for 0-65536
> cache appears valid but isn't 0

Then it's totally fine.

For the 0-65536 problem, mind to provide the following dump?

# btrfs ins dump-tree -t fst <device>

I'm afraid since the fs is somewhat old, there may be some corner case 
btrfs-check is not handling properly.

Thanks,
Qu

> 
> is still there.
> 
> Archange
> 
> 

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-12 10:01                       ` Qu Wenruo
@ 2024-09-12 10:04                         ` Archange
  2024-09-12 10:23                           ` Qu Wenruo
  0 siblings, 1 reply; 24+ messages in thread
From: Archange @ 2024-09-12 10:04 UTC (permalink / raw)
  To: Qu Wenruo, Qu Wenruo, linux-btrfs

Le 12/09/2024 à 14:01, Qu Wenruo a écrit :
> 在 2024/9/12 19:27, Archange 写道:
>> […]
>>
>> [3/7] checking free space tree
>> there is no free space entry for 0-65536
>> cache appears valid but isn't 0
>
> Then it's totally fine.
>
> For the 0-65536 problem, mind to provide the following dump?
>
> # btrfs ins dump-tree -t fst <device>
>
> I'm afraid since the fs is somewhat old, there may be some corner case 
> btrfs-check is not handling properly.

ERROR: unexpected tree id suffix of 'fst': t

Archange


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-12 10:04                         ` Archange
@ 2024-09-12 10:23                           ` Qu Wenruo
  2024-09-12 12:13                             ` Archange
  0 siblings, 1 reply; 24+ messages in thread
From: Qu Wenruo @ 2024-09-12 10:23 UTC (permalink / raw)
  To: Archange, Qu Wenruo, linux-btrfs



在 2024/9/12 19:34, Archange 写道:
> Le 12/09/2024 à 14:01, Qu Wenruo a écrit :
>> 在 2024/9/12 19:27, Archange 写道:
>>> […]
>>>
>>> [3/7] checking free space tree
>>> there is no free space entry for 0-65536
>>> cache appears valid but isn't 0
>>
>> Then it's totally fine.
>>
>> For the 0-65536 problem, mind to provide the following dump?
>>
>> # btrfs ins dump-tree -t fst <device>
>>
>> I'm afraid since the fs is somewhat old, there may be some corner case
>> btrfs-check is not handling properly.
>
> ERROR: unexpected tree id suffix of 'fst': t

My bad, it should be "btrfs ins dump-tree -t free-space <device>".

And if possible, also "btrfs ins dump-tree -t extent <device>" just in case.

Thanks,
Qu
>
> Archange
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-12 10:23                           ` Qu Wenruo
@ 2024-09-12 12:13                             ` Archange
  2024-09-12 21:42                               ` Qu Wenruo
  0 siblings, 1 reply; 24+ messages in thread
From: Archange @ 2024-09-12 12:13 UTC (permalink / raw)
  To: Qu Wenruo, Qu Wenruo, linux-btrfs

Le 12/09/2024 à 14:23, Qu Wenruo a écrit :
> 在 2024/9/12 19:34, Archange 写道:
>> Le 12/09/2024 à 14:01, Qu Wenruo a écrit :
>>> 在 2024/9/12 19:27, Archange 写道:
>>>> […]
>>>>
>>>> [3/7] checking free space tree
>>>> there is no free space entry for 0-65536
>>>> cache appears valid but isn't 0
>>>
>>> Then it's totally fine.
>>>
>>> For the 0-65536 problem, mind to provide the following dump?
>>>
>>> # btrfs ins dump-tree -t fst <device>
>>>
>>> I'm afraid since the fs is somewhat old, there may be some corner case
>>> btrfs-check is not handling properly.
>>
>> ERROR: unexpected tree id suffix of 'fst': t
>
> My bad, it should be "btrfs ins dump-tree -t free-space <device>".

The output is too big for an email, so uploaded here:

https://paste.xinu.at/XtR8/

> And if possible, also "btrfs ins dump-tree -t extent <device>" just in 
> case.

Same thing (even bigger), also output on the terminal and while 
redirecting to a file was quite different (but maybe that’s more because 
something changed between the two calls), so here are:

– the cli run : https://paste.xinu.at/9vs/

– the file run: https://paste.xinu.at/XpzhbZ/

Regards,
Archange


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-12 12:13                             ` Archange
@ 2024-09-12 21:42                               ` Qu Wenruo
  2024-09-13  5:25                                 ` Archange
  0 siblings, 1 reply; 24+ messages in thread
From: Qu Wenruo @ 2024-09-12 21:42 UTC (permalink / raw)
  To: Archange, Qu Wenruo, linux-btrfs



在 2024/9/12 21:43, Archange 写道:
> Le 12/09/2024 à 14:23, Qu Wenruo a écrit :
>> 在 2024/9/12 19:34, Archange 写道:
>>> Le 12/09/2024 à 14:01, Qu Wenruo a écrit :
>>>> 在 2024/9/12 19:27, Archange 写道:
>>>>> […]
>>>>>
>>>>> [3/7] checking free space tree
>>>>> there is no free space entry for 0-65536
>>>>> cache appears valid but isn't 0
>>>>
>>>> Then it's totally fine.
>>>>
>>>> For the 0-65536 problem, mind to provide the following dump?
>>>>
>>>> # btrfs ins dump-tree -t fst <device>
>>>>
>>>> I'm afraid since the fs is somewhat old, there may be some corner case
>>>> btrfs-check is not handling properly.
>>>
>>> ERROR: unexpected tree id suffix of 'fst': t
>>
>> My bad, it should be "btrfs ins dump-tree -t free-space <device>".
> 
> The output is too big for an email, so uploaded here:
> 
> https://paste.xinu.at/XtR8/
> 
>> And if possible, also "btrfs ins dump-tree -t extent <device>" just in 
>> case.
> 
> Same thing (even bigger), also output on the terminal and while 
> redirecting to a file was quite different (but maybe that’s more because 
> something changed between the two calls), so here are:
> 
> – the cli run : https://paste.xinu.at/9vs/
> 
> – the file run: https://paste.xinu.at/XpzhbZ/

Thanks a lot.

This indeed shows a very old filesystem, and for a long long time, we no 
longer create any block group at logical bytenr 0, thus it shows an 
corner case that older fs layout doesn't exclude the first 1MiB.

And it's indeed a false alert.

In that case, as long as you still have unallocated space, you can just 
relocate the system chunks:

# btrfs balacne start -s <mnt>

Which should move the system chunks to new locations and will not 
utilize the first 1MiB reserved space.

Thanks,
Qu
> 
> Regards,
> Archange
> 

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-12 21:42                               ` Qu Wenruo
@ 2024-09-13  5:25                                 ` Archange
  2024-09-13  5:29                                   ` Qu Wenruo
  0 siblings, 1 reply; 24+ messages in thread
From: Archange @ 2024-09-13  5:25 UTC (permalink / raw)
  To: Qu Wenruo, Qu Wenruo, linux-btrfs


Le 13/09/2024 à 01:42, Qu Wenruo a écrit :
>
>
> 在 2024/9/12 21:43, Archange 写道:
>> Le 12/09/2024 à 14:23, Qu Wenruo a écrit :
>>> 在 2024/9/12 19:34, Archange 写道:
>>>> Le 12/09/2024 à 14:01, Qu Wenruo a écrit :
>>>>> 在 2024/9/12 19:27, Archange 写道:
>>>>>> […]
>>>>>>
>>>>>> [3/7] checking free space tree
>>>>>> there is no free space entry for 0-65536
>>>>>> cache appears valid but isn't 0
>>>>>
>>>>> Then it's totally fine.
>>>>>
>>>>> For the 0-65536 problem, mind to provide the following dump?
>>>>>
>>>>> # btrfs ins dump-tree -t fst <device>
>>>>>
>>>>> I'm afraid since the fs is somewhat old, there may be some corner 
>>>>> case
>>>>> btrfs-check is not handling properly.
>>>>
>>>> ERROR: unexpected tree id suffix of 'fst': t
>>>
>>> My bad, it should be "btrfs ins dump-tree -t free-space <device>".
>>
>> The output is too big for an email, so uploaded here:
>>
>> https://paste.xinu.at/XtR8/
>>
>>> And if possible, also "btrfs ins dump-tree -t extent <device>" just 
>>> in case.
>>
>> Same thing (even bigger), also output on the terminal and while 
>> redirecting to a file was quite different (but maybe that’s more 
>> because something changed between the two calls), so here are:
>>
>> – the cli run : https://paste.xinu.at/9vs/
>>
>> – the file run: https://paste.xinu.at/XpzhbZ/
>
> Thanks a lot.
>
> This indeed shows a very old filesystem, and for a long long time, we 
> no longer create any block group at logical bytenr 0, thus it shows an 
> corner case that older fs layout doesn't exclude the first 1MiB.

IIRC this file system was created in 2016.

> And it's indeed a false alert.
>
> In that case, as long as you still have unallocated space, you can 
> just relocate the system chunks:
>
> # btrfs balacne start -s <mnt>
>
> Which should move the system chunks to new locations and will not 
> utilize the first 1MiB reserved space.

# btrfs balance start -s /
ERROR: Refusing to explicitly operate on system chunks.
Pass --force if you really want to do that.

According to https://btrfs.readthedocs.io/en/latest/btrfs-balance.html, 
-s requires -f, so I guess I should continue with that?

Regards,
Archange


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-13  5:25                                 ` Archange
@ 2024-09-13  5:29                                   ` Qu Wenruo
  2024-09-13  5:54                                     ` Archange
  0 siblings, 1 reply; 24+ messages in thread
From: Qu Wenruo @ 2024-09-13  5:29 UTC (permalink / raw)
  To: Archange, Qu Wenruo, linux-btrfs



在 2024/9/13 14:55, Archange 写道:
[...]
>>
>> This indeed shows a very old filesystem, and for a long long time, we
>> no longer create any block group at logical bytenr 0, thus it shows an
>> corner case that older fs layout doesn't exclude the first 1MiB.
>
> IIRC this file system was created in 2016.

That explains why the first chunk starts at exactly bytenr 0.

Newer fs will start at bytenr 1M, so it won't touch the reserved range.

>
>> And it's indeed a false alert.
>>
>> In that case, as long as you still have unallocated space, you can
>> just relocate the system chunks:
>>
>> # btrfs balacne start -s <mnt>
>>
>> Which should move the system chunks to new locations and will not
>> utilize the first 1MiB reserved space.
>
> # btrfs balance start -s /
> ERROR: Refusing to explicitly operate on system chunks.
> Pass --force if you really want to do that.
>
> According to https://btrfs.readthedocs.io/en/latest/btrfs-balance.html,
> -s requires -f, so I guess I should continue with that?

Yes.

And I also recommend to convert your metadata and system chunks to DUP,
if there are enough unallocated space.
(If have more devices then RAID1).

It looks like the old mkfs defaults to SINGLE for SSDs, but nowadays we
keep DUP no matter if it's SSD or not.

Thanks,
Qu
>
> Regards,
> Archange
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-13  5:29                                   ` Qu Wenruo
@ 2024-09-13  5:54                                     ` Archange
  2024-09-13  6:12                                       ` Archange
  2024-09-13  6:41                                       ` Qu Wenruo
  0 siblings, 2 replies; 24+ messages in thread
From: Archange @ 2024-09-13  5:54 UTC (permalink / raw)
  To: Qu Wenruo, Qu Wenruo, linux-btrfs

Le 13/09/2024 à 09:29, Qu Wenruo a écrit :
> 在 2024/9/13 14:55, Archange 写道:
> [...]
>>> And it's indeed a false alert.
>>>
>>> In that case, as long as you still have unallocated space, you can
>>> just relocate the system chunks:
>>>
>>> # btrfs balacne start -s <mnt>
>>>
>>> Which should move the system chunks to new locations and will not
>>> utilize the first 1MiB reserved space.
>>
>> # btrfs balance start -s /
>> ERROR: Refusing to explicitly operate on system chunks.
>> Pass --force if you really want to do that.
>>
>> According to https://btrfs.readthedocs.io/en/latest/btrfs-balance.html,
>> -s requires -f, so I guess I should continue with that?
>
> Yes.

Hum, no success:

# btrfs balance start -s --force /
ERROR: error during balancing '/': No space left on device
There may be more info in syslog - try dmesg | tail

# dmesg
[ 2919.917607] BTRFS info (device dm-0): balance: start -f -s
[ 2919.918105] BTRFS info (device dm-0): 1 enospc errors during balance
[ 2919.918108] BTRFS info (device dm-0): balance: ended with status: -28

Indeed,

# btrfs filesystem show /dev/mapper/root
Label: 'root'  uuid: e6614f01-6f56-4776-8b0a-c260089c35e7
     Total devices 1 FS bytes used 439.69GiB
     devid    1 size 476.87GiB used 476.87GiB path /dev/mapper/root

There is unused space though, but not sure how to reclaim it.

$ btrfs filesystem df /
Data, single: total=472.87GiB, used=438.21GiB
System, single: total=4.00MiB, used=80.00KiB
Metadata, single: total=4.00GiB, used=1.48GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

As advised on the balance page, I’ve tried to run with `usage=0` as 
filter (for both m, s and d), but the result is always:

Done, had to relocate 0 out of 480 chunks

> And I also recommend to convert your metadata and system chunks to DUP,
> if there are enough unallocated space.
> (If have more devices then RAID1).
>
> It looks like the old mkfs defaults to SINGLE for SSDs, but nowadays we
> keep DUP no matter if it's SSD or not.

Alright, but I guess I need to solve -ENOSPC first…

Thanks,
Archange


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-13  5:54                                     ` Archange
@ 2024-09-13  6:12                                       ` Archange
  2024-09-13  6:41                                       ` Qu Wenruo
  1 sibling, 0 replies; 24+ messages in thread
From: Archange @ 2024-09-13  6:12 UTC (permalink / raw)
  To: Qu Wenruo, Qu Wenruo, linux-btrfs

Le 13/09/2024 à 09:54, Archange a écrit :
> Le 13/09/2024 à 09:29, Qu Wenruo a écrit :
>> 在 2024/9/13 14:55, Archange 写道:
>> [...]
>>>> And it's indeed a false alert.
>>>>
>>>> In that case, as long as you still have unallocated space, you can
>>>> just relocate the system chunks:
>>>>
>>>> # btrfs balacne start -s <mnt>
>>>>
>>>> Which should move the system chunks to new locations and will not
>>>> utilize the first 1MiB reserved space.
>>>
>>> # btrfs balance start -s /
>>> ERROR: Refusing to explicitly operate on system chunks.
>>> Pass --force if you really want to do that.
>>>
>>> According to https://btrfs.readthedocs.io/en/latest/btrfs-balance.html,
>>> -s requires -f, so I guess I should continue with that?
>>
>> Yes.
>
> Hum, no success:
>
> # btrfs balance start -s --force /
> ERROR: error during balancing '/': No space left on device
> There may be more info in syslog - try dmesg | tail
>
> # dmesg
> [ 2919.917607] BTRFS info (device dm-0): balance: start -f -s
> [ 2919.918105] BTRFS info (device dm-0): 1 enospc errors during balance
> [ 2919.918108] BTRFS info (device dm-0): balance: ended with status: -28
>
> Indeed,
>
> # btrfs filesystem show /dev/mapper/root
> Label: 'root'  uuid: e6614f01-6f56-4776-8b0a-c260089c35e7
>     Total devices 1 FS bytes used 439.69GiB
>     devid    1 size 476.87GiB used 476.87GiB path /dev/mapper/root
>
> There is unused space though, but not sure how to reclaim it.
>
> $ btrfs filesystem df /
> Data, single: total=472.87GiB, used=438.21GiB
> System, single: total=4.00MiB, used=80.00KiB
> Metadata, single: total=4.00GiB, used=1.48GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> As advised on the balance page, I’ve tried to run with `usage=0` as 
> filter (for both m, s and d), but the result is always:
>
> Done, had to relocate 0 out of 480 chunks

OK, ramping usage until something was moved did the trick.

Thanks again for your help,
Archange


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: Critical error from Tree-checker
  2024-09-13  5:54                                     ` Archange
  2024-09-13  6:12                                       ` Archange
@ 2024-09-13  6:41                                       ` Qu Wenruo
  1 sibling, 0 replies; 24+ messages in thread
From: Qu Wenruo @ 2024-09-13  6:41 UTC (permalink / raw)
  To: Archange, Qu Wenruo, linux-btrfs



在 2024/9/13 15:24, Archange 写道:
> Le 13/09/2024 à 09:29, Qu Wenruo a écrit :
>> 在 2024/9/13 14:55, Archange 写道:
>> [...]
>>>> And it's indeed a false alert.
>>>>
>>>> In that case, as long as you still have unallocated space, you can
>>>> just relocate the system chunks:
>>>>
>>>> # btrfs balacne start -s <mnt>
>>>>
>>>> Which should move the system chunks to new locations and will not
>>>> utilize the first 1MiB reserved space.
>>>
>>> # btrfs balance start -s /
>>> ERROR: Refusing to explicitly operate on system chunks.
>>> Pass --force if you really want to do that.
>>>
>>> According to https://btrfs.readthedocs.io/en/latest/btrfs-balance.html,
>>> -s requires -f, so I guess I should continue with that?
>>
>> Yes.
> 
> Hum, no success:
> 
> # btrfs balance start -s --force /
> ERROR: error during balancing '/': No space left on device
> There may be more info in syslog - try dmesg | tail
> 
> # dmesg
> [ 2919.917607] BTRFS info (device dm-0): balance: start -f -s
> [ 2919.918105] BTRFS info (device dm-0): 1 enospc errors during balance
> [ 2919.918108] BTRFS info (device dm-0): balance: ended with status: -28
> 
> Indeed,
> 
> # btrfs filesystem show /dev/mapper/root
> Label: 'root'  uuid: e6614f01-6f56-4776-8b0a-c260089c35e7
>      Total devices 1 FS bytes used 439.69GiB
>      devid    1 size 476.87GiB used 476.87GiB path /dev/mapper/root
> 
> There is unused space though, but not sure how to reclaim it.
> 
> $ btrfs filesystem df /
> Data, single: total=472.87GiB, used=438.21GiB
> System, single: total=4.00MiB, used=80.00KiB
> Metadata, single: total=4.00GiB, used=1.48GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B

This means most of the "free" space is inside data chunks.

You will need to free up some continuous data space, or use different 
"usage=" filters for data and metadata, or both.

Thankfully your metadata has enough free space, so not an urgent situation.

Thanks,
Qu
> 
> As advised on the balance page, I’ve tried to run with `usage=0` as 
> filter (for both m, s and d), but the result is always:
> 
> Done, had to relocate 0 out of 480 chunks
> 
>> And I also recommend to convert your metadata and system chunks to DUP,
>> if there are enough unallocated space.
>> (If have more devices then RAID1).
>>
>> It looks like the old mkfs defaults to SINGLE for SSDs, but nowadays we
>> keep DUP no matter if it's SSD or not.
> 
> Alright, but I guess I need to solve -ENOSPC first…
> 
> Thanks,
> Archange
> 

^ permalink raw reply	[flat|nested] 24+ messages in thread

end of thread, other threads:[~2024-09-13  6:41 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-09-10 21:05 Critical error from Tree-checker Archange
2024-09-10 21:28 ` Qu Wenruo
2024-09-10 21:37   ` Qu Wenruo
2024-09-11 19:55     ` Archange
2024-09-11 20:54       ` Qu Wenruo
2024-09-11 21:20         ` Archange
2024-09-11 21:23           ` Qu Wenruo
2024-09-11 21:42             ` Qu Wenruo
2024-09-11 22:06               ` Archange
2024-09-11 22:05             ` Archange
2024-09-11 22:34               ` Qu Wenruo
2024-09-12  8:21                 ` Archange
2024-09-12  8:25                   ` Qu Wenruo
2024-09-12  9:57                     ` Archange
2024-09-12 10:01                       ` Qu Wenruo
2024-09-12 10:04                         ` Archange
2024-09-12 10:23                           ` Qu Wenruo
2024-09-12 12:13                             ` Archange
2024-09-12 21:42                               ` Qu Wenruo
2024-09-13  5:25                                 ` Archange
2024-09-13  5:29                                   ` Qu Wenruo
2024-09-13  5:54                                     ` Archange
2024-09-13  6:12                                       ` Archange
2024-09-13  6:41                                       ` Qu Wenruo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox