Re: Critical error from Tree-checker

[Archange <archange@archlinux.org>]

Le 11/09/2024 à 01:37, Qu Wenruo a écrit :
> 在 2024/9/11 06:58, Qu Wenruo 写道:
>> 在 2024/9/11 06:35, Archange 写道:
>>> Hi there,
>>>
>>> Since today, my system started randomly becoming read-only. At that
>>> point I can still run dmesg in an open terminal, so I’ve seen it was
>>> related to a btrfs error, but did not try anything since I could not
>>> open a web browser anymore. But I’ve seen the error to be “BTRFS
>>> critical” and related to a “corrupt leaf”.
>>>
>>> I’ve tried to run `btrfs scrub` on the device after rebooting, and in
>>> fact it aborted almost right away triggering the same error in dmesg
>>> (but not turning the system read-only, so I can copy paste it here):
>>>
>>> [  365.268769] BTRFS info (device dm-0): scrub: started on devid 1
>>> [  385.788000] page: refcount:3 mapcount:0 mapping:00000000d0054cae
>>> index:0x9678888 pfn:0x11ce15
>>> [  385.788015] memcg:ffff9fc94db8f000
>>> [  385.788021] aops:btree_aops [btrfs] ino:1
>>> [  385.788235] flags:
>>> 0x2ffffa000004020(lru|private|node=0|zone=2|lastcpupid=0x1ffff)
>>> [  385.788248] raw: 02ffffa000004020 ffffea9a8574ff88 ffffea9a847385c8
>>> ffff9fc95b8365b0
>>> [  385.788255] raw: 0000000009678888 ffff9fc9ae554000 00000003ffffffff
>>> ffff9fc94db8f000
>>> [  385.788259] page dumped because: eb page dump
>>> [  385.788264] BTRFS critical (device dm-0): corrupt leaf:
>>> block=646267305984 slot=92 extent bytenr=1182031872 len=106496 invalid
>>> data ref objectid value 257
>>
>> Full dmesg please.
>>
>> Normally it should dump the full content of the tree block, to help
>> debugging the problem.
>
> Nevermind, the code doesn't dump the full leaf for debug anyway.
>
> In that case please dump that corrupted leaf by:
>
>  # btrfs ins dump-tree -b 1182031872 /dev/dm-0

Sorry for the delay, in the meantime my computer went unbootable: the 
initramfs went missing, then some systemd files… Last time such things 
happened was when my btrfs went out of free space, but there is plenty 
currently, so I guess this is related to this other kind of btrfs issue 
I’m facing. Now everything seems in order and I’m back to my emails.

Here is the output of the asked command:

# btrfs ins dump-tree -b 1182031872 /dev/dm-0
btrfs-progs v6.10.1
checksum verify failed on 1182031872 wanted 0x00000000 found 0x21b9544e
ERROR: failed to read tree block 1182031872

Some additional informations:

1. I was able to save the log the next time it went read-only, it is a 
bit different:

[ 4588.750188] page: refcount:4 mapcount:0 mapping:00000000d0054cae 
index:0x967c1f0 pfn:0x35077d
[ 4588.750203] memcg:ffff9fc9400ae000
[ 4588.750208] aops:btree_aops [btrfs] ino:1
[ 4588.750407] flags: 
0x2ffff8000004000(private|node=0|zone=2|lastcpupid=0x1ffff)
[ 4588.750419] raw: 02ffff8000004000 0000000000000000 dead000000000122 
ffff9fc95b8365b0
[ 4588.750425] raw: 000000000967c1f0 ffff9fcafdcc2690 00000004ffffffff 
ffff9fc9400ae000
[ 4588.750428] page dumped because: eb page dump
[ 4588.750433] BTRFS critical (device dm-0): corrupt leaf: 
block=646327500800 slot=105 extent bytenr=11287011328 len=114688 invalid 
data ref objectid value 258
[ 4588.750451] BTRFS error (device dm-0): read time tree block 
corruption detected on logical 646327500800 mirror 1
[ 4588.750524] BTRFS error (device dm-0): failed to run delayed ref for 
logical 11285897216 num_bytes 36864 type 178 action 1 ref_mod 1: -5
[ 4588.750542] BTRFS error (device dm-0 state A): Transaction aborted 
(error -5)
[ 4588.750549] BTRFS: error (device dm-0 state A) in 
btrfs_run_delayed_refs:2207: errno=-5 IO failure
[ 4588.750559] BTRFS info (device dm-0 state EA): forced readonly

2. I’ve thus decided to run

# btrfs ins dump-tree -b 11287011328 /dev/dm-0
btrfs-progs v6.10.1
checksum verify failed on 11287011328 wanted 0x00000000 found 0x2c7de3ac
ERROR: failed to read tree block 11287011328

and

# btrfs ins dump-tree -b 11285897216 /dev/dm-0
btrfs-progs v6.10.1
checksum verify failed on 11285897216 wanted 0x28b52ffd found 0xa166b670
ERROR: failed to read tree block 11285897216

3. There is some information on kernel loading

[   12.793025] Btrfs loaded, zoned=yes, fsverity=yes
[   12.886212] BTRFS: device label root devid 1 transid 6065022 
/dev/mapper/root (254:0) scanned by mount (252)
[   12.887845] BTRFS info (device dm-0): first mount of filesystem 
e6614f01-6f56-4776-8b0a-c260089c35e7
[   12.887874] BTRFS info (device dm-0): using crc32c (crc32c-intel) 
checksum algorithm
[   12.887885] BTRFS info (device dm-0): disk space caching is enabled
[   12.907001] BTRFS warning (device dm-0): devid 1 physical 0 len 
4194304 inside the reserved space
[   12.910034] BTRFS info (device dm-0): bdev /dev/mapper/root errs: wr 
0, rd 0, flush 0, corrupt 4, gen 0

(Especially the “corrupt 4” I guess, but the warning above might also be 
relevant?)

4. I’ve run a full scrub while the disk was mounted on another system, 
which returned no error.

5. I’ve also run a check from that system while the fs was not mounted:

# btrfs check /dev/mapper/rootext
Opening filesystem to check...
Checking filesystem on /dev/mapper/rootext
UUID: e6614f01-6f56-4776-8b0a-c260089c35e7
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 480039342080 bytes used, no error found
total csum bytes: 466668868
total tree bytes: 1618149376
total fs tree bytes: 898007040
total extent tree bytes: 138395648
btree space waste bytes: 331985228
file data blocks allocated: 517562126336
  referenced 492533391360

Waiting for further debugging instructions.

Thanks,
Archange