system drive corruption, btrfs check failure

Linux Btrfs filesystem development
 help / color / mirror / Atom feed

* system drive corruption, btrfs check failure
@ 2024-03-29 17:30 Jared Van Bortel
  2024-03-29 23:42 ` Qu Wenruo
  0 siblings, 1 reply; 4+ messages in thread
From: Jared Van Bortel @ 2024-03-29 17:30 UTC (permalink / raw)
  To: Btrfs BTRFS

[-- Attachment #1: Type: text/plain, Size: 2330 bytes --]

Hi,

Yesterday I ran `pacman -Syu` to update my Arch Linux installation. I
saw a lot of complaints from ldconfig, and programs started crashing.
Thinking it was related to having only 7GiB of free space available, I
tried deleting some large files and reinstalling the affected
packages. I saw no clear improvement from this, and eventually decided
to shut my computer down.

I booted memtest, and it completed a full pass without errors. I then
booted a live USB and ran `btrfs check --readonly /dev/nvme0n1p2`, and
saw a long list of errors, realizing my filesystem is most likely
beyond repair.

Basic information (RAID1 metadata, single data):
```
Label: 'System'  uuid: 76721faa-8c32-4e70-8a9e-859dece0aec1
Total devices 2 FS bytes used 2.18TiB
devid    1 size 422.63GiB used 422.63GiB path /dev/nvme0n1p2
devid    2 size 1.82TiB used 1.82TiB path /dev/nvme1n1
```
The installed kernel is linux-zen 6.6.10 with a few patches. The live
USB I'm using has the Arch Linux 6.4.7-arch1-1 kernel. Full `btrfs
check` log and smartctl information is attached.

There are three main errors. One:
```
ref mismatch on [1248293634048 16384] extent item 1, found 0
tree extent[1248293634048, 16384] parent 2368656916480 has no tree block found
incorrect global backref count on 1248293634048 found 1 wanted 0
backpointer mismatch on [1248293634048 16384]
owner ref check failed [1248293634048 16384]
```

Two:
```
ref mismatch on [1261902016512 4096] extent item 2, found 1
data extent[1261902016512, 4096] bytenr mimsmatch, extent item bytenr
1261902016512 file item bytenr 0
data extent[1261902016512, 4096] referencer count mismatch (parent
2369673248768) wanted 1 have 0
backpointer mismatch on [1261902016512 4096]
```

Three:
```
block group 1342751899648 has wrong amount of free space, free space
cache has 34193408 block group has 42893312
failed to load free space cache for block group 1342751899648
```

And this warning:
```
[4/7] checking fs roots
warning line 3916
```

I bought some replacement disks that I can install alongside the old
ones, but I don't have a recent backup of the full FS. It seems to
mount readonly without issue.

What's the best way to recover the data that's left? And is there any
clue here as to what went wrong? I'm really not sure. If this is a
drive failure, it seems premature.

Thanks,
Jared

[-- Attachment #2: smart_nvme0n1.txt --]
[-- Type: text/plain, Size: 3425 bytes --]

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.4.7-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       SAMSUNG MZ1WV480HCGL-000MV
Serial Number:                      S1Y0NY0HC02614
Firmware Version:                   BXU87M9Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Controller ID:                      5197
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          480,103,981,056 [480 GB]
Namespace 1 Utilization:            470,449,700,864 [470 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            000000 0000002538
Local Time is:                      Fri Mar 29 17:08:41 2024 UTC
Firmware Updates (0x07):            3 Slots, Slot 1 R/O
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0014):     DS_Mngmt Sav/Sel_Feat
Log Page Attributes (0x01):         S/H_per_NS

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     8.00W       -        -    0  0  0  0       30      30

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        33 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    1%
Data Units Read:                    93,578,386 [47.9 TB]
Data Units Written:                 190,894,684 [97.7 TB]
Host Read Commands:                 946,743,056
Host Write Commands:                2,347,043,730
Controller Busy Time:               227,928
Power Cycles:                       477
Power On Hours:                     33,750
Unsafe Shutdowns:                   109
Media and Data Integrity Errors:    10
Error Information Log Entries:      620
Warning  Comp. Temperature Time:    87
Critical Comp. Temperature Time:    49

Error Information (NVMe Log 0x01, 16 of 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0        620     0  0x001d  0x4004  0x000            0     0     -
  1        619     0  0x001c  0x4212  0x028            0   255     -
  2        618     0  0x0002  0x4004  0x000            0     0     -
  3        617     0  0x001d  0x4004  0x000            0     0     -
  4        616     0  0x001c  0x4212  0x028            0   255     -
  5        615     0  0x0002  0x4004  0x000            0     0     -
  6        614     0  0x001d  0x4004  0x000            0     0     -
  7        613     0  0x001c  0x4212  0x028            0   255     -
  8        612     0  0x0002  0x4004  0x000            0     0     -
  9        611     0  0x001d  0x4004  0x000            0     0     -
 10        610     0  0x001c  0x4212  0x028            0   255     -
 11        609     0  0x0002  0x4004  0x000            0     0     -
 12        608     0  0x001d  0x4004  0x000            0     0     -
 13        607     0  0x001c  0x4212  0x028            0   255     -
 14        606     0  0x0002  0x4004  0x000            0     0     -
 15        605     0  0xa01a  0x4004  0x000            0     1     -
... (48 entries not read)

[-- Attachment #3: check.log.gz --]
[-- Type: application/x-gzip, Size: 66526 bytes --]

[-- Attachment #4: smart_nvme1n1.txt --]
[-- Type: text/plain, Size: 2850 bytes --]

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.4.7-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       PCIe SSD
Serial Number:                      21051220002754
Firmware Version:                   ECFM53.0
PCI Vendor/Subsystem ID:            0x1987
IEEE OUI Identifier:                0x6479a7
Total NVM Capacity:                 2,000,398,934,016 [2.00 TB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          2,000,398,934,016 [2.00 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            6479a7 4d10203702
Local Time is:                      Fri Mar 29 17:09:07 2024 UTC
Firmware Updates (0x12):            1 Slot, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005d):     Comp DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x08):         Telmtry_Lg
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     75 Celsius
Critical Comp. Temp. Threshold:     80 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     7.13W       -        -    0  0  0  0        0       0
 1 +     5.29W       -        -    1  1  1  1        0       0
 2 +     4.36W       -        -    2  2  2  2        0       0
 3 -   0.0490W       -        -    3  3  3  3     2000    2000
 4 -   0.0018W       -        -    4  4  4  4    25000   25000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        29 Celsius
Available Spare:                    100%
Available Spare Threshold:          5%
Percentage Used:                    20%
Data Units Read:                    155,463,764 [79.5 TB]
Data Units Written:                 246,041,613 [125 TB]
Host Read Commands:                 936,230,375
Host Write Commands:                1,683,141,962
Controller Busy Time:               5,320
Power Cycles:                       338
Power On Hours:                     20,034
Unsafe Shutdowns:                   77
Media and Data Integrity Errors:    0
Error Information Log Entries:      661
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 63 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0        661     0  0x5003  0x4004  0x028            0     0     -

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: system drive corruption, btrfs check failure
  2024-03-29 17:30 system drive corruption, btrfs check failure Jared Van Bortel
@ 2024-03-29 23:42 ` Qu Wenruo
  2024-05-19  2:17   ` Jared Van Bortel
  0 siblings, 1 reply; 4+ messages in thread
From: Qu Wenruo @ 2024-03-29 23:42 UTC (permalink / raw)
  To: Jared Van Bortel, Btrfs BTRFS



在 2024/3/30 04:00, Jared Van Bortel 写道:
> Hi,
>
> Yesterday I ran `pacman -Syu` to update my Arch Linux installation. I
> saw a lot of complaints from ldconfig, and programs started crashing.
> Thinking it was related to having only 7GiB of free space available, I
> tried deleting some large files and reinstalling the affected
> packages. I saw no clear improvement from this, and eventually decided
> to shut my computer down.

Do you have any dmesg of that incident?

>
> I booted memtest, and it completed a full pass without errors. I then
> booted a live USB and ran `btrfs check --readonly /dev/nvme0n1p2`, and
> saw a long list of errors, realizing my filesystem is most likely
> beyond repair.
>
> Basic information (RAID1 metadata, single data):
> ```
> Label: 'System'  uuid: 76721faa-8c32-4e70-8a9e-859dece0aec1
> Total devices 2 FS bytes used 2.18TiB
> devid    1 size 422.63GiB used 422.63GiB path /dev/nvme0n1p2
> devid    2 size 1.82TiB used 1.82TiB path /dev/nvme1n1
> ```
> The installed kernel is linux-zen 6.6.10 with a few patches. The live
> USB I'm using has the Arch Linux 6.4.7-arch1-1 kernel. Full `btrfs
> check` log and smartctl information is attached.
>
> There are three main errors. One:
> ```
> ref mismatch on [1248293634048 16384] extent item 1, found 0
> tree extent[1248293634048, 16384] parent 2368656916480 has no tree block found
> incorrect global backref count on 1248293634048 found 1 wanted 0
> backpointer mismatch on [1248293634048 16384]
> owner ref check failed [1248293634048 16384]
> ```
>
> Two:
> ```
> ref mismatch on [1261902016512 4096] extent item 2, found 1
> data extent[1261902016512, 4096] bytenr mimsmatch, extent item bytenr
> 1261902016512 file item bytenr 0
> data extent[1261902016512, 4096] referencer count mismatch (parent
> 2369673248768) wanted 1 have 0
> backpointer mismatch on [1261902016512 4096]
> ```

Corrupted extent tree, this can lead to fs falling back to read-only
halfway.

>
> Three:
> ```
> block group 1342751899648 has wrong amount of free space, free space
> cache has 34193408 block group has 42893312
> failed to load free space cache for block group 1342751899648
> ```

This is not that uncommon if extent  tree is already corrupted.

But unfortunately, this may not be the direct/root cause of the corruption.

Thus I'd prefer to have the initial dmesg.

>
> And this warning:
> ```
> [4/7] checking fs roots
> warning line 3916
> ```
>
> I bought some replacement disks that I can install alongside the old
> ones, but I don't have a recent backup of the full FS. It seems to
> mount readonly without issue.

The fs tree is mostly fine, so you can mount it RO and grab your data.

>
> What's the best way to recover the data that's left? And is there any
> clue here as to what went wrong? I'm really not sure. If this is a
> drive failure, it seems premature.

It's hard to say. The old original mode check output is not that helpful
to locate the root cause.

Mind to run "btrfs check --mode=lowmem" on that fs, and save both stderr
and stdout?

Thanks,
Qu

>
> Thanks,
> Jared

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: system drive corruption, btrfs check failure
  2024-03-29 23:42 ` Qu Wenruo
@ 2024-05-19  2:17   ` Jared Van Bortel
  2024-05-19  3:34     ` Qu Wenruo
  0 siblings, 1 reply; 4+ messages in thread
From: Jared Van Bortel @ 2024-05-19  2:17 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Btrfs BTRFS

On Sat, 2024-03-30 at 10:12 +1030, Qu Wenruo wrote:
> 
> 
> 在 2024/3/30 04:00, Jared Van Bortel 写道:
> > Hi,
> > 
> > Yesterday I ran `pacman -Syu` to update my Arch Linux installation.
> > I
> > saw a lot of complaints from ldconfig, and programs started
> > crashing.
> > Thinking it was related to having only 7GiB of free space available,
> > I
> > tried deleting some large files and reinstalling the affected
> > packages. I saw no clear improvement from this, and eventually
> > decided
> > to shut my computer down.
> 
> Do you have any dmesg of that incident?

Hi, sorry for the delay. I finally got around to running the lowmem
check on the old drives.

Firstly, there was nothing relevant in dmesg. When I first saw your
reply, I checked the system journal from the time of the incident and
there was nothing disk-related from the kernel between mount and
shutdown - medium errors, btrfs I/O errors, or anything like that.

> 
> > 
> > I booted memtest, and it completed a full pass without errors. I
> > then
> > booted a live USB and ran `btrfs check --readonly /dev/nvme0n1p2`,
> > and
> > saw a long list of errors, realizing my filesystem is most likely
> > beyond repair.
> > 
> > Basic information (RAID1 metadata, single data):
> > ```
> > Label: 'System'  uuid: 76721faa-8c32-4e70-8a9e-859dece0aec1
> > Total devices 2 FS bytes used 2.18TiB
> > devid    1 size 422.63GiB used 422.63GiB path /dev/nvme0n1p2
> > devid    2 size 1.82TiB used 1.82TiB path /dev/nvme1n1
> > ```
> > The installed kernel is linux-zen 6.6.10 with a few patches. The
> > live
> > USB I'm using has the Arch Linux 6.4.7-arch1-1 kernel. Full `btrfs
> > check` log and smartctl information is attached.
> > 
> > There are three main errors. One:
> > ```
> > ref mismatch on [1248293634048 16384] extent item 1, found 0
> > tree extent[1248293634048, 16384] parent 2368656916480 has no tree
> > block found
> > incorrect global backref count on 1248293634048 found 1 wanted 0
> > backpointer mismatch on [1248293634048 16384]
> > owner ref check failed [1248293634048 16384]
> > ```
> > 
> > Two:
> > ```
> > ref mismatch on [1261902016512 4096] extent item 2, found 1
> > data extent[1261902016512, 4096] bytenr mimsmatch, extent item
> > bytenr
> > 1261902016512 file item bytenr 0
> > data extent[1261902016512, 4096] referencer count mismatch (parent
> > 2369673248768) wanted 1 have 0
> > backpointer mismatch on [1261902016512 4096]
> > ```
> 
> Corrupted extent tree, this can lead to fs falling back to read-only
> halfway.

This fs actually still mounts writable without any issue, FWIW. Although
the error counters are not zeroed:

bdev /dev/nvme0n1p2 errs: wr 51, rd 0, flush 0, corrupt 0, gen 5

It's not clear to me when these errors occurred - wouldn't they have
been logged to dmesg at the time?

> > 
> > Three:
> > ```
> > block group 1342751899648 has wrong amount of free space, free space
> > cache has 34193408 block group has 42893312
> > failed to load free space cache for block group 1342751899648
> > ```
> 
> This is not that uncommon if extent  tree is already corrupted.
> 
> But unfortunately, this may not be the direct/root cause of the
> corruption.
> 
> Thus I'd prefer to have the initial dmesg.
> 
> > 
> > And this warning:
> > ```
> > [4/7] checking fs roots
> > warning line 3916
> > ```
> > 
> > I bought some replacement disks that I can install alongside the old
> > ones, but I don't have a recent backup of the full FS. It seems to
> > mount readonly without issue.
> 
> The fs tree is mostly fine, so you can mount it RO and grab your data.
> 
> > 
> > What's the best way to recover the data that's left? And is there
> > any
> > clue here as to what went wrong? I'm really not sure. If this is a
> > drive failure, it seems premature.
> 
> It's hard to say. The old original mode check output is not that
> helpful
> to locate the root cause.
> 
> Mind to run "btrfs check --mode=lowmem" on that fs, and save both
> stderr
> and stdout?

Here is the output:

$ sudo btrfs check --mode=lowmem /dev/nvme0n1p2
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p2
UUID: 76721faa-8c32-4e70-8a9e-859dece0aec1
[1/7] checking root items
[2/7] checking extents
ERROR: shared extent[1248293634048 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248304054272 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248304807936 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248315129856 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248324468736 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248588103680 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248685195264 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248901791744 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248939687936 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248953860096 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248962543616 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248966279168 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248966934528 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1248978468864 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1249032519680 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1249075806208 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1249310851072 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1249319256064 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1249401454592 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1249661976576 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1249796259840 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1249802420224 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1250339143680 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1250870542336 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1250975170560 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1251038543872 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1251216883712 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1251319889920 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1251321462784 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1251324887040 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1251325100032 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1253394546688 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1254622068736 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1255368310784 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1255986282496 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1255986331648 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1256068186112 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1256145797120 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1256190967808 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1256305262592 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1256650244096 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1256696086528 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1258225631232 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1258967351296 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1258967367680 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1258967400448 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1258967482368 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1258967515136 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1258967531520 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1258967564288 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1259028234240 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1259345543168 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1259767791616 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260607963136 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260611665920 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260611862528 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260617236480 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260620627968 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260707020800 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260716212224 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260718686208 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260725223424 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260836290560 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260837158912 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260838322176 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260840239104 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260841402368 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260841910272 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260842811392 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260843696128 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260848300032 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260850085888 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260851412992 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260852494336 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260854067200 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260856000512 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260856590336 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260857556992 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260859555840 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260861636608 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260861898752 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260863062016 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[1260863815680 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent 1284548759552 referencer lost (parent: 1252056268800)
ERROR: shared extent 1287628115968 referencer lost (parent: 1252056268800)
ERROR: shared extent 1287628156928 referencer lost (parent: 1252056268800)
ERROR: shared extent 1291874332672 referencer lost (parent: 1252056268800)
ERROR: shared extent 1291892744192 referencer lost (parent: 1252056268800)
ERROR: shared extent 1291893682176 referencer lost (parent: 1252056268800)
ERROR: shared extent 1302641364992 referencer lost (parent: 1252056268800)
ERROR: shared extent 1302646538240 referencer lost (parent: 1252056268800)
ERROR: shared extent 1302646906880 referencer lost (parent: 1252056268800)
ERROR: shared extent 1302725738496 referencer lost (parent: 1252056268800)
ERROR: shared extent 1307336536064 referencer lost (parent: 1252056268800)
ERROR: shared extent 1307421573120 referencer lost (parent: 1252056268800)
ERROR: shared extent 1318061346816 referencer lost (parent: 1252056268800)
ERROR: shared extent 1321886617600 referencer lost (parent: 1252056268800)
ERROR: shared extent 1548210819072 referencer lost (parent: 1252056268800)
ERROR: shared extent 1564826959872 referencer lost (parent: 1252056268800)
ERROR: shared extent 1609847508992 referencer lost (parent: 1252056268800)
ERROR: shared extent 1609915379712 referencer lost (parent: 1252056268800)
ERROR: shared extent 1611724808192 referencer lost (parent: 1252056268800)
ERROR: shared extent 1611749105664 referencer lost (parent: 1252056268800)
ERROR: shared extent 1611749363712 referencer lost (parent: 1252056268800)
ERROR: shared extent 1620880302080 referencer lost (parent: 1252056268800)
ERROR: shared extent 1620899688448 referencer lost (parent: 1252056268800)
ERROR: shared extent 1620944224256 referencer lost (parent: 1252056268800)
ERROR: shared extent 1623958888448 referencer lost (parent: 1252056268800)
ERROR: shared extent 1624013213696 referencer lost (parent: 1252056268800)
ERROR: shared extent 1624013307904 referencer lost (parent: 1252056268800)
ERROR: shared extent[2368230686720 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2368279707648 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2368409845760 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2368417906688 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2368656982016 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369673248768 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369795997696 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369801355264 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369801568256 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369801666560 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369807466496 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369807548416 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369808187392 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369808678912 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369808990208 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369809530880 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369809661952 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369809743872 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369811316736 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369811365888 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369811562496 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369811906560 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369811972096 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369812234240 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369812365312 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369813331968 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369813610496 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369814020096 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369816100864 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369816231936 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369816788992 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369817116672 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369817182208 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369817247744 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369817296896 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369817395200 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369817509888 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369817542656 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369817853952 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369818001408 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369818116096 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369818148864 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369818263552 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369818492928 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369818640384 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369818869760 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369819623424 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369819869184 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369821540352 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369821622272 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369821949952 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369822572544 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369823145984 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369823195136 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369823621120 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369824112640 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2369863860224 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2370334277632 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[2370373042176 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949035106304 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949037793280 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949039235072 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949053538304 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949401337856 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949457502208 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949804662784 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949804761088 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949804777472 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949804793856 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: shared extent[4949849210880 16384] lost its parent (parent: 2368656916480, level: 0)
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
block group 1342751899648 has wrong amount of free space, free space cache has 34193408 block group has 42893312
failed to load free space cache for block group 1342751899648
block group 1343825641472 has wrong amount of free space, free space cache has 23257088 block group has 25903104
failed to load free space cache for block group 1343825641472
block group 1351341834240 has wrong amount of free space, free space cache has 22396928 block group has 42348544
failed to load free space cache for block group 1351341834240
block group 1566090199040 has wrong amount of free space, free space cache has 48562176 block group has 50651136
failed to load free space cache for block group 1566090199040
block group 1572532649984 has wrong amount of free space, free space cache has 12173312 block group has 15945728
failed to load free space cache for block group 1572532649984
block group 1580048842752 has wrong amount of free space, free space cache has 29745152 block group has 33087488
failed to load free space cache for block group 1580048842752
block group 1584343810048 has wrong amount of free space, free space cache has 56512512 block group has 58601472
failed to load free space cache for block group 1584343810048
block group 1602597421056 has wrong amount of free space, free space cache has 87953408 block group has 90349568
failed to load free space cache for block group 1602597421056
block group 1744331341824 has wrong amount of free space, free space cache has 339968 block group has 393216
failed to load free space cache for block group 1744331341824
block group 2666675568640 has wrong amount of free space, free space cache has 602112 block group has 782336
failed to load free space cache for block group 2666675568640
block group 2909341220864 has wrong amount of free space, free space cache has 65536 block group has 151552
failed to load free space cache for block group 2909341220864
block group 3904699891712 has wrong amount of free space, free space cache has 172032 block group has 221184
failed to load free space cache for block group 3904699891712
block group 3941207113728 has wrong amount of free space, free space cache has 1728512 block group has 1826816
failed to load free space cache for block group 3941207113728
block group 4085088518144 has wrong amount of free space, free space cache has 5697536 block group has 5754880
failed to load free space cache for block group 4085088518144
block group 4241854824448 has wrong amount of free space, free space cache has 23293952 block group has 28966912
failed to load free space cache for block group 4241854824448
block group 4838855278592 has wrong amount of free space, free space cache has 86016 block group has 118784
failed to load free space cache for block group 4838855278592
block group 4847445213184 has wrong amount of free space, free space cache has 49152 block group has 110592
failed to load free space cache for block group 4847445213184
block group 4897911078912 has wrong amount of free space, free space cache has 7475200 block group has 7577600
failed to load free space cache for block group 4897911078912
block group 5010008178688 has wrong amount of free space, free space cache has 69632 block group has 106496
failed to load free space cache for block group 5010008178688
block group 5062655082496 has wrong amount of free space, free space cache has 5836800 block group has 5890048
failed to load free space cache for block group 5062655082496
block group 5268813512704 has wrong amount of free space, free space cache has 135168 block group has 221184
failed to load free space cache for block group 5268813512704
[4/7] checking fs roots
ERROR: root 259 EXTENT_DATA[1522634 4096] gap exists, expected: EXTENT_DATA[1522634 128]
ERROR: root 259 EXTENT_DATA[1522636 4096] gap exists, expected: EXTENT_DATA[1522636 128]
ERROR: root 407 EXTENT_DATA[398831 4096] gap exists, expected: EXTENT_DATA[398831 25]
ERROR: root 407 EXTENT_DATA[398973 4096] gap exists, expected: EXTENT_DATA[398973 25]
ERROR: root 407 EXTENT_DATA[398975 4096] gap exists, expected: EXTENT_DATA[398975 25]
ERROR: root 407 EXTENT_DATA[398976 4096] gap exists, expected: EXTENT_DATA[398976 25]
ERROR: root 407 EXTENT_DATA[418307 4096] gap exists, expected: EXTENT_DATA[418307 25]
ERROR: root 407 EXTENT_DATA[418316 4096] gap exists, expected: EXTENT_DATA[418316 25]
ERROR: root 407 EXTENT_DATA[418317 4096] gap exists, expected: EXTENT_DATA[418317 25]
ERROR: root 407 EXTENT_DATA[420660 4096] gap exists, expected: EXTENT_DATA[420660 25]
ERROR: root 407 EXTENT_DATA[420673 4096] gap exists, expected: EXTENT_DATA[420673 25]
ERROR: root 407 EXTENT_DATA[439382 4096] gap exists, expected: EXTENT_DATA[439382 25]
ERROR: root 407 EXTENT_DATA[439383 4096] gap exists, expected: EXTENT_DATA[439383 25]
ERROR: root 407 EXTENT_DATA[451252 4096] gap exists, expected: EXTENT_DATA[451252 25]
ERROR: root 407 EXTENT_DATA[451264 4096] gap exists, expected: EXTENT_DATA[451264 25]
ERROR: root 407 EXTENT_DATA[451265 4096] gap exists, expected: EXTENT_DATA[451265 25]
ERROR: root 407 EXTENT_DATA[452326 4096] gap exists, expected: EXTENT_DATA[452326 25]
ERROR: root 407 EXTENT_DATA[452332 4096] gap exists, expected: EXTENT_DATA[452332 25]
ERROR: root 407 EXTENT_DATA[452339 4096] gap exists, expected: EXTENT_DATA[452339 25]
ERROR: root 407 EXTENT_DATA[4293157 4096] gap exists, expected: EXTENT_DATA[4293157 25]
ERROR: root 407 EXTENT_DATA[4293570 4096] gap exists, expected: EXTENT_DATA[4293570 25]
ERROR: root 407 EXTENT_DATA[4293571 4096] gap exists, expected: EXTENT_DATA[4293571 25]
ERROR: root 407 EXTENT_DATA[4293572 4096] gap exists, expected: EXTENT_DATA[4293572 25]
ERROR: root 407 EXTENT_DATA[4302136 4096] gap exists, expected: EXTENT_DATA[4302136 25]
ERROR: root 407 EXTENT_DATA[4302148 4096] gap exists, expected: EXTENT_DATA[4302148 25]
ERROR: root 407 EXTENT_DATA[4302149 4096] gap exists, expected: EXTENT_DATA[4302149 25]
ERROR: root 407 EXTENT_DATA[4302150 4096] gap exists, expected: EXTENT_DATA[4302150 25]
ERROR: root 407 EXTENT_DATA[5970391 4096] gap exists, expected: EXTENT_DATA[5970391 25]
ERROR: errors found in fs roots
found 2397613547520 bytes used, error(s) found
total csum bytes: 1840478932
total tree bytes: 13337329664
total fs tree bytes: 10208378880
total extent tree bytes: 874070016
btree space waste bytes: 2240708820
file data blocks allocated: 24819271946240
 referenced 2695187488768

Hopefully that means something to you. I'm still curious to know to what
degree I should still trust these drives if I were to wipe the fs and
start over. I suppose I could run a SMART test or something, right?

Thanks,
Jared

> 
> Thanks,
> Qu
> 
> > 
> > Thanks,
> > Jared

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: system drive corruption, btrfs check failure
  2024-05-19  2:17   ` Jared Van Bortel
@ 2024-05-19  3:34     ` Qu Wenruo
  0 siblings, 0 replies; 4+ messages in thread
From: Qu Wenruo @ 2024-05-19  3:34 UTC (permalink / raw)
  To: Jared Van Bortel, Qu Wenruo; +Cc: Btrfs BTRFS



在 2024/5/19 11:47, Jared Van Bortel 写道:
> On Sat, 2024-03-30 at 10:12 +1030, Qu Wenruo wrote:
>>
>>
[...]
>>
>> Do you have any dmesg of that incident?
> 
> Hi, sorry for the delay. I finally got around to running the lowmem
> check on the old drives.
> 
> Firstly, there was nothing relevant in dmesg. When I first saw your
> reply, I checked the system journal from the time of the incident and
> there was nothing disk-related from the kernel between mount and
> shutdown - medium errors, btrfs I/O errors, or anything like that.

I believe the fs flipped RO due to extent tree corruption, thus nothing 
can really be recorded into the journal.

[...]
>>
>> Corrupted extent tree, this can lead to fs falling back to read-only
>> halfway.
> 
> This fs actually still mounts writable without any issue, FWIW. Although
> the error counters are not zeroed:

Only when the corrupted extent backref got modified the kernel would 
throw a lot of errors, and then flip to RO.

> 
> bdev /dev/nvme0n1p2 errs: wr 51, rd 0, flush 0, corrupt 0, gen 5

This shows two new findings:

1. There is some write failures, which may or may not be the root cause, 
but I'll be extra cautious.

And even if it's the root cause, it just exposed some error corner cases 
where btrfs kernel module didn't handle it correctly.
At least even we hit some write errors, we should not lead to some 
corrupted extent tree.

> 
> It's not clear to me when these errors occurred - wouldn't they have
> been logged to dmesg at the time?

As explained, if the root fs flipped RO, no journal can be really 
written onto disk.

[...]
>> Mind to run "btrfs check --mode=lowmem" on that fs, and save both
>> stderr
>> and stdout?
> 
> Here is the output:
> 
> $ sudo btrfs check --mode=lowmem /dev/nvme0n1p2
> Opening filesystem to check...
> Checking filesystem on /dev/nvme0n1p2
> UUID: 76721faa-8c32-4e70-8a9e-859dece0aec1
> [1/7] checking root items
> [2/7] checking extents
[...]
> ERROR: shared extent[1249401454592 16384] lost its parent (parent: 2368656916480, level: 0)
[...]
> ERROR: shared extent 1624013307904 referencer lost (parent: 1252056268800)
[...]

So yes, there is something wrong with the extent tree.

The "lost its parent" line means for the shared metadata backref item, 
the parent can not be found.

Thus although you can still mount the fs, if you delete or COW that 
extent, the fs would flip RO.

The later "referencer lost" is for data backref item. It is  mostly just 
caused by the previous "lost its parent" line.



> ERROR: errors found in extent allocation tree or chunk allocation
> [3/7] checking free space cache
> block group 1342751899648 has wrong amount of free space, free space cache has 34193408 block group has 42893312
> failed to load free space cache for block group 1342751899648
> block group 1343825641472 has wrong amount of free space, free space cache has 23257088 block group has 25903104
> failed to load free space cache for block group 1343825641472
> block group 1351341834240 has wrong amount of free space, free space cache has 22396928 block group has 42348544
> failed to load free space cache for block group 1351341834240
> block group 1566090199040 has wrong amount of free space, free space cache has 48562176 block group has 50651136
> failed to load free space cache for block group 1566090199040
> block group 1572532649984 has wrong amount of free space, free space cache has 12173312 block group has 15945728
> failed to load free space cache for block group 1572532649984
> block group 1580048842752 has wrong amount of free space, free space cache has 29745152 block group has 33087488
> failed to load free space cache for block group 1580048842752
> block group 1584343810048 has wrong amount of free space, free space cache has 56512512 block group has 58601472
> failed to load free space cache for block group 1584343810048
> block group 1602597421056 has wrong amount of free space, free space cache has 87953408 block group has 90349568
> failed to load free space cache for block group 1602597421056
> block group 1744331341824 has wrong amount of free space, free space cache has 339968 block group has 393216
> failed to load free space cache for block group 1744331341824
> block group 2666675568640 has wrong amount of free space, free space cache has 602112 block group has 782336
> failed to load free space cache for block group 2666675568640
> block group 2909341220864 has wrong amount of free space, free space cache has 65536 block group has 151552
> failed to load free space cache for block group 2909341220864
> block group 3904699891712 has wrong amount of free space, free space cache has 172032 block group has 221184
> failed to load free space cache for block group 3904699891712
> block group 3941207113728 has wrong amount of free space, free space cache has 1728512 block group has 1826816
> failed to load free space cache for block group 3941207113728
> block group 4085088518144 has wrong amount of free space, free space cache has 5697536 block group has 5754880
> failed to load free space cache for block group 4085088518144
> block group 4241854824448 has wrong amount of free space, free space cache has 23293952 block group has 28966912
> failed to load free space cache for block group 4241854824448
> block group 4838855278592 has wrong amount of free space, free space cache has 86016 block group has 118784
> failed to load free space cache for block group 4838855278592
> block group 4847445213184 has wrong amount of free space, free space cache has 49152 block group has 110592
> failed to load free space cache for block group 4847445213184
> block group 4897911078912 has wrong amount of free space, free space cache has 7475200 block group has 7577600
> failed to load free space cache for block group 4897911078912
> block group 5010008178688 has wrong amount of free space, free space cache has 69632 block group has 106496
> failed to load free space cache for block group 5010008178688
> block group 5062655082496 has wrong amount of free space, free space cache has 5836800 block group has 5890048
> failed to load free space cache for block group 5062655082496
> block group 5268813512704 has wrong amount of free space, free space cache has 135168 block group has 221184
> failed to load free space cache for block group 5268813512704

This is minor error, but I'm not sure if it would lead to more problems.

Overall recommended to go v2 space cache.

> [4/7] checking fs roots
> ERROR: root 259 EXTENT_DATA[1522634 4096] gap exists, expected: EXTENT_DATA[1522634 128]
> ERROR: root 259 EXTENT_DATA[1522636 4096] gap exists, expected: EXTENT_DATA[1522636 128]
> ERROR: root 407 EXTENT_DATA[398831 4096] gap exists, expected: EXTENT_DATA[398831 25]
> ERROR: root 407 EXTENT_DATA[398973 4096] gap exists, expected: EXTENT_DATA[398973 25]
> ERROR: root 407 EXTENT_DATA[398975 4096] gap exists, expected: EXTENT_DATA[398975 25]
> ERROR: root 407 EXTENT_DATA[398976 4096] gap exists, expected: EXTENT_DATA[398976 25]
> ERROR: root 407 EXTENT_DATA[418307 4096] gap exists, expected: EXTENT_DATA[418307 25]
> ERROR: root 407 EXTENT_DATA[418316 4096] gap exists, expected: EXTENT_DATA[418316 25]
> ERROR: root 407 EXTENT_DATA[418317 4096] gap exists, expected: EXTENT_DATA[418317 25]
> ERROR: root 407 EXTENT_DATA[420660 4096] gap exists, expected: EXTENT_DATA[420660 25]
> ERROR: root 407 EXTENT_DATA[420673 4096] gap exists, expected: EXTENT_DATA[420673 25]
> ERROR: root 407 EXTENT_DATA[439382 4096] gap exists, expected: EXTENT_DATA[439382 25]
> ERROR: root 407 EXTENT_DATA[439383 4096] gap exists, expected: EXTENT_DATA[439383 25]
> ERROR: root 407 EXTENT_DATA[451252 4096] gap exists, expected: EXTENT_DATA[451252 25]
> ERROR: root 407 EXTENT_DATA[451264 4096] gap exists, expected: EXTENT_DATA[451264 25]
> ERROR: root 407 EXTENT_DATA[451265 4096] gap exists, expected: EXTENT_DATA[451265 25]
> ERROR: root 407 EXTENT_DATA[452326 4096] gap exists, expected: EXTENT_DATA[452326 25]
> ERROR: root 407 EXTENT_DATA[452332 4096] gap exists, expected: EXTENT_DATA[452332 25]
> ERROR: root 407 EXTENT_DATA[452339 4096] gap exists, expected: EXTENT_DATA[452339 25]
> ERROR: root 407 EXTENT_DATA[4293157 4096] gap exists, expected: EXTENT_DATA[4293157 25]
> ERROR: root 407 EXTENT_DATA[4293570 4096] gap exists, expected: EXTENT_DATA[4293570 25]
> ERROR: root 407 EXTENT_DATA[4293571 4096] gap exists, expected: EXTENT_DATA[4293571 25]
> ERROR: root 407 EXTENT_DATA[4293572 4096] gap exists, expected: EXTENT_DATA[4293572 25]
> ERROR: root 407 EXTENT_DATA[4302136 4096] gap exists, expected: EXTENT_DATA[4302136 25]
> ERROR: root 407 EXTENT_DATA[4302148 4096] gap exists, expected: EXTENT_DATA[4302148 25]
> ERROR: root 407 EXTENT_DATA[4302149 4096] gap exists, expected: EXTENT_DATA[4302149 25]
> ERROR: root 407 EXTENT_DATA[4302150 4096] gap exists, expected: EXTENT_DATA[4302150 25]
> ERROR: root 407 EXTENT_DATA[5970391 4096] gap exists, expected: EXTENT_DATA[5970391 25]

Minor problems, would not cause anything wrong, kernel can easily handle it.

So far the fs looks pretty old (v1 cache, no NO_HOLES feature).
Thus the corruption may be caused by some older kernel bugs.

After backing up critical info, you can try "btrfs check --repair" to 
see if it can fix the extent tree corruption.

> ERROR: errors found in fs roots
> found 2397613547520 bytes used, error(s) found
> total csum bytes: 1840478932
> total tree bytes: 13337329664
> total fs tree bytes: 10208378880
> total extent tree bytes: 874070016
> btree space waste bytes: 2240708820
> file data blocks allocated: 24819271946240
>   referenced 2695187488768
> 
> 
> Hopefully that means something to you. I'm still curious to know to what
> degree I should still trust these drives if I were to wipe the fs and
> start over. I suppose I could run a SMART test or something, right?

So far I do not think there is something related to the device.
The only thing that can lead to the drive is the write errors.

The extent tree corruption so far looks more like a bug in (possibly 
older) kernel.

At least the minor problems would never occur for the new default mkfs 
options.

Thanks,
Qu

> 
> Thanks,
> Jared
> 
>>
>> Thanks,
>> Qu
>>
>>>
>>> Thanks,
>>> Jared
> 
> 

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2024-05-19  3:34 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-03-29 17:30 system drive corruption, btrfs check failure Jared Van Bortel
2024-03-29 23:42 ` Qu Wenruo
2024-05-19  2:17   ` Jared Van Bortel
2024-05-19  3:34     ` Qu Wenruo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox