How to remove an unremovable file and directory?

How to remove an unremovable file and directory?

[Henri Hyyryläinen]

Hello,

I hope this is the right place to ask about a filesystem problem. Really 
shortly put, I have a file that both exists and doesn't and prevents the 
containing directory from being deleted. No matter what variant of rm 
and inode based deletion I try I get an error about the file not 
existing, and I also cannot try to read the file, but if I try to delete 
the directory I get an error that it is not empty (so the file kind of 
exists). Trying to ls the directory also gives a file doesn't exist error.

Here's what btrfs check found, which I hope does better in illustrating 
the problem:

> root 5 inode 25953213 errors 200, dir isize wrong
> root 5 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item

I've tried everything I've found suggested including a full scrub, 
balance with -dusage=75 -musage=75, resetting file attributes, deleting 
through the find command, and even some repair mount flags that don't 
seem to exist for btrfs. What I haven't tried is a full rebalance with 
no filters, but I did not try that yet as it would take quite a long 
time and if it only moves data blocks around without recomputing 
directory items, it doesn't seem like the right tool to fix my problem. 
So I'm pretty much stuck and to me it seems like my only option is to 
run btrfs check with the repair flag, but as that has big warnings on it 
I thought I would try asking here first (sorry if this is not the right 
experts group to ask). So is there still something I can try or am I 
finally "allowed" to use the repair command? Here's the full output I 
got from btrfs check:

> Opening filesystem to check...
> Checking filesystem on /dev/sdc
> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
> [1/8] checking log skipped (none written)
> [2/8] checking root items
> [3/8] checking extents
> [4/8] checking free space tree
> We have a space info key for a block group that doesn't exist
> [5/8] checking fs roots
> root 5 inode 25953213 errors 200, dir isize wrong
> root 5 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14428 inode 25953213 errors 200, dir isize wrong
> root 14428 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14451 inode 25953213 errors 200, dir isize wrong
> root 14451 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14475 inode 25953213 errors 200, dir isize wrong
> root 14475 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14499 inode 25953213 errors 200, dir isize wrong
> root 14499 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14523 inode 25953213 errors 200, dir isize wrong
> root 14523 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14544 inode 25953213 errors 200, dir isize wrong
> root 14544 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14545 inode 25953213 errors 200, dir isize wrong
> root 14545 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14546 inode 25953213 errors 200, dir isize wrong
> root 14546 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14547 inode 25953213 errors 200, dir isize wrong
> root 14547 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14548 inode 25953213 errors 200, dir isize wrong
> root 14548 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14549 inode 25953213 errors 200, dir isize wrong
> root 14549 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> root 14550 inode 25953213 errors 200, dir isize wrong
> root 14550 inode 27166085 errors 2000, link count wrong
>         unresolved ref dir 25953213 index 7 namelen 33 name 
> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> ERROR: errors found in fs roots
> found 49400296812544 bytes used, error(s) found
> total csum bytes: 48179330432
> total tree bytes: 65067483136
> total fs tree bytes: 12107431936
> total extent tree bytes: 3194437632
> btree space waste bytes: 4558984171
> file data blocks allocated: 76487982252032
>  referenced 60030799097856

So hopefully if I'm reading things right, running a repair would delete 
just that one file and directory (which itself is a backup so I will not 
miss that file at all)?

I do not have enough disk space to copy off the entire filesystem and 
rebuild from scratch, without doing something like rebalancing all data 
from raid10 to single and then removing half the disks, but I assume 
that would take at least 4 weeks to process (as I just replaced a disk 
which took like a week).

As to what originally caused the corruption, I think it was probably 
faulty RAM, because up to to like 3 weeks ago I had one really bad RAM 
stick in my computer where a certain memory region always had 
incorrectly reading bytes. I had seen intermittent quite high csum 
errors in monthly scrubs pretty randomly, which thankfully could almost 
always be corrected so I didn't have any major problems even though I 
had like totally broken RAM in my computer for who knows how long. So 
btrfs was able to protect my data quite impressively from bad RAM.

Sorry for getting a bit sidetracked there, but what should I do in this 
situation?

- Henri Hyyryläinen

Re: How to remove an unremovable file and directory?

[Qu Wenruo]

在 2025/9/30 02:41, Henri Hyyryläinen 写道:
> Hello,
> 
> I hope this is the right place to ask about a filesystem problem. Really 
> shortly put, I have a file that both exists and doesn't and prevents the 
> containing directory from being deleted. No matter what variant of rm 
> and inode based deletion I try I get an error about the file not 
> existing, and I also cannot try to read the file, but if I try to delete 
> the directory I get an error that it is not empty (so the file kind of 
> exists). Trying to ls the directory also gives a file doesn't exist error.
> 
> Here's what btrfs check found, which I hope does better in illustrating 
> the problem:
> 
>> root 5 inode 25953213 errors 200, dir isize wrong
>> root 5 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
> 
> I've tried everything I've found suggested including a full scrub, 
> balance with -dusage=75 -musage=75, resetting file attributes, deleting 
> through the find command, and even some repair mount flags that don't 
> seem to exist for btrfs.

The fs is corrupted, thus none of those will help.
I'm more interested in how the corruption happened.

Did you use some tools other than btrfs kernel module and btrfs-progs?
Like ntfs2btrfs or winbtrfs?

IIRC certain versions have some bugs related to extent tree, but should 
not cause this problem.


The other possibility is hardware memory bitflip, which is more common 
than you thought (almostly one report per month)

In that case, a full memtest is always recommended, or you will hit all 
kinds of weird corruptions in the future anyway.


With a full memtest proving the memory hardware is fine, then "btrfs 
check --repair" should be able to fix it.

Thanks,
Qu


> What I haven't tried is a full rebalance with 
> no filters, but I did not try that yet as it would take quite a long 
> time and if it only moves data blocks around without recomputing 
> directory items, it doesn't seem like the right tool to fix my problem. 
> So I'm pretty much stuck and to me it seems like my only option is to 
> run btrfs check with the repair flag, but as that has big warnings on it 
> I thought I would try asking here first (sorry if this is not the right 
> experts group to ask). So is there still something I can try or am I 
> finally "allowed" to use the repair command? Here's the full output I 
> got from btrfs check:
> 
>> Opening filesystem to check...
>> Checking filesystem on /dev/sdc
>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>> [1/8] checking log skipped (none written)
>> [2/8] checking root items
>> [3/8] checking extents
>> [4/8] checking free space tree
>> We have a space info key for a block group that doesn't exist
>> [5/8] checking fs roots
>> root 5 inode 25953213 errors 200, dir isize wrong
>> root 5 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14428 inode 25953213 errors 200, dir isize wrong
>> root 14428 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14451 inode 25953213 errors 200, dir isize wrong
>> root 14451 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14475 inode 25953213 errors 200, dir isize wrong
>> root 14475 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14499 inode 25953213 errors 200, dir isize wrong
>> root 14499 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14523 inode 25953213 errors 200, dir isize wrong
>> root 14523 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14544 inode 25953213 errors 200, dir isize wrong
>> root 14544 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14545 inode 25953213 errors 200, dir isize wrong
>> root 14545 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14546 inode 25953213 errors 200, dir isize wrong
>> root 14546 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14547 inode 25953213 errors 200, dir isize wrong
>> root 14547 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14548 inode 25953213 errors 200, dir isize wrong
>> root 14548 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14549 inode 25953213 errors 200, dir isize wrong
>> root 14549 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> root 14550 inode 25953213 errors 200, dir isize wrong
>> root 14550 inode 27166085 errors 2000, link count wrong
>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>> ERROR: errors found in fs roots
>> found 49400296812544 bytes used, error(s) found
>> total csum bytes: 48179330432
>> total tree bytes: 65067483136
>> total fs tree bytes: 12107431936
>> total extent tree bytes: 3194437632
>> btree space waste bytes: 4558984171
>> file data blocks allocated: 76487982252032
>>  referenced 60030799097856
> 
> So hopefully if I'm reading things right, running a repair would delete 
> just that one file and directory (which itself is a backup so I will not 
> miss that file at all)?
> 
> I do not have enough disk space to copy off the entire filesystem and 
> rebuild from scratch, without doing something like rebalancing all data 
> from raid10 to single and then removing half the disks, but I assume 
> that would take at least 4 weeks to process (as I just replaced a disk 
> which took like a week).
> 
> As to what originally caused the corruption, I think it was probably 
> faulty RAM, because up to to like 3 weeks ago I had one really bad RAM 
> stick in my computer where a certain memory region always had 
> incorrectly reading bytes. I had seen intermittent quite high csum 
> errors in monthly scrubs pretty randomly, which thankfully could almost 
> always be corrected so I didn't have any major problems even though I 
> had like totally broken RAM in my computer for who knows how long. So 
> btrfs was able to protect my data quite impressively from bad RAM.
> 
> Sorry for getting a bit sidetracked there, but what should I do in this 
> situation?
> 
> - Henri Hyyryläinen
> 
> 

Re: How to remove an unremovable file and directory?

[Henri Hyyryläinen]

Thank you for the reply.

> The fs is corrupted, thus none of those will help.
> I'm more interested in how the corruption happened.
>
> Did you use some tools other than btrfs kernel module and btrfs-progs?
> Like ntfs2btrfs or winbtrfs?
>
> IIRC certain versions have some bugs related to extent tree, but 
> should not cause this problem. 

I did not use those tools. I only have Fedora 42 installed on this computer.

> The other possibility is hardware memory bitflip, which is more common 
> than you thought (almostly one report per month)
>
> In that case, a full memtest is always recommended, or you will hit 
> all kinds of weird corruptions in the future anyway. 

Yes, that seems the most likely explanation. I switched my motherboard, 
RAM, and CPU a few weeks ago, and when building a new setup out of  the 
old components and 3 different drives (including one brand new NVME SSD) 
I noticed that each time I ran a btrfs scrub on any of those I got a 
bunch of csum errors. I initially thought that maybe the drives or 
something were faulty, but when I ran memtest it discovered very quickly 
a faulty single RAM stick. That RAM had been in my main system in a 
faulty state for months probably at least. The different computer I 
built from the old parts immediately stopped failing scrubs when I 
removed the faulty RAM stick.

So yeah, I think the corruption would have been caused by that RAM 
stick. I have now completely different RAM in my system and I've run the 
same memtest multiple times and it has not found any problem, so I think 
I have now fully good RAM and won't have the same corruption issue again.

One potentially relevant detail is that I replaced one of my disks just 
before I discovered the corruption with a bigger hard drive (at this 
point I had already swapped my RAM so there shouldn't be memory 
corruption problems during the replace). I started a btrfs replace 
operation, but I had a scrub running which was then canceled. That 
cancel for some reason also canceled the replace operation. When I 
noticed that I then restarted the replace and then let it run to 
completion (though I think the replace got canceled twice as I didn't 
realize after the first one what canceled it, so I needed to restart the 
replace multiple times). After completing the replace command without an 
error message I shut down my computer and removed the replaced drive. 
Upon booting my filesystem could not mount without the degraded flag. 
Somehow the replace had copied the right amount of data to the new 
drive, but for some reason the old drive was still marked as used. I had 
to then do a btrfs remove on that device which took a really long time 
but eventually succeeded as it seemingly had to rebuild the missing 
drive from raid data (even though all of the necessary data should have 
been copied already). So it seems like the btrfs replace operation 
doesn't fully work correctly if it is interrupted.

That could be a relevant detail, but as my filesystem only had one 
corrupt folder I somewhat doubt that as I would expect a lot more things 
to have gone wrong if the replace messed up the data. So to me that 
seems only like a minor bug in the replace operation, but one that did 
not cause this corruption.

> With a full memtest proving the memory hardware is fine, then "btrfs 
> check --repair" should be able to fix it. 

Yes, my RAM should be totally fine now.

Thanks, I will try running that repair tonight, and hoping everything is 
then fine.


- Henri Hyyryläinen

Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>
>
> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>> Hello,
>>
>> I hope this is the right place to ask about a filesystem problem. 
>> Really shortly put, I have a file that both exists and doesn't and 
>> prevents the containing directory from being deleted. No matter what 
>> variant of rm and inode based deletion I try I get an error about the 
>> file not existing, and I also cannot try to read the file, but if I 
>> try to delete the directory I get an error that it is not empty (so 
>> the file kind of exists). Trying to ls the directory also gives a 
>> file doesn't exist error.
>>
>> Here's what btrfs check found, which I hope does better in 
>> illustrating the problem:
>>
>>> root 5 inode 25953213 errors 200, dir isize wrong
>>> root 5 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>
>> I've tried everything I've found suggested including a full scrub, 
>> balance with -dusage=75 -musage=75, resetting file attributes, 
>> deleting through the find command, and even some repair mount flags 
>> that don't seem to exist for btrfs.
>
> The fs is corrupted, thus none of those will help.
> I'm more interested in how the corruption happened.
>
> Did you use some tools other than btrfs kernel module and btrfs-progs?
> Like ntfs2btrfs or winbtrfs?
>
> IIRC certain versions have some bugs related to extent tree, but 
> should not cause this problem.
>
>
> The other possibility is hardware memory bitflip, which is more common 
> than you thought (almostly one report per month)
>
> In that case, a full memtest is always recommended, or you will hit 
> all kinds of weird corruptions in the future anyway.
>
>
> With a full memtest proving the memory hardware is fine, then "btrfs 
> check --repair" should be able to fix it.
>
> Thanks,
> Qu
>
>
>> What I haven't tried is a full rebalance with no filters, but I did 
>> not try that yet as it would take quite a long time and if it only 
>> moves data blocks around without recomputing directory items, it 
>> doesn't seem like the right tool to fix my problem. So I'm pretty 
>> much stuck and to me it seems like my only option is to run btrfs 
>> check with the repair flag, but as that has big warnings on it I 
>> thought I would try asking here first (sorry if this is not the right 
>> experts group to ask). So is there still something I can try or am I 
>> finally "allowed" to use the repair command? Here's the full output I 
>> got from btrfs check:
>>
>>> Opening filesystem to check...
>>> Checking filesystem on /dev/sdc
>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>> [1/8] checking log skipped (none written)
>>> [2/8] checking root items
>>> [3/8] checking extents
>>> [4/8] checking free space tree
>>> We have a space info key for a block group that doesn't exist
>>> [5/8] checking fs roots
>>> root 5 inode 25953213 errors 200, dir isize wrong
>>> root 5 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> ERROR: errors found in fs roots
>>> found 49400296812544 bytes used, error(s) found
>>> total csum bytes: 48179330432
>>> total tree bytes: 65067483136
>>> total fs tree bytes: 12107431936
>>> total extent tree bytes: 3194437632
>>> btree space waste bytes: 4558984171
>>> file data blocks allocated: 76487982252032
>>>  referenced 60030799097856
>>
>> So hopefully if I'm reading things right, running a repair would 
>> delete just that one file and directory (which itself is a backup so 
>> I will not miss that file at all)?
>>
>> I do not have enough disk space to copy off the entire filesystem and 
>> rebuild from scratch, without doing something like rebalancing all 
>> data from raid10 to single and then removing half the disks, but I 
>> assume that would take at least 4 weeks to process (as I just 
>> replaced a disk which took like a week).
>>
>> As to what originally caused the corruption, I think it was probably 
>> faulty RAM, because up to to like 3 weeks ago I had one really bad 
>> RAM stick in my computer where a certain memory region always had 
>> incorrectly reading bytes. I had seen intermittent quite high csum 
>> errors in monthly scrubs pretty randomly, which thankfully could 
>> almost always be corrected so I didn't have any major problems even 
>> though I had like totally broken RAM in my computer for who knows how 
>> long. So btrfs was able to protect my data quite impressively from 
>> bad RAM.
>>
>> Sorry for getting a bit sidetracked there, but what should I do in 
>> this situation?
>>
>> - Henri Hyyryläinen
>>
>>
>

Re: How to remove an unremovable file and directory?

[Henri Hyyryläinen]

Hello again.

It took over 3 days, but the btrfs check --repair has now completed 
seemingly successfully. I mostly saw output about the file being placed 
in lost+found and and directory size being corrected. However, there 
were some messages about mismatch of used bytes.

Unfortunately it seems like the situation has gotten worse since the 
repair, because now I cannot mount the filesystem at all. Instead I get 
an error like this:

> BTRFS error (device sdc): dev extent physical offset 19977638903808 on 
> devid 4 doesn't have corresponding chunk
> BTRFS error (device sdc): failed to verify dev extents against chunks: 
> -117
> BTRFS error (device sdc): open_ctree failed: -117
Even if I remove that one problematic device physically from my 
computer, the filesystem still refuses to mount with the same error. 
Maybe the problems with the device replace are again showing up with the 
actual size of the hard drive not being used correctly? I cannot try to 
remove the device slack as I cannot mount the filesystem.

I did try to run a repair again, and this time I got a bunch of messages 
like:

> repair deleting extent record: key [65795546775552,169,0]
> adding new tree backref on start 65795546775552 len 16384 parent 
> 65811674234880 root 65811674234880
> adding new tree backref on start 65795546775552 len 16384 parent 0 
> root 14499
> adding new tree backref on start 65795546775552 len 16384 parent 
> 65791012274176 root 65791012274176
> adding new tree backref on start 65795546775552 len 16384 parent 
> 65806385807360 root 65806385807360
> Repaired extent references for 65795546775552
> ref mismatch on [65795548233728 16384] extent item 5, found 4
But the filesystem still refuses to mount with the exact same error. I 
did not let the repair run entirely as it would have likely taken 
another 3 days. What should I do? This time I'm not finding any good 
information on what to do. For now, I've started the repair again, but 
it doesn't exactly sound like it is even fixing anything now. Still, 
I'll let it continue. The output so far is:

> [1/8] checking log skipped (none written)
> [2/8] checking root items
> Fixed 0 roots.
> [3/8] checking extents
> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
> chunk.
> super bytes used 49454738989056 mismatches actual used 49454738923520
> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
> chunk.
> super bytes used 49454739005440 mismatches actual used 49454738956288
> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
> chunk.
> super bytes used 49454739021824 mismatches actual used 49454738972672
> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
> chunk.
> super bytes used 49454739005440 mismatches actual used 49454738972672


If I was able to somehow remove that one logically corrupt devid from 
the filesystem, or somehow correct the size, that would hopefully allow 
me to rebuild from the raid10 data then, but I can't do those with the 
unmountable filesystem.


- Henri Hyyryläinen

Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>
>
> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>> Hello,
>>
>> I hope this is the right place to ask about a filesystem problem. 
>> Really shortly put, I have a file that both exists and doesn't and 
>> prevents the containing directory from being deleted. No matter what 
>> variant of rm and inode based deletion I try I get an error about the 
>> file not existing, and I also cannot try to read the file, but if I 
>> try to delete the directory I get an error that it is not empty (so 
>> the file kind of exists). Trying to ls the directory also gives a 
>> file doesn't exist error.
>>
>> Here's what btrfs check found, which I hope does better in 
>> illustrating the problem:
>>
>>> root 5 inode 25953213 errors 200, dir isize wrong
>>> root 5 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>
>> I've tried everything I've found suggested including a full scrub, 
>> balance with -dusage=75 -musage=75, resetting file attributes, 
>> deleting through the find command, and even some repair mount flags 
>> that don't seem to exist for btrfs.
>
> The fs is corrupted, thus none of those will help.
> I'm more interested in how the corruption happened.
>
> Did you use some tools other than btrfs kernel module and btrfs-progs?
> Like ntfs2btrfs or winbtrfs?
>
> IIRC certain versions have some bugs related to extent tree, but 
> should not cause this problem.
>
>
> The other possibility is hardware memory bitflip, which is more common 
> than you thought (almostly one report per month)
>
> In that case, a full memtest is always recommended, or you will hit 
> all kinds of weird corruptions in the future anyway.
>
>
> With a full memtest proving the memory hardware is fine, then "btrfs 
> check --repair" should be able to fix it.
>
> Thanks,
> Qu
>
>
>> What I haven't tried is a full rebalance with no filters, but I did 
>> not try that yet as it would take quite a long time and if it only 
>> moves data blocks around without recomputing directory items, it 
>> doesn't seem like the right tool to fix my problem. So I'm pretty 
>> much stuck and to me it seems like my only option is to run btrfs 
>> check with the repair flag, but as that has big warnings on it I 
>> thought I would try asking here first (sorry if this is not the right 
>> experts group to ask). So is there still something I can try or am I 
>> finally "allowed" to use the repair command? Here's the full output I 
>> got from btrfs check:
>>
>>> Opening filesystem to check...
>>> Checking filesystem on /dev/sdc
>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>> [1/8] checking log skipped (none written)
>>> [2/8] checking root items
>>> [3/8] checking extents
>>> [4/8] checking free space tree
>>> We have a space info key for a block group that doesn't exist
>>> [5/8] checking fs roots
>>> root 5 inode 25953213 errors 200, dir isize wrong
>>> root 5 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> ERROR: errors found in fs roots
>>> found 49400296812544 bytes used, error(s) found
>>> total csum bytes: 48179330432
>>> total tree bytes: 65067483136
>>> total fs tree bytes: 12107431936
>>> total extent tree bytes: 3194437632
>>> btree space waste bytes: 4558984171
>>> file data blocks allocated: 76487982252032
>>>  referenced 60030799097856
>>
>> So hopefully if I'm reading things right, running a repair would 
>> delete just that one file and directory (which itself is a backup so 
>> I will not miss that file at all)?
>>
>> I do not have enough disk space to copy off the entire filesystem and 
>> rebuild from scratch, without doing something like rebalancing all 
>> data from raid10 to single and then removing half the disks, but I 
>> assume that would take at least 4 weeks to process (as I just 
>> replaced a disk which took like a week).
>>
>> As to what originally caused the corruption, I think it was probably 
>> faulty RAM, because up to to like 3 weeks ago I had one really bad 
>> RAM stick in my computer where a certain memory region always had 
>> incorrectly reading bytes. I had seen intermittent quite high csum 
>> errors in monthly scrubs pretty randomly, which thankfully could 
>> almost always be corrected so I didn't have any major problems even 
>> though I had like totally broken RAM in my computer for who knows how 
>> long. So btrfs was able to protect my data quite impressively from 
>> bad RAM.
>>
>> Sorry for getting a bit sidetracked there, but what should I do in 
>> this situation?
>>
>> - Henri Hyyryläinen
>>
>>
>

Re: How to remove an unremovable file and directory?

[Qu Wenruo]

在 2025/10/5 04:13, Henri Hyyryläinen 写道:
> Hello again.
> 
> It took over 3 days, but the btrfs check --repair has now completed 
> seemingly successfully. I mostly saw output about the file being placed 
> in lost+found and and directory size being corrected. However, there 
> were some messages about mismatch of used bytes.
> 
> Unfortunately it seems like the situation has gotten worse since the 
> repair, because now I cannot mount the filesystem at all. Instead I get 
> an error like this:
> 
>> BTRFS error (device sdc): dev extent physical offset 19977638903808 on 
>> devid 4 doesn't have corresponding chunk
>> BTRFS error (device sdc): failed to verify dev extents against chunks: 
>> -117
>> BTRFS error (device sdc): open_ctree failed: -117
> Even if I remove that one problematic device physically from my 
> computer, the filesystem still refuses to mount with the same error. 
> Maybe the problems with the device replace are again showing up with the 
> actual size of the hard drive not being used correctly? I cannot try to 
> remove the device slack as I cannot mount the filesystem.

Nope, this is a different problem, and not related to dev replace.

Unfortunately btrfs check has not implemented any repair for that.

Overall if the dev extent is found but not corresponding chunk, it 
should still be fine but some space unavailable.

But the kernel is overly cautious on chunk tree, as it's a very 
important and basic functionality.

Please provide the full "btrfs check --readonly" output so that we can 
evaluate and add the missing repair functionality.

Thanks,
Qu

> 
> I did try to run a repair again, and this time I got a bunch of messages 
> like:
> 
>> repair deleting extent record: key [65795546775552,169,0]
>> adding new tree backref on start 65795546775552 len 16384 parent 
>> 65811674234880 root 65811674234880
>> adding new tree backref on start 65795546775552 len 16384 parent 0 
>> root 14499
>> adding new tree backref on start 65795546775552 len 16384 parent 
>> 65791012274176 root 65791012274176
>> adding new tree backref on start 65795546775552 len 16384 parent 
>> 65806385807360 root 65806385807360
>> Repaired extent references for 65795546775552
>> ref mismatch on [65795548233728 16384] extent item 5, found 4

That's fixing some backref mismatch, which you can ignore unless "btrfs 
check --reaonly" later reports new problems.

> But the filesystem still refuses to mount with the exact same error. I 
> did not let the repair run entirely as it would have likely taken 
> another 3 days. What should I do? This time I'm not finding any good 
> information on what to do. For now, I've started the repair again, but 
> it doesn't exactly sound like it is even fixing anything now. Still, 
> I'll let it continue. The output so far is:
> 
>> [1/8] checking log skipped (none written)
>> [2/8] checking root items
>> Fixed 0 roots.
>> [3/8] checking extents
>> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
>> chunk.
>> super bytes used 49454738989056 mismatches actual used 49454738923520
>> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
>> chunk.
>> super bytes used 49454739005440 mismatches actual used 49454738956288
>> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
>> chunk.
>> super bytes used 49454739021824 mismatches actual used 49454738972672
>> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
>> chunk.
>> super bytes used 49454739005440 mismatches actual used 49454738972672
> 
> 
> If I was able to somehow remove that one logically corrupt devid from 
> the filesystem, or somehow correct the size, that would hopefully allow 
> me to rebuild from the raid10 data then, but I can't do those with the 
> unmountable filesystem.
> 
> 
> - Henri Hyyryläinen
> 
> Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>>
>>
>> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>>> Hello,
>>>
>>> I hope this is the right place to ask about a filesystem problem. 
>>> Really shortly put, I have a file that both exists and doesn't and 
>>> prevents the containing directory from being deleted. No matter what 
>>> variant of rm and inode based deletion I try I get an error about the 
>>> file not existing, and I also cannot try to read the file, but if I 
>>> try to delete the directory I get an error that it is not empty (so 
>>> the file kind of exists). Trying to ls the directory also gives a 
>>> file doesn't exist error.
>>>
>>> Here's what btrfs check found, which I hope does better in 
>>> illustrating the problem:
>>>
>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>
>>> I've tried everything I've found suggested including a full scrub, 
>>> balance with -dusage=75 -musage=75, resetting file attributes, 
>>> deleting through the find command, and even some repair mount flags 
>>> that don't seem to exist for btrfs.
>>
>> The fs is corrupted, thus none of those will help.
>> I'm more interested in how the corruption happened.
>>
>> Did you use some tools other than btrfs kernel module and btrfs-progs?
>> Like ntfs2btrfs or winbtrfs?
>>
>> IIRC certain versions have some bugs related to extent tree, but 
>> should not cause this problem.
>>
>>
>> The other possibility is hardware memory bitflip, which is more common 
>> than you thought (almostly one report per month)
>>
>> In that case, a full memtest is always recommended, or you will hit 
>> all kinds of weird corruptions in the future anyway.
>>
>>
>> With a full memtest proving the memory hardware is fine, then "btrfs 
>> check --repair" should be able to fix it.
>>
>> Thanks,
>> Qu
>>
>>
>>> What I haven't tried is a full rebalance with no filters, but I did 
>>> not try that yet as it would take quite a long time and if it only 
>>> moves data blocks around without recomputing directory items, it 
>>> doesn't seem like the right tool to fix my problem. So I'm pretty 
>>> much stuck and to me it seems like my only option is to run btrfs 
>>> check with the repair flag, but as that has big warnings on it I 
>>> thought I would try asking here first (sorry if this is not the right 
>>> experts group to ask). So is there still something I can try or am I 
>>> finally "allowed" to use the repair command? Here's the full output I 
>>> got from btrfs check:
>>>
>>>> Opening filesystem to check...
>>>> Checking filesystem on /dev/sdc
>>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>> [1/8] checking log skipped (none written)
>>>> [2/8] checking root items
>>>> [3/8] checking extents
>>>> [4/8] checking free space tree
>>>> We have a space info key for a block group that doesn't exist
>>>> [5/8] checking fs roots
>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>> ERROR: errors found in fs roots
>>>> found 49400296812544 bytes used, error(s) found
>>>> total csum bytes: 48179330432
>>>> total tree bytes: 65067483136
>>>> total fs tree bytes: 12107431936
>>>> total extent tree bytes: 3194437632
>>>> btree space waste bytes: 4558984171
>>>> file data blocks allocated: 76487982252032
>>>>  referenced 60030799097856
>>>
>>> So hopefully if I'm reading things right, running a repair would 
>>> delete just that one file and directory (which itself is a backup so 
>>> I will not miss that file at all)?
>>>
>>> I do not have enough disk space to copy off the entire filesystem and 
>>> rebuild from scratch, without doing something like rebalancing all 
>>> data from raid10 to single and then removing half the disks, but I 
>>> assume that would take at least 4 weeks to process (as I just 
>>> replaced a disk which took like a week).
>>>
>>> As to what originally caused the corruption, I think it was probably 
>>> faulty RAM, because up to to like 3 weeks ago I had one really bad 
>>> RAM stick in my computer where a certain memory region always had 
>>> incorrectly reading bytes. I had seen intermittent quite high csum 
>>> errors in monthly scrubs pretty randomly, which thankfully could 
>>> almost always be corrected so I didn't have any major problems even 
>>> though I had like totally broken RAM in my computer for who knows how 
>>> long. So btrfs was able to protect my data quite impressively from 
>>> bad RAM.
>>>
>>> Sorry for getting a bit sidetracked there, but what should I do in 
>>> this situation?
>>>
>>> - Henri Hyyryläinen
>>>
>>>
>>

Re: How to remove an unremovable file and directory?

[Qu Wenruo]

在 2025/10/5 07:14, Qu Wenruo 写道:
> 
> 
> 在 2025/10/5 04:13, Henri Hyyryläinen 写道:
>> Hello again.
>>
>> It took over 3 days, but the btrfs check --repair has now completed 
>> seemingly successfully. I mostly saw output about the file being 
>> placed in lost+found and and directory size being corrected. However, 
>> there were some messages about mismatch of used bytes.
>>
>> Unfortunately it seems like the situation has gotten worse since the 
>> repair, because now I cannot mount the filesystem at all. Instead I 
>> get an error like this:
>>
>>> BTRFS error (device sdc): dev extent physical offset 19977638903808 
>>> on devid 4 doesn't have corresponding chunk
>>> BTRFS error (device sdc): failed to verify dev extents against 
>>> chunks: -117
>>> BTRFS error (device sdc): open_ctree failed: -117
>> Even if I remove that one problematic device physically from my 
>> computer, the filesystem still refuses to mount with the same error. 
>> Maybe the problems with the device replace are again showing up with 
>> the actual size of the hard drive not being used correctly? I cannot 
>> try to remove the device slack as I cannot mount the filesystem.
> 
> Nope, this is a different problem, and not related to dev replace.
> 
> Unfortunately btrfs check has not implemented any repair for that.
> 
> Overall if the dev extent is found but not corresponding chunk, it 
> should still be fine but some space unavailable.
> 
> But the kernel is overly cautious on chunk tree, as it's a very 
> important and basic functionality.
> 
> Please provide the full "btrfs check --readonly" output so that we can 
> evaluate and add the missing repair functionality.

One extra thing, at the time of the initial "btrfs check --repair", 
there is no running dev-replace/balance or whatever running?

Although those operations will be automatically paused by unmount,
btrfs check is not going to be able to handle some of those paused 
operations.
> 
> Thanks,
> Qu
> 
>>
>> I did try to run a repair again, and this time I got a bunch of 
>> messages like:
>>
>>> repair deleting extent record: key [65795546775552,169,0]
>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>> 65811674234880 root 65811674234880
>>> adding new tree backref on start 65795546775552 len 16384 parent 0 
>>> root 14499
>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>> 65791012274176 root 65791012274176
>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>> 65806385807360 root 65806385807360
>>> Repaired extent references for 65795546775552
>>> ref mismatch on [65795548233728 16384] extent item 5, found 4
> 
> That's fixing some backref mismatch, which you can ignore unless "btrfs 
> check --reaonly" later reports new problems.
> 
>> But the filesystem still refuses to mount with the exact same error. I 
>> did not let the repair run entirely as it would have likely taken 
>> another 3 days. What should I do? This time I'm not finding any good 
>> information on what to do. For now, I've started the repair again, but 
>> it doesn't exactly sound like it is even fixing anything now. Still, 
>> I'll let it continue. The output so far is:
>>
>>> [1/8] checking log skipped (none written)
>>> [2/8] checking root items
>>> Fixed 0 roots.
>>> [3/8] checking extents
>>> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
>>> chunk.
>>> super bytes used 49454738989056 mismatches actual used 49454738923520
>>> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
>>> chunk.
>>> super bytes used 49454739005440 mismatches actual used 49454738956288
>>> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
>>> chunk.
>>> super bytes used 49454739021824 mismatches actual used 49454738972672
>>> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
>>> chunk.
>>> super bytes used 49454739005440 mismatches actual used 49454738972672
>>
>>
>> If I was able to somehow remove that one logically corrupt devid from 
>> the filesystem, or somehow correct the size, that would hopefully 
>> allow me to rebuild from the raid10 data then, but I can't do those 
>> with the unmountable filesystem.
>>
>>
>> - Henri Hyyryläinen
>>
>> Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>>>
>>>
>>> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>>>> Hello,
>>>>
>>>> I hope this is the right place to ask about a filesystem problem. 
>>>> Really shortly put, I have a file that both exists and doesn't and 
>>>> prevents the containing directory from being deleted. No matter what 
>>>> variant of rm and inode based deletion I try I get an error about 
>>>> the file not existing, and I also cannot try to read the file, but 
>>>> if I try to delete the directory I get an error that it is not empty 
>>>> (so the file kind of exists). Trying to ls the directory also gives 
>>>> a file doesn't exist error.
>>>>
>>>> Here's what btrfs check found, which I hope does better in 
>>>> illustrating the problem:
>>>>
>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>
>>>> I've tried everything I've found suggested including a full scrub, 
>>>> balance with -dusage=75 -musage=75, resetting file attributes, 
>>>> deleting through the find command, and even some repair mount flags 
>>>> that don't seem to exist for btrfs.
>>>
>>> The fs is corrupted, thus none of those will help.
>>> I'm more interested in how the corruption happened.
>>>
>>> Did you use some tools other than btrfs kernel module and btrfs-progs?
>>> Like ntfs2btrfs or winbtrfs?
>>>
>>> IIRC certain versions have some bugs related to extent tree, but 
>>> should not cause this problem.
>>>
>>>
>>> The other possibility is hardware memory bitflip, which is more 
>>> common than you thought (almostly one report per month)
>>>
>>> In that case, a full memtest is always recommended, or you will hit 
>>> all kinds of weird corruptions in the future anyway.
>>>
>>>
>>> With a full memtest proving the memory hardware is fine, then "btrfs 
>>> check --repair" should be able to fix it.
>>>
>>> Thanks,
>>> Qu
>>>
>>>
>>>> What I haven't tried is a full rebalance with no filters, but I did 
>>>> not try that yet as it would take quite a long time and if it only 
>>>> moves data blocks around without recomputing directory items, it 
>>>> doesn't seem like the right tool to fix my problem. So I'm pretty 
>>>> much stuck and to me it seems like my only option is to run btrfs 
>>>> check with the repair flag, but as that has big warnings on it I 
>>>> thought I would try asking here first (sorry if this is not the 
>>>> right experts group to ask). So is there still something I can try 
>>>> or am I finally "allowed" to use the repair command? Here's the full 
>>>> output I got from btrfs check:
>>>>
>>>>> Opening filesystem to check...
>>>>> Checking filesystem on /dev/sdc
>>>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>>> [1/8] checking log skipped (none written)
>>>>> [2/8] checking root items
>>>>> [3/8] checking extents
>>>>> [4/8] checking free space tree
>>>>> We have a space info key for a block group that doesn't exist
>>>>> [5/8] checking fs roots
>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>> ERROR: errors found in fs roots
>>>>> found 49400296812544 bytes used, error(s) found
>>>>> total csum bytes: 48179330432
>>>>> total tree bytes: 65067483136
>>>>> total fs tree bytes: 12107431936
>>>>> total extent tree bytes: 3194437632
>>>>> btree space waste bytes: 4558984171
>>>>> file data blocks allocated: 76487982252032
>>>>>  referenced 60030799097856
>>>>
>>>> So hopefully if I'm reading things right, running a repair would 
>>>> delete just that one file and directory (which itself is a backup so 
>>>> I will not miss that file at all)?
>>>>
>>>> I do not have enough disk space to copy off the entire filesystem 
>>>> and rebuild from scratch, without doing something like rebalancing 
>>>> all data from raid10 to single and then removing half the disks, but 
>>>> I assume that would take at least 4 weeks to process (as I just 
>>>> replaced a disk which took like a week).
>>>>
>>>> As to what originally caused the corruption, I think it was probably 
>>>> faulty RAM, because up to to like 3 weeks ago I had one really bad 
>>>> RAM stick in my computer where a certain memory region always had 
>>>> incorrectly reading bytes. I had seen intermittent quite high csum 
>>>> errors in monthly scrubs pretty randomly, which thankfully could 
>>>> almost always be corrected so I didn't have any major problems even 
>>>> though I had like totally broken RAM in my computer for who knows 
>>>> how long. So btrfs was able to protect my data quite impressively 
>>>> from bad RAM.
>>>>
>>>> Sorry for getting a bit sidetracked there, but what should I do in 
>>>> this situation?
>>>>
>>>> - Henri Hyyryläinen
>>>>
>>>>
>>>
> 
> 

Re: How to remove an unremovable file and directory?

[Henri Hyyryläinen]

> One extra thing, at the time of the initial "btrfs check --repair", 
> there is no running dev-replace/balance or whatever running?
>
> Although those operations will be automatically paused by unmount,
> btrfs check is not going to be able to handle some of those paused 
> operations. 

Is there a way for me to check that without mounting the filesystem? As 
far as I could find, none of the balance / scrub commands allow working 
on an unmounted filesystem. So I couldn't find out if I had any in 
canceled state.

Though, I'm pretty sure I let the last scrub and balance operation I 
tried to fully complete before starting using the check and repair 
commands. But I'm not absolutely certain that I didn't try one of those 
a last time and didn't let it fully complete.

I'll start that "btrfs check --readonly" command now. And I'll report 
back once it is done (hopefully by the morning in my timezone).

- Henri Hyyryläinen

Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
>
>
> 在 2025/10/5 07:14, Qu Wenruo 写道:
>>
>>
>> 在 2025/10/5 04:13, Henri Hyyryläinen 写道:
>>> Hello again.
>>>
>>> It took over 3 days, but the btrfs check --repair has now completed 
>>> seemingly successfully. I mostly saw output about the file being 
>>> placed in lost+found and and directory size being corrected. 
>>> However, there were some messages about mismatch of used bytes.
>>>
>>> Unfortunately it seems like the situation has gotten worse since the 
>>> repair, because now I cannot mount the filesystem at all. Instead I 
>>> get an error like this:
>>>
>>>> BTRFS error (device sdc): dev extent physical offset 19977638903808 
>>>> on devid 4 doesn't have corresponding chunk
>>>> BTRFS error (device sdc): failed to verify dev extents against 
>>>> chunks: -117
>>>> BTRFS error (device sdc): open_ctree failed: -117
>>> Even if I remove that one problematic device physically from my 
>>> computer, the filesystem still refuses to mount with the same error. 
>>> Maybe the problems with the device replace are again showing up with 
>>> the actual size of the hard drive not being used correctly? I cannot 
>>> try to remove the device slack as I cannot mount the filesystem.
>>
>> Nope, this is a different problem, and not related to dev replace.
>>
>> Unfortunately btrfs check has not implemented any repair for that.
>>
>> Overall if the dev extent is found but not corresponding chunk, it 
>> should still be fine but some space unavailable.
>>
>> But the kernel is overly cautious on chunk tree, as it's a very 
>> important and basic functionality.
>>
>> Please provide the full "btrfs check --readonly" output so that we 
>> can evaluate and add the missing repair functionality.
>
> One extra thing, at the time of the initial "btrfs check --repair", 
> there is no running dev-replace/balance or whatever running?
>
> Although those operations will be automatically paused by unmount,
> btrfs check is not going to be able to handle some of those paused 
> operations.
>>
>> Thanks,
>> Qu
>>
>>>
>>> I did try to run a repair again, and this time I got a bunch of 
>>> messages like:
>>>
>>>> repair deleting extent record: key [65795546775552,169,0]
>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>> 65811674234880 root 65811674234880
>>>> adding new tree backref on start 65795546775552 len 16384 parent 0 
>>>> root 14499
>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>> 65791012274176 root 65791012274176
>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>> 65806385807360 root 65806385807360
>>>> Repaired extent references for 65795546775552
>>>> ref mismatch on [65795548233728 16384] extent item 5, found 4
>>
>> That's fixing some backref mismatch, which you can ignore unless 
>> "btrfs check --reaonly" later reports new problems.
>>
>>> But the filesystem still refuses to mount with the exact same error. 
>>> I did not let the repair run entirely as it would have likely taken 
>>> another 3 days. What should I do? This time I'm not finding any good 
>>> information on what to do. For now, I've started the repair again, 
>>> but it doesn't exactly sound like it is even fixing anything now. 
>>> Still, I'll let it continue. The output so far is:
>>>
>>>> [1/8] checking log skipped (none written)
>>>> [2/8] checking root items
>>>> Fixed 0 roots.
>>>> [3/8] checking extents
>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>> relative chunk.
>>>> super bytes used 49454738989056 mismatches actual used 49454738923520
>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>> relative chunk.
>>>> super bytes used 49454739005440 mismatches actual used 49454738956288
>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>> relative chunk.
>>>> super bytes used 49454739021824 mismatches actual used 49454738972672
>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>> relative chunk.
>>>> super bytes used 49454739005440 mismatches actual used 49454738972672
>>>
>>>
>>> If I was able to somehow remove that one logically corrupt devid 
>>> from the filesystem, or somehow correct the size, that would 
>>> hopefully allow me to rebuild from the raid10 data then, but I can't 
>>> do those with the unmountable filesystem.
>>>
>>>
>>> - Henri Hyyryläinen
>>>
>>> Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>>>>
>>>>
>>>> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>>>>> Hello,
>>>>>
>>>>> I hope this is the right place to ask about a filesystem problem. 
>>>>> Really shortly put, I have a file that both exists and doesn't and 
>>>>> prevents the containing directory from being deleted. No matter 
>>>>> what variant of rm and inode based deletion I try I get an error 
>>>>> about the file not existing, and I also cannot try to read the 
>>>>> file, but if I try to delete the directory I get an error that it 
>>>>> is not empty (so the file kind of exists). Trying to ls the 
>>>>> directory also gives a file doesn't exist error.
>>>>>
>>>>> Here's what btrfs check found, which I hope does better in 
>>>>> illustrating the problem:
>>>>>
>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>
>>>>> I've tried everything I've found suggested including a full scrub, 
>>>>> balance with -dusage=75 -musage=75, resetting file attributes, 
>>>>> deleting through the find command, and even some repair mount 
>>>>> flags that don't seem to exist for btrfs.
>>>>
>>>> The fs is corrupted, thus none of those will help.
>>>> I'm more interested in how the corruption happened.
>>>>
>>>> Did you use some tools other than btrfs kernel module and btrfs-progs?
>>>> Like ntfs2btrfs or winbtrfs?
>>>>
>>>> IIRC certain versions have some bugs related to extent tree, but 
>>>> should not cause this problem.
>>>>
>>>>
>>>> The other possibility is hardware memory bitflip, which is more 
>>>> common than you thought (almostly one report per month)
>>>>
>>>> In that case, a full memtest is always recommended, or you will hit 
>>>> all kinds of weird corruptions in the future anyway.
>>>>
>>>>
>>>> With a full memtest proving the memory hardware is fine, then 
>>>> "btrfs check --repair" should be able to fix it.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>
>>>>> What I haven't tried is a full rebalance with no filters, but I 
>>>>> did not try that yet as it would take quite a long time and if it 
>>>>> only moves data blocks around without recomputing directory items, 
>>>>> it doesn't seem like the right tool to fix my problem. So I'm 
>>>>> pretty much stuck and to me it seems like my only option is to run 
>>>>> btrfs check with the repair flag, but as that has big warnings on 
>>>>> it I thought I would try asking here first (sorry if this is not 
>>>>> the right experts group to ask). So is there still something I can 
>>>>> try or am I finally "allowed" to use the repair command? Here's 
>>>>> the full output I got from btrfs check:
>>>>>
>>>>>> Opening filesystem to check...
>>>>>> Checking filesystem on /dev/sdc
>>>>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>>>> [1/8] checking log skipped (none written)
>>>>>> [2/8] checking root items
>>>>>> [3/8] checking extents
>>>>>> [4/8] checking free space tree
>>>>>> We have a space info key for a block group that doesn't exist
>>>>>> [5/8] checking fs roots
>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>>>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>> ERROR: errors found in fs roots
>>>>>> found 49400296812544 bytes used, error(s) found
>>>>>> total csum bytes: 48179330432
>>>>>> total tree bytes: 65067483136
>>>>>> total fs tree bytes: 12107431936
>>>>>> total extent tree bytes: 3194437632
>>>>>> btree space waste bytes: 4558984171
>>>>>> file data blocks allocated: 76487982252032
>>>>>>  referenced 60030799097856
>>>>>
>>>>> So hopefully if I'm reading things right, running a repair would 
>>>>> delete just that one file and directory (which itself is a backup 
>>>>> so I will not miss that file at all)?
>>>>>
>>>>> I do not have enough disk space to copy off the entire filesystem 
>>>>> and rebuild from scratch, without doing something like rebalancing 
>>>>> all data from raid10 to single and then removing half the disks, 
>>>>> but I assume that would take at least 4 weeks to process (as I 
>>>>> just replaced a disk which took like a week).
>>>>>
>>>>> As to what originally caused the corruption, I think it was 
>>>>> probably faulty RAM, because up to to like 3 weeks ago I had one 
>>>>> really bad RAM stick in my computer where a certain memory region 
>>>>> always had incorrectly reading bytes. I had seen intermittent 
>>>>> quite high csum errors in monthly scrubs pretty randomly, which 
>>>>> thankfully could almost always be corrected so I didn't have any 
>>>>> major problems even though I had like totally broken RAM in my 
>>>>> computer for who knows how long. So btrfs was able to protect my 
>>>>> data quite impressively from bad RAM.
>>>>>
>>>>> Sorry for getting a bit sidetracked there, but what should I do in 
>>>>> this situation?
>>>>>
>>>>> - Henri Hyyryläinen
>>>>>
>>>>>
>>>>
>>
>>
>

Re: How to remove an unremovable file and directory?

[Qu Wenruo]

在 2025/10/5 08:23, Henri Hyyryläinen 写道:
>> One extra thing, at the time of the initial "btrfs check --repair", 
>> there is no running dev-replace/balance or whatever running?
>>
>> Although those operations will be automatically paused by unmount,
>> btrfs check is not going to be able to handle some of those paused 
>> operations. 
> 
> Is there a way for me to check that without mounting the filesystem? As 
> far as I could find, none of the balance / scrub commands allow working 
> on an unmounted filesystem. So I couldn't find out if I had any in 
> canceled state.

# btrfs ins dump-tree -t root <device>
> 
> Though, I'm pretty sure I let the last scrub and balance operation I 
> tried to fully complete before starting using the check and repair 
> commands. But I'm not absolutely certain that I didn't try one of those 
> a last time and didn't let it fully complete.
> 
> I'll start that "btrfs check --readonly" command now. And I'll report 
> back once it is done (hopefully by the morning in my timezone).
> 
> - Henri Hyyryläinen
> 
> Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
>>
>>
>> 在 2025/10/5 07:14, Qu Wenruo 写道:
>>>
>>>
>>> 在 2025/10/5 04:13, Henri Hyyryläinen 写道:
>>>> Hello again.
>>>>
>>>> It took over 3 days, but the btrfs check --repair has now completed 
>>>> seemingly successfully. I mostly saw output about the file being 
>>>> placed in lost+found and and directory size being corrected. 
>>>> However, there were some messages about mismatch of used bytes.
>>>>
>>>> Unfortunately it seems like the situation has gotten worse since the 
>>>> repair, because now I cannot mount the filesystem at all. Instead I 
>>>> get an error like this:
>>>>
>>>>> BTRFS error (device sdc): dev extent physical offset 19977638903808 
>>>>> on devid 4 doesn't have corresponding chunk
>>>>> BTRFS error (device sdc): failed to verify dev extents against 
>>>>> chunks: -117
>>>>> BTRFS error (device sdc): open_ctree failed: -117
>>>> Even if I remove that one problematic device physically from my 
>>>> computer, the filesystem still refuses to mount with the same error. 
>>>> Maybe the problems with the device replace are again showing up with 
>>>> the actual size of the hard drive not being used correctly? I cannot 
>>>> try to remove the device slack as I cannot mount the filesystem.
>>>
>>> Nope, this is a different problem, and not related to dev replace.
>>>
>>> Unfortunately btrfs check has not implemented any repair for that.
>>>
>>> Overall if the dev extent is found but not corresponding chunk, it 
>>> should still be fine but some space unavailable.
>>>
>>> But the kernel is overly cautious on chunk tree, as it's a very 
>>> important and basic functionality.
>>>
>>> Please provide the full "btrfs check --readonly" output so that we 
>>> can evaluate and add the missing repair functionality.
>>
>> One extra thing, at the time of the initial "btrfs check --repair", 
>> there is no running dev-replace/balance or whatever running?
>>
>> Although those operations will be automatically paused by unmount,
>> btrfs check is not going to be able to handle some of those paused 
>> operations.
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>> I did try to run a repair again, and this time I got a bunch of 
>>>> messages like:
>>>>
>>>>> repair deleting extent record: key [65795546775552,169,0]
>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>> 65811674234880 root 65811674234880
>>>>> adding new tree backref on start 65795546775552 len 16384 parent 0 
>>>>> root 14499
>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>> 65791012274176 root 65791012274176
>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>> 65806385807360 root 65806385807360
>>>>> Repaired extent references for 65795546775552
>>>>> ref mismatch on [65795548233728 16384] extent item 5, found 4
>>>
>>> That's fixing some backref mismatch, which you can ignore unless 
>>> "btrfs check --reaonly" later reports new problems.
>>>
>>>> But the filesystem still refuses to mount with the exact same error. 
>>>> I did not let the repair run entirely as it would have likely taken 
>>>> another 3 days. What should I do? This time I'm not finding any good 
>>>> information on what to do. For now, I've started the repair again, 
>>>> but it doesn't exactly sound like it is even fixing anything now. 
>>>> Still, I'll let it continue. The output so far is:
>>>>
>>>>> [1/8] checking log skipped (none written)
>>>>> [2/8] checking root items
>>>>> Fixed 0 roots.
>>>>> [3/8] checking extents
>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>> relative chunk.
>>>>> super bytes used 49454738989056 mismatches actual used 49454738923520
>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>> relative chunk.
>>>>> super bytes used 49454739005440 mismatches actual used 49454738956288
>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>> relative chunk.
>>>>> super bytes used 49454739021824 mismatches actual used 49454738972672
>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>> relative chunk.
>>>>> super bytes used 49454739005440 mismatches actual used 49454738972672
>>>>
>>>>
>>>> If I was able to somehow remove that one logically corrupt devid 
>>>> from the filesystem, or somehow correct the size, that would 
>>>> hopefully allow me to rebuild from the raid10 data then, but I can't 
>>>> do those with the unmountable filesystem.
>>>>
>>>>
>>>> - Henri Hyyryläinen
>>>>
>>>> Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>>>>>
>>>>>
>>>>> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>>>>>> Hello,
>>>>>>
>>>>>> I hope this is the right place to ask about a filesystem problem. 
>>>>>> Really shortly put, I have a file that both exists and doesn't and 
>>>>>> prevents the containing directory from being deleted. No matter 
>>>>>> what variant of rm and inode based deletion I try I get an error 
>>>>>> about the file not existing, and I also cannot try to read the 
>>>>>> file, but if I try to delete the directory I get an error that it 
>>>>>> is not empty (so the file kind of exists). Trying to ls the 
>>>>>> directory also gives a file doesn't exist error.
>>>>>>
>>>>>> Here's what btrfs check found, which I hope does better in 
>>>>>> illustrating the problem:
>>>>>>
>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>
>>>>>> I've tried everything I've found suggested including a full scrub, 
>>>>>> balance with -dusage=75 -musage=75, resetting file attributes, 
>>>>>> deleting through the find command, and even some repair mount 
>>>>>> flags that don't seem to exist for btrfs.
>>>>>
>>>>> The fs is corrupted, thus none of those will help.
>>>>> I'm more interested in how the corruption happened.
>>>>>
>>>>> Did you use some tools other than btrfs kernel module and btrfs-progs?
>>>>> Like ntfs2btrfs or winbtrfs?
>>>>>
>>>>> IIRC certain versions have some bugs related to extent tree, but 
>>>>> should not cause this problem.
>>>>>
>>>>>
>>>>> The other possibility is hardware memory bitflip, which is more 
>>>>> common than you thought (almostly one report per month)
>>>>>
>>>>> In that case, a full memtest is always recommended, or you will hit 
>>>>> all kinds of weird corruptions in the future anyway.
>>>>>
>>>>>
>>>>> With a full memtest proving the memory hardware is fine, then 
>>>>> "btrfs check --repair" should be able to fix it.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>
>>>>>> What I haven't tried is a full rebalance with no filters, but I 
>>>>>> did not try that yet as it would take quite a long time and if it 
>>>>>> only moves data blocks around without recomputing directory items, 
>>>>>> it doesn't seem like the right tool to fix my problem. So I'm 
>>>>>> pretty much stuck and to me it seems like my only option is to run 
>>>>>> btrfs check with the repair flag, but as that has big warnings on 
>>>>>> it I thought I would try asking here first (sorry if this is not 
>>>>>> the right experts group to ask). So is there still something I can 
>>>>>> try or am I finally "allowed" to use the repair command? Here's 
>>>>>> the full output I got from btrfs check:
>>>>>>
>>>>>>> Opening filesystem to check...
>>>>>>> Checking filesystem on /dev/sdc
>>>>>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>>>>> [1/8] checking log skipped (none written)
>>>>>>> [2/8] checking root items
>>>>>>> [3/8] checking extents
>>>>>>> [4/8] checking free space tree
>>>>>>> We have a space info key for a block group that doesn't exist
>>>>>>> [5/8] checking fs roots
>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>>>>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>> ERROR: errors found in fs roots
>>>>>>> found 49400296812544 bytes used, error(s) found
>>>>>>> total csum bytes: 48179330432
>>>>>>> total tree bytes: 65067483136
>>>>>>> total fs tree bytes: 12107431936
>>>>>>> total extent tree bytes: 3194437632
>>>>>>> btree space waste bytes: 4558984171
>>>>>>> file data blocks allocated: 76487982252032
>>>>>>>  referenced 60030799097856
>>>>>>
>>>>>> So hopefully if I'm reading things right, running a repair would 
>>>>>> delete just that one file and directory (which itself is a backup 
>>>>>> so I will not miss that file at all)?
>>>>>>
>>>>>> I do not have enough disk space to copy off the entire filesystem 
>>>>>> and rebuild from scratch, without doing something like rebalancing 
>>>>>> all data from raid10 to single and then removing half the disks, 
>>>>>> but I assume that would take at least 4 weeks to process (as I 
>>>>>> just replaced a disk which took like a week).
>>>>>>
>>>>>> As to what originally caused the corruption, I think it was 
>>>>>> probably faulty RAM, because up to to like 3 weeks ago I had one 
>>>>>> really bad RAM stick in my computer where a certain memory region 
>>>>>> always had incorrectly reading bytes. I had seen intermittent 
>>>>>> quite high csum errors in monthly scrubs pretty randomly, which 
>>>>>> thankfully could almost always be corrected so I didn't have any 
>>>>>> major problems even though I had like totally broken RAM in my 
>>>>>> computer for who knows how long. So btrfs was able to protect my 
>>>>>> data quite impressively from bad RAM.
>>>>>>
>>>>>> Sorry for getting a bit sidetracked there, but what should I do in 
>>>>>> this situation?
>>>>>>
>>>>>> - Henri Hyyryläinen
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>

Re: How to remove an unremovable file and directory?

[Henri Hyyryläinen]

Thanks for the command.

I tried reading through the data but I could not determine if there was 
any information related to scrub or balance status.

Here's the full output I got:

> btrfs-progs v6.16.1
> root tree
> leaf 227587769614336 items 61 free space 3547 generation 2035509 owner 
> ROOT_TREE
> leaf 227587769614336 flags 0x1(WRITTEN) backref revision 1
> fs uuid 2b4ad16d-e456-4adf-960b-dca43560b98b
> chunk uuid 0e317678-73cc-4485-ab6b-987285e19681
>     item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
>         generation 2035509 root_dirid 0 bytenr 227587769630720 
> byte_limit 0 bytes_used 3319316480
>         last_snapshot 0 flags 0x0(none) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2035509
>         uuid 00000000-0000-0000-0000-000000000000
>         parent_uuid 00000000-0000-0000-0000-000000000000
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 0 otransid 0 stransid 0 rtransid 0
>         ctime 0.0 (1970-01-01 02:00:00)
>         otime 0.0 (1970-01-01 02:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439
>         generation 2025974 root_dirid 0 bytenr 227594196631552 
> byte_limit 0 bytes_used 21889024
>         last_snapshot 0 flags 0x0(none) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 2 generation_v2 2025974
>         uuid 00000000-0000-0000-0000-000000000000
>         parent_uuid 00000000-0000-0000-0000-000000000000
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 0 otransid 0 stransid 0 rtransid 0
>         ctime 0.0 (1970-01-01 02:00:00)
>         otime 0.0 (1970-01-01 02:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17
>         index 0 namelen 7 name: default
>     item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439
>         generation 2025975 root_dirid 256 bytenr 227573939142656 
> byte_limit 0 bytes_used 6603997184
>         last_snapshot 2025951 flags 0x0(none) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025975
>         uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         parent_uuid 00000000-0000-0000-0000-000000000000
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2025953 otransid 0 stransid 0 rtransid 0
>         ctime 1759258850.523837328 (2025-09-30 22:00:50)
>         otime 1679586168.0 (2023-03-23 17:42:48)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 4 key (FS_TREE ROOT_REF 256) itemoff 14921 itemsize 28
>         root ref key dirid 256 sequence 6 name .snapshots
>     item 5 key (ROOT_TREE_DIR INODE_ITEM 0) itemoff 14761 itemsize 160
>         generation 3 transid 0 size 0 nbytes 16384
>         block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>         sequence 0 flags 0x0(none)
>         atime 1679586169.0 (2023-03-23 17:42:49)
>         ctime 1679586169.0 (2023-03-23 17:42:49)
>         mtime 1679586169.0 (2023-03-23 17:42:49)
>         otime 1679586169.0 (2023-03-23 17:42:49)
>     item 6 key (ROOT_TREE_DIR INODE_REF 6) itemoff 14749 itemsize 12
>         index 0 namelen 2 name: ..
>     item 7 key (ROOT_TREE_DIR DIR_ITEM 2378154706) itemoff 14712 
> itemsize 37
>         location key (FS_TREE ROOT_ITEM 18446744073709551615) type DIR
>         transid 0 data_len 0 name_len 7
>         name: default
>     item 8 key (CSUM_TREE ROOT_ITEM 0) itemoff 14273 itemsize 439
>         generation 2025954 root_dirid 0 bytenr 227588044062720 
> byte_limit 0 bytes_used 49791238144
>         last_snapshot 0 flags 0x0(none) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025954
>         uuid 00000000-0000-0000-0000-000000000000
>         parent_uuid 00000000-0000-0000-0000-000000000000
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 0 otransid 0 stransid 0 rtransid 0
>         ctime 0.0 (1970-01-01 02:00:00)
>         otime 0.0 (1970-01-01 02:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 9 key (UUID_TREE ROOT_ITEM 0) itemoff 13834 itemsize 439
>         generation 2025954 root_dirid 0 bytenr 227587802447872 
> byte_limit 0 bytes_used 16384
>         last_snapshot 0 flags 0x0(none) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 0 generation_v2 2025954
>         uuid 00000000-0000-0000-0000-000000000000
>         parent_uuid 00000000-0000-0000-0000-000000000000
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 0 otransid 0 stransid 0 rtransid 0
>         ctime 0.0 (1970-01-01 02:00:00)
>         otime 0.0 (1970-01-01 02:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 10 key (256 ROOT_ITEM 0) itemoff 13395 itemsize 439
>         generation 2025954 root_dirid 256 bytenr 227587802316800 
> byte_limit 0 bytes_used 49152
>         last_snapshot 2015386 flags 0x0(none) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 1 generation_v2 2025954
>         uuid 56f85f76-1eb8-1245-a35b-8a9307d5c68f
>         parent_uuid 00000000-0000-0000-0000-000000000000
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2025954 otransid 31 stransid 0 rtransid 0
>         ctime 1759260258.238143629 (2025-09-30 22:24:18)
>         otime 1679588253.995834574 (2023-03-23 18:17:33)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 11 key (256 ROOT_BACKREF 5) itemoff 13367 itemsize 28
>         root backref key dirid 256 sequence 6 name .snapshots
>     item 12 key (256 ROOT_REF 14428) itemoff 13341 itemsize 26
>         root ref key dirid 28589 sequence 2 name snapshot
>     item 13 key (256 ROOT_REF 14451) itemoff 13315 itemsize 26
>         root ref key dirid 28635 sequence 2 name snapshot
>     item 14 key (256 ROOT_REF 14475) itemoff 13289 itemsize 26
>         root ref key dirid 28683 sequence 2 name snapshot
>     item 15 key (256 ROOT_REF 14499) itemoff 13263 itemsize 26
>         root ref key dirid 28731 sequence 2 name snapshot
>     item 16 key (256 ROOT_REF 14523) itemoff 13237 itemsize 26
>         root ref key dirid 28779 sequence 2 name snapshot
>     item 17 key (256 ROOT_REF 14547) itemoff 13211 itemsize 26
>         root ref key dirid 28827 sequence 2 name snapshot
>     item 18 key (256 ROOT_REF 14559) itemoff 13185 itemsize 26
>         root ref key dirid 28851 sequence 2 name snapshot
>     item 19 key (256 ROOT_REF 14583) itemoff 13159 itemsize 26
>         root ref key dirid 28899 sequence 2 name snapshot
>     item 20 key (256 ROOT_REF 14607) itemoff 13133 itemsize 26
>         root ref key dirid 28947 sequence 2 name snapshot
>     item 21 key (256 ROOT_REF 14631) itemoff 13107 itemsize 26
>         root ref key dirid 28995 sequence 2 name snapshot
>     item 22 key (256 ROOT_REF 14640) itemoff 13081 itemsize 26
>         root ref key dirid 29013 sequence 2 name snapshot
>     item 23 key (256 ROOT_REF 14641) itemoff 13055 itemsize 26
>         root ref key dirid 29015 sequence 2 name snapshot
>     item 24 key (256 ROOT_REF 14642) itemoff 13029 itemsize 26
>         root ref key dirid 29017 sequence 2 name snapshot
>     item 25 key (256 ROOT_REF 14643) itemoff 13003 itemsize 26
>         root ref key dirid 29019 sequence 2 name snapshot
>     item 26 key (256 ROOT_REF 14644) itemoff 12977 itemsize 26
>         root ref key dirid 29021 sequence 2 name snapshot
>     item 27 key (256 ROOT_REF 14645) itemoff 12951 itemsize 26
>         root ref key dirid 29023 sequence 2 name snapshot
>     item 28 key (14428 ROOT_ITEM 1849346) itemoff 12512 itemsize 439
>         generation 2025978 root_dirid 256 bytenr 71398781173760 
> byte_limit 0 bytes_used 8815476736
>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025978
>         uuid 97cd122d-c519-8746-bdfe-13479bfefb9d
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 1849346 otransid 1849346 stransid 0 rtransid 0
>         ctime 1758406304.841464812 (2025-09-21 01:11:44)
>         otime 1758406304.840394467 (2025-09-21 01:11:44)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 29 key (14428 ROOT_BACKREF 256) itemoff 12486 itemsize 26
>         root backref key dirid 28589 sequence 2 name snapshot
>     item 30 key (14451 ROOT_ITEM 1899154) itemoff 12047 itemsize 439
>         generation 2025981 root_dirid 256 bytenr 71398809468928 
> byte_limit 0 bytes_used 6747799552
>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025981
>         uuid c5e2ba38-c982-8a4f-99e5-bb9ff8c5f7cd
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 1897416 otransid 1899154 stransid 0 rtransid 0
>         ctime 1758484800.817458724 (2025-09-21 23:00:00)
>         otime 1758488400.101635440 (2025-09-22 00:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 31 key (14451 ROOT_BACKREF 256) itemoff 12021 itemsize 26
>         root backref key dirid 28635 sequence 2 name snapshot
>     item 32 key (14475 ROOT_ITEM 1942246) itemoff 11582 itemsize 439
>         generation 2025984 root_dirid 256 bytenr 71398831259648 
> byte_limit 0 bytes_used 6755549184
>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025984
>         uuid 38b9ea09-54ef-5140-bddc-83dc884c31b3
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 1940067 otransid 1942246 stransid 0 rtransid 0
>         ctime 1758570000.151997903 (2025-09-22 22:40:00)
>         otime 1758574800.777676964 (2025-09-23 00:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 33 key (14475 ROOT_BACKREF 256) itemoff 11556 itemsize 26
>         root backref key dirid 28683 sequence 2 name snapshot
>     item 34 key (14499 ROOT_ITEM 1977241) itemoff 11117 itemsize 439
>         generation 2025986 root_dirid 256 bytenr 71398873153536 
> byte_limit 0 bytes_used 6538182656
>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025986
>         uuid 703f0298-831b-1e4d-b1f1-fc37ff582770
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 1977241 otransid 1977241 stransid 0 rtransid 0
>         ctime 1758661201.518446292 (2025-09-24 00:00:01)
>         otime 1758661204.161207119 (2025-09-24 00:00:04)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 35 key (14499 ROOT_BACKREF 256) itemoff 11091 itemsize 26
>         root backref key dirid 28731 sequence 2 name snapshot
>     item 36 key (14523 ROOT_ITEM 2001507) itemoff 10652 itemsize 439
>         generation 2025988 root_dirid 256 bytenr 71398873497600 
> byte_limit 0 bytes_used 6561398784
>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025988
>         uuid 9672abd0-7e7b-ab4f-9e0f-7183e58bb6c4
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2001507 otransid 2001507 stransid 0 rtransid 0
>         ctime 1758747600.712874426 (2025-09-25 00:00:00)
>         otime 1758747600.840460363 (2025-09-25 00:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 37 key (14523 ROOT_BACKREF 256) itemoff 10626 itemsize 26
>         root backref key dirid 28779 sequence 2 name snapshot
>     item 38 key (14547 ROOT_ITEM 2017206) itemoff 10187 itemsize 439
>         generation 2025990 root_dirid 256 bytenr 71398872924160 
> byte_limit 0 bytes_used 6570868736
>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025990
>         uuid 671b4b4f-fd99-e140-8535-fce0a16babfb
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2017206 otransid 2017206 stransid 0 rtransid 0
>         ctime 1758834001.88320469 (2025-09-26 00:00:01)
>         otime 1758834001.271433540 (2025-09-26 00:00:01)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 39 key (14547 ROOT_BACKREF 256) itemoff 10161 itemsize 26
>         root backref key dirid 28827 sequence 2 name snapshot
>     item 40 key (14559 ROOT_ITEM 2019735) itemoff 9722 itemsize 439
>         generation 2025992 root_dirid 256 bytenr 75423231606784 
> byte_limit 0 bytes_used 6576078848
>         last_snapshot 2019735 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025992
>         uuid 3ff52125-57ee-6140-87a5-4fb848ec4638
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2019711 otransid 2019735 stransid 0 rtransid 0
>         ctime 1758919659.690576855 (2025-09-26 23:47:39)
>         otime 1758920400.700930626 (2025-09-27 00:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 41 key (14559 ROOT_BACKREF 256) itemoff 9696 itemsize 26
>         root backref key dirid 28851 sequence 2 name snapshot
>     item 42 key (14583 ROOT_ITEM 2022097) itemoff 9257 itemsize 439
>         generation 2025994 root_dirid 256 bytenr 65812067024896 
> byte_limit 0 bytes_used 6557646848
>         last_snapshot 2022097 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025994
>         uuid 860f0635-f296-a04b-9fc5-bd85d480799c
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2022097 otransid 2022097 stransid 0 rtransid 0
>         ctime 1759006819.37527411 (2025-09-28 00:00:19)
>         otime 1759006820.861182212 (2025-09-28 00:00:20)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 43 key (14583 ROOT_BACKREF 256) itemoff 9231 itemsize 26
>         root backref key dirid 28899 sequence 2 name snapshot
>     item 44 key (14607 ROOT_ITEM 2024585) itemoff 8792 itemsize 439
>         generation 2025996 root_dirid 256 bytenr 227573944696832 
> byte_limit 0 bytes_used 6583533568
>         last_snapshot 2024585 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025996
>         uuid af3524d0-8a9a-894c-b5b7-705b10114326
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2024474 otransid 2024585 stransid 0 rtransid 0
>         ctime 1759089642.387498362 (2025-09-28 23:00:42)
>         otime 1759093200.412309185 (2025-09-29 00:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 45 key (14607 ROOT_BACKREF 256) itemoff 8766 itemsize 26
>         root backref key dirid 28947 sequence 2 name snapshot
>     item 46 key (14631 ROOT_ITEM 2025813) itemoff 8327 itemsize 439
>         generation 2025998 root_dirid 256 bytenr 178141351657472 
> byte_limit 0 bytes_used 6602981376
>         last_snapshot 2025813 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2025998
>         uuid 6f085114-93b0-d146-8997-9a41ff62f8db
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2025812 otransid 2025813 stransid 0 rtransid 0
>         ctime 1759177902.175859896 (2025-09-29 23:31:42)
>         otime 1759210423.120386595 (2025-09-30 08:33:43)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 47 key (14631 ROOT_BACKREF 256) itemoff 8301 itemsize 26
>         root backref key dirid 28995 sequence 2 name snapshot
>     item 48 key (14640 ROOT_ITEM 2025865) itemoff 7862 itemsize 439
>         generation 2026000 root_dirid 256 bytenr 194611771719680 
> byte_limit 0 bytes_used 6603735040
>         last_snapshot 2025865 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2026000
>         uuid a48f03d9-4847-654a-af1d-5e0eaf6ecd0c
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2025863 otransid 2025865 stransid 0 rtransid 0
>         ctime 1759237252.139079795 (2025-09-30 16:00:52)
>         otime 1759240800.56609287 (2025-09-30 17:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 49 key (14640 ROOT_BACKREF 256) itemoff 7836 itemsize 26
>         root backref key dirid 29013 sequence 2 name snapshot
>     item 50 key (14641 ROOT_ITEM 2025870) itemoff 7397 itemsize 439
>         generation 2026002 root_dirid 256 bytenr 65801342844928 
> byte_limit 0 bytes_used 6603292672
>         last_snapshot 2025870 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2026002
>         uuid 56e7aa4e-e166-504d-9a55-8c8b074c0fd9
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2025868 otransid 2025870 stransid 0 rtransid 0
>         ctime 1759241717.101400473 (2025-09-30 17:15:17)
>         otime 1759244400.232224532 (2025-09-30 18:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 51 key (14641 ROOT_BACKREF 256) itemoff 7371 itemsize 26
>         root backref key dirid 29015 sequence 2 name snapshot
>     item 52 key (14642 ROOT_ITEM 2025874) itemoff 6932 itemsize 439
>         generation 2026004 root_dirid 256 bytenr 187252895498240 
> byte_limit 0 bytes_used 6603341824
>         last_snapshot 2025874 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2026004
>         uuid f1761dfd-ec60-724a-9af0-dea952cc6771
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2025872 otransid 2025874 stransid 0 rtransid 0
>         ctime 1759244441.841998010 (2025-09-30 18:00:41)
>         otime 1759248000.122051877 (2025-09-30 19:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 53 key (14642 ROOT_BACKREF 256) itemoff 6906 itemsize 26
>         root backref key dirid 29017 sequence 2 name snapshot
>     item 54 key (14643 ROOT_ITEM 2025904) itemoff 6467 itemsize 439
>         generation 2026006 root_dirid 256 bytenr 227587769647104 
> byte_limit 0 bytes_used 6603587584
>         last_snapshot 2025904 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2026006
>         uuid 68fd5037-6c8a-c74e-b08f-cff4cdccf752
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2025904 otransid 2025904 stransid 0 rtransid 0
>         ctime 1759251597.815452999 (2025-09-30 19:59:57)
>         otime 1759251600.884521740 (2025-09-30 20:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 55 key (14643 ROOT_BACKREF 256) itemoff 6441 itemsize 26
>         root backref key dirid 29019 sequence 2 name snapshot
>     item 56 key (14644 ROOT_ITEM 2025922) itemoff 6002 itemsize 439
>         generation 2026008 root_dirid 256 bytenr 227594191208448 
> byte_limit 0 bytes_used 6603653120
>         last_snapshot 2025922 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2026008
>         uuid 07ccd2ae-f51d-2c46-82df-05e16964b2e5
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2025921 otransid 2025922 stransid 0 rtransid 0
>         ctime 1759254975.20573006 (2025-09-30 20:56:15)
>         otime 1759255200.196692319 (2025-09-30 21:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 57 key (14644 ROOT_BACKREF 256) itemoff 5976 itemsize 26
>         root backref key dirid 29021 sequence 2 name snapshot
>     item 58 key (14645 ROOT_ITEM 2025951) itemoff 5537 itemsize 439
>         generation 2026010 root_dirid 256 bytenr 193309358817280 
> byte_limit 0 bytes_used 6603964416
>         last_snapshot 2025951 flags 0x1(RDONLY) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 3 generation_v2 2026010
>         uuid b46ce558-2c39-3944-91d4-b2b229f04a16
>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 2025950 otransid 2025951 stransid 0 rtransid 0
>         ctime 1759257514.988583086 (2025-09-30 21:38:34)
>         otime 1759258800.482340435 (2025-09-30 22:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)
>     item 59 key (14645 ROOT_BACKREF 256) itemoff 5511 itemsize 26
>         root backref key dirid 29023 sequence 2 name snapshot
>     item 60 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 5072 itemsize 439
>         generation 2018764 root_dirid 256 bytenr 71399055278080 
> byte_limit 0 bytes_used 16384
>         last_snapshot 0 flags 0x0(none) refs 1
>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>         level 0 generation_v2 2018764
>         uuid 00000000-0000-0000-0000-000000000000
>         parent_uuid 00000000-0000-0000-0000-000000000000
>         received_uuid 00000000-0000-0000-0000-000000000000
>         ctransid 0 otransid 0 stransid 0 rtransid 0
>         ctime 0.0 (1970-01-01 02:00:00)
>         otime 0.0 (1970-01-01 02:00:00)
>         stime 0.0 (1970-01-01 02:00:00)
>         rtime 0.0 (1970-01-01 02:00:00)

Qu Wenruo kirjoitti 5.10.2025 klo 1.07:
>
>
> 在 2025/10/5 08:23, Henri Hyyryläinen 写道:
>>> One extra thing, at the time of the initial "btrfs check --repair", 
>>> there is no running dev-replace/balance or whatever running?
>>>
>>> Although those operations will be automatically paused by unmount,
>>> btrfs check is not going to be able to handle some of those paused 
>>> operations. 
>>
>> Is there a way for me to check that without mounting the filesystem? 
>> As far as I could find, none of the balance / scrub commands allow 
>> working on an unmounted filesystem. So I couldn't find out if I had 
>> any in canceled state.
>
> # btrfs ins dump-tree -t root <device>
>>
>> Though, I'm pretty sure I let the last scrub and balance operation I 
>> tried to fully complete before starting using the check and repair 
>> commands. But I'm not absolutely certain that I didn't try one of 
>> those a last time and didn't let it fully complete.
>>
>> I'll start that "btrfs check --readonly" command now. And I'll report 
>> back once it is done (hopefully by the morning in my timezone).
>>
>> - Henri Hyyryläinen
>>
>> Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
>>>
>>>
>>> 在 2025/10/5 07:14, Qu Wenruo 写道:
>>>>
>>>>
>>>> 在 2025/10/5 04:13, Henri Hyyryläinen 写道:
>>>>> Hello again.
>>>>>
>>>>> It took over 3 days, but the btrfs check --repair has now 
>>>>> completed seemingly successfully. I mostly saw output about the 
>>>>> file being placed in lost+found and and directory size being 
>>>>> corrected. However, there were some messages about mismatch of 
>>>>> used bytes.
>>>>>
>>>>> Unfortunately it seems like the situation has gotten worse since 
>>>>> the repair, because now I cannot mount the filesystem at all. 
>>>>> Instead I get an error like this:
>>>>>
>>>>>> BTRFS error (device sdc): dev extent physical offset 
>>>>>> 19977638903808 on devid 4 doesn't have corresponding chunk
>>>>>> BTRFS error (device sdc): failed to verify dev extents against 
>>>>>> chunks: -117
>>>>>> BTRFS error (device sdc): open_ctree failed: -117
>>>>> Even if I remove that one problematic device physically from my 
>>>>> computer, the filesystem still refuses to mount with the same 
>>>>> error. Maybe the problems with the device replace are again 
>>>>> showing up with the actual size of the hard drive not being used 
>>>>> correctly? I cannot try to remove the device slack as I cannot 
>>>>> mount the filesystem.
>>>>
>>>> Nope, this is a different problem, and not related to dev replace.
>>>>
>>>> Unfortunately btrfs check has not implemented any repair for that.
>>>>
>>>> Overall if the dev extent is found but not corresponding chunk, it 
>>>> should still be fine but some space unavailable.
>>>>
>>>> But the kernel is overly cautious on chunk tree, as it's a very 
>>>> important and basic functionality.
>>>>
>>>> Please provide the full "btrfs check --readonly" output so that we 
>>>> can evaluate and add the missing repair functionality.
>>>
>>> One extra thing, at the time of the initial "btrfs check --repair", 
>>> there is no running dev-replace/balance or whatever running?
>>>
>>> Although those operations will be automatically paused by unmount,
>>> btrfs check is not going to be able to handle some of those paused 
>>> operations.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>>
>>>>> I did try to run a repair again, and this time I got a bunch of 
>>>>> messages like:
>>>>>
>>>>>> repair deleting extent record: key [65795546775552,169,0]
>>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>>> 65811674234880 root 65811674234880
>>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>>> 0 root 14499
>>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>>> 65791012274176 root 65791012274176
>>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>>> 65806385807360 root 65806385807360
>>>>>> Repaired extent references for 65795546775552
>>>>>> ref mismatch on [65795548233728 16384] extent item 5, found 4
>>>>
>>>> That's fixing some backref mismatch, which you can ignore unless 
>>>> "btrfs check --reaonly" later reports new problems.
>>>>
>>>>> But the filesystem still refuses to mount with the exact same 
>>>>> error. I did not let the repair run entirely as it would have 
>>>>> likely taken another 3 days. What should I do? This time I'm not 
>>>>> finding any good information on what to do. For now, I've started 
>>>>> the repair again, but it doesn't exactly sound like it is even 
>>>>> fixing anything now. Still, I'll let it continue. The output so 
>>>>> far is:
>>>>>
>>>>>> [1/8] checking log skipped (none written)
>>>>>> [2/8] checking root items
>>>>>> Fixed 0 roots.
>>>>>> [3/8] checking extents
>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>> relative chunk.
>>>>>> super bytes used 49454738989056 mismatches actual used 
>>>>>> 49454738923520
>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>> relative chunk.
>>>>>> super bytes used 49454739005440 mismatches actual used 
>>>>>> 49454738956288
>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>> relative chunk.
>>>>>> super bytes used 49454739021824 mismatches actual used 
>>>>>> 49454738972672
>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>> relative chunk.
>>>>>> super bytes used 49454739005440 mismatches actual used 
>>>>>> 49454738972672
>>>>>
>>>>>
>>>>> If I was able to somehow remove that one logically corrupt devid 
>>>>> from the filesystem, or somehow correct the size, that would 
>>>>> hopefully allow me to rebuild from the raid10 data then, but I 
>>>>> can't do those with the unmountable filesystem.
>>>>>
>>>>>
>>>>> - Henri Hyyryläinen
>>>>>
>>>>> Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>>>>>>
>>>>>>
>>>>>> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I hope this is the right place to ask about a filesystem 
>>>>>>> problem. Really shortly put, I have a file that both exists and 
>>>>>>> doesn't and prevents the containing directory from being 
>>>>>>> deleted. No matter what variant of rm and inode based deletion I 
>>>>>>> try I get an error about the file not existing, and I also 
>>>>>>> cannot try to read the file, but if I try to delete the 
>>>>>>> directory I get an error that it is not empty (so the file kind 
>>>>>>> of exists). Trying to ls the directory also gives a file doesn't 
>>>>>>> exist error.
>>>>>>>
>>>>>>> Here's what btrfs check found, which I hope does better in 
>>>>>>> illustrating the problem:
>>>>>>>
>>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>
>>>>>>> I've tried everything I've found suggested including a full 
>>>>>>> scrub, balance with -dusage=75 -musage=75, resetting file 
>>>>>>> attributes, deleting through the find command, and even some 
>>>>>>> repair mount flags that don't seem to exist for btrfs.
>>>>>>
>>>>>> The fs is corrupted, thus none of those will help.
>>>>>> I'm more interested in how the corruption happened.
>>>>>>
>>>>>> Did you use some tools other than btrfs kernel module and 
>>>>>> btrfs-progs?
>>>>>> Like ntfs2btrfs or winbtrfs?
>>>>>>
>>>>>> IIRC certain versions have some bugs related to extent tree, but 
>>>>>> should not cause this problem.
>>>>>>
>>>>>>
>>>>>> The other possibility is hardware memory bitflip, which is more 
>>>>>> common than you thought (almostly one report per month)
>>>>>>
>>>>>> In that case, a full memtest is always recommended, or you will 
>>>>>> hit all kinds of weird corruptions in the future anyway.
>>>>>>
>>>>>>
>>>>>> With a full memtest proving the memory hardware is fine, then 
>>>>>> "btrfs check --repair" should be able to fix it.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>
>>>>>>> What I haven't tried is a full rebalance with no filters, but I 
>>>>>>> did not try that yet as it would take quite a long time and if 
>>>>>>> it only moves data blocks around without recomputing directory 
>>>>>>> items, it doesn't seem like the right tool to fix my problem. So 
>>>>>>> I'm pretty much stuck and to me it seems like my only option is 
>>>>>>> to run btrfs check with the repair flag, but as that has big 
>>>>>>> warnings on it I thought I would try asking here first (sorry if 
>>>>>>> this is not the right experts group to ask). So is there still 
>>>>>>> something I can try or am I finally "allowed" to use the repair 
>>>>>>> command? Here's the full output I got from btrfs check:
>>>>>>>
>>>>>>>> Opening filesystem to check...
>>>>>>>> Checking filesystem on /dev/sdc
>>>>>>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>>>>>> [1/8] checking log skipped (none written)
>>>>>>>> [2/8] checking root items
>>>>>>>> [3/8] checking extents
>>>>>>>> [4/8] checking free space tree
>>>>>>>> We have a space info key for a block group that doesn't exist
>>>>>>>> [5/8] checking fs roots
>>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>>>>>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>> ERROR: errors found in fs roots
>>>>>>>> found 49400296812544 bytes used, error(s) found
>>>>>>>> total csum bytes: 48179330432
>>>>>>>> total tree bytes: 65067483136
>>>>>>>> total fs tree bytes: 12107431936
>>>>>>>> total extent tree bytes: 3194437632
>>>>>>>> btree space waste bytes: 4558984171
>>>>>>>> file data blocks allocated: 76487982252032
>>>>>>>>  referenced 60030799097856
>>>>>>>
>>>>>>> So hopefully if I'm reading things right, running a repair would 
>>>>>>> delete just that one file and directory (which itself is a 
>>>>>>> backup so I will not miss that file at all)?
>>>>>>>
>>>>>>> I do not have enough disk space to copy off the entire 
>>>>>>> filesystem and rebuild from scratch, without doing something 
>>>>>>> like rebalancing all data from raid10 to single and then 
>>>>>>> removing half the disks, but I assume that would take at least 4 
>>>>>>> weeks to process (as I just replaced a disk which took like a 
>>>>>>> week).
>>>>>>>
>>>>>>> As to what originally caused the corruption, I think it was 
>>>>>>> probably faulty RAM, because up to to like 3 weeks ago I had one 
>>>>>>> really bad RAM stick in my computer where a certain memory 
>>>>>>> region always had incorrectly reading bytes. I had seen 
>>>>>>> intermittent quite high csum errors in monthly scrubs pretty 
>>>>>>> randomly, which thankfully could almost always be corrected so I 
>>>>>>> didn't have any major problems even though I had like totally 
>>>>>>> broken RAM in my computer for who knows how long. So btrfs was 
>>>>>>> able to protect my data quite impressively from bad RAM.
>>>>>>>
>>>>>>> Sorry for getting a bit sidetracked there, but what should I do 
>>>>>>> in this situation?
>>>>>>>
>>>>>>> - Henri Hyyryläinen
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>

Re: How to remove an unremovable file and directory?

[Qu Wenruo]

在 2025/10/5 09:06, Henri Hyyryläinen 写道:
> Thanks for the command.
> 
> I tried reading through the data but I could not determine if there was 
> any information related to scrub or balance status.

It's fine. No persistent item for balance/dev-replace. So it's not involved.

Waiting for the readonly check result.

And meanwhile, please prepare an environment to compile btrfs-progs.

If there is only one stray dev extent, I'll create a simple dirty fix so 
that you can mount the fs again.

Thanks,
Qu

> 
> Here's the full output I got:
> 
>> btrfs-progs v6.16.1
>> root tree
>> leaf 227587769614336 items 61 free space 3547 generation 2035509 owner 
>> ROOT_TREE
>> leaf 227587769614336 flags 0x1(WRITTEN) backref revision 1
>> fs uuid 2b4ad16d-e456-4adf-960b-dca43560b98b
>> chunk uuid 0e317678-73cc-4485-ab6b-987285e19681
>>     item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
>>         generation 2035509 root_dirid 0 bytenr 227587769630720 
>> byte_limit 0 bytes_used 3319316480
>>         last_snapshot 0 flags 0x0(none) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2035509
>>         uuid 00000000-0000-0000-0000-000000000000
>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>         ctime 0.0 (1970-01-01 02:00:00)
>>         otime 0.0 (1970-01-01 02:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439
>>         generation 2025974 root_dirid 0 bytenr 227594196631552 
>> byte_limit 0 bytes_used 21889024
>>         last_snapshot 0 flags 0x0(none) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 2 generation_v2 2025974
>>         uuid 00000000-0000-0000-0000-000000000000
>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>         ctime 0.0 (1970-01-01 02:00:00)
>>         otime 0.0 (1970-01-01 02:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17
>>         index 0 namelen 7 name: default
>>     item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439
>>         generation 2025975 root_dirid 256 bytenr 227573939142656 
>> byte_limit 0 bytes_used 6603997184
>>         last_snapshot 2025951 flags 0x0(none) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025975
>>         uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2025953 otransid 0 stransid 0 rtransid 0
>>         ctime 1759258850.523837328 (2025-09-30 22:00:50)
>>         otime 1679586168.0 (2023-03-23 17:42:48)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 4 key (FS_TREE ROOT_REF 256) itemoff 14921 itemsize 28
>>         root ref key dirid 256 sequence 6 name .snapshots
>>     item 5 key (ROOT_TREE_DIR INODE_ITEM 0) itemoff 14761 itemsize 160
>>         generation 3 transid 0 size 0 nbytes 16384
>>         block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>>         sequence 0 flags 0x0(none)
>>         atime 1679586169.0 (2023-03-23 17:42:49)
>>         ctime 1679586169.0 (2023-03-23 17:42:49)
>>         mtime 1679586169.0 (2023-03-23 17:42:49)
>>         otime 1679586169.0 (2023-03-23 17:42:49)
>>     item 6 key (ROOT_TREE_DIR INODE_REF 6) itemoff 14749 itemsize 12
>>         index 0 namelen 2 name: ..
>>     item 7 key (ROOT_TREE_DIR DIR_ITEM 2378154706) itemoff 14712 
>> itemsize 37
>>         location key (FS_TREE ROOT_ITEM 18446744073709551615) type DIR
>>         transid 0 data_len 0 name_len 7
>>         name: default
>>     item 8 key (CSUM_TREE ROOT_ITEM 0) itemoff 14273 itemsize 439
>>         generation 2025954 root_dirid 0 bytenr 227588044062720 
>> byte_limit 0 bytes_used 49791238144
>>         last_snapshot 0 flags 0x0(none) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025954
>>         uuid 00000000-0000-0000-0000-000000000000
>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>         ctime 0.0 (1970-01-01 02:00:00)
>>         otime 0.0 (1970-01-01 02:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 9 key (UUID_TREE ROOT_ITEM 0) itemoff 13834 itemsize 439
>>         generation 2025954 root_dirid 0 bytenr 227587802447872 
>> byte_limit 0 bytes_used 16384
>>         last_snapshot 0 flags 0x0(none) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 0 generation_v2 2025954
>>         uuid 00000000-0000-0000-0000-000000000000
>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>         ctime 0.0 (1970-01-01 02:00:00)
>>         otime 0.0 (1970-01-01 02:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 10 key (256 ROOT_ITEM 0) itemoff 13395 itemsize 439
>>         generation 2025954 root_dirid 256 bytenr 227587802316800 
>> byte_limit 0 bytes_used 49152
>>         last_snapshot 2015386 flags 0x0(none) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 1 generation_v2 2025954
>>         uuid 56f85f76-1eb8-1245-a35b-8a9307d5c68f
>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2025954 otransid 31 stransid 0 rtransid 0
>>         ctime 1759260258.238143629 (2025-09-30 22:24:18)
>>         otime 1679588253.995834574 (2023-03-23 18:17:33)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 11 key (256 ROOT_BACKREF 5) itemoff 13367 itemsize 28
>>         root backref key dirid 256 sequence 6 name .snapshots
>>     item 12 key (256 ROOT_REF 14428) itemoff 13341 itemsize 26
>>         root ref key dirid 28589 sequence 2 name snapshot
>>     item 13 key (256 ROOT_REF 14451) itemoff 13315 itemsize 26
>>         root ref key dirid 28635 sequence 2 name snapshot
>>     item 14 key (256 ROOT_REF 14475) itemoff 13289 itemsize 26
>>         root ref key dirid 28683 sequence 2 name snapshot
>>     item 15 key (256 ROOT_REF 14499) itemoff 13263 itemsize 26
>>         root ref key dirid 28731 sequence 2 name snapshot
>>     item 16 key (256 ROOT_REF 14523) itemoff 13237 itemsize 26
>>         root ref key dirid 28779 sequence 2 name snapshot
>>     item 17 key (256 ROOT_REF 14547) itemoff 13211 itemsize 26
>>         root ref key dirid 28827 sequence 2 name snapshot
>>     item 18 key (256 ROOT_REF 14559) itemoff 13185 itemsize 26
>>         root ref key dirid 28851 sequence 2 name snapshot
>>     item 19 key (256 ROOT_REF 14583) itemoff 13159 itemsize 26
>>         root ref key dirid 28899 sequence 2 name snapshot
>>     item 20 key (256 ROOT_REF 14607) itemoff 13133 itemsize 26
>>         root ref key dirid 28947 sequence 2 name snapshot
>>     item 21 key (256 ROOT_REF 14631) itemoff 13107 itemsize 26
>>         root ref key dirid 28995 sequence 2 name snapshot
>>     item 22 key (256 ROOT_REF 14640) itemoff 13081 itemsize 26
>>         root ref key dirid 29013 sequence 2 name snapshot
>>     item 23 key (256 ROOT_REF 14641) itemoff 13055 itemsize 26
>>         root ref key dirid 29015 sequence 2 name snapshot
>>     item 24 key (256 ROOT_REF 14642) itemoff 13029 itemsize 26
>>         root ref key dirid 29017 sequence 2 name snapshot
>>     item 25 key (256 ROOT_REF 14643) itemoff 13003 itemsize 26
>>         root ref key dirid 29019 sequence 2 name snapshot
>>     item 26 key (256 ROOT_REF 14644) itemoff 12977 itemsize 26
>>         root ref key dirid 29021 sequence 2 name snapshot
>>     item 27 key (256 ROOT_REF 14645) itemoff 12951 itemsize 26
>>         root ref key dirid 29023 sequence 2 name snapshot
>>     item 28 key (14428 ROOT_ITEM 1849346) itemoff 12512 itemsize 439
>>         generation 2025978 root_dirid 256 bytenr 71398781173760 
>> byte_limit 0 bytes_used 8815476736
>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025978
>>         uuid 97cd122d-c519-8746-bdfe-13479bfefb9d
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 1849346 otransid 1849346 stransid 0 rtransid 0
>>         ctime 1758406304.841464812 (2025-09-21 01:11:44)
>>         otime 1758406304.840394467 (2025-09-21 01:11:44)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 29 key (14428 ROOT_BACKREF 256) itemoff 12486 itemsize 26
>>         root backref key dirid 28589 sequence 2 name snapshot
>>     item 30 key (14451 ROOT_ITEM 1899154) itemoff 12047 itemsize 439
>>         generation 2025981 root_dirid 256 bytenr 71398809468928 
>> byte_limit 0 bytes_used 6747799552
>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025981
>>         uuid c5e2ba38-c982-8a4f-99e5-bb9ff8c5f7cd
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 1897416 otransid 1899154 stransid 0 rtransid 0
>>         ctime 1758484800.817458724 (2025-09-21 23:00:00)
>>         otime 1758488400.101635440 (2025-09-22 00:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 31 key (14451 ROOT_BACKREF 256) itemoff 12021 itemsize 26
>>         root backref key dirid 28635 sequence 2 name snapshot
>>     item 32 key (14475 ROOT_ITEM 1942246) itemoff 11582 itemsize 439
>>         generation 2025984 root_dirid 256 bytenr 71398831259648 
>> byte_limit 0 bytes_used 6755549184
>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025984
>>         uuid 38b9ea09-54ef-5140-bddc-83dc884c31b3
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 1940067 otransid 1942246 stransid 0 rtransid 0
>>         ctime 1758570000.151997903 (2025-09-22 22:40:00)
>>         otime 1758574800.777676964 (2025-09-23 00:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 33 key (14475 ROOT_BACKREF 256) itemoff 11556 itemsize 26
>>         root backref key dirid 28683 sequence 2 name snapshot
>>     item 34 key (14499 ROOT_ITEM 1977241) itemoff 11117 itemsize 439
>>         generation 2025986 root_dirid 256 bytenr 71398873153536 
>> byte_limit 0 bytes_used 6538182656
>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025986
>>         uuid 703f0298-831b-1e4d-b1f1-fc37ff582770
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 1977241 otransid 1977241 stransid 0 rtransid 0
>>         ctime 1758661201.518446292 (2025-09-24 00:00:01)
>>         otime 1758661204.161207119 (2025-09-24 00:00:04)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 35 key (14499 ROOT_BACKREF 256) itemoff 11091 itemsize 26
>>         root backref key dirid 28731 sequence 2 name snapshot
>>     item 36 key (14523 ROOT_ITEM 2001507) itemoff 10652 itemsize 439
>>         generation 2025988 root_dirid 256 bytenr 71398873497600 
>> byte_limit 0 bytes_used 6561398784
>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025988
>>         uuid 9672abd0-7e7b-ab4f-9e0f-7183e58bb6c4
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2001507 otransid 2001507 stransid 0 rtransid 0
>>         ctime 1758747600.712874426 (2025-09-25 00:00:00)
>>         otime 1758747600.840460363 (2025-09-25 00:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 37 key (14523 ROOT_BACKREF 256) itemoff 10626 itemsize 26
>>         root backref key dirid 28779 sequence 2 name snapshot
>>     item 38 key (14547 ROOT_ITEM 2017206) itemoff 10187 itemsize 439
>>         generation 2025990 root_dirid 256 bytenr 71398872924160 
>> byte_limit 0 bytes_used 6570868736
>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025990
>>         uuid 671b4b4f-fd99-e140-8535-fce0a16babfb
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2017206 otransid 2017206 stransid 0 rtransid 0
>>         ctime 1758834001.88320469 (2025-09-26 00:00:01)
>>         otime 1758834001.271433540 (2025-09-26 00:00:01)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 39 key (14547 ROOT_BACKREF 256) itemoff 10161 itemsize 26
>>         root backref key dirid 28827 sequence 2 name snapshot
>>     item 40 key (14559 ROOT_ITEM 2019735) itemoff 9722 itemsize 439
>>         generation 2025992 root_dirid 256 bytenr 75423231606784 
>> byte_limit 0 bytes_used 6576078848
>>         last_snapshot 2019735 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025992
>>         uuid 3ff52125-57ee-6140-87a5-4fb848ec4638
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2019711 otransid 2019735 stransid 0 rtransid 0
>>         ctime 1758919659.690576855 (2025-09-26 23:47:39)
>>         otime 1758920400.700930626 (2025-09-27 00:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 41 key (14559 ROOT_BACKREF 256) itemoff 9696 itemsize 26
>>         root backref key dirid 28851 sequence 2 name snapshot
>>     item 42 key (14583 ROOT_ITEM 2022097) itemoff 9257 itemsize 439
>>         generation 2025994 root_dirid 256 bytenr 65812067024896 
>> byte_limit 0 bytes_used 6557646848
>>         last_snapshot 2022097 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025994
>>         uuid 860f0635-f296-a04b-9fc5-bd85d480799c
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2022097 otransid 2022097 stransid 0 rtransid 0
>>         ctime 1759006819.37527411 (2025-09-28 00:00:19)
>>         otime 1759006820.861182212 (2025-09-28 00:00:20)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 43 key (14583 ROOT_BACKREF 256) itemoff 9231 itemsize 26
>>         root backref key dirid 28899 sequence 2 name snapshot
>>     item 44 key (14607 ROOT_ITEM 2024585) itemoff 8792 itemsize 439
>>         generation 2025996 root_dirid 256 bytenr 227573944696832 
>> byte_limit 0 bytes_used 6583533568
>>         last_snapshot 2024585 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025996
>>         uuid af3524d0-8a9a-894c-b5b7-705b10114326
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2024474 otransid 2024585 stransid 0 rtransid 0
>>         ctime 1759089642.387498362 (2025-09-28 23:00:42)
>>         otime 1759093200.412309185 (2025-09-29 00:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 45 key (14607 ROOT_BACKREF 256) itemoff 8766 itemsize 26
>>         root backref key dirid 28947 sequence 2 name snapshot
>>     item 46 key (14631 ROOT_ITEM 2025813) itemoff 8327 itemsize 439
>>         generation 2025998 root_dirid 256 bytenr 178141351657472 
>> byte_limit 0 bytes_used 6602981376
>>         last_snapshot 2025813 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2025998
>>         uuid 6f085114-93b0-d146-8997-9a41ff62f8db
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2025812 otransid 2025813 stransid 0 rtransid 0
>>         ctime 1759177902.175859896 (2025-09-29 23:31:42)
>>         otime 1759210423.120386595 (2025-09-30 08:33:43)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 47 key (14631 ROOT_BACKREF 256) itemoff 8301 itemsize 26
>>         root backref key dirid 28995 sequence 2 name snapshot
>>     item 48 key (14640 ROOT_ITEM 2025865) itemoff 7862 itemsize 439
>>         generation 2026000 root_dirid 256 bytenr 194611771719680 
>> byte_limit 0 bytes_used 6603735040
>>         last_snapshot 2025865 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2026000
>>         uuid a48f03d9-4847-654a-af1d-5e0eaf6ecd0c
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2025863 otransid 2025865 stransid 0 rtransid 0
>>         ctime 1759237252.139079795 (2025-09-30 16:00:52)
>>         otime 1759240800.56609287 (2025-09-30 17:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 49 key (14640 ROOT_BACKREF 256) itemoff 7836 itemsize 26
>>         root backref key dirid 29013 sequence 2 name snapshot
>>     item 50 key (14641 ROOT_ITEM 2025870) itemoff 7397 itemsize 439
>>         generation 2026002 root_dirid 256 bytenr 65801342844928 
>> byte_limit 0 bytes_used 6603292672
>>         last_snapshot 2025870 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2026002
>>         uuid 56e7aa4e-e166-504d-9a55-8c8b074c0fd9
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2025868 otransid 2025870 stransid 0 rtransid 0
>>         ctime 1759241717.101400473 (2025-09-30 17:15:17)
>>         otime 1759244400.232224532 (2025-09-30 18:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 51 key (14641 ROOT_BACKREF 256) itemoff 7371 itemsize 26
>>         root backref key dirid 29015 sequence 2 name snapshot
>>     item 52 key (14642 ROOT_ITEM 2025874) itemoff 6932 itemsize 439
>>         generation 2026004 root_dirid 256 bytenr 187252895498240 
>> byte_limit 0 bytes_used 6603341824
>>         last_snapshot 2025874 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2026004
>>         uuid f1761dfd-ec60-724a-9af0-dea952cc6771
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2025872 otransid 2025874 stransid 0 rtransid 0
>>         ctime 1759244441.841998010 (2025-09-30 18:00:41)
>>         otime 1759248000.122051877 (2025-09-30 19:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 53 key (14642 ROOT_BACKREF 256) itemoff 6906 itemsize 26
>>         root backref key dirid 29017 sequence 2 name snapshot
>>     item 54 key (14643 ROOT_ITEM 2025904) itemoff 6467 itemsize 439
>>         generation 2026006 root_dirid 256 bytenr 227587769647104 
>> byte_limit 0 bytes_used 6603587584
>>         last_snapshot 2025904 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2026006
>>         uuid 68fd5037-6c8a-c74e-b08f-cff4cdccf752
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2025904 otransid 2025904 stransid 0 rtransid 0
>>         ctime 1759251597.815452999 (2025-09-30 19:59:57)
>>         otime 1759251600.884521740 (2025-09-30 20:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 55 key (14643 ROOT_BACKREF 256) itemoff 6441 itemsize 26
>>         root backref key dirid 29019 sequence 2 name snapshot
>>     item 56 key (14644 ROOT_ITEM 2025922) itemoff 6002 itemsize 439
>>         generation 2026008 root_dirid 256 bytenr 227594191208448 
>> byte_limit 0 bytes_used 6603653120
>>         last_snapshot 2025922 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2026008
>>         uuid 07ccd2ae-f51d-2c46-82df-05e16964b2e5
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2025921 otransid 2025922 stransid 0 rtransid 0
>>         ctime 1759254975.20573006 (2025-09-30 20:56:15)
>>         otime 1759255200.196692319 (2025-09-30 21:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 57 key (14644 ROOT_BACKREF 256) itemoff 5976 itemsize 26
>>         root backref key dirid 29021 sequence 2 name snapshot
>>     item 58 key (14645 ROOT_ITEM 2025951) itemoff 5537 itemsize 439
>>         generation 2026010 root_dirid 256 bytenr 193309358817280 
>> byte_limit 0 bytes_used 6603964416
>>         last_snapshot 2025951 flags 0x1(RDONLY) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 3 generation_v2 2026010
>>         uuid b46ce558-2c39-3944-91d4-b2b229f04a16
>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 2025950 otransid 2025951 stransid 0 rtransid 0
>>         ctime 1759257514.988583086 (2025-09-30 21:38:34)
>>         otime 1759258800.482340435 (2025-09-30 22:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
>>     item 59 key (14645 ROOT_BACKREF 256) itemoff 5511 itemsize 26
>>         root backref key dirid 29023 sequence 2 name snapshot
>>     item 60 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 5072 itemsize 439
>>         generation 2018764 root_dirid 256 bytenr 71399055278080 
>> byte_limit 0 bytes_used 16384
>>         last_snapshot 0 flags 0x0(none) refs 1
>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>         level 0 generation_v2 2018764
>>         uuid 00000000-0000-0000-0000-000000000000
>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>         received_uuid 00000000-0000-0000-0000-000000000000
>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>         ctime 0.0 (1970-01-01 02:00:00)
>>         otime 0.0 (1970-01-01 02:00:00)
>>         stime 0.0 (1970-01-01 02:00:00)
>>         rtime 0.0 (1970-01-01 02:00:00)
> 
> Qu Wenruo kirjoitti 5.10.2025 klo 1.07:
>>
>>
>> 在 2025/10/5 08:23, Henri Hyyryläinen 写道:
>>>> One extra thing, at the time of the initial "btrfs check --repair", 
>>>> there is no running dev-replace/balance or whatever running?
>>>>
>>>> Although those operations will be automatically paused by unmount,
>>>> btrfs check is not going to be able to handle some of those paused 
>>>> operations. 
>>>
>>> Is there a way for me to check that without mounting the filesystem? 
>>> As far as I could find, none of the balance / scrub commands allow 
>>> working on an unmounted filesystem. So I couldn't find out if I had 
>>> any in canceled state.
>>
>> # btrfs ins dump-tree -t root <device>
>>>
>>> Though, I'm pretty sure I let the last scrub and balance operation I 
>>> tried to fully complete before starting using the check and repair 
>>> commands. But I'm not absolutely certain that I didn't try one of 
>>> those a last time and didn't let it fully complete.
>>>
>>> I'll start that "btrfs check --readonly" command now. And I'll report 
>>> back once it is done (hopefully by the morning in my timezone).
>>>
>>> - Henri Hyyryläinen
>>>
>>> Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
>>>>
>>>>
>>>> 在 2025/10/5 07:14, Qu Wenruo 写道:
>>>>>
>>>>>
>>>>> 在 2025/10/5 04:13, Henri Hyyryläinen 写道:
>>>>>> Hello again.
>>>>>>
>>>>>> It took over 3 days, but the btrfs check --repair has now 
>>>>>> completed seemingly successfully. I mostly saw output about the 
>>>>>> file being placed in lost+found and and directory size being 
>>>>>> corrected. However, there were some messages about mismatch of 
>>>>>> used bytes.
>>>>>>
>>>>>> Unfortunately it seems like the situation has gotten worse since 
>>>>>> the repair, because now I cannot mount the filesystem at all. 
>>>>>> Instead I get an error like this:
>>>>>>
>>>>>>> BTRFS error (device sdc): dev extent physical offset 
>>>>>>> 19977638903808 on devid 4 doesn't have corresponding chunk
>>>>>>> BTRFS error (device sdc): failed to verify dev extents against 
>>>>>>> chunks: -117
>>>>>>> BTRFS error (device sdc): open_ctree failed: -117
>>>>>> Even if I remove that one problematic device physically from my 
>>>>>> computer, the filesystem still refuses to mount with the same 
>>>>>> error. Maybe the problems with the device replace are again 
>>>>>> showing up with the actual size of the hard drive not being used 
>>>>>> correctly? I cannot try to remove the device slack as I cannot 
>>>>>> mount the filesystem.
>>>>>
>>>>> Nope, this is a different problem, and not related to dev replace.
>>>>>
>>>>> Unfortunately btrfs check has not implemented any repair for that.
>>>>>
>>>>> Overall if the dev extent is found but not corresponding chunk, it 
>>>>> should still be fine but some space unavailable.
>>>>>
>>>>> But the kernel is overly cautious on chunk tree, as it's a very 
>>>>> important and basic functionality.
>>>>>
>>>>> Please provide the full "btrfs check --readonly" output so that we 
>>>>> can evaluate and add the missing repair functionality.
>>>>
>>>> One extra thing, at the time of the initial "btrfs check --repair", 
>>>> there is no running dev-replace/balance or whatever running?
>>>>
>>>> Although those operations will be automatically paused by unmount,
>>>> btrfs check is not going to be able to handle some of those paused 
>>>> operations.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>>
>>>>>> I did try to run a repair again, and this time I got a bunch of 
>>>>>> messages like:
>>>>>>
>>>>>>> repair deleting extent record: key [65795546775552,169,0]
>>>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>>>> 65811674234880 root 65811674234880
>>>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>>>> 0 root 14499
>>>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>>>> 65791012274176 root 65791012274176
>>>>>>> adding new tree backref on start 65795546775552 len 16384 parent 
>>>>>>> 65806385807360 root 65806385807360
>>>>>>> Repaired extent references for 65795546775552
>>>>>>> ref mismatch on [65795548233728 16384] extent item 5, found 4
>>>>>
>>>>> That's fixing some backref mismatch, which you can ignore unless 
>>>>> "btrfs check --reaonly" later reports new problems.
>>>>>
>>>>>> But the filesystem still refuses to mount with the exact same 
>>>>>> error. I did not let the repair run entirely as it would have 
>>>>>> likely taken another 3 days. What should I do? This time I'm not 
>>>>>> finding any good information on what to do. For now, I've started 
>>>>>> the repair again, but it doesn't exactly sound like it is even 
>>>>>> fixing anything now. Still, I'll let it continue. The output so 
>>>>>> far is:
>>>>>>
>>>>>>> [1/8] checking log skipped (none written)
>>>>>>> [2/8] checking root items
>>>>>>> Fixed 0 roots.
>>>>>>> [3/8] checking extents
>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>> relative chunk.
>>>>>>> super bytes used 49454738989056 mismatches actual used 
>>>>>>> 49454738923520
>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>> relative chunk.
>>>>>>> super bytes used 49454739005440 mismatches actual used 
>>>>>>> 49454738956288
>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>> relative chunk.
>>>>>>> super bytes used 49454739021824 mismatches actual used 
>>>>>>> 49454738972672
>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>> relative chunk.
>>>>>>> super bytes used 49454739005440 mismatches actual used 
>>>>>>> 49454738972672
>>>>>>
>>>>>>
>>>>>> If I was able to somehow remove that one logically corrupt devid 
>>>>>> from the filesystem, or somehow correct the size, that would 
>>>>>> hopefully allow me to rebuild from the raid10 data then, but I 
>>>>>> can't do those with the unmountable filesystem.
>>>>>>
>>>>>>
>>>>>> - Henri Hyyryläinen
>>>>>>
>>>>>> Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>>>>>>>
>>>>>>>
>>>>>>> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I hope this is the right place to ask about a filesystem 
>>>>>>>> problem. Really shortly put, I have a file that both exists and 
>>>>>>>> doesn't and prevents the containing directory from being 
>>>>>>>> deleted. No matter what variant of rm and inode based deletion I 
>>>>>>>> try I get an error about the file not existing, and I also 
>>>>>>>> cannot try to read the file, but if I try to delete the 
>>>>>>>> directory I get an error that it is not empty (so the file kind 
>>>>>>>> of exists). Trying to ls the directory also gives a file doesn't 
>>>>>>>> exist error.
>>>>>>>>
>>>>>>>> Here's what btrfs check found, which I hope does better in 
>>>>>>>> illustrating the problem:
>>>>>>>>
>>>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>
>>>>>>>> I've tried everything I've found suggested including a full 
>>>>>>>> scrub, balance with -dusage=75 -musage=75, resetting file 
>>>>>>>> attributes, deleting through the find command, and even some 
>>>>>>>> repair mount flags that don't seem to exist for btrfs.
>>>>>>>
>>>>>>> The fs is corrupted, thus none of those will help.
>>>>>>> I'm more interested in how the corruption happened.
>>>>>>>
>>>>>>> Did you use some tools other than btrfs kernel module and btrfs- 
>>>>>>> progs?
>>>>>>> Like ntfs2btrfs or winbtrfs?
>>>>>>>
>>>>>>> IIRC certain versions have some bugs related to extent tree, but 
>>>>>>> should not cause this problem.
>>>>>>>
>>>>>>>
>>>>>>> The other possibility is hardware memory bitflip, which is more 
>>>>>>> common than you thought (almostly one report per month)
>>>>>>>
>>>>>>> In that case, a full memtest is always recommended, or you will 
>>>>>>> hit all kinds of weird corruptions in the future anyway.
>>>>>>>
>>>>>>>
>>>>>>> With a full memtest proving the memory hardware is fine, then 
>>>>>>> "btrfs check --repair" should be able to fix it.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>>
>>>>>>>> What I haven't tried is a full rebalance with no filters, but I 
>>>>>>>> did not try that yet as it would take quite a long time and if 
>>>>>>>> it only moves data blocks around without recomputing directory 
>>>>>>>> items, it doesn't seem like the right tool to fix my problem. So 
>>>>>>>> I'm pretty much stuck and to me it seems like my only option is 
>>>>>>>> to run btrfs check with the repair flag, but as that has big 
>>>>>>>> warnings on it I thought I would try asking here first (sorry if 
>>>>>>>> this is not the right experts group to ask). So is there still 
>>>>>>>> something I can try or am I finally "allowed" to use the repair 
>>>>>>>> command? Here's the full output I got from btrfs check:
>>>>>>>>
>>>>>>>>> Opening filesystem to check...
>>>>>>>>> Checking filesystem on /dev/sdc
>>>>>>>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>>>>>>> [1/8] checking log skipped (none written)
>>>>>>>>> [2/8] checking root items
>>>>>>>>> [3/8] checking extents
>>>>>>>>> [4/8] checking free space tree
>>>>>>>>> We have a space info key for a block group that doesn't exist
>>>>>>>>> [5/8] checking fs roots
>>>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>>>>>>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>>>>>>>> ERROR: errors found in fs roots
>>>>>>>>> found 49400296812544 bytes used, error(s) found
>>>>>>>>> total csum bytes: 48179330432
>>>>>>>>> total tree bytes: 65067483136
>>>>>>>>> total fs tree bytes: 12107431936
>>>>>>>>> total extent tree bytes: 3194437632
>>>>>>>>> btree space waste bytes: 4558984171
>>>>>>>>> file data blocks allocated: 76487982252032
>>>>>>>>>  referenced 60030799097856
>>>>>>>>
>>>>>>>> So hopefully if I'm reading things right, running a repair would 
>>>>>>>> delete just that one file and directory (which itself is a 
>>>>>>>> backup so I will not miss that file at all)?
>>>>>>>>
>>>>>>>> I do not have enough disk space to copy off the entire 
>>>>>>>> filesystem and rebuild from scratch, without doing something 
>>>>>>>> like rebalancing all data from raid10 to single and then 
>>>>>>>> removing half the disks, but I assume that would take at least 4 
>>>>>>>> weeks to process (as I just replaced a disk which took like a 
>>>>>>>> week).
>>>>>>>>
>>>>>>>> As to what originally caused the corruption, I think it was 
>>>>>>>> probably faulty RAM, because up to to like 3 weeks ago I had one 
>>>>>>>> really bad RAM stick in my computer where a certain memory 
>>>>>>>> region always had incorrectly reading bytes. I had seen 
>>>>>>>> intermittent quite high csum errors in monthly scrubs pretty 
>>>>>>>> randomly, which thankfully could almost always be corrected so I 
>>>>>>>> didn't have any major problems even though I had like totally 
>>>>>>>> broken RAM in my computer for who knows how long. So btrfs was 
>>>>>>>> able to protect my data quite impressively from bad RAM.
>>>>>>>>
>>>>>>>> Sorry for getting a bit sidetracked there, but what should I do 
>>>>>>>> in this situation?
>>>>>>>>
>>>>>>>> - Henri Hyyryläinen
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>
> 

Re: How to remove an unremovable file and directory?

[Henri Hyyryläinen]

Hello,

The check has completed and, probably thanks to the second repair, it 
did not find much. Here's the full output:

> [1/8] checking log skipped (none written)
> [2/8] checking root items
> [3/8] checking extents
> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
> chunk.
> [4/8] checking free space cache
> [5/8] checking fs roots
> [6/8] checking only csums items (without verifying data)
> [7/8] checking root refs
> [8/8] checking quota groups skipped (not enabled on this FS)
> Opening filesystem to check...
> Checking filesystem on /dev/sdb
> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
> found 49454062784512 bytes used, no error found
> total csum bytes: 48225005724
> total tree bytes: 72333164544
> total fs tree bytes: 19189743616
> total extent tree bytes: 3319316480
> btree space waste bytes: 5998261217
> file data blocks allocated: 119737771229184
>  referenced 72125752061952


- Henri Hyyryläinen

Qu Wenruo kirjoitti 5.10.2025 klo 1.41:
>
>
> 在 2025/10/5 09:06, Henri Hyyryläinen 写道:
>> Thanks for the command.
>>
>> I tried reading through the data but I could not determine if there 
>> was any information related to scrub or balance status.
>
> It's fine. No persistent item for balance/dev-replace. So it's not 
> involved.
>
> Waiting for the readonly check result.
>
> And meanwhile, please prepare an environment to compile btrfs-progs.
>
> If there is only one stray dev extent, I'll create a simple dirty fix 
> so that you can mount the fs again.
>
> Thanks,
> Qu
>
>>
>> Here's the full output I got:
>>
>>> btrfs-progs v6.16.1
>>> root tree
>>> leaf 227587769614336 items 61 free space 3547 generation 2035509 
>>> owner ROOT_TREE
>>> leaf 227587769614336 flags 0x1(WRITTEN) backref revision 1
>>> fs uuid 2b4ad16d-e456-4adf-960b-dca43560b98b
>>> chunk uuid 0e317678-73cc-4485-ab6b-987285e19681
>>>     item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
>>>         generation 2035509 root_dirid 0 bytenr 227587769630720 
>>> byte_limit 0 bytes_used 3319316480
>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2035509
>>>         uuid 00000000-0000-0000-0000-000000000000
>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>         otime 0.0 (1970-01-01 02:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439
>>>         generation 2025974 root_dirid 0 bytenr 227594196631552 
>>> byte_limit 0 bytes_used 21889024
>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 2 generation_v2 2025974
>>>         uuid 00000000-0000-0000-0000-000000000000
>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>         otime 0.0 (1970-01-01 02:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17
>>>         index 0 namelen 7 name: default
>>>     item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439
>>>         generation 2025975 root_dirid 256 bytenr 227573939142656 
>>> byte_limit 0 bytes_used 6603997184
>>>         last_snapshot 2025951 flags 0x0(none) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025975
>>>         uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2025953 otransid 0 stransid 0 rtransid 0
>>>         ctime 1759258850.523837328 (2025-09-30 22:00:50)
>>>         otime 1679586168.0 (2023-03-23 17:42:48)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 4 key (FS_TREE ROOT_REF 256) itemoff 14921 itemsize 28
>>>         root ref key dirid 256 sequence 6 name .snapshots
>>>     item 5 key (ROOT_TREE_DIR INODE_ITEM 0) itemoff 14761 itemsize 160
>>>         generation 3 transid 0 size 0 nbytes 16384
>>>         block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>>>         sequence 0 flags 0x0(none)
>>>         atime 1679586169.0 (2023-03-23 17:42:49)
>>>         ctime 1679586169.0 (2023-03-23 17:42:49)
>>>         mtime 1679586169.0 (2023-03-23 17:42:49)
>>>         otime 1679586169.0 (2023-03-23 17:42:49)
>>>     item 6 key (ROOT_TREE_DIR INODE_REF 6) itemoff 14749 itemsize 12
>>>         index 0 namelen 2 name: ..
>>>     item 7 key (ROOT_TREE_DIR DIR_ITEM 2378154706) itemoff 14712 
>>> itemsize 37
>>>         location key (FS_TREE ROOT_ITEM 18446744073709551615) type DIR
>>>         transid 0 data_len 0 name_len 7
>>>         name: default
>>>     item 8 key (CSUM_TREE ROOT_ITEM 0) itemoff 14273 itemsize 439
>>>         generation 2025954 root_dirid 0 bytenr 227588044062720 
>>> byte_limit 0 bytes_used 49791238144
>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025954
>>>         uuid 00000000-0000-0000-0000-000000000000
>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>         otime 0.0 (1970-01-01 02:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 9 key (UUID_TREE ROOT_ITEM 0) itemoff 13834 itemsize 439
>>>         generation 2025954 root_dirid 0 bytenr 227587802447872 
>>> byte_limit 0 bytes_used 16384
>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 0 generation_v2 2025954
>>>         uuid 00000000-0000-0000-0000-000000000000
>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>         otime 0.0 (1970-01-01 02:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 10 key (256 ROOT_ITEM 0) itemoff 13395 itemsize 439
>>>         generation 2025954 root_dirid 256 bytenr 227587802316800 
>>> byte_limit 0 bytes_used 49152
>>>         last_snapshot 2015386 flags 0x0(none) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 1 generation_v2 2025954
>>>         uuid 56f85f76-1eb8-1245-a35b-8a9307d5c68f
>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2025954 otransid 31 stransid 0 rtransid 0
>>>         ctime 1759260258.238143629 (2025-09-30 22:24:18)
>>>         otime 1679588253.995834574 (2023-03-23 18:17:33)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 11 key (256 ROOT_BACKREF 5) itemoff 13367 itemsize 28
>>>         root backref key dirid 256 sequence 6 name .snapshots
>>>     item 12 key (256 ROOT_REF 14428) itemoff 13341 itemsize 26
>>>         root ref key dirid 28589 sequence 2 name snapshot
>>>     item 13 key (256 ROOT_REF 14451) itemoff 13315 itemsize 26
>>>         root ref key dirid 28635 sequence 2 name snapshot
>>>     item 14 key (256 ROOT_REF 14475) itemoff 13289 itemsize 26
>>>         root ref key dirid 28683 sequence 2 name snapshot
>>>     item 15 key (256 ROOT_REF 14499) itemoff 13263 itemsize 26
>>>         root ref key dirid 28731 sequence 2 name snapshot
>>>     item 16 key (256 ROOT_REF 14523) itemoff 13237 itemsize 26
>>>         root ref key dirid 28779 sequence 2 name snapshot
>>>     item 17 key (256 ROOT_REF 14547) itemoff 13211 itemsize 26
>>>         root ref key dirid 28827 sequence 2 name snapshot
>>>     item 18 key (256 ROOT_REF 14559) itemoff 13185 itemsize 26
>>>         root ref key dirid 28851 sequence 2 name snapshot
>>>     item 19 key (256 ROOT_REF 14583) itemoff 13159 itemsize 26
>>>         root ref key dirid 28899 sequence 2 name snapshot
>>>     item 20 key (256 ROOT_REF 14607) itemoff 13133 itemsize 26
>>>         root ref key dirid 28947 sequence 2 name snapshot
>>>     item 21 key (256 ROOT_REF 14631) itemoff 13107 itemsize 26
>>>         root ref key dirid 28995 sequence 2 name snapshot
>>>     item 22 key (256 ROOT_REF 14640) itemoff 13081 itemsize 26
>>>         root ref key dirid 29013 sequence 2 name snapshot
>>>     item 23 key (256 ROOT_REF 14641) itemoff 13055 itemsize 26
>>>         root ref key dirid 29015 sequence 2 name snapshot
>>>     item 24 key (256 ROOT_REF 14642) itemoff 13029 itemsize 26
>>>         root ref key dirid 29017 sequence 2 name snapshot
>>>     item 25 key (256 ROOT_REF 14643) itemoff 13003 itemsize 26
>>>         root ref key dirid 29019 sequence 2 name snapshot
>>>     item 26 key (256 ROOT_REF 14644) itemoff 12977 itemsize 26
>>>         root ref key dirid 29021 sequence 2 name snapshot
>>>     item 27 key (256 ROOT_REF 14645) itemoff 12951 itemsize 26
>>>         root ref key dirid 29023 sequence 2 name snapshot
>>>     item 28 key (14428 ROOT_ITEM 1849346) itemoff 12512 itemsize 439
>>>         generation 2025978 root_dirid 256 bytenr 71398781173760 
>>> byte_limit 0 bytes_used 8815476736
>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025978
>>>         uuid 97cd122d-c519-8746-bdfe-13479bfefb9d
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 1849346 otransid 1849346 stransid 0 rtransid 0
>>>         ctime 1758406304.841464812 (2025-09-21 01:11:44)
>>>         otime 1758406304.840394467 (2025-09-21 01:11:44)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 29 key (14428 ROOT_BACKREF 256) itemoff 12486 itemsize 26
>>>         root backref key dirid 28589 sequence 2 name snapshot
>>>     item 30 key (14451 ROOT_ITEM 1899154) itemoff 12047 itemsize 439
>>>         generation 2025981 root_dirid 256 bytenr 71398809468928 
>>> byte_limit 0 bytes_used 6747799552
>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025981
>>>         uuid c5e2ba38-c982-8a4f-99e5-bb9ff8c5f7cd
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 1897416 otransid 1899154 stransid 0 rtransid 0
>>>         ctime 1758484800.817458724 (2025-09-21 23:00:00)
>>>         otime 1758488400.101635440 (2025-09-22 00:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 31 key (14451 ROOT_BACKREF 256) itemoff 12021 itemsize 26
>>>         root backref key dirid 28635 sequence 2 name snapshot
>>>     item 32 key (14475 ROOT_ITEM 1942246) itemoff 11582 itemsize 439
>>>         generation 2025984 root_dirid 256 bytenr 71398831259648 
>>> byte_limit 0 bytes_used 6755549184
>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025984
>>>         uuid 38b9ea09-54ef-5140-bddc-83dc884c31b3
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 1940067 otransid 1942246 stransid 0 rtransid 0
>>>         ctime 1758570000.151997903 (2025-09-22 22:40:00)
>>>         otime 1758574800.777676964 (2025-09-23 00:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 33 key (14475 ROOT_BACKREF 256) itemoff 11556 itemsize 26
>>>         root backref key dirid 28683 sequence 2 name snapshot
>>>     item 34 key (14499 ROOT_ITEM 1977241) itemoff 11117 itemsize 439
>>>         generation 2025986 root_dirid 256 bytenr 71398873153536 
>>> byte_limit 0 bytes_used 6538182656
>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025986
>>>         uuid 703f0298-831b-1e4d-b1f1-fc37ff582770
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 1977241 otransid 1977241 stransid 0 rtransid 0
>>>         ctime 1758661201.518446292 (2025-09-24 00:00:01)
>>>         otime 1758661204.161207119 (2025-09-24 00:00:04)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 35 key (14499 ROOT_BACKREF 256) itemoff 11091 itemsize 26
>>>         root backref key dirid 28731 sequence 2 name snapshot
>>>     item 36 key (14523 ROOT_ITEM 2001507) itemoff 10652 itemsize 439
>>>         generation 2025988 root_dirid 256 bytenr 71398873497600 
>>> byte_limit 0 bytes_used 6561398784
>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025988
>>>         uuid 9672abd0-7e7b-ab4f-9e0f-7183e58bb6c4
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2001507 otransid 2001507 stransid 0 rtransid 0
>>>         ctime 1758747600.712874426 (2025-09-25 00:00:00)
>>>         otime 1758747600.840460363 (2025-09-25 00:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 37 key (14523 ROOT_BACKREF 256) itemoff 10626 itemsize 26
>>>         root backref key dirid 28779 sequence 2 name snapshot
>>>     item 38 key (14547 ROOT_ITEM 2017206) itemoff 10187 itemsize 439
>>>         generation 2025990 root_dirid 256 bytenr 71398872924160 
>>> byte_limit 0 bytes_used 6570868736
>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025990
>>>         uuid 671b4b4f-fd99-e140-8535-fce0a16babfb
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2017206 otransid 2017206 stransid 0 rtransid 0
>>>         ctime 1758834001.88320469 (2025-09-26 00:00:01)
>>>         otime 1758834001.271433540 (2025-09-26 00:00:01)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 39 key (14547 ROOT_BACKREF 256) itemoff 10161 itemsize 26
>>>         root backref key dirid 28827 sequence 2 name snapshot
>>>     item 40 key (14559 ROOT_ITEM 2019735) itemoff 9722 itemsize 439
>>>         generation 2025992 root_dirid 256 bytenr 75423231606784 
>>> byte_limit 0 bytes_used 6576078848
>>>         last_snapshot 2019735 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025992
>>>         uuid 3ff52125-57ee-6140-87a5-4fb848ec4638
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2019711 otransid 2019735 stransid 0 rtransid 0
>>>         ctime 1758919659.690576855 (2025-09-26 23:47:39)
>>>         otime 1758920400.700930626 (2025-09-27 00:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 41 key (14559 ROOT_BACKREF 256) itemoff 9696 itemsize 26
>>>         root backref key dirid 28851 sequence 2 name snapshot
>>>     item 42 key (14583 ROOT_ITEM 2022097) itemoff 9257 itemsize 439
>>>         generation 2025994 root_dirid 256 bytenr 65812067024896 
>>> byte_limit 0 bytes_used 6557646848
>>>         last_snapshot 2022097 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025994
>>>         uuid 860f0635-f296-a04b-9fc5-bd85d480799c
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2022097 otransid 2022097 stransid 0 rtransid 0
>>>         ctime 1759006819.37527411 (2025-09-28 00:00:19)
>>>         otime 1759006820.861182212 (2025-09-28 00:00:20)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 43 key (14583 ROOT_BACKREF 256) itemoff 9231 itemsize 26
>>>         root backref key dirid 28899 sequence 2 name snapshot
>>>     item 44 key (14607 ROOT_ITEM 2024585) itemoff 8792 itemsize 439
>>>         generation 2025996 root_dirid 256 bytenr 227573944696832 
>>> byte_limit 0 bytes_used 6583533568
>>>         last_snapshot 2024585 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025996
>>>         uuid af3524d0-8a9a-894c-b5b7-705b10114326
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2024474 otransid 2024585 stransid 0 rtransid 0
>>>         ctime 1759089642.387498362 (2025-09-28 23:00:42)
>>>         otime 1759093200.412309185 (2025-09-29 00:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 45 key (14607 ROOT_BACKREF 256) itemoff 8766 itemsize 26
>>>         root backref key dirid 28947 sequence 2 name snapshot
>>>     item 46 key (14631 ROOT_ITEM 2025813) itemoff 8327 itemsize 439
>>>         generation 2025998 root_dirid 256 bytenr 178141351657472 
>>> byte_limit 0 bytes_used 6602981376
>>>         last_snapshot 2025813 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2025998
>>>         uuid 6f085114-93b0-d146-8997-9a41ff62f8db
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2025812 otransid 2025813 stransid 0 rtransid 0
>>>         ctime 1759177902.175859896 (2025-09-29 23:31:42)
>>>         otime 1759210423.120386595 (2025-09-30 08:33:43)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 47 key (14631 ROOT_BACKREF 256) itemoff 8301 itemsize 26
>>>         root backref key dirid 28995 sequence 2 name snapshot
>>>     item 48 key (14640 ROOT_ITEM 2025865) itemoff 7862 itemsize 439
>>>         generation 2026000 root_dirid 256 bytenr 194611771719680 
>>> byte_limit 0 bytes_used 6603735040
>>>         last_snapshot 2025865 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2026000
>>>         uuid a48f03d9-4847-654a-af1d-5e0eaf6ecd0c
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2025863 otransid 2025865 stransid 0 rtransid 0
>>>         ctime 1759237252.139079795 (2025-09-30 16:00:52)
>>>         otime 1759240800.56609287 (2025-09-30 17:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 49 key (14640 ROOT_BACKREF 256) itemoff 7836 itemsize 26
>>>         root backref key dirid 29013 sequence 2 name snapshot
>>>     item 50 key (14641 ROOT_ITEM 2025870) itemoff 7397 itemsize 439
>>>         generation 2026002 root_dirid 256 bytenr 65801342844928 
>>> byte_limit 0 bytes_used 6603292672
>>>         last_snapshot 2025870 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2026002
>>>         uuid 56e7aa4e-e166-504d-9a55-8c8b074c0fd9
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2025868 otransid 2025870 stransid 0 rtransid 0
>>>         ctime 1759241717.101400473 (2025-09-30 17:15:17)
>>>         otime 1759244400.232224532 (2025-09-30 18:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 51 key (14641 ROOT_BACKREF 256) itemoff 7371 itemsize 26
>>>         root backref key dirid 29015 sequence 2 name snapshot
>>>     item 52 key (14642 ROOT_ITEM 2025874) itemoff 6932 itemsize 439
>>>         generation 2026004 root_dirid 256 bytenr 187252895498240 
>>> byte_limit 0 bytes_used 6603341824
>>>         last_snapshot 2025874 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2026004
>>>         uuid f1761dfd-ec60-724a-9af0-dea952cc6771
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2025872 otransid 2025874 stransid 0 rtransid 0
>>>         ctime 1759244441.841998010 (2025-09-30 18:00:41)
>>>         otime 1759248000.122051877 (2025-09-30 19:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 53 key (14642 ROOT_BACKREF 256) itemoff 6906 itemsize 26
>>>         root backref key dirid 29017 sequence 2 name snapshot
>>>     item 54 key (14643 ROOT_ITEM 2025904) itemoff 6467 itemsize 439
>>>         generation 2026006 root_dirid 256 bytenr 227587769647104 
>>> byte_limit 0 bytes_used 6603587584
>>>         last_snapshot 2025904 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2026006
>>>         uuid 68fd5037-6c8a-c74e-b08f-cff4cdccf752
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2025904 otransid 2025904 stransid 0 rtransid 0
>>>         ctime 1759251597.815452999 (2025-09-30 19:59:57)
>>>         otime 1759251600.884521740 (2025-09-30 20:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 55 key (14643 ROOT_BACKREF 256) itemoff 6441 itemsize 26
>>>         root backref key dirid 29019 sequence 2 name snapshot
>>>     item 56 key (14644 ROOT_ITEM 2025922) itemoff 6002 itemsize 439
>>>         generation 2026008 root_dirid 256 bytenr 227594191208448 
>>> byte_limit 0 bytes_used 6603653120
>>>         last_snapshot 2025922 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2026008
>>>         uuid 07ccd2ae-f51d-2c46-82df-05e16964b2e5
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2025921 otransid 2025922 stransid 0 rtransid 0
>>>         ctime 1759254975.20573006 (2025-09-30 20:56:15)
>>>         otime 1759255200.196692319 (2025-09-30 21:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 57 key (14644 ROOT_BACKREF 256) itemoff 5976 itemsize 26
>>>         root backref key dirid 29021 sequence 2 name snapshot
>>>     item 58 key (14645 ROOT_ITEM 2025951) itemoff 5537 itemsize 439
>>>         generation 2026010 root_dirid 256 bytenr 193309358817280 
>>> byte_limit 0 bytes_used 6603964416
>>>         last_snapshot 2025951 flags 0x1(RDONLY) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 3 generation_v2 2026010
>>>         uuid b46ce558-2c39-3944-91d4-b2b229f04a16
>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 2025950 otransid 2025951 stransid 0 rtransid 0
>>>         ctime 1759257514.988583086 (2025-09-30 21:38:34)
>>>         otime 1759258800.482340435 (2025-09-30 22:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>     item 59 key (14645 ROOT_BACKREF 256) itemoff 5511 itemsize 26
>>>         root backref key dirid 29023 sequence 2 name snapshot
>>>     item 60 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 5072 itemsize 439
>>>         generation 2018764 root_dirid 256 bytenr 71399055278080 
>>> byte_limit 0 bytes_used 16384
>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>         level 0 generation_v2 2018764
>>>         uuid 00000000-0000-0000-0000-000000000000
>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>         otime 0.0 (1970-01-01 02:00:00)
>>>         stime 0.0 (1970-01-01 02:00:00)
>>>         rtime 0.0 (1970-01-01 02:00:00)
>>
>> Qu Wenruo kirjoitti 5.10.2025 klo 1.07:
>>>
>>>
>>> 在 2025/10/5 08:23, Henri Hyyryläinen 写道:
>>>>> One extra thing, at the time of the initial "btrfs check 
>>>>> --repair", there is no running dev-replace/balance or whatever 
>>>>> running?
>>>>>
>>>>> Although those operations will be automatically paused by unmount,
>>>>> btrfs check is not going to be able to handle some of those paused 
>>>>> operations. 
>>>>
>>>> Is there a way for me to check that without mounting the 
>>>> filesystem? As far as I could find, none of the balance / scrub 
>>>> commands allow working on an unmounted filesystem. So I couldn't 
>>>> find out if I had any in canceled state.
>>>
>>> # btrfs ins dump-tree -t root <device>
>>>>
>>>> Though, I'm pretty sure I let the last scrub and balance operation 
>>>> I tried to fully complete before starting using the check and 
>>>> repair commands. But I'm not absolutely certain that I didn't try 
>>>> one of those a last time and didn't let it fully complete.
>>>>
>>>> I'll start that "btrfs check --readonly" command now. And I'll 
>>>> report back once it is done (hopefully by the morning in my timezone).
>>>>
>>>> - Henri Hyyryläinen
>>>>
>>>> Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
>>>>>
>>>>>
>>>>> 在 2025/10/5 07:14, Qu Wenruo 写道:
>>>>>>
>>>>>>
>>>>>> 在 2025/10/5 04:13, Henri Hyyryläinen 写道:
>>>>>>> Hello again.
>>>>>>>
>>>>>>> It took over 3 days, but the btrfs check --repair has now 
>>>>>>> completed seemingly successfully. I mostly saw output about the 
>>>>>>> file being placed in lost+found and and directory size being 
>>>>>>> corrected. However, there were some messages about mismatch of 
>>>>>>> used bytes.
>>>>>>>
>>>>>>> Unfortunately it seems like the situation has gotten worse since 
>>>>>>> the repair, because now I cannot mount the filesystem at all. 
>>>>>>> Instead I get an error like this:
>>>>>>>
>>>>>>>> BTRFS error (device sdc): dev extent physical offset 
>>>>>>>> 19977638903808 on devid 4 doesn't have corresponding chunk
>>>>>>>> BTRFS error (device sdc): failed to verify dev extents against 
>>>>>>>> chunks: -117
>>>>>>>> BTRFS error (device sdc): open_ctree failed: -117
>>>>>>> Even if I remove that one problematic device physically from my 
>>>>>>> computer, the filesystem still refuses to mount with the same 
>>>>>>> error. Maybe the problems with the device replace are again 
>>>>>>> showing up with the actual size of the hard drive not being used 
>>>>>>> correctly? I cannot try to remove the device slack as I cannot 
>>>>>>> mount the filesystem.
>>>>>>
>>>>>> Nope, this is a different problem, and not related to dev replace.
>>>>>>
>>>>>> Unfortunately btrfs check has not implemented any repair for that.
>>>>>>
>>>>>> Overall if the dev extent is found but not corresponding chunk, 
>>>>>> it should still be fine but some space unavailable.
>>>>>>
>>>>>> But the kernel is overly cautious on chunk tree, as it's a very 
>>>>>> important and basic functionality.
>>>>>>
>>>>>> Please provide the full "btrfs check --readonly" output so that 
>>>>>> we can evaluate and add the missing repair functionality.
>>>>>
>>>>> One extra thing, at the time of the initial "btrfs check 
>>>>> --repair", there is no running dev-replace/balance or whatever 
>>>>> running?
>>>>>
>>>>> Although those operations will be automatically paused by unmount,
>>>>> btrfs check is not going to be able to handle some of those paused 
>>>>> operations.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>>
>>>>>>> I did try to run a repair again, and this time I got a bunch of 
>>>>>>> messages like:
>>>>>>>
>>>>>>>> repair deleting extent record: key [65795546775552,169,0]
>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>> parent 65811674234880 root 65811674234880
>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>> parent 0 root 14499
>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>> parent 65791012274176 root 65791012274176
>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>> parent 65806385807360 root 65806385807360
>>>>>>>> Repaired extent references for 65795546775552
>>>>>>>> ref mismatch on [65795548233728 16384] extent item 5, found 4
>>>>>>
>>>>>> That's fixing some backref mismatch, which you can ignore unless 
>>>>>> "btrfs check --reaonly" later reports new problems.
>>>>>>
>>>>>>> But the filesystem still refuses to mount with the exact same 
>>>>>>> error. I did not let the repair run entirely as it would have 
>>>>>>> likely taken another 3 days. What should I do? This time I'm not 
>>>>>>> finding any good information on what to do. For now, I've 
>>>>>>> started the repair again, but it doesn't exactly sound like it 
>>>>>>> is even fixing anything now. Still, I'll let it continue. The 
>>>>>>> output so far is:
>>>>>>>
>>>>>>>> [1/8] checking log skipped (none written)
>>>>>>>> [2/8] checking root items
>>>>>>>> Fixed 0 roots.
>>>>>>>> [3/8] checking extents
>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>> relative chunk.
>>>>>>>> super bytes used 49454738989056 mismatches actual used 
>>>>>>>> 49454738923520
>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>> relative chunk.
>>>>>>>> super bytes used 49454739005440 mismatches actual used 
>>>>>>>> 49454738956288
>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>> relative chunk.
>>>>>>>> super bytes used 49454739021824 mismatches actual used 
>>>>>>>> 49454738972672
>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>> relative chunk.
>>>>>>>> super bytes used 49454739005440 mismatches actual used 
>>>>>>>> 49454738972672
>>>>>>>
>>>>>>>
>>>>>>> If I was able to somehow remove that one logically corrupt devid 
>>>>>>> from the filesystem, or somehow correct the size, that would 
>>>>>>> hopefully allow me to rebuild from the raid10 data then, but I 
>>>>>>> can't do those with the unmountable filesystem.
>>>>>>>
>>>>>>>
>>>>>>> - Henri Hyyryläinen
>>>>>>>
>>>>>>> Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>>>>>>>>
>>>>>>>>
>>>>>>>> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I hope this is the right place to ask about a filesystem 
>>>>>>>>> problem. Really shortly put, I have a file that both exists 
>>>>>>>>> and doesn't and prevents the containing directory from being 
>>>>>>>>> deleted. No matter what variant of rm and inode based deletion 
>>>>>>>>> I try I get an error about the file not existing, and I also 
>>>>>>>>> cannot try to read the file, but if I try to delete the 
>>>>>>>>> directory I get an error that it is not empty (so the file 
>>>>>>>>> kind of exists). Trying to ls the directory also gives a file 
>>>>>>>>> doesn't exist error.
>>>>>>>>>
>>>>>>>>> Here's what btrfs check found, which I hope does better in 
>>>>>>>>> illustrating the problem:
>>>>>>>>>
>>>>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>
>>>>>>>>> I've tried everything I've found suggested including a full 
>>>>>>>>> scrub, balance with -dusage=75 -musage=75, resetting file 
>>>>>>>>> attributes, deleting through the find command, and even some 
>>>>>>>>> repair mount flags that don't seem to exist for btrfs.
>>>>>>>>
>>>>>>>> The fs is corrupted, thus none of those will help.
>>>>>>>> I'm more interested in how the corruption happened.
>>>>>>>>
>>>>>>>> Did you use some tools other than btrfs kernel module and 
>>>>>>>> btrfs- progs?
>>>>>>>> Like ntfs2btrfs or winbtrfs?
>>>>>>>>
>>>>>>>> IIRC certain versions have some bugs related to extent tree, 
>>>>>>>> but should not cause this problem.
>>>>>>>>
>>>>>>>>
>>>>>>>> The other possibility is hardware memory bitflip, which is more 
>>>>>>>> common than you thought (almostly one report per month)
>>>>>>>>
>>>>>>>> In that case, a full memtest is always recommended, or you will 
>>>>>>>> hit all kinds of weird corruptions in the future anyway.
>>>>>>>>
>>>>>>>>
>>>>>>>> With a full memtest proving the memory hardware is fine, then 
>>>>>>>> "btrfs check --repair" should be able to fix it.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>
>>>>>>>>> What I haven't tried is a full rebalance with no filters, but 
>>>>>>>>> I did not try that yet as it would take quite a long time and 
>>>>>>>>> if it only moves data blocks around without recomputing 
>>>>>>>>> directory items, it doesn't seem like the right tool to fix my 
>>>>>>>>> problem. So I'm pretty much stuck and to me it seems like my 
>>>>>>>>> only option is to run btrfs check with the repair flag, but as 
>>>>>>>>> that has big warnings on it I thought I would try asking here 
>>>>>>>>> first (sorry if this is not the right experts group to ask). 
>>>>>>>>> So is there still something I can try or am I finally 
>>>>>>>>> "allowed" to use the repair command? Here's the full output I 
>>>>>>>>> got from btrfs check:
>>>>>>>>>
>>>>>>>>>> Opening filesystem to check...
>>>>>>>>>> Checking filesystem on /dev/sdc
>>>>>>>>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>>>>>>>> [1/8] checking log skipped (none written)
>>>>>>>>>> [2/8] checking root items
>>>>>>>>>> [3/8] checking extents
>>>>>>>>>> [4/8] checking free space tree
>>>>>>>>>> We have a space info key for a block group that doesn't exist
>>>>>>>>>> [5/8] checking fs roots
>>>>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>> item
>>>>>>>>>> ERROR: errors found in fs roots
>>>>>>>>>> found 49400296812544 bytes used, error(s) found
>>>>>>>>>> total csum bytes: 48179330432
>>>>>>>>>> total tree bytes: 65067483136
>>>>>>>>>> total fs tree bytes: 12107431936
>>>>>>>>>> total extent tree bytes: 3194437632
>>>>>>>>>> btree space waste bytes: 4558984171
>>>>>>>>>> file data blocks allocated: 76487982252032
>>>>>>>>>>  referenced 60030799097856
>>>>>>>>>
>>>>>>>>> So hopefully if I'm reading things right, running a repair 
>>>>>>>>> would delete just that one file and directory (which itself is 
>>>>>>>>> a backup so I will not miss that file at all)?
>>>>>>>>>
>>>>>>>>> I do not have enough disk space to copy off the entire 
>>>>>>>>> filesystem and rebuild from scratch, without doing something 
>>>>>>>>> like rebalancing all data from raid10 to single and then 
>>>>>>>>> removing half the disks, but I assume that would take at least 
>>>>>>>>> 4 weeks to process (as I just replaced a disk which took like 
>>>>>>>>> a week).
>>>>>>>>>
>>>>>>>>> As to what originally caused the corruption, I think it was 
>>>>>>>>> probably faulty RAM, because up to to like 3 weeks ago I had 
>>>>>>>>> one really bad RAM stick in my computer where a certain memory 
>>>>>>>>> region always had incorrectly reading bytes. I had seen 
>>>>>>>>> intermittent quite high csum errors in monthly scrubs pretty 
>>>>>>>>> randomly, which thankfully could almost always be corrected so 
>>>>>>>>> I didn't have any major problems even though I had like 
>>>>>>>>> totally broken RAM in my computer for who knows how long. So 
>>>>>>>>> btrfs was able to protect my data quite impressively from bad 
>>>>>>>>> RAM.
>>>>>>>>>
>>>>>>>>> Sorry for getting a bit sidetracked there, but what should I 
>>>>>>>>> do in this situation?
>>>>>>>>>
>>>>>>>>> - Henri Hyyryläinen
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>
>

Re: How to remove an unremovable file and directory?

[Qu Wenruo]

在 2025/10/5 18:50, Henri Hyyryläinen 写道:
> Hello,
> 
> The check has completed and, probably thanks to the second repair, it 
> did not find much. Here's the full output:
> 
>> [1/8] checking log skipped (none written)
>> [2/8] checking root items
>> [3/8] checking extents
>> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
>> chunk.

Great news, we only need to delete that item.


Try btrfs-corrupt-block (I know this sounds dangerous) command, which 
may not be included by a lot of distros, thus recommended to build a 
btrfs-progs and run it (without installing)

# ./btrfs-corrupt-block -d 4,204,19977638903808 -r 4 <device>

If it worked, there should be no message output. Otherwise something 
went wrong.

Normally even if the command failed, it shouldn't cause extra damages 
since everything else in your fs is still fine.

Thanks,
Qu

>> [4/8] checking free space cache
>> [5/8] checking fs roots
>> [6/8] checking only csums items (without verifying data)
>> [7/8] checking root refs
>> [8/8] checking quota groups skipped (not enabled on this FS)
>> Opening filesystem to check...
>> Checking filesystem on /dev/sdb
>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>> found 49454062784512 bytes used, no error found
>> total csum bytes: 48225005724
>> total tree bytes: 72333164544
>> total fs tree bytes: 19189743616
>> total extent tree bytes: 3319316480
>> btree space waste bytes: 5998261217
>> file data blocks allocated: 119737771229184
>>  referenced 72125752061952
> 
> 
> - Henri Hyyryläinen
> 
> Qu Wenruo kirjoitti 5.10.2025 klo 1.41:
>>
>>
>> 在 2025/10/5 09:06, Henri Hyyryläinen 写道:
>>> Thanks for the command.
>>>
>>> I tried reading through the data but I could not determine if there 
>>> was any information related to scrub or balance status.
>>
>> It's fine. No persistent item for balance/dev-replace. So it's not 
>> involved.
>>
>> Waiting for the readonly check result.
>>
>> And meanwhile, please prepare an environment to compile btrfs-progs.
>>
>> If there is only one stray dev extent, I'll create a simple dirty fix 
>> so that you can mount the fs again.
>>
>> Thanks,
>> Qu
>>
>>>
>>> Here's the full output I got:
>>>
>>>> btrfs-progs v6.16.1
>>>> root tree
>>>> leaf 227587769614336 items 61 free space 3547 generation 2035509 
>>>> owner ROOT_TREE
>>>> leaf 227587769614336 flags 0x1(WRITTEN) backref revision 1
>>>> fs uuid 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>> chunk uuid 0e317678-73cc-4485-ab6b-987285e19681
>>>>     item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
>>>>         generation 2035509 root_dirid 0 bytenr 227587769630720 
>>>> byte_limit 0 bytes_used 3319316480
>>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2035509
>>>>         uuid 00000000-0000-0000-0000-000000000000
>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>>         otime 0.0 (1970-01-01 02:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439
>>>>         generation 2025974 root_dirid 0 bytenr 227594196631552 
>>>> byte_limit 0 bytes_used 21889024
>>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 2 generation_v2 2025974
>>>>         uuid 00000000-0000-0000-0000-000000000000
>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>>         otime 0.0 (1970-01-01 02:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17
>>>>         index 0 namelen 7 name: default
>>>>     item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439
>>>>         generation 2025975 root_dirid 256 bytenr 227573939142656 
>>>> byte_limit 0 bytes_used 6603997184
>>>>         last_snapshot 2025951 flags 0x0(none) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025975
>>>>         uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2025953 otransid 0 stransid 0 rtransid 0
>>>>         ctime 1759258850.523837328 (2025-09-30 22:00:50)
>>>>         otime 1679586168.0 (2023-03-23 17:42:48)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 4 key (FS_TREE ROOT_REF 256) itemoff 14921 itemsize 28
>>>>         root ref key dirid 256 sequence 6 name .snapshots
>>>>     item 5 key (ROOT_TREE_DIR INODE_ITEM 0) itemoff 14761 itemsize 160
>>>>         generation 3 transid 0 size 0 nbytes 16384
>>>>         block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>>>>         sequence 0 flags 0x0(none)
>>>>         atime 1679586169.0 (2023-03-23 17:42:49)
>>>>         ctime 1679586169.0 (2023-03-23 17:42:49)
>>>>         mtime 1679586169.0 (2023-03-23 17:42:49)
>>>>         otime 1679586169.0 (2023-03-23 17:42:49)
>>>>     item 6 key (ROOT_TREE_DIR INODE_REF 6) itemoff 14749 itemsize 12
>>>>         index 0 namelen 2 name: ..
>>>>     item 7 key (ROOT_TREE_DIR DIR_ITEM 2378154706) itemoff 14712 
>>>> itemsize 37
>>>>         location key (FS_TREE ROOT_ITEM 18446744073709551615) type DIR
>>>>         transid 0 data_len 0 name_len 7
>>>>         name: default
>>>>     item 8 key (CSUM_TREE ROOT_ITEM 0) itemoff 14273 itemsize 439
>>>>         generation 2025954 root_dirid 0 bytenr 227588044062720 
>>>> byte_limit 0 bytes_used 49791238144
>>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025954
>>>>         uuid 00000000-0000-0000-0000-000000000000
>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>>         otime 0.0 (1970-01-01 02:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 9 key (UUID_TREE ROOT_ITEM 0) itemoff 13834 itemsize 439
>>>>         generation 2025954 root_dirid 0 bytenr 227587802447872 
>>>> byte_limit 0 bytes_used 16384
>>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 0 generation_v2 2025954
>>>>         uuid 00000000-0000-0000-0000-000000000000
>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>>         otime 0.0 (1970-01-01 02:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 10 key (256 ROOT_ITEM 0) itemoff 13395 itemsize 439
>>>>         generation 2025954 root_dirid 256 bytenr 227587802316800 
>>>> byte_limit 0 bytes_used 49152
>>>>         last_snapshot 2015386 flags 0x0(none) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 1 generation_v2 2025954
>>>>         uuid 56f85f76-1eb8-1245-a35b-8a9307d5c68f
>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2025954 otransid 31 stransid 0 rtransid 0
>>>>         ctime 1759260258.238143629 (2025-09-30 22:24:18)
>>>>         otime 1679588253.995834574 (2023-03-23 18:17:33)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 11 key (256 ROOT_BACKREF 5) itemoff 13367 itemsize 28
>>>>         root backref key dirid 256 sequence 6 name .snapshots
>>>>     item 12 key (256 ROOT_REF 14428) itemoff 13341 itemsize 26
>>>>         root ref key dirid 28589 sequence 2 name snapshot
>>>>     item 13 key (256 ROOT_REF 14451) itemoff 13315 itemsize 26
>>>>         root ref key dirid 28635 sequence 2 name snapshot
>>>>     item 14 key (256 ROOT_REF 14475) itemoff 13289 itemsize 26
>>>>         root ref key dirid 28683 sequence 2 name snapshot
>>>>     item 15 key (256 ROOT_REF 14499) itemoff 13263 itemsize 26
>>>>         root ref key dirid 28731 sequence 2 name snapshot
>>>>     item 16 key (256 ROOT_REF 14523) itemoff 13237 itemsize 26
>>>>         root ref key dirid 28779 sequence 2 name snapshot
>>>>     item 17 key (256 ROOT_REF 14547) itemoff 13211 itemsize 26
>>>>         root ref key dirid 28827 sequence 2 name snapshot
>>>>     item 18 key (256 ROOT_REF 14559) itemoff 13185 itemsize 26
>>>>         root ref key dirid 28851 sequence 2 name snapshot
>>>>     item 19 key (256 ROOT_REF 14583) itemoff 13159 itemsize 26
>>>>         root ref key dirid 28899 sequence 2 name snapshot
>>>>     item 20 key (256 ROOT_REF 14607) itemoff 13133 itemsize 26
>>>>         root ref key dirid 28947 sequence 2 name snapshot
>>>>     item 21 key (256 ROOT_REF 14631) itemoff 13107 itemsize 26
>>>>         root ref key dirid 28995 sequence 2 name snapshot
>>>>     item 22 key (256 ROOT_REF 14640) itemoff 13081 itemsize 26
>>>>         root ref key dirid 29013 sequence 2 name snapshot
>>>>     item 23 key (256 ROOT_REF 14641) itemoff 13055 itemsize 26
>>>>         root ref key dirid 29015 sequence 2 name snapshot
>>>>     item 24 key (256 ROOT_REF 14642) itemoff 13029 itemsize 26
>>>>         root ref key dirid 29017 sequence 2 name snapshot
>>>>     item 25 key (256 ROOT_REF 14643) itemoff 13003 itemsize 26
>>>>         root ref key dirid 29019 sequence 2 name snapshot
>>>>     item 26 key (256 ROOT_REF 14644) itemoff 12977 itemsize 26
>>>>         root ref key dirid 29021 sequence 2 name snapshot
>>>>     item 27 key (256 ROOT_REF 14645) itemoff 12951 itemsize 26
>>>>         root ref key dirid 29023 sequence 2 name snapshot
>>>>     item 28 key (14428 ROOT_ITEM 1849346) itemoff 12512 itemsize 439
>>>>         generation 2025978 root_dirid 256 bytenr 71398781173760 
>>>> byte_limit 0 bytes_used 8815476736
>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025978
>>>>         uuid 97cd122d-c519-8746-bdfe-13479bfefb9d
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 1849346 otransid 1849346 stransid 0 rtransid 0
>>>>         ctime 1758406304.841464812 (2025-09-21 01:11:44)
>>>>         otime 1758406304.840394467 (2025-09-21 01:11:44)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 29 key (14428 ROOT_BACKREF 256) itemoff 12486 itemsize 26
>>>>         root backref key dirid 28589 sequence 2 name snapshot
>>>>     item 30 key (14451 ROOT_ITEM 1899154) itemoff 12047 itemsize 439
>>>>         generation 2025981 root_dirid 256 bytenr 71398809468928 
>>>> byte_limit 0 bytes_used 6747799552
>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025981
>>>>         uuid c5e2ba38-c982-8a4f-99e5-bb9ff8c5f7cd
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 1897416 otransid 1899154 stransid 0 rtransid 0
>>>>         ctime 1758484800.817458724 (2025-09-21 23:00:00)
>>>>         otime 1758488400.101635440 (2025-09-22 00:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 31 key (14451 ROOT_BACKREF 256) itemoff 12021 itemsize 26
>>>>         root backref key dirid 28635 sequence 2 name snapshot
>>>>     item 32 key (14475 ROOT_ITEM 1942246) itemoff 11582 itemsize 439
>>>>         generation 2025984 root_dirid 256 bytenr 71398831259648 
>>>> byte_limit 0 bytes_used 6755549184
>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025984
>>>>         uuid 38b9ea09-54ef-5140-bddc-83dc884c31b3
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 1940067 otransid 1942246 stransid 0 rtransid 0
>>>>         ctime 1758570000.151997903 (2025-09-22 22:40:00)
>>>>         otime 1758574800.777676964 (2025-09-23 00:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 33 key (14475 ROOT_BACKREF 256) itemoff 11556 itemsize 26
>>>>         root backref key dirid 28683 sequence 2 name snapshot
>>>>     item 34 key (14499 ROOT_ITEM 1977241) itemoff 11117 itemsize 439
>>>>         generation 2025986 root_dirid 256 bytenr 71398873153536 
>>>> byte_limit 0 bytes_used 6538182656
>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025986
>>>>         uuid 703f0298-831b-1e4d-b1f1-fc37ff582770
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 1977241 otransid 1977241 stransid 0 rtransid 0
>>>>         ctime 1758661201.518446292 (2025-09-24 00:00:01)
>>>>         otime 1758661204.161207119 (2025-09-24 00:00:04)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 35 key (14499 ROOT_BACKREF 256) itemoff 11091 itemsize 26
>>>>         root backref key dirid 28731 sequence 2 name snapshot
>>>>     item 36 key (14523 ROOT_ITEM 2001507) itemoff 10652 itemsize 439
>>>>         generation 2025988 root_dirid 256 bytenr 71398873497600 
>>>> byte_limit 0 bytes_used 6561398784
>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025988
>>>>         uuid 9672abd0-7e7b-ab4f-9e0f-7183e58bb6c4
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2001507 otransid 2001507 stransid 0 rtransid 0
>>>>         ctime 1758747600.712874426 (2025-09-25 00:00:00)
>>>>         otime 1758747600.840460363 (2025-09-25 00:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 37 key (14523 ROOT_BACKREF 256) itemoff 10626 itemsize 26
>>>>         root backref key dirid 28779 sequence 2 name snapshot
>>>>     item 38 key (14547 ROOT_ITEM 2017206) itemoff 10187 itemsize 439
>>>>         generation 2025990 root_dirid 256 bytenr 71398872924160 
>>>> byte_limit 0 bytes_used 6570868736
>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025990
>>>>         uuid 671b4b4f-fd99-e140-8535-fce0a16babfb
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2017206 otransid 2017206 stransid 0 rtransid 0
>>>>         ctime 1758834001.88320469 (2025-09-26 00:00:01)
>>>>         otime 1758834001.271433540 (2025-09-26 00:00:01)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 39 key (14547 ROOT_BACKREF 256) itemoff 10161 itemsize 26
>>>>         root backref key dirid 28827 sequence 2 name snapshot
>>>>     item 40 key (14559 ROOT_ITEM 2019735) itemoff 9722 itemsize 439
>>>>         generation 2025992 root_dirid 256 bytenr 75423231606784 
>>>> byte_limit 0 bytes_used 6576078848
>>>>         last_snapshot 2019735 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025992
>>>>         uuid 3ff52125-57ee-6140-87a5-4fb848ec4638
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2019711 otransid 2019735 stransid 0 rtransid 0
>>>>         ctime 1758919659.690576855 (2025-09-26 23:47:39)
>>>>         otime 1758920400.700930626 (2025-09-27 00:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 41 key (14559 ROOT_BACKREF 256) itemoff 9696 itemsize 26
>>>>         root backref key dirid 28851 sequence 2 name snapshot
>>>>     item 42 key (14583 ROOT_ITEM 2022097) itemoff 9257 itemsize 439
>>>>         generation 2025994 root_dirid 256 bytenr 65812067024896 
>>>> byte_limit 0 bytes_used 6557646848
>>>>         last_snapshot 2022097 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025994
>>>>         uuid 860f0635-f296-a04b-9fc5-bd85d480799c
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2022097 otransid 2022097 stransid 0 rtransid 0
>>>>         ctime 1759006819.37527411 (2025-09-28 00:00:19)
>>>>         otime 1759006820.861182212 (2025-09-28 00:00:20)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 43 key (14583 ROOT_BACKREF 256) itemoff 9231 itemsize 26
>>>>         root backref key dirid 28899 sequence 2 name snapshot
>>>>     item 44 key (14607 ROOT_ITEM 2024585) itemoff 8792 itemsize 439
>>>>         generation 2025996 root_dirid 256 bytenr 227573944696832 
>>>> byte_limit 0 bytes_used 6583533568
>>>>         last_snapshot 2024585 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025996
>>>>         uuid af3524d0-8a9a-894c-b5b7-705b10114326
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2024474 otransid 2024585 stransid 0 rtransid 0
>>>>         ctime 1759089642.387498362 (2025-09-28 23:00:42)
>>>>         otime 1759093200.412309185 (2025-09-29 00:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 45 key (14607 ROOT_BACKREF 256) itemoff 8766 itemsize 26
>>>>         root backref key dirid 28947 sequence 2 name snapshot
>>>>     item 46 key (14631 ROOT_ITEM 2025813) itemoff 8327 itemsize 439
>>>>         generation 2025998 root_dirid 256 bytenr 178141351657472 
>>>> byte_limit 0 bytes_used 6602981376
>>>>         last_snapshot 2025813 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2025998
>>>>         uuid 6f085114-93b0-d146-8997-9a41ff62f8db
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2025812 otransid 2025813 stransid 0 rtransid 0
>>>>         ctime 1759177902.175859896 (2025-09-29 23:31:42)
>>>>         otime 1759210423.120386595 (2025-09-30 08:33:43)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 47 key (14631 ROOT_BACKREF 256) itemoff 8301 itemsize 26
>>>>         root backref key dirid 28995 sequence 2 name snapshot
>>>>     item 48 key (14640 ROOT_ITEM 2025865) itemoff 7862 itemsize 439
>>>>         generation 2026000 root_dirid 256 bytenr 194611771719680 
>>>> byte_limit 0 bytes_used 6603735040
>>>>         last_snapshot 2025865 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2026000
>>>>         uuid a48f03d9-4847-654a-af1d-5e0eaf6ecd0c
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2025863 otransid 2025865 stransid 0 rtransid 0
>>>>         ctime 1759237252.139079795 (2025-09-30 16:00:52)
>>>>         otime 1759240800.56609287 (2025-09-30 17:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 49 key (14640 ROOT_BACKREF 256) itemoff 7836 itemsize 26
>>>>         root backref key dirid 29013 sequence 2 name snapshot
>>>>     item 50 key (14641 ROOT_ITEM 2025870) itemoff 7397 itemsize 439
>>>>         generation 2026002 root_dirid 256 bytenr 65801342844928 
>>>> byte_limit 0 bytes_used 6603292672
>>>>         last_snapshot 2025870 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2026002
>>>>         uuid 56e7aa4e-e166-504d-9a55-8c8b074c0fd9
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2025868 otransid 2025870 stransid 0 rtransid 0
>>>>         ctime 1759241717.101400473 (2025-09-30 17:15:17)
>>>>         otime 1759244400.232224532 (2025-09-30 18:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 51 key (14641 ROOT_BACKREF 256) itemoff 7371 itemsize 26
>>>>         root backref key dirid 29015 sequence 2 name snapshot
>>>>     item 52 key (14642 ROOT_ITEM 2025874) itemoff 6932 itemsize 439
>>>>         generation 2026004 root_dirid 256 bytenr 187252895498240 
>>>> byte_limit 0 bytes_used 6603341824
>>>>         last_snapshot 2025874 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2026004
>>>>         uuid f1761dfd-ec60-724a-9af0-dea952cc6771
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2025872 otransid 2025874 stransid 0 rtransid 0
>>>>         ctime 1759244441.841998010 (2025-09-30 18:00:41)
>>>>         otime 1759248000.122051877 (2025-09-30 19:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 53 key (14642 ROOT_BACKREF 256) itemoff 6906 itemsize 26
>>>>         root backref key dirid 29017 sequence 2 name snapshot
>>>>     item 54 key (14643 ROOT_ITEM 2025904) itemoff 6467 itemsize 439
>>>>         generation 2026006 root_dirid 256 bytenr 227587769647104 
>>>> byte_limit 0 bytes_used 6603587584
>>>>         last_snapshot 2025904 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2026006
>>>>         uuid 68fd5037-6c8a-c74e-b08f-cff4cdccf752
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2025904 otransid 2025904 stransid 0 rtransid 0
>>>>         ctime 1759251597.815452999 (2025-09-30 19:59:57)
>>>>         otime 1759251600.884521740 (2025-09-30 20:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 55 key (14643 ROOT_BACKREF 256) itemoff 6441 itemsize 26
>>>>         root backref key dirid 29019 sequence 2 name snapshot
>>>>     item 56 key (14644 ROOT_ITEM 2025922) itemoff 6002 itemsize 439
>>>>         generation 2026008 root_dirid 256 bytenr 227594191208448 
>>>> byte_limit 0 bytes_used 6603653120
>>>>         last_snapshot 2025922 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2026008
>>>>         uuid 07ccd2ae-f51d-2c46-82df-05e16964b2e5
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2025921 otransid 2025922 stransid 0 rtransid 0
>>>>         ctime 1759254975.20573006 (2025-09-30 20:56:15)
>>>>         otime 1759255200.196692319 (2025-09-30 21:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 57 key (14644 ROOT_BACKREF 256) itemoff 5976 itemsize 26
>>>>         root backref key dirid 29021 sequence 2 name snapshot
>>>>     item 58 key (14645 ROOT_ITEM 2025951) itemoff 5537 itemsize 439
>>>>         generation 2026010 root_dirid 256 bytenr 193309358817280 
>>>> byte_limit 0 bytes_used 6603964416
>>>>         last_snapshot 2025951 flags 0x1(RDONLY) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 3 generation_v2 2026010
>>>>         uuid b46ce558-2c39-3944-91d4-b2b229f04a16
>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 2025950 otransid 2025951 stransid 0 rtransid 0
>>>>         ctime 1759257514.988583086 (2025-09-30 21:38:34)
>>>>         otime 1759258800.482340435 (2025-09-30 22:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>     item 59 key (14645 ROOT_BACKREF 256) itemoff 5511 itemsize 26
>>>>         root backref key dirid 29023 sequence 2 name snapshot
>>>>     item 60 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 5072 itemsize 439
>>>>         generation 2018764 root_dirid 256 bytenr 71399055278080 
>>>> byte_limit 0 bytes_used 16384
>>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>         level 0 generation_v2 2018764
>>>>         uuid 00000000-0000-0000-0000-000000000000
>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>>         otime 0.0 (1970-01-01 02:00:00)
>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>
>>> Qu Wenruo kirjoitti 5.10.2025 klo 1.07:
>>>>
>>>>
>>>> 在 2025/10/5 08:23, Henri Hyyryläinen 写道:
>>>>>> One extra thing, at the time of the initial "btrfs check -- 
>>>>>> repair", there is no running dev-replace/balance or whatever running?
>>>>>>
>>>>>> Although those operations will be automatically paused by unmount,
>>>>>> btrfs check is not going to be able to handle some of those paused 
>>>>>> operations. 
>>>>>
>>>>> Is there a way for me to check that without mounting the 
>>>>> filesystem? As far as I could find, none of the balance / scrub 
>>>>> commands allow working on an unmounted filesystem. So I couldn't 
>>>>> find out if I had any in canceled state.
>>>>
>>>> # btrfs ins dump-tree -t root <device>
>>>>>
>>>>> Though, I'm pretty sure I let the last scrub and balance operation 
>>>>> I tried to fully complete before starting using the check and 
>>>>> repair commands. But I'm not absolutely certain that I didn't try 
>>>>> one of those a last time and didn't let it fully complete.
>>>>>
>>>>> I'll start that "btrfs check --readonly" command now. And I'll 
>>>>> report back once it is done (hopefully by the morning in my timezone).
>>>>>
>>>>> - Henri Hyyryläinen
>>>>>
>>>>> Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
>>>>>>
>>>>>>
>>>>>> 在 2025/10/5 07:14, Qu Wenruo 写道:
>>>>>>>
>>>>>>>
>>>>>>> 在 2025/10/5 04:13, Henri Hyyryläinen 写道:
>>>>>>>> Hello again.
>>>>>>>>
>>>>>>>> It took over 3 days, but the btrfs check --repair has now 
>>>>>>>> completed seemingly successfully. I mostly saw output about the 
>>>>>>>> file being placed in lost+found and and directory size being 
>>>>>>>> corrected. However, there were some messages about mismatch of 
>>>>>>>> used bytes.
>>>>>>>>
>>>>>>>> Unfortunately it seems like the situation has gotten worse since 
>>>>>>>> the repair, because now I cannot mount the filesystem at all. 
>>>>>>>> Instead I get an error like this:
>>>>>>>>
>>>>>>>>> BTRFS error (device sdc): dev extent physical offset 
>>>>>>>>> 19977638903808 on devid 4 doesn't have corresponding chunk
>>>>>>>>> BTRFS error (device sdc): failed to verify dev extents against 
>>>>>>>>> chunks: -117
>>>>>>>>> BTRFS error (device sdc): open_ctree failed: -117
>>>>>>>> Even if I remove that one problematic device physically from my 
>>>>>>>> computer, the filesystem still refuses to mount with the same 
>>>>>>>> error. Maybe the problems with the device replace are again 
>>>>>>>> showing up with the actual size of the hard drive not being used 
>>>>>>>> correctly? I cannot try to remove the device slack as I cannot 
>>>>>>>> mount the filesystem.
>>>>>>>
>>>>>>> Nope, this is a different problem, and not related to dev replace.
>>>>>>>
>>>>>>> Unfortunately btrfs check has not implemented any repair for that.
>>>>>>>
>>>>>>> Overall if the dev extent is found but not corresponding chunk, 
>>>>>>> it should still be fine but some space unavailable.
>>>>>>>
>>>>>>> But the kernel is overly cautious on chunk tree, as it's a very 
>>>>>>> important and basic functionality.
>>>>>>>
>>>>>>> Please provide the full "btrfs check --readonly" output so that 
>>>>>>> we can evaluate and add the missing repair functionality.
>>>>>>
>>>>>> One extra thing, at the time of the initial "btrfs check -- 
>>>>>> repair", there is no running dev-replace/balance or whatever running?
>>>>>>
>>>>>> Although those operations will be automatically paused by unmount,
>>>>>> btrfs check is not going to be able to handle some of those paused 
>>>>>> operations.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>>>
>>>>>>>> I did try to run a repair again, and this time I got a bunch of 
>>>>>>>> messages like:
>>>>>>>>
>>>>>>>>> repair deleting extent record: key [65795546775552,169,0]
>>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>>> parent 65811674234880 root 65811674234880
>>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>>> parent 0 root 14499
>>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>>> parent 65791012274176 root 65791012274176
>>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>>> parent 65806385807360 root 65806385807360
>>>>>>>>> Repaired extent references for 65795546775552
>>>>>>>>> ref mismatch on [65795548233728 16384] extent item 5, found 4
>>>>>>>
>>>>>>> That's fixing some backref mismatch, which you can ignore unless 
>>>>>>> "btrfs check --reaonly" later reports new problems.
>>>>>>>
>>>>>>>> But the filesystem still refuses to mount with the exact same 
>>>>>>>> error. I did not let the repair run entirely as it would have 
>>>>>>>> likely taken another 3 days. What should I do? This time I'm not 
>>>>>>>> finding any good information on what to do. For now, I've 
>>>>>>>> started the repair again, but it doesn't exactly sound like it 
>>>>>>>> is even fixing anything now. Still, I'll let it continue. The 
>>>>>>>> output so far is:
>>>>>>>>
>>>>>>>>> [1/8] checking log skipped (none written)
>>>>>>>>> [2/8] checking root items
>>>>>>>>> Fixed 0 roots.
>>>>>>>>> [3/8] checking extents
>>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>>> relative chunk.
>>>>>>>>> super bytes used 49454738989056 mismatches actual used 
>>>>>>>>> 49454738923520
>>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>>> relative chunk.
>>>>>>>>> super bytes used 49454739005440 mismatches actual used 
>>>>>>>>> 49454738956288
>>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>>> relative chunk.
>>>>>>>>> super bytes used 49454739021824 mismatches actual used 
>>>>>>>>> 49454738972672
>>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>>> relative chunk.
>>>>>>>>> super bytes used 49454739005440 mismatches actual used 
>>>>>>>>> 49454738972672
>>>>>>>>
>>>>>>>>
>>>>>>>> If I was able to somehow remove that one logically corrupt devid 
>>>>>>>> from the filesystem, or somehow correct the size, that would 
>>>>>>>> hopefully allow me to rebuild from the raid10 data then, but I 
>>>>>>>> can't do those with the unmountable filesystem.
>>>>>>>>
>>>>>>>>
>>>>>>>> - Henri Hyyryläinen
>>>>>>>>
>>>>>>>> Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ��� 2025/9/30 02:41, Henri Hyyryläinen 写道:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I hope this is the right place to ask about a filesystem 
>>>>>>>>>> problem. Really shortly put, I have a file that both exists 
>>>>>>>>>> and doesn't and prevents the containing directory from being 
>>>>>>>>>> deleted. No matter what variant of rm and inode based deletion 
>>>>>>>>>> I try I get an error about the file not existing, and I also 
>>>>>>>>>> cannot try to read the file, but if I try to delete the 
>>>>>>>>>> directory I get an error that it is not empty (so the file 
>>>>>>>>>> kind of exists). Trying to ls the directory also gives a file 
>>>>>>>>>> doesn't exist error.
>>>>>>>>>>
>>>>>>>>>> Here's what btrfs check found, which I hope does better in 
>>>>>>>>>> illustrating the problem:
>>>>>>>>>>
>>>>>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>
>>>>>>>>>> I've tried everything I've found suggested including a full 
>>>>>>>>>> scrub, balance with -dusage=75 -musage=75, resetting file 
>>>>>>>>>> attributes, deleting through the find command, and even some 
>>>>>>>>>> repair mount flags that don't seem to exist for btrfs.
>>>>>>>>>
>>>>>>>>> The fs is corrupted, thus none of those will help.
>>>>>>>>> I'm more interested in how the corruption happened.
>>>>>>>>>
>>>>>>>>> Did you use some tools other than btrfs kernel module and 
>>>>>>>>> btrfs- progs?
>>>>>>>>> Like ntfs2btrfs or winbtrfs?
>>>>>>>>>
>>>>>>>>> IIRC certain versions have some bugs related to extent tree, 
>>>>>>>>> but should not cause this problem.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The other possibility is hardware memory bitflip, which is more 
>>>>>>>>> common than you thought (almostly one report per month)
>>>>>>>>>
>>>>>>>>> In that case, a full memtest is always recommended, or you will 
>>>>>>>>> hit all kinds of weird corruptions in the future anyway.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> With a full memtest proving the memory hardware is fine, then 
>>>>>>>>> "btrfs check --repair" should be able to fix it.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> What I haven't tried is a full rebalance with no filters, but 
>>>>>>>>>> I did not try that yet as it would take quite a long time and 
>>>>>>>>>> if it only moves data blocks around without recomputing 
>>>>>>>>>> directory items, it doesn't seem like the right tool to fix my 
>>>>>>>>>> problem. So I'm pretty much stuck and to me it seems like my 
>>>>>>>>>> only option is to run btrfs check with the repair flag, but as 
>>>>>>>>>> that has big warnings on it I thought I would try asking here 
>>>>>>>>>> first (sorry if this is not the right experts group to ask). 
>>>>>>>>>> So is there still something I can try or am I finally 
>>>>>>>>>> "allowed" to use the repair command? Here's the full output I 
>>>>>>>>>> got from btrfs check:
>>>>>>>>>>
>>>>>>>>>>> Opening filesystem to check...
>>>>>>>>>>> Checking filesystem on /dev/sdc
>>>>>>>>>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>>>>>>>>> [1/8] checking log skipped (none written)
>>>>>>>>>>> [2/8] checking root items
>>>>>>>>>>> [3/8] checking extents
>>>>>>>>>>> [4/8] checking free space tree
>>>>>>>>>>> We have a space info key for a block group that doesn't exist
>>>>>>>>>>> [5/8] checking fs roots
>>>>>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir 
>>>>>>>>>>> item
>>>>>>>>>>> ERROR: errors found in fs roots
>>>>>>>>>>> found 49400296812544 bytes used, error(s) found
>>>>>>>>>>> total csum bytes: 48179330432
>>>>>>>>>>> total tree bytes: 65067483136
>>>>>>>>>>> total fs tree bytes: 12107431936
>>>>>>>>>>> total extent tree bytes: 3194437632
>>>>>>>>>>> btree space waste bytes: 4558984171
>>>>>>>>>>> file data blocks allocated: 76487982252032
>>>>>>>>>>>  referenced 60030799097856
>>>>>>>>>>
>>>>>>>>>> So hopefully if I'm reading things right, running a repair 
>>>>>>>>>> would delete just that one file and directory (which itself is 
>>>>>>>>>> a backup so I will not miss that file at all)?
>>>>>>>>>>
>>>>>>>>>> I do not have enough disk space to copy off the entire 
>>>>>>>>>> filesystem and rebuild from scratch, without doing something 
>>>>>>>>>> like rebalancing all data from raid10 to single and then 
>>>>>>>>>> removing half the disks, but I assume that would take at least 
>>>>>>>>>> 4 weeks to process (as I just replaced a disk which took like 
>>>>>>>>>> a week).
>>>>>>>>>>
>>>>>>>>>> As to what originally caused the corruption, I think it was 
>>>>>>>>>> probably faulty RAM, because up to to like 3 weeks ago I had 
>>>>>>>>>> one really bad RAM stick in my computer where a certain memory 
>>>>>>>>>> region always had incorrectly reading bytes. I had seen 
>>>>>>>>>> intermittent quite high csum errors in monthly scrubs pretty 
>>>>>>>>>> randomly, which thankfully could almost always be corrected so 
>>>>>>>>>> I didn't have any major problems even though I had like 
>>>>>>>>>> totally broken RAM in my computer for who knows how long. So 
>>>>>>>>>> btrfs was able to protect my data quite impressively from bad 
>>>>>>>>>> RAM.
>>>>>>>>>>
>>>>>>>>>> Sorry for getting a bit sidetracked there, but what should I 
>>>>>>>>>> do in this situation?
>>>>>>>>>>
>>>>>>>>>> - Henri Hyyryläinen
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>

Re: How to remove an unremovable file and directory?

[Henri Hyyryläinen]

Thank you so much. After compiling that program and running it like 
that, I can now mount my filesystem.

It took a little bit to mount due to free space cache rebuild, but as 
far as I can tell from a quick look, everything is fine and all the used 
disk sizes are correct.

- Henri Hyyryläinen

Qu Wenruo kirjoitti 5.10.2025 klo 11.37:
>
>
> 在 2025/10/5 18:50, Henri Hyyryläinen 写道:
>> Hello,
>>
>> The check has completed and, probably thanks to the second repair, it 
>> did not find much. Here's the full output:
>>
>>> [1/8] checking log skipped (none written)
>>> [2/8] checking root items
>>> [3/8] checking extents
>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>> relative chunk.
>
> Great news, we only need to delete that item.
>
>
> Try btrfs-corrupt-block (I know this sounds dangerous) command, which 
> may not be included by a lot of distros, thus recommended to build a 
> btrfs-progs and run it (without installing)
>
> # ./btrfs-corrupt-block -d 4,204,19977638903808 -r 4 <device>
>
> If it worked, there should be no message output. Otherwise something 
> went wrong.
>
> Normally even if the command failed, it shouldn't cause extra damages 
> since everything else in your fs is still fine.
>
> Thanks,
> Qu
>
>>> [4/8] checking free space cache
>>> [5/8] checking fs roots
>>> [6/8] checking only csums items (without verifying data)
>>> [7/8] checking root refs
>>> [8/8] checking quota groups skipped (not enabled on this FS)
>>> Opening filesystem to check...
>>> Checking filesystem on /dev/sdb
>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>> found 49454062784512 bytes used, no error found
>>> total csum bytes: 48225005724
>>> total tree bytes: 72333164544
>>> total fs tree bytes: 19189743616
>>> total extent tree bytes: 3319316480
>>> btree space waste bytes: 5998261217
>>> file data blocks allocated: 119737771229184
>>>  referenced 72125752061952
>>
>>
>> - Henri Hyyryläinen
>>
>> Qu Wenruo kirjoitti 5.10.2025 klo 1.41:
>>>
>>>
>>> 在 2025/10/5 09:06, Henri Hyyryläinen 写道:
>>>> Thanks for the command.
>>>>
>>>> I tried reading through the data but I could not determine if there 
>>>> was any information related to scrub or balance status.
>>>
>>> It's fine. No persistent item for balance/dev-replace. So it's not 
>>> involved.
>>>
>>> Waiting for the readonly check result.
>>>
>>> And meanwhile, please prepare an environment to compile btrfs-progs.
>>>
>>> If there is only one stray dev extent, I'll create a simple dirty 
>>> fix so that you can mount the fs again.
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>> Here's the full output I got:
>>>>
>>>>> btrfs-progs v6.16.1
>>>>> root tree
>>>>> leaf 227587769614336 items 61 free space 3547 generation 2035509 
>>>>> owner ROOT_TREE
>>>>> leaf 227587769614336 flags 0x1(WRITTEN) backref revision 1
>>>>> fs uuid 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>>> chunk uuid 0e317678-73cc-4485-ab6b-987285e19681
>>>>>     item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
>>>>>         generation 2035509 root_dirid 0 bytenr 227587769630720 
>>>>> byte_limit 0 bytes_used 3319316480
>>>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2035509
>>>>>         uuid 00000000-0000-0000-0000-000000000000
>>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>>>         otime 0.0 (1970-01-01 02:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439
>>>>>         generation 2025974 root_dirid 0 bytenr 227594196631552 
>>>>> byte_limit 0 bytes_used 21889024
>>>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 2 generation_v2 2025974
>>>>>         uuid 00000000-0000-0000-0000-000000000000
>>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>>>         otime 0.0 (1970-01-01 02:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17
>>>>>         index 0 namelen 7 name: default
>>>>>     item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439
>>>>>         generation 2025975 root_dirid 256 bytenr 227573939142656 
>>>>> byte_limit 0 bytes_used 6603997184
>>>>>         last_snapshot 2025951 flags 0x0(none) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025975
>>>>>         uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2025953 otransid 0 stransid 0 rtransid 0
>>>>>         ctime 1759258850.523837328 (2025-09-30 22:00:50)
>>>>>         otime 1679586168.0 (2023-03-23 17:42:48)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 4 key (FS_TREE ROOT_REF 256) itemoff 14921 itemsize 28
>>>>>         root ref key dirid 256 sequence 6 name .snapshots
>>>>>     item 5 key (ROOT_TREE_DIR INODE_ITEM 0) itemoff 14761 itemsize 
>>>>> 160
>>>>>         generation 3 transid 0 size 0 nbytes 16384
>>>>>         block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>>>>>         sequence 0 flags 0x0(none)
>>>>>         atime 1679586169.0 (2023-03-23 17:42:49)
>>>>>         ctime 1679586169.0 (2023-03-23 17:42:49)
>>>>>         mtime 1679586169.0 (2023-03-23 17:42:49)
>>>>>         otime 1679586169.0 (2023-03-23 17:42:49)
>>>>>     item 6 key (ROOT_TREE_DIR INODE_REF 6) itemoff 14749 itemsize 12
>>>>>         index 0 namelen 2 name: ..
>>>>>     item 7 key (ROOT_TREE_DIR DIR_ITEM 2378154706) itemoff 14712 
>>>>> itemsize 37
>>>>>         location key (FS_TREE ROOT_ITEM 18446744073709551615) type 
>>>>> DIR
>>>>>         transid 0 data_len 0 name_len 7
>>>>>         name: default
>>>>>     item 8 key (CSUM_TREE ROOT_ITEM 0) itemoff 14273 itemsize 439
>>>>>         generation 2025954 root_dirid 0 bytenr 227588044062720 
>>>>> byte_limit 0 bytes_used 49791238144
>>>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025954
>>>>>         uuid 00000000-0000-0000-0000-000000000000
>>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>>>         otime 0.0 (1970-01-01 02:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 9 key (UUID_TREE ROOT_ITEM 0) itemoff 13834 itemsize 439
>>>>>         generation 2025954 root_dirid 0 bytenr 227587802447872 
>>>>> byte_limit 0 bytes_used 16384
>>>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 0 generation_v2 2025954
>>>>>         uuid 00000000-0000-0000-0000-000000000000
>>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>>>         otime 0.0 (1970-01-01 02:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 10 key (256 ROOT_ITEM 0) itemoff 13395 itemsize 439
>>>>>         generation 2025954 root_dirid 256 bytenr 227587802316800 
>>>>> byte_limit 0 bytes_used 49152
>>>>>         last_snapshot 2015386 flags 0x0(none) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 1 generation_v2 2025954
>>>>>         uuid 56f85f76-1eb8-1245-a35b-8a9307d5c68f
>>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2025954 otransid 31 stransid 0 rtransid 0
>>>>>         ctime 1759260258.238143629 (2025-09-30 22:24:18)
>>>>>         otime 1679588253.995834574 (2023-03-23 18:17:33)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 11 key (256 ROOT_BACKREF 5) itemoff 13367 itemsize 28
>>>>>         root backref key dirid 256 sequence 6 name .snapshots
>>>>>     item 12 key (256 ROOT_REF 14428) itemoff 13341 itemsize 26
>>>>>         root ref key dirid 28589 sequence 2 name snapshot
>>>>>     item 13 key (256 ROOT_REF 14451) itemoff 13315 itemsize 26
>>>>>         root ref key dirid 28635 sequence 2 name snapshot
>>>>>     item 14 key (256 ROOT_REF 14475) itemoff 13289 itemsize 26
>>>>>         root ref key dirid 28683 sequence 2 name snapshot
>>>>>     item 15 key (256 ROOT_REF 14499) itemoff 13263 itemsize 26
>>>>>         root ref key dirid 28731 sequence 2 name snapshot
>>>>>     item 16 key (256 ROOT_REF 14523) itemoff 13237 itemsize 26
>>>>>         root ref key dirid 28779 sequence 2 name snapshot
>>>>>     item 17 key (256 ROOT_REF 14547) itemoff 13211 itemsize 26
>>>>>         root ref key dirid 28827 sequence 2 name snapshot
>>>>>     item 18 key (256 ROOT_REF 14559) itemoff 13185 itemsize 26
>>>>>         root ref key dirid 28851 sequence 2 name snapshot
>>>>>     item 19 key (256 ROOT_REF 14583) itemoff 13159 itemsize 26
>>>>>         root ref key dirid 28899 sequence 2 name snapshot
>>>>>     item 20 key (256 ROOT_REF 14607) itemoff 13133 itemsize 26
>>>>>         root ref key dirid 28947 sequence 2 name snapshot
>>>>>     item 21 key (256 ROOT_REF 14631) itemoff 13107 itemsize 26
>>>>>         root ref key dirid 28995 sequence 2 name snapshot
>>>>>     item 22 key (256 ROOT_REF 14640) itemoff 13081 itemsize 26
>>>>>         root ref key dirid 29013 sequence 2 name snapshot
>>>>>     item 23 key (256 ROOT_REF 14641) itemoff 13055 itemsize 26
>>>>>         root ref key dirid 29015 sequence 2 name snapshot
>>>>>     item 24 key (256 ROOT_REF 14642) itemoff 13029 itemsize 26
>>>>>         root ref key dirid 29017 sequence 2 name snapshot
>>>>>     item 25 key (256 ROOT_REF 14643) itemoff 13003 itemsize 26
>>>>>         root ref key dirid 29019 sequence 2 name snapshot
>>>>>     item 26 key (256 ROOT_REF 14644) itemoff 12977 itemsize 26
>>>>>         root ref key dirid 29021 sequence 2 name snapshot
>>>>>     item 27 key (256 ROOT_REF 14645) itemoff 12951 itemsize 26
>>>>>         root ref key dirid 29023 sequence 2 name snapshot
>>>>>     item 28 key (14428 ROOT_ITEM 1849346) itemoff 12512 itemsize 439
>>>>>         generation 2025978 root_dirid 256 bytenr 71398781173760 
>>>>> byte_limit 0 bytes_used 8815476736
>>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025978
>>>>>         uuid 97cd122d-c519-8746-bdfe-13479bfefb9d
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 1849346 otransid 1849346 stransid 0 rtransid 0
>>>>>         ctime 1758406304.841464812 (2025-09-21 01:11:44)
>>>>>         otime 1758406304.840394467 (2025-09-21 01:11:44)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 29 key (14428 ROOT_BACKREF 256) itemoff 12486 itemsize 26
>>>>>         root backref key dirid 28589 sequence 2 name snapshot
>>>>>     item 30 key (14451 ROOT_ITEM 1899154) itemoff 12047 itemsize 439
>>>>>         generation 2025981 root_dirid 256 bytenr 71398809468928 
>>>>> byte_limit 0 bytes_used 6747799552
>>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025981
>>>>>         uuid c5e2ba38-c982-8a4f-99e5-bb9ff8c5f7cd
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 1897416 otransid 1899154 stransid 0 rtransid 0
>>>>>         ctime 1758484800.817458724 (2025-09-21 23:00:00)
>>>>>         otime 1758488400.101635440 (2025-09-22 00:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 31 key (14451 ROOT_BACKREF 256) itemoff 12021 itemsize 26
>>>>>         root backref key dirid 28635 sequence 2 name snapshot
>>>>>     item 32 key (14475 ROOT_ITEM 1942246) itemoff 11582 itemsize 439
>>>>>         generation 2025984 root_dirid 256 bytenr 71398831259648 
>>>>> byte_limit 0 bytes_used 6755549184
>>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025984
>>>>>         uuid 38b9ea09-54ef-5140-bddc-83dc884c31b3
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 1940067 otransid 1942246 stransid 0 rtransid 0
>>>>>         ctime 1758570000.151997903 (2025-09-22 22:40:00)
>>>>>         otime 1758574800.777676964 (2025-09-23 00:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 33 key (14475 ROOT_BACKREF 256) itemoff 11556 itemsize 26
>>>>>         root backref key dirid 28683 sequence 2 name snapshot
>>>>>     item 34 key (14499 ROOT_ITEM 1977241) itemoff 11117 itemsize 439
>>>>>         generation 2025986 root_dirid 256 bytenr 71398873153536 
>>>>> byte_limit 0 bytes_used 6538182656
>>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025986
>>>>>         uuid 703f0298-831b-1e4d-b1f1-fc37ff582770
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 1977241 otransid 1977241 stransid 0 rtransid 0
>>>>>         ctime 1758661201.518446292 (2025-09-24 00:00:01)
>>>>>         otime 1758661204.161207119 (2025-09-24 00:00:04)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 35 key (14499 ROOT_BACKREF 256) itemoff 11091 itemsize 26
>>>>>         root backref key dirid 28731 sequence 2 name snapshot
>>>>>     item 36 key (14523 ROOT_ITEM 2001507) itemoff 10652 itemsize 439
>>>>>         generation 2025988 root_dirid 256 bytenr 71398873497600 
>>>>> byte_limit 0 bytes_used 6561398784
>>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025988
>>>>>         uuid 9672abd0-7e7b-ab4f-9e0f-7183e58bb6c4
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2001507 otransid 2001507 stransid 0 rtransid 0
>>>>>         ctime 1758747600.712874426 (2025-09-25 00:00:00)
>>>>>         otime 1758747600.840460363 (2025-09-25 00:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 37 key (14523 ROOT_BACKREF 256) itemoff 10626 itemsize 26
>>>>>         root backref key dirid 28779 sequence 2 name snapshot
>>>>>     item 38 key (14547 ROOT_ITEM 2017206) itemoff 10187 itemsize 439
>>>>>         generation 2025990 root_dirid 256 bytenr 71398872924160 
>>>>> byte_limit 0 bytes_used 6570868736
>>>>>         last_snapshot 2018745 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025990
>>>>>         uuid 671b4b4f-fd99-e140-8535-fce0a16babfb
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2017206 otransid 2017206 stransid 0 rtransid 0
>>>>>         ctime 1758834001.88320469 (2025-09-26 00:00:01)
>>>>>         otime 1758834001.271433540 (2025-09-26 00:00:01)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 39 key (14547 ROOT_BACKREF 256) itemoff 10161 itemsize 26
>>>>>         root backref key dirid 28827 sequence 2 name snapshot
>>>>>     item 40 key (14559 ROOT_ITEM 2019735) itemoff 9722 itemsize 439
>>>>>         generation 2025992 root_dirid 256 bytenr 75423231606784 
>>>>> byte_limit 0 bytes_used 6576078848
>>>>>         last_snapshot 2019735 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025992
>>>>>         uuid 3ff52125-57ee-6140-87a5-4fb848ec4638
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2019711 otransid 2019735 stransid 0 rtransid 0
>>>>>         ctime 1758919659.690576855 (2025-09-26 23:47:39)
>>>>>         otime 1758920400.700930626 (2025-09-27 00:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 41 key (14559 ROOT_BACKREF 256) itemoff 9696 itemsize 26
>>>>>         root backref key dirid 28851 sequence 2 name snapshot
>>>>>     item 42 key (14583 ROOT_ITEM 2022097) itemoff 9257 itemsize 439
>>>>>         generation 2025994 root_dirid 256 bytenr 65812067024896 
>>>>> byte_limit 0 bytes_used 6557646848
>>>>>         last_snapshot 2022097 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025994
>>>>>         uuid 860f0635-f296-a04b-9fc5-bd85d480799c
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2022097 otransid 2022097 stransid 0 rtransid 0
>>>>>         ctime 1759006819.37527411 (2025-09-28 00:00:19)
>>>>>         otime 1759006820.861182212 (2025-09-28 00:00:20)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 43 key (14583 ROOT_BACKREF 256) itemoff 9231 itemsize 26
>>>>>         root backref key dirid 28899 sequence 2 name snapshot
>>>>>     item 44 key (14607 ROOT_ITEM 2024585) itemoff 8792 itemsize 439
>>>>>         generation 2025996 root_dirid 256 bytenr 227573944696832 
>>>>> byte_limit 0 bytes_used 6583533568
>>>>>         last_snapshot 2024585 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025996
>>>>>         uuid af3524d0-8a9a-894c-b5b7-705b10114326
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2024474 otransid 2024585 stransid 0 rtransid 0
>>>>>         ctime 1759089642.387498362 (2025-09-28 23:00:42)
>>>>>         otime 1759093200.412309185 (2025-09-29 00:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 45 key (14607 ROOT_BACKREF 256) itemoff 8766 itemsize 26
>>>>>         root backref key dirid 28947 sequence 2 name snapshot
>>>>>     item 46 key (14631 ROOT_ITEM 2025813) itemoff 8327 itemsize 439
>>>>>         generation 2025998 root_dirid 256 bytenr 178141351657472 
>>>>> byte_limit 0 bytes_used 6602981376
>>>>>         last_snapshot 2025813 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2025998
>>>>>         uuid 6f085114-93b0-d146-8997-9a41ff62f8db
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2025812 otransid 2025813 stransid 0 rtransid 0
>>>>>         ctime 1759177902.175859896 (2025-09-29 23:31:42)
>>>>>         otime 1759210423.120386595 (2025-09-30 08:33:43)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 47 key (14631 ROOT_BACKREF 256) itemoff 8301 itemsize 26
>>>>>         root backref key dirid 28995 sequence 2 name snapshot
>>>>>     item 48 key (14640 ROOT_ITEM 2025865) itemoff 7862 itemsize 439
>>>>>         generation 2026000 root_dirid 256 bytenr 194611771719680 
>>>>> byte_limit 0 bytes_used 6603735040
>>>>>         last_snapshot 2025865 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2026000
>>>>>         uuid a48f03d9-4847-654a-af1d-5e0eaf6ecd0c
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2025863 otransid 2025865 stransid 0 rtransid 0
>>>>>         ctime 1759237252.139079795 (2025-09-30 16:00:52)
>>>>>         otime 1759240800.56609287 (2025-09-30 17:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 49 key (14640 ROOT_BACKREF 256) itemoff 7836 itemsize 26
>>>>>         root backref key dirid 29013 sequence 2 name snapshot
>>>>>     item 50 key (14641 ROOT_ITEM 2025870) itemoff 7397 itemsize 439
>>>>>         generation 2026002 root_dirid 256 bytenr 65801342844928 
>>>>> byte_limit 0 bytes_used 6603292672
>>>>>         last_snapshot 2025870 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2026002
>>>>>         uuid 56e7aa4e-e166-504d-9a55-8c8b074c0fd9
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2025868 otransid 2025870 stransid 0 rtransid 0
>>>>>         ctime 1759241717.101400473 (2025-09-30 17:15:17)
>>>>>         otime 1759244400.232224532 (2025-09-30 18:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 51 key (14641 ROOT_BACKREF 256) itemoff 7371 itemsize 26
>>>>>         root backref key dirid 29015 sequence 2 name snapshot
>>>>>     item 52 key (14642 ROOT_ITEM 2025874) itemoff 6932 itemsize 439
>>>>>         generation 2026004 root_dirid 256 bytenr 187252895498240 
>>>>> byte_limit 0 bytes_used 6603341824
>>>>>         last_snapshot 2025874 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2026004
>>>>>         uuid f1761dfd-ec60-724a-9af0-dea952cc6771
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2025872 otransid 2025874 stransid 0 rtransid 0
>>>>>         ctime 1759244441.841998010 (2025-09-30 18:00:41)
>>>>>         otime 1759248000.122051877 (2025-09-30 19:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 53 key (14642 ROOT_BACKREF 256) itemoff 6906 itemsize 26
>>>>>         root backref key dirid 29017 sequence 2 name snapshot
>>>>>     item 54 key (14643 ROOT_ITEM 2025904) itemoff 6467 itemsize 439
>>>>>         generation 2026006 root_dirid 256 bytenr 227587769647104 
>>>>> byte_limit 0 bytes_used 6603587584
>>>>>         last_snapshot 2025904 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2026006
>>>>>         uuid 68fd5037-6c8a-c74e-b08f-cff4cdccf752
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2025904 otransid 2025904 stransid 0 rtransid 0
>>>>>         ctime 1759251597.815452999 (2025-09-30 19:59:57)
>>>>>         otime 1759251600.884521740 (2025-09-30 20:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 55 key (14643 ROOT_BACKREF 256) itemoff 6441 itemsize 26
>>>>>         root backref key dirid 29019 sequence 2 name snapshot
>>>>>     item 56 key (14644 ROOT_ITEM 2025922) itemoff 6002 itemsize 439
>>>>>         generation 2026008 root_dirid 256 bytenr 227594191208448 
>>>>> byte_limit 0 bytes_used 6603653120
>>>>>         last_snapshot 2025922 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2026008
>>>>>         uuid 07ccd2ae-f51d-2c46-82df-05e16964b2e5
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2025921 otransid 2025922 stransid 0 rtransid 0
>>>>>         ctime 1759254975.20573006 (2025-09-30 20:56:15)
>>>>>         otime 1759255200.196692319 (2025-09-30 21:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 57 key (14644 ROOT_BACKREF 256) itemoff 5976 itemsize 26
>>>>>         root backref key dirid 29021 sequence 2 name snapshot
>>>>>     item 58 key (14645 ROOT_ITEM 2025951) itemoff 5537 itemsize 439
>>>>>         generation 2026010 root_dirid 256 bytenr 193309358817280 
>>>>> byte_limit 0 bytes_used 6603964416
>>>>>         last_snapshot 2025951 flags 0x1(RDONLY) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 3 generation_v2 2026010
>>>>>         uuid b46ce558-2c39-3944-91d4-b2b229f04a16
>>>>>         parent_uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 2025950 otransid 2025951 stransid 0 rtransid 0
>>>>>         ctime 1759257514.988583086 (2025-09-30 21:38:34)
>>>>>         otime 1759258800.482340435 (2025-09-30 22:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>>     item 59 key (14645 ROOT_BACKREF 256) itemoff 5511 itemsize 26
>>>>>         root backref key dirid 29023 sequence 2 name snapshot
>>>>>     item 60 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 5072 
>>>>> itemsize 439
>>>>>         generation 2018764 root_dirid 256 bytenr 71399055278080 
>>>>> byte_limit 0 bytes_used 16384
>>>>>         last_snapshot 0 flags 0x0(none) refs 1
>>>>>         drop_progress key (0 UNKNOWN.0 0) drop_level 0
>>>>>         level 0 generation_v2 2018764
>>>>>         uuid 00000000-0000-0000-0000-000000000000
>>>>>         parent_uuid 00000000-0000-0000-0000-000000000000
>>>>>         received_uuid 00000000-0000-0000-0000-000000000000
>>>>>         ctransid 0 otransid 0 stransid 0 rtransid 0
>>>>>         ctime 0.0 (1970-01-01 02:00:00)
>>>>>         otime 0.0 (1970-01-01 02:00:00)
>>>>>         stime 0.0 (1970-01-01 02:00:00)
>>>>>         rtime 0.0 (1970-01-01 02:00:00)
>>>>
>>>> Qu Wenruo kirjoitti 5.10.2025 klo 1.07:
>>>>>
>>>>>
>>>>> 在 2025/10/5 08:23, Henri Hyyryläinen 写道:
>>>>>>> One extra thing, at the time of the initial "btrfs check -- 
>>>>>>> repair", there is no running dev-replace/balance or whatever 
>>>>>>> running?
>>>>>>>
>>>>>>> Although those operations will be automatically paused by unmount,
>>>>>>> btrfs check is not going to be able to handle some of those 
>>>>>>> paused operations. 
>>>>>>
>>>>>> Is there a way for me to check that without mounting the 
>>>>>> filesystem? As far as I could find, none of the balance / scrub 
>>>>>> commands allow working on an unmounted filesystem. So I couldn't 
>>>>>> find out if I had any in canceled state.
>>>>>
>>>>> # btrfs ins dump-tree -t root <device>
>>>>>>
>>>>>> Though, I'm pretty sure I let the last scrub and balance 
>>>>>> operation I tried to fully complete before starting using the 
>>>>>> check and repair commands. But I'm not absolutely certain that I 
>>>>>> didn't try one of those a last time and didn't let it fully 
>>>>>> complete.
>>>>>>
>>>>>> I'll start that "btrfs check --readonly" command now. And I'll 
>>>>>> report back once it is done (hopefully by the morning in my 
>>>>>> timezone).
>>>>>>
>>>>>> - Henri Hyyryläinen
>>>>>>
>>>>>> Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
>>>>>>>
>>>>>>>
>>>>>>> 在 2025/10/5 07:14, Qu Wenruo 写道:
>>>>>>>>
>>>>>>>>
>>>>>>>> 在 2025/10/5 04:13, Henri Hyyryläinen 写道:
>>>>>>>>> Hello again.
>>>>>>>>>
>>>>>>>>> It took over 3 days, but the btrfs check --repair has now 
>>>>>>>>> completed seemingly successfully. I mostly saw output about 
>>>>>>>>> the file being placed in lost+found and and directory size 
>>>>>>>>> being corrected. However, there were some messages about 
>>>>>>>>> mismatch of used bytes.
>>>>>>>>>
>>>>>>>>> Unfortunately it seems like the situation has gotten worse 
>>>>>>>>> since the repair, because now I cannot mount the filesystem at 
>>>>>>>>> all. Instead I get an error like this:
>>>>>>>>>
>>>>>>>>>> BTRFS error (device sdc): dev extent physical offset 
>>>>>>>>>> 19977638903808 on devid 4 doesn't have corresponding chunk
>>>>>>>>>> BTRFS error (device sdc): failed to verify dev extents 
>>>>>>>>>> against chunks: -117
>>>>>>>>>> BTRFS error (device sdc): open_ctree failed: -117
>>>>>>>>> Even if I remove that one problematic device physically from 
>>>>>>>>> my computer, the filesystem still refuses to mount with the 
>>>>>>>>> same error. Maybe the problems with the device replace are 
>>>>>>>>> again showing up with the actual size of the hard drive not 
>>>>>>>>> being used correctly? I cannot try to remove the device slack 
>>>>>>>>> as I cannot mount the filesystem.
>>>>>>>>
>>>>>>>> Nope, this is a different problem, and not related to dev replace.
>>>>>>>>
>>>>>>>> Unfortunately btrfs check has not implemented any repair for that.
>>>>>>>>
>>>>>>>> Overall if the dev extent is found but not corresponding chunk, 
>>>>>>>> it should still be fine but some space unavailable.
>>>>>>>>
>>>>>>>> But the kernel is overly cautious on chunk tree, as it's a very 
>>>>>>>> important and basic functionality.
>>>>>>>>
>>>>>>>> Please provide the full "btrfs check --readonly" output so that 
>>>>>>>> we can evaluate and add the missing repair functionality.
>>>>>>>
>>>>>>> One extra thing, at the time of the initial "btrfs check -- 
>>>>>>> repair", there is no running dev-replace/balance or whatever 
>>>>>>> running?
>>>>>>>
>>>>>>> Although those operations will be automatically paused by unmount,
>>>>>>> btrfs check is not going to be able to handle some of those 
>>>>>>> paused operations.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I did try to run a repair again, and this time I got a bunch 
>>>>>>>>> of messages like:
>>>>>>>>>
>>>>>>>>>> repair deleting extent record: key [65795546775552,169,0]
>>>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>>>> parent 65811674234880 root 65811674234880
>>>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>>>> parent 0 root 14499
>>>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>>>> parent 65791012274176 root 65791012274176
>>>>>>>>>> adding new tree backref on start 65795546775552 len 16384 
>>>>>>>>>> parent 65806385807360 root 65806385807360
>>>>>>>>>> Repaired extent references for 65795546775552
>>>>>>>>>> ref mismatch on [65795548233728 16384] extent item 5, found 4
>>>>>>>>
>>>>>>>> That's fixing some backref mismatch, which you can ignore 
>>>>>>>> unless "btrfs check --reaonly" later reports new problems.
>>>>>>>>
>>>>>>>>> But the filesystem still refuses to mount with the exact same 
>>>>>>>>> error. I did not let the repair run entirely as it would have 
>>>>>>>>> likely taken another 3 days. What should I do? This time I'm 
>>>>>>>>> not finding any good information on what to do. For now, I've 
>>>>>>>>> started the repair again, but it doesn't exactly sound like it 
>>>>>>>>> is even fixing anything now. Still, I'll let it continue. The 
>>>>>>>>> output so far is:
>>>>>>>>>
>>>>>>>>>> [1/8] checking log skipped (none written)
>>>>>>>>>> [2/8] checking root items
>>>>>>>>>> Fixed 0 roots.
>>>>>>>>>> [3/8] checking extents
>>>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>>>> relative chunk.
>>>>>>>>>> super bytes used 49454738989056 mismatches actual used 
>>>>>>>>>> 49454738923520
>>>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>>>> relative chunk.
>>>>>>>>>> super bytes used 49454739005440 mismatches actual used 
>>>>>>>>>> 49454738956288
>>>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>>>> relative chunk.
>>>>>>>>>> super bytes used 49454739021824 mismatches actual used 
>>>>>>>>>> 49454738972672
>>>>>>>>>> Device extent[4, 19977638903808, 1073741824] didn't find the 
>>>>>>>>>> relative chunk.
>>>>>>>>>> super bytes used 49454739005440 mismatches actual used 
>>>>>>>>>> 49454738972672
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If I was able to somehow remove that one logically corrupt 
>>>>>>>>> devid from the filesystem, or somehow correct the size, that 
>>>>>>>>> would hopefully allow me to rebuild from the raid10 data then, 
>>>>>>>>> but I can't do those with the unmountable filesystem.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - Henri Hyyryläinen
>>>>>>>>>
>>>>>>>>> Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ��� 2025/9/30 02:41, Henri Hyyryläinen 写道:
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I hope this is the right place to ask about a filesystem 
>>>>>>>>>>> problem. Really shortly put, I have a file that both exists 
>>>>>>>>>>> and doesn't and prevents the containing directory from being 
>>>>>>>>>>> deleted. No matter what variant of rm and inode based 
>>>>>>>>>>> deletion I try I get an error about the file not existing, 
>>>>>>>>>>> and I also cannot try to read the file, but if I try to 
>>>>>>>>>>> delete the directory I get an error that it is not empty (so 
>>>>>>>>>>> the file kind of exists). Trying to ls the directory also 
>>>>>>>>>>> gives a file doesn't exist error.
>>>>>>>>>>>
>>>>>>>>>>> Here's what btrfs check found, which I hope does better in 
>>>>>>>>>>> illustrating the problem:
>>>>>>>>>>>
>>>>>>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>
>>>>>>>>>>> I've tried everything I've found suggested including a full 
>>>>>>>>>>> scrub, balance with -dusage=75 -musage=75, resetting file 
>>>>>>>>>>> attributes, deleting through the find command, and even some 
>>>>>>>>>>> repair mount flags that don't seem to exist for btrfs.
>>>>>>>>>>
>>>>>>>>>> The fs is corrupted, thus none of those will help.
>>>>>>>>>> I'm more interested in how the corruption happened.
>>>>>>>>>>
>>>>>>>>>> Did you use some tools other than btrfs kernel module and 
>>>>>>>>>> btrfs- progs?
>>>>>>>>>> Like ntfs2btrfs or winbtrfs?
>>>>>>>>>>
>>>>>>>>>> IIRC certain versions have some bugs related to extent tree, 
>>>>>>>>>> but should not cause this problem.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The other possibility is hardware memory bitflip, which is 
>>>>>>>>>> more common than you thought (almostly one report per month)
>>>>>>>>>>
>>>>>>>>>> In that case, a full memtest is always recommended, or you 
>>>>>>>>>> will hit all kinds of weird corruptions in the future anyway.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> With a full memtest proving the memory hardware is fine, then 
>>>>>>>>>> "btrfs check --repair" should be able to fix it.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Qu
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> What I haven't tried is a full rebalance with no filters, 
>>>>>>>>>>> but I did not try that yet as it would take quite a long 
>>>>>>>>>>> time and if it only moves data blocks around without 
>>>>>>>>>>> recomputing directory items, it doesn't seem like the right 
>>>>>>>>>>> tool to fix my problem. So I'm pretty much stuck and to me 
>>>>>>>>>>> it seems like my only option is to run btrfs check with the 
>>>>>>>>>>> repair flag, but as that has big warnings on it I thought I 
>>>>>>>>>>> would try asking here first (sorry if this is not the right 
>>>>>>>>>>> experts group to ask). So is there still something I can try 
>>>>>>>>>>> or am I finally "allowed" to use the repair command? Here's 
>>>>>>>>>>> the full output I got from btrfs check:
>>>>>>>>>>>
>>>>>>>>>>>> Opening filesystem to check...
>>>>>>>>>>>> Checking filesystem on /dev/sdc
>>>>>>>>>>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>>>>>>>>>>> [1/8] checking log skipped (none written)
>>>>>>>>>>>> [2/8] checking root items
>>>>>>>>>>>> [3/8] checking extents
>>>>>>>>>>>> [4/8] checking free space tree
>>>>>>>>>>>> We have a space info key for a block group that doesn't exist
>>>>>>>>>>>> [5/8] checking fs roots
>>>>>>>>>>>> root 5 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 5 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>>>>>>>>>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>>>>>>>>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>>>>>>>>>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no 
>>>>>>>>>>>> dir item
>>>>>>>>>>>> ERROR: errors found in fs roots
>>>>>>>>>>>> found 49400296812544 bytes used, error(s) found
>>>>>>>>>>>> total csum bytes: 48179330432
>>>>>>>>>>>> total tree bytes: 65067483136
>>>>>>>>>>>> total fs tree bytes: 12107431936
>>>>>>>>>>>> total extent tree bytes: 3194437632
>>>>>>>>>>>> btree space waste bytes: 4558984171
>>>>>>>>>>>> file data blocks allocated: 76487982252032
>>>>>>>>>>>>  referenced 60030799097856
>>>>>>>>>>>
>>>>>>>>>>> So hopefully if I'm reading things right, running a repair 
>>>>>>>>>>> would delete just that one file and directory (which itself 
>>>>>>>>>>> is a backup so I will not miss that file at all)?
>>>>>>>>>>>
>>>>>>>>>>> I do not have enough disk space to copy off the entire 
>>>>>>>>>>> filesystem and rebuild from scratch, without doing something 
>>>>>>>>>>> like rebalancing all data from raid10 to single and then 
>>>>>>>>>>> removing half the disks, but I assume that would take at 
>>>>>>>>>>> least 4 weeks to process (as I just replaced a disk which 
>>>>>>>>>>> took like a week).
>>>>>>>>>>>
>>>>>>>>>>> As to what originally caused the corruption, I think it was 
>>>>>>>>>>> probably faulty RAM, because up to to like 3 weeks ago I had 
>>>>>>>>>>> one really bad RAM stick in my computer where a certain 
>>>>>>>>>>> memory region always had incorrectly reading bytes. I had 
>>>>>>>>>>> seen intermittent quite high csum errors in monthly scrubs 
>>>>>>>>>>> pretty randomly, which thankfully could almost always be 
>>>>>>>>>>> corrected so I didn't have any major problems even though I 
>>>>>>>>>>> had like totally broken RAM in my computer for who knows how 
>>>>>>>>>>> long. So btrfs was able to protect my data quite 
>>>>>>>>>>> impressively from bad RAM.
>>>>>>>>>>>
>>>>>>>>>>> Sorry for getting a bit sidetracked there, but what should I 
>>>>>>>>>>> do in this situation?
>>>>>>>>>>>
>>>>>>>>>>> - Henri Hyyryläinen
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>