Re: How to remove an unremovable file and directory?

["Henri Hyyryläinen" <henri.hyyrylainen@gmail.com>]

Hello again.

It took over 3 days, but the btrfs check --repair has now completed 
seemingly successfully. I mostly saw output about the file being placed 
in lost+found and and directory size being corrected. However, there 
were some messages about mismatch of used bytes.

Unfortunately it seems like the situation has gotten worse since the 
repair, because now I cannot mount the filesystem at all. Instead I get 
an error like this:

> BTRFS error (device sdc): dev extent physical offset 19977638903808 on 
> devid 4 doesn't have corresponding chunk
> BTRFS error (device sdc): failed to verify dev extents against chunks: 
> -117
> BTRFS error (device sdc): open_ctree failed: -117
Even if I remove that one problematic device physically from my 
computer, the filesystem still refuses to mount with the same error. 
Maybe the problems with the device replace are again showing up with the 
actual size of the hard drive not being used correctly? I cannot try to 
remove the device slack as I cannot mount the filesystem.

I did try to run a repair again, and this time I got a bunch of messages 
like:

> repair deleting extent record: key [65795546775552,169,0]
> adding new tree backref on start 65795546775552 len 16384 parent 
> 65811674234880 root 65811674234880
> adding new tree backref on start 65795546775552 len 16384 parent 0 
> root 14499
> adding new tree backref on start 65795546775552 len 16384 parent 
> 65791012274176 root 65791012274176
> adding new tree backref on start 65795546775552 len 16384 parent 
> 65806385807360 root 65806385807360
> Repaired extent references for 65795546775552
> ref mismatch on [65795548233728 16384] extent item 5, found 4
But the filesystem still refuses to mount with the exact same error. I 
did not let the repair run entirely as it would have likely taken 
another 3 days. What should I do? This time I'm not finding any good 
information on what to do. For now, I've started the repair again, but 
it doesn't exactly sound like it is even fixing anything now. Still, 
I'll let it continue. The output so far is:

> [1/8] checking log skipped (none written)
> [2/8] checking root items
> Fixed 0 roots.
> [3/8] checking extents
> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
> chunk.
> super bytes used 49454738989056 mismatches actual used 49454738923520
> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
> chunk.
> super bytes used 49454739005440 mismatches actual used 49454738956288
> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
> chunk.
> super bytes used 49454739021824 mismatches actual used 49454738972672
> Device extent[4, 19977638903808, 1073741824] didn't find the relative 
> chunk.
> super bytes used 49454739005440 mismatches actual used 49454738972672


If I was able to somehow remove that one logically corrupt devid from 
the filesystem, or somehow correct the size, that would hopefully allow 
me to rebuild from the raid10 data then, but I can't do those with the 
unmountable filesystem.


- Henri Hyyryläinen

Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>
>
> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>> Hello,
>>
>> I hope this is the right place to ask about a filesystem problem. 
>> Really shortly put, I have a file that both exists and doesn't and 
>> prevents the containing directory from being deleted. No matter what 
>> variant of rm and inode based deletion I try I get an error about the 
>> file not existing, and I also cannot try to read the file, but if I 
>> try to delete the directory I get an error that it is not empty (so 
>> the file kind of exists). Trying to ls the directory also gives a 
>> file doesn't exist error.
>>
>> Here's what btrfs check found, which I hope does better in 
>> illustrating the problem:
>>
>>> root 5 inode 25953213 errors 200, dir isize wrong
>>> root 5 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>
>> I've tried everything I've found suggested including a full scrub, 
>> balance with -dusage=75 -musage=75, resetting file attributes, 
>> deleting through the find command, and even some repair mount flags 
>> that don't seem to exist for btrfs.
>
> The fs is corrupted, thus none of those will help.
> I'm more interested in how the corruption happened.
>
> Did you use some tools other than btrfs kernel module and btrfs-progs?
> Like ntfs2btrfs or winbtrfs?
>
> IIRC certain versions have some bugs related to extent tree, but 
> should not cause this problem.
>
>
> The other possibility is hardware memory bitflip, which is more common 
> than you thought (almostly one report per month)
>
> In that case, a full memtest is always recommended, or you will hit 
> all kinds of weird corruptions in the future anyway.
>
>
> With a full memtest proving the memory hardware is fine, then "btrfs 
> check --repair" should be able to fix it.
>
> Thanks,
> Qu
>
>
>> What I haven't tried is a full rebalance with no filters, but I did 
>> not try that yet as it would take quite a long time and if it only 
>> moves data blocks around without recomputing directory items, it 
>> doesn't seem like the right tool to fix my problem. So I'm pretty 
>> much stuck and to me it seems like my only option is to run btrfs 
>> check with the repair flag, but as that has big warnings on it I 
>> thought I would try asking here first (sorry if this is not the right 
>> experts group to ask). So is there still something I can try or am I 
>> finally "allowed" to use the repair command? Here's the full output I 
>> got from btrfs check:
>>
>>> Opening filesystem to check...
>>> Checking filesystem on /dev/sdc
>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>> [1/8] checking log skipped (none written)
>>> [2/8] checking root items
>>> [3/8] checking extents
>>> [4/8] checking free space tree
>>> We have a space info key for a block group that doesn't exist
>>> [5/8] checking fs roots
>>> root 5 inode 25953213 errors 200, dir isize wrong
>>> root 5 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> ERROR: errors found in fs roots
>>> found 49400296812544 bytes used, error(s) found
>>> total csum bytes: 48179330432
>>> total tree bytes: 65067483136
>>> total fs tree bytes: 12107431936
>>> total extent tree bytes: 3194437632
>>> btree space waste bytes: 4558984171
>>> file data blocks allocated: 76487982252032
>>>  referenced 60030799097856
>>
>> So hopefully if I'm reading things right, running a repair would 
>> delete just that one file and directory (which itself is a backup so 
>> I will not miss that file at all)?
>>
>> I do not have enough disk space to copy off the entire filesystem and 
>> rebuild from scratch, without doing something like rebalancing all 
>> data from raid10 to single and then removing half the disks, but I 
>> assume that would take at least 4 weeks to process (as I just 
>> replaced a disk which took like a week).
>>
>> As to what originally caused the corruption, I think it was probably 
>> faulty RAM, because up to to like 3 weeks ago I had one really bad 
>> RAM stick in my computer where a certain memory region always had 
>> incorrectly reading bytes. I had seen intermittent quite high csum 
>> errors in monthly scrubs pretty randomly, which thankfully could 
>> almost always be corrected so I didn't have any major problems even 
>> though I had like totally broken RAM in my computer for who knows how 
>> long. So btrfs was able to protect my data quite impressively from 
>> bad RAM.
>>
>> Sorry for getting a bit sidetracked there, but what should I do in 
>> this situation?
>>
>> - Henri Hyyryläinen
>>
>>
>