Re: How to remove an unremovable file and directory?

["Henri Hyyryläinen" <henri.hyyrylainen@gmail.com>]

Thank you for the reply.

> The fs is corrupted, thus none of those will help.
> I'm more interested in how the corruption happened.
>
> Did you use some tools other than btrfs kernel module and btrfs-progs?
> Like ntfs2btrfs or winbtrfs?
>
> IIRC certain versions have some bugs related to extent tree, but 
> should not cause this problem. 

I did not use those tools. I only have Fedora 42 installed on this computer.

> The other possibility is hardware memory bitflip, which is more common 
> than you thought (almostly one report per month)
>
> In that case, a full memtest is always recommended, or you will hit 
> all kinds of weird corruptions in the future anyway. 

Yes, that seems the most likely explanation. I switched my motherboard, 
RAM, and CPU a few weeks ago, and when building a new setup out of  the 
old components and 3 different drives (including one brand new NVME SSD) 
I noticed that each time I ran a btrfs scrub on any of those I got a 
bunch of csum errors. I initially thought that maybe the drives or 
something were faulty, but when I ran memtest it discovered very quickly 
a faulty single RAM stick. That RAM had been in my main system in a 
faulty state for months probably at least. The different computer I 
built from the old parts immediately stopped failing scrubs when I 
removed the faulty RAM stick.

So yeah, I think the corruption would have been caused by that RAM 
stick. I have now completely different RAM in my system and I've run the 
same memtest multiple times and it has not found any problem, so I think 
I have now fully good RAM and won't have the same corruption issue again.

One potentially relevant detail is that I replaced one of my disks just 
before I discovered the corruption with a bigger hard drive (at this 
point I had already swapped my RAM so there shouldn't be memory 
corruption problems during the replace). I started a btrfs replace 
operation, but I had a scrub running which was then canceled. That 
cancel for some reason also canceled the replace operation. When I 
noticed that I then restarted the replace and then let it run to 
completion (though I think the replace got canceled twice as I didn't 
realize after the first one what canceled it, so I needed to restart the 
replace multiple times). After completing the replace command without an 
error message I shut down my computer and removed the replaced drive. 
Upon booting my filesystem could not mount without the degraded flag. 
Somehow the replace had copied the right amount of data to the new 
drive, but for some reason the old drive was still marked as used. I had 
to then do a btrfs remove on that device which took a really long time 
but eventually succeeded as it seemingly had to rebuild the missing 
drive from raid data (even though all of the necessary data should have 
been copied already). So it seems like the btrfs replace operation 
doesn't fully work correctly if it is interrupted.

That could be a relevant detail, but as my filesystem only had one 
corrupt folder I somewhat doubt that as I would expect a lot more things 
to have gone wrong if the replace messed up the data. So to me that 
seems only like a minor bug in the replace operation, but one that did 
not cause this corruption.

> With a full memtest proving the memory hardware is fine, then "btrfs 
> check --repair" should be able to fix it. 

Yes, my RAM should be totally fine now.

Thanks, I will try running that repair tonight, and hoping everything is 
then fine.


- Henri Hyyryläinen

Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
>
>
> 在 2025/9/30 02:41, Henri Hyyryläinen 写道:
>> Hello,
>>
>> I hope this is the right place to ask about a filesystem problem. 
>> Really shortly put, I have a file that both exists and doesn't and 
>> prevents the containing directory from being deleted. No matter what 
>> variant of rm and inode based deletion I try I get an error about the 
>> file not existing, and I also cannot try to read the file, but if I 
>> try to delete the directory I get an error that it is not empty (so 
>> the file kind of exists). Trying to ls the directory also gives a 
>> file doesn't exist error.
>>
>> Here's what btrfs check found, which I hope does better in 
>> illustrating the problem:
>>
>>> root 5 inode 25953213 errors 200, dir isize wrong
>>> root 5 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>
>> I've tried everything I've found suggested including a full scrub, 
>> balance with -dusage=75 -musage=75, resetting file attributes, 
>> deleting through the find command, and even some repair mount flags 
>> that don't seem to exist for btrfs.
>
> The fs is corrupted, thus none of those will help.
> I'm more interested in how the corruption happened.
>
> Did you use some tools other than btrfs kernel module and btrfs-progs?
> Like ntfs2btrfs or winbtrfs?
>
> IIRC certain versions have some bugs related to extent tree, but 
> should not cause this problem.
>
>
> The other possibility is hardware memory bitflip, which is more common 
> than you thought (almostly one report per month)
>
> In that case, a full memtest is always recommended, or you will hit 
> all kinds of weird corruptions in the future anyway.
>
>
> With a full memtest proving the memory hardware is fine, then "btrfs 
> check --repair" should be able to fix it.
>
> Thanks,
> Qu
>
>
>> What I haven't tried is a full rebalance with no filters, but I did 
>> not try that yet as it would take quite a long time and if it only 
>> moves data blocks around without recomputing directory items, it 
>> doesn't seem like the right tool to fix my problem. So I'm pretty 
>> much stuck and to me it seems like my only option is to run btrfs 
>> check with the repair flag, but as that has big warnings on it I 
>> thought I would try asking here first (sorry if this is not the right 
>> experts group to ask). So is there still something I can try or am I 
>> finally "allowed" to use the repair command? Here's the full output I 
>> got from btrfs check:
>>
>>> Opening filesystem to check...
>>> Checking filesystem on /dev/sdc
>>> UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
>>> [1/8] checking log skipped (none written)
>>> [2/8] checking root items
>>> [3/8] checking extents
>>> [4/8] checking free space tree
>>> We have a space info key for a block group that doesn't exist
>>> [5/8] checking fs roots
>>> root 5 inode 25953213 errors 200, dir isize wrong
>>> root 5 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14428 inode 25953213 errors 200, dir isize wrong
>>> root 14428 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14451 inode 25953213 errors 200, dir isize wrong
>>> root 14451 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14475 inode 25953213 errors 200, dir isize wrong
>>> root 14475 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14499 inode 25953213 errors 200, dir isize wrong
>>> root 14499 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14523 inode 25953213 errors 200, dir isize wrong
>>> root 14523 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14544 inode 25953213 errors 200, dir isize wrong
>>> root 14544 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14545 inode 25953213 errors 200, dir isize wrong
>>> root 14545 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14546 inode 25953213 errors 200, dir isize wrong
>>> root 14546 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14547 inode 25953213 errors 200, dir isize wrong
>>> root 14547 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14548 inode 25953213 errors 200, dir isize wrong
>>> root 14548 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14549 inode 25953213 errors 200, dir isize wrong
>>> root 14549 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> root 14550 inode 25953213 errors 200, dir isize wrong
>>> root 14550 inode 27166085 errors 2000, link count wrong
>>>         unresolved ref dir 25953213 index 7 namelen 33 name 
>>> Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
>>> ERROR: errors found in fs roots
>>> found 49400296812544 bytes used, error(s) found
>>> total csum bytes: 48179330432
>>> total tree bytes: 65067483136
>>> total fs tree bytes: 12107431936
>>> total extent tree bytes: 3194437632
>>> btree space waste bytes: 4558984171
>>> file data blocks allocated: 76487982252032
>>>  referenced 60030799097856
>>
>> So hopefully if I'm reading things right, running a repair would 
>> delete just that one file and directory (which itself is a backup so 
>> I will not miss that file at all)?
>>
>> I do not have enough disk space to copy off the entire filesystem and 
>> rebuild from scratch, without doing something like rebalancing all 
>> data from raid10 to single and then removing half the disks, but I 
>> assume that would take at least 4 weeks to process (as I just 
>> replaced a disk which took like a week).
>>
>> As to what originally caused the corruption, I think it was probably 
>> faulty RAM, because up to to like 3 weeks ago I had one really bad 
>> RAM stick in my computer where a certain memory region always had 
>> incorrectly reading bytes. I had seen intermittent quite high csum 
>> errors in monthly scrubs pretty randomly, which thankfully could 
>> almost always be corrected so I didn't have any major problems even 
>> though I had like totally broken RAM in my computer for who knows how 
>> long. So btrfs was able to protect my data quite impressively from 
>> bad RAM.
>>
>> Sorry for getting a bit sidetracked there, but what should I do in 
>> this situation?
>>
>> - Henri Hyyryläinen
>>
>>
>