Can btrfs repair this ?

Can btrfs repair this ?

[Joe Salmeri]

Hi,

I was running openSUSE Tumbleweed build 20231001 when I first found this 
issue but updated to TW build 20231031 the other day and it still 
reports the same issue.

Kernel 6.5.9-1 btrfsprogs 6.5.1-1.2 Device Samsung 860 EVO 500 GB 
Partion #5 root btrfs filesystem, no RAID or other drives

I run "btrfs device stats /" about once a week and no problems are reported.

I run "btrfs scrub start /"   regularly too and no problem are reported

I ran "btrfs check --readonly --force /dev/sda5" the other day and got 
the following errors:

         Opening filesystem to check...

         WARNING: filesystem mounted, continuing because of --force
         Checking filesystem on /dev/sda5
         UUID: 7591d83f-f78e-402b-afe5-fab23dad0ffe
         [1/7] checking root items
         [2/7] checking extents
         [3/7] checking free space cache
         [4/7] checking fs roots
         root 262 inode 31996735 errors 1, no inode item
                 unresolved ref dir 132030 index 769 namelen 36 name 
02179466-b671-4313-8fa5-0eb87d716f92 filetype 2 errors 5, no dir item, 
no inode ref
                 unresolved ref dir 132030 index 769 namelen 36 name 
77ef9cd4-0efe-46af-bf7f-47f582851e16 filetype 2 errors 2, no dir index
         ERROR: errors found in fs roots
         found 33034690560 bytes used, error(s) found
         total csum bytes: 28819244
         total tree bytes: 986251264
         total fs tree bytes: 876134400
         total extent tree bytes: 62521344
         btree space waste bytes: 277302161
         file data blocks allocated: 141608800256
          referenced 39054090240

Running "find -inum 31996735" identifies the item is is complaining about as

         /usr/bin/find: File system loop detected; 
‘./.snapshots/1/snapshot’ is part of the same file system loop as ‘.’.
         /usr/bin/find: 
‘./home/denise/.config/skypeforlinux/blob_storage/02179466-b671-4313-8fa5-0eb87d716f92’: 
No such file or directory

Running "ls -al /home/denise/.config/skypeforlinux/blob_storage/" also 
shows that this is correct item

     drwx------ 1 denise joe-denise   72 Nov  1 22:49 .
     drwxr-xr-x 1 denise joe-denise 3.7K Nov  1 20:07 ..
     d????????? ? ?      ?             ?            ? 
02179466-b671-4313-8fa5-0eb87d716f92

When I originally ran btrfs check there were actually a bunch of other 
items listed, however, I have timeline snapshots turned on for the 
/@home subvolume and all the other items were because of that item in 
each of the other snapshots.

I removed all the other "home" snapshots and now btrfs check only 
reports that one item as shown above.

I have heard that btrfs check --repair is generally not recommended but 
I have been unable to find a way to have btrfs remove the item it is 
complaining about.

I have tried rmdir, rm -rf, as well as find -inum 31996735 -delete and 
all report the same issue with not found.

If I understand correctly, the parent directory entry ( so 
/home/denise/.config/skypeforlinux/blob_storage/ ) has the entry for 
/home/denise/.config/skypeforlinux/blob_storage/02179466-b671-4313-8fa5-0eb87d716f92 
with inode of 31996735 but it doesn't really exist.

I do not consider this a HW issue because btrfs stats, scrub, and smart 
do not report any errors and I also track all the smart info ( health, 
reallocated sector, wear leveling, etc ) for SSDs and there are no 
errors reported and I am not having any other issues.

I suspect that this occurred the other day when Skype crashed. The item 
is not needed, I just cannot figure out how to remove it.

So, is it possible for me to remove this item and if so how do I do it ?

-- 
Regards,

Joe

Re: Can btrfs repair this ?

[Qu Wenruo]

On 2023/11/6 07:08, Joe Salmeri wrote:
> Hi,
>
> I was running openSUSE Tumbleweed build 20231001 when I first found this
> issue but updated to TW build 20231031 the other day and it still
> reports the same issue.
>
> Kernel 6.5.9-1 btrfsprogs 6.5.1-1.2 Device Samsung 860 EVO 500 GB
> Partion #5 root btrfs filesystem, no RAID or other drives
>
> I run "btrfs device stats /" about once a week and no problems are
> reported.
>
> I run "btrfs scrub start /"   regularly too and no problem are reported
>
> I ran "btrfs check --readonly --force /dev/sda5" the other day and got
> the following errors:
>
>          Opening filesystem to check...
>
>          WARNING: filesystem mounted, continuing because of --force
>          Checking filesystem on /dev/sda5
>          UUID: 7591d83f-f78e-402b-afe5-fab23dad0ffe
>          [1/7] checking root items
>          [2/7] checking extents
>          [3/7] checking free space cache
>          [4/7] checking fs roots
>          root 262 inode 31996735 errors 1, no inode item
>                  unresolved ref dir 132030 index 769 namelen 36 name
> 02179466-b671-4313-8fa5-0eb87d716f92 filetype 2 errors 5, no dir item,
> no inode ref
>                  unresolved ref dir 132030 index 769 namelen 36 name
> 77ef9cd4-0efe-46af-bf7f-47f582851e16 filetype 2 errors 2, no dir index
>          ERROR: errors found in fs roots
>          found 33034690560 bytes used, error(s) found
>          total csum bytes: 28819244
>          total tree bytes: 986251264
>          total fs tree bytes: 876134400
>          total extent tree bytes: 62521344
>          btree space waste bytes: 277302161
>          file data blocks allocated: 141608800256
>           referenced 39054090240
>
> Running "find -inum 31996735" identifies the item is is complaining
> about as
>
>          /usr/bin/find: File system loop detected;
> ‘./.snapshots/1/snapshot’ is part of the same file system loop as ‘.’.
>          /usr/bin/find:
> ‘./home/denise/.config/skypeforlinux/blob_storage/02179466-b671-4313-8fa5-0eb87d716f92’: No such file or directory
>
> Running "ls -al /home/denise/.config/skypeforlinux/blob_storage/" also
> shows that this is correct item
>
>      drwx------ 1 denise joe-denise   72 Nov  1 22:49 .
>      drwxr-xr-x 1 denise joe-denise 3.7K Nov  1 20:07 ..
>      d????????? ? ?      ?             ?            ?
> 02179466-b671-4313-8fa5-0eb87d716f92
>
> When I originally ran btrfs check there were actually a bunch of other
> items listed, however, I have timeline snapshots turned on for the
> /@home subvolume and all the other items were because of that item in
> each of the other snapshots.
>
> I removed all the other "home" snapshots and now btrfs check only
> reports that one item as shown above.
>
> I have heard that btrfs check --repair is generally not recommended but
> I have been unable to find a way to have btrfs remove the item it is
> complaining about.

--repair can fix the problem.

But for your particular problem, please also do a memtest just in case.

This problem looks like a bad hash, which may be caused by memory bitflip.

Thanks,
Qu
>
> I have tried rmdir, rm -rf, as well as find -inum 31996735 -delete and
> all report the same issue with not found.
>
> If I understand correctly, the parent directory entry ( so
> /home/denise/.config/skypeforlinux/blob_storage/ ) has the entry for
> /home/denise/.config/skypeforlinux/blob_storage/02179466-b671-4313-8fa5-0eb87d716f92 with inode of 31996735 but it doesn't really exist.
>
> I do not consider this a HW issue because btrfs stats, scrub, and smart
> do not report any errors and I also track all the smart info ( health,
> reallocated sector, wear leveling, etc ) for SSDs and there are no
> errors reported and I am not having any other issues.
>
> I suspect that this occurred the other day when Skype crashed. The item
> is not needed, I just cannot figure out how to remove it.
>
> So, is it possible for me to remove this item and if so how do I do it ?
>

Re: Can btrfs repair this ?

[Joe Salmeri]

On 11/5/23 18:45, Qu Wenruo wrote:
>
>
> On 2023/11/6 07:08, Joe Salmeri wrote:
>> Hi,
>>
>> I was running openSUSE Tumbleweed build 20231001 when I first found this
>> issue but updated to TW build 20231031 the other day and it still
>> reports the same issue.
>>
>> Kernel 6.5.9-1 btrfsprogs 6.5.1-1.2 Device Samsung 860 EVO 500 GB
>> Partion #5 root btrfs filesystem, no RAID or other drives
>>
>> I run "btrfs device stats /" about once a week and no problems are
>> reported.
>>
>> I run "btrfs scrub start /"   regularly too and no problem are reported
>>
>> I ran "btrfs check --readonly --force /dev/sda5" the other day and got
>> the following errors:
>>
>>          Opening filesystem to check...
>>
>>          WARNING: filesystem mounted, continuing because of --force
>>          Checking filesystem on /dev/sda5
>>          UUID: 7591d83f-f78e-402b-afe5-fab23dad0ffe
>>          [1/7] checking root items
>>          [2/7] checking extents
>>          [3/7] checking free space cache
>>          [4/7] checking fs roots
>>          root 262 inode 31996735 errors 1, no inode item
>>                  unresolved ref dir 132030 index 769 namelen 36 name
>> 02179466-b671-4313-8fa5-0eb87d716f92 filetype 2 errors 5, no dir item,
>> no inode ref
>>                  unresolved ref dir 132030 index 769 namelen 36 name
>> 77ef9cd4-0efe-46af-bf7f-47f582851e16 filetype 2 errors 2, no dir index
>>          ERROR: errors found in fs roots
>>          found 33034690560 bytes used, error(s) found
>>          total csum bytes: 28819244
>>          total tree bytes: 986251264
>>          total fs tree bytes: 876134400
>>          total extent tree bytes: 62521344
>>          btree space waste bytes: 277302161
>>          file data blocks allocated: 141608800256
>>           referenced 39054090240
>>
>> Running "find -inum 31996735" identifies the item is is complaining
>> about as
>>
>>          /usr/bin/find: File system loop detected;
>> ‘./.snapshots/1/snapshot’ is part of the same file system loop as ‘.’.
>>          /usr/bin/find:
>> ‘./home/denise/.config/skypeforlinux/blob_storage/02179466-b671-4313-8fa5-0eb87d716f92’: 
>> No such file or directory
>>
>> Running "ls -al /home/denise/.config/skypeforlinux/blob_storage/" also
>> shows that this is correct item
>>
>>      drwx------ 1 denise joe-denise   72 Nov  1 22:49 .
>>      drwxr-xr-x 1 denise joe-denise 3.7K Nov  1 20:07 ..
>>      d????????? ? ?      ?             ?            ?
>> 02179466-b671-4313-8fa5-0eb87d716f92
>>
>> When I originally ran btrfs check there were actually a bunch of other
>> items listed, however, I have timeline snapshots turned on for the
>> /@home subvolume and all the other items were because of that item in
>> each of the other snapshots.
>>
>> I removed all the other "home" snapshots and now btrfs check only
>> reports that one item as shown above.
>>
>> I have heard that btrfs check --repair is generally not recommended but
>> I have been unable to find a way to have btrfs remove the item it is
>> complaining about.
>
> --repair can fix the problem.
>
> But for your particular problem, please also do a memtest just in case.
>
> This problem looks like a bad hash, which may be caused by memory 
> bitflip.
>
I ran the memtests last night multiple times and no problems reported.

I have heard that --repair usage is usually not recommended.

I cannot afford to have this system have to be rebuilt right now.

Is it definitely safe in this case ?   Otherwise, I would just ignore it 
since I know why it exists and that it is not a hw problem.

Since this is the root fs, I assume I have to boot a USB live 
environment and do the repair from there or can it be done on the 
mounted btrfs root fs with the --force option ?

>
>> I have tried rmdir, rm -rf, as well as find -inum 31996735 -delete and
>> all report the same issue with not found.
>>
>> If I understand correctly, the parent directory entry ( so
>> /home/denise/.config/skypeforlinux/blob_storage/ ) has the entry for
>> /home/denise/.config/skypeforlinux/blob_storage/02179466-b671-4313-8fa5-0eb87d716f92 
>> with inode of 31996735 but it doesn't really exist.
>>
>> I do not consider this a HW issue because btrfs stats, scrub, and smart
>> do not report any errors and I also track all the smart info ( health,
>> reallocated sector, wear leveling, etc ) for SSDs and there are no
>> errors reported and I am not having any other issues.
>>
>> I suspect that this occurred the other day when Skype crashed. The item
>> is not needed, I just cannot figure out how to remove it.
>>
>> So, is it possible for me to remove this item and if so how do I do it ?
>>
I find it interesting that interesting that I can remove a snapshot which contains the item using the normal delete snapshot functionality but that none of the built in tools have a way to do it.



-- 
Regards,

Joe

Re: Can btrfs repair this ?

[Qu Wenruo]

On 2023/11/7 06:26, Joe Salmeri wrote:
>
> On 11/5/23 18:45, Qu Wenruo wrote:
>>
>>
>> On 2023/11/6 07:08, Joe Salmeri wrote:
>>> Hi,
>>>
>>> I was running openSUSE Tumbleweed build 20231001 when I first found this
>>> issue but updated to TW build 20231031 the other day and it still
>>> reports the same issue.
>>>
>>> Kernel 6.5.9-1 btrfsprogs 6.5.1-1.2 Device Samsung 860 EVO 500 GB
>>> Partion #5 root btrfs filesystem, no RAID or other drives
>>>
>>> I run "btrfs device stats /" about once a week and no problems are
>>> reported.
>>>
>>> I run "btrfs scrub start /"   regularly too and no problem are reported
>>>
>>> I ran "btrfs check --readonly --force /dev/sda5" the other day and got
>>> the following errors:
>>>
>>>          Opening filesystem to check...
>>>
>>>          WARNING: filesystem mounted, continuing because of --force
>>>          Checking filesystem on /dev/sda5
>>>          UUID: 7591d83f-f78e-402b-afe5-fab23dad0ffe
>>>          [1/7] checking root items
>>>          [2/7] checking extents
>>>          [3/7] checking free space cache
>>>          [4/7] checking fs roots
>>>          root 262 inode 31996735 errors 1, no inode item
>>>                  unresolved ref dir 132030 index 769 namelen 36 name
>>> 02179466-b671-4313-8fa5-0eb87d716f92 filetype 2 errors 5, no dir item,
>>> no inode ref
>>>                  unresolved ref dir 132030 index 769 namelen 36 name
>>> 77ef9cd4-0efe-46af-bf7f-47f582851e16 filetype 2 errors 2, no dir index
>>>          ERROR: errors found in fs roots
>>>          found 33034690560 bytes used, error(s) found
>>>          total csum bytes: 28819244
>>>          total tree bytes: 986251264
>>>          total fs tree bytes: 876134400
>>>          total extent tree bytes: 62521344
>>>          btree space waste bytes: 277302161
>>>          file data blocks allocated: 141608800256
>>>           referenced 39054090240
>>>
>>> Running "find -inum 31996735" identifies the item is is complaining
>>> about as
>>>
>>>          /usr/bin/find: File system loop detected;
>>> ‘./.snapshots/1/snapshot’ is part of the same file system loop as ‘.’.
>>>          /usr/bin/find:
>>> ‘./home/denise/.config/skypeforlinux/blob_storage/02179466-b671-4313-8fa5-0eb87d716f92’: No such file or directory
>>>
>>> Running "ls -al /home/denise/.config/skypeforlinux/blob_storage/" also
>>> shows that this is correct item
>>>
>>>      drwx------ 1 denise joe-denise   72 Nov  1 22:49 .
>>>      drwxr-xr-x 1 denise joe-denise 3.7K Nov  1 20:07 ..
>>>      d????????? ? ?      ?             ?            ?
>>> 02179466-b671-4313-8fa5-0eb87d716f92
>>>
>>> When I originally ran btrfs check there were actually a bunch of other
>>> items listed, however, I have timeline snapshots turned on for the
>>> /@home subvolume and all the other items were because of that item in
>>> each of the other snapshots.
>>>
>>> I removed all the other "home" snapshots and now btrfs check only
>>> reports that one item as shown above.
>>>
>>> I have heard that btrfs check --repair is generally not recommended but
>>> I have been unable to find a way to have btrfs remove the item it is
>>> complaining about.
>>
>> --repair can fix the problem.
>>
>> But for your particular problem, please also do a memtest just in case.
>>
>> This problem looks like a bad hash, which may be caused by memory
>> bitflip.
>>
> I ran the memtests last night multiple times and no problems reported.
>
> I have heard that --repair usage is usually not recommended.
>
> I cannot afford to have this system have to be rebuilt right now.
>
> Is it definitely safe in this case ?   Otherwise, I would just ignore it
> since I know why it exists and that it is not a hw problem.

At least from your output, it won't make things any worse.

Thanks,
Qu

>
> Since this is the root fs, I assume I have to boot a USB live
> environment and do the repair from there or can it be done on the
> mounted btrfs root fs with the --force option ?
>
>>
>>> I have tried rmdir, rm -rf, as well as find -inum 31996735 -delete and
>>> all report the same issue with not found.
>>>
>>> If I understand correctly, the parent directory entry ( so
>>> /home/denise/.config/skypeforlinux/blob_storage/ ) has the entry for
>>> /home/denise/.config/skypeforlinux/blob_storage/02179466-b671-4313-8fa5-0eb87d716f92 with inode of 31996735 but it doesn't really exist.
>>>
>>> I do not consider this a HW issue because btrfs stats, scrub, and smart
>>> do not report any errors and I also track all the smart info ( health,
>>> reallocated sector, wear leveling, etc ) for SSDs and there are no
>>> errors reported and I am not having any other issues.
>>>
>>> I suspect that this occurred the other day when Skype crashed. The item
>>> is not needed, I just cannot figure out how to remove it.
>>>
>>> So, is it possible for me to remove this item and if so how do I do it ?
>>>
> I find it interesting that interesting that I can remove a snapshot
> which contains the item using the normal delete snapshot functionality
> but that none of the built in tools have a way to do it.
>
>
>