Metadata corruption detected, fatal error -- couldn't map inode, err = 117

Metadata corruption detected, fatal error -- couldn't map inode, err = 117

[Phillip Ferentinos]

Hello,

Looking for opinions on recovering a filesystem that does not
successfully repair from xfs_repair. On the first xfs_repair, I was
prompted to mount the disk to replay the log which I did successfully.
I created an image of the disk with ddrescue and am attempting to
recover the data. Unfortunately, I do not have a recent backup of this
disk.

The final output of xfs_repair is:

Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
rebuilding directory inode 12955326179
Metadata corruption detected at 0x46fa05, inode 0x38983bd88 dinode

fatal error -- couldn't map inode 15192014216, err = 117

The full log is:
https://raw.githubusercontent.com/phillipjf/xfs_debug/main/xfs_repair_1.log

Based on another discussion (https://narkive.com/4dDxIees.10), I've
included the specific inode:
https://raw.githubusercontent.com/phillipjf/xfs_debug/main/xfs_db_01.log

I also cannot create a metadump due to the following issue:
https://raw.githubusercontent.com/phillipjf/xfs_debug/main/xfs_metadump_01.log.

Any advice would be greatly appreciated. If any more information is
needed, hopefully I can provide it.

Thanks in advance

Re: Metadata corruption detected, fatal error -- couldn't map inode, err = 117

[Dave Chinner]

On Thu, Dec 21, 2023 at 08:05:43PM -0600, Phillip Ferentinos wrote:
> Hello,
> 
> Looking for opinions on recovering a filesystem that does not
> successfully repair from xfs_repair.

What version of xfs_repair?

> On the first xfs_repair, I was
> prompted to mount the disk to replay the log which I did successfully.
> I created an image of the disk with ddrescue and am attempting to
> recover the data. Unfortunately, I do not have a recent backup of this
> disk.

There is lots of random damage all over the filesystem. What caused
this damage to occur? I generally only see this sort of widespread
damage when RAID devices (hardware or software) go bad...

Keep in mind that regardless of whether xfs_repair returns the
filesystem to a consistent state, the data in the filesystem is
still going to be badly corrupted. If you don't have backups, then
there's a high probability of significant data loss here....

> The final output of xfs_repair is:
> 
> Phase 5 - rebuild AG headers and trees...
>         - reset superblock...
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
> rebuilding directory inode 12955326179
> Metadata corruption detected at 0x46fa05, inode 0x38983bd88 dinode

Can you run 'gdb xfs_repair' and run the command 'l *0x46fa05' to
dump the line of code that the error was detected at? You probably
need the distro debug package for xfsprogs installed to do this.

> fatal error -- couldn't map inode 15192014216, err = 117
> 
> The full log is:
> https://raw.githubusercontent.com/phillipjf/xfs_debug/main/xfs_repair_1.log

That's messy.

> Based on another discussion (https://narkive.com/4dDxIees.10), I've
> included the specific inode:
> https://raw.githubusercontent.com/phillipjf/xfs_debug/main/xfs_db_01.log

Nothing obviously wrong with that inode in the image file - it's a
directory inode in node format that looks to be internally
consistent.  But that inode has been processed earlier in the repair
process, so maybe it's bad in memory as a result of trying to fix
some other problem. Very hard to say given how much other stuff is
broken and is getting either trashed, re-initialised or repaired up
to that point....

> I also cannot create a metadump due to the following issue:
> https://raw.githubusercontent.com/phillipjf/xfs_debug/main/xfs_metadump_01.log.

No surprise, metadump has to traverse the metadata in order to dump
it, and if the metadata is corrupt then the traversals can fail
leading to a failed dump. The more badly damaged the filesystem is,
the more likely a metadump failure is.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Metadata corruption detected, fatal error -- couldn't map inode, err = 117

[Phillip Ferentinos]

On Tue, Dec 26, 2023 at 2:36 PM Dave Chinner <david@fromorbit.com> wrote:
>
> On Thu, Dec 21, 2023 at 08:05:43PM -0600, Phillip Ferentinos wrote:
> > Hello,
> >
> > Looking for opinions on recovering a filesystem that does not
> > successfully repair from xfs_repair.
>
> What version of xfs_repair?

# xfs_repair -V
xfs_repair version 6.1.1

> > On the first xfs_repair, I was
> > prompted to mount the disk to replay the log which I did successfully.
> > I created an image of the disk with ddrescue and am attempting to
> > recover the data. Unfortunately, I do not have a recent backup of this
> > disk.
>
> There is lots of random damage all over the filesystem. What caused
> this damage to occur? I generally only see this sort of widespread
> damage when RAID devices (hardware or software) go bad...
>
> Keep in mind that regardless of whether xfs_repair returns the
> filesystem to a consistent state, the data in the filesystem is
> still going to be badly corrupted. If you don't have backups, then
> there's a high probability of significant data loss here....

My best guess is a power outage causing an unclean shutdown. I am
running Unraid:
# uname -a
Linux Tower 6.1.64-Unraid #1 SMP PREEMPT_DYNAMIC Wed Nov 29 12:48:16
PST 2023 x86_64 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz GenuineIntel
GNU/Linux

I was able to load the disk img in UFS Explorer on an Ubuntu machine
and, as far as I can tell, all the data is in the img but it reports a
handful of bad objects.
https://github.com/phillipjf/xfs_debug/blob/main/ufs-explorer.png

> > The final output of xfs_repair is:
> >
> > Phase 5 - rebuild AG headers and trees...
> >         - reset superblock...
> > Phase 6 - check inode connectivity...
> >         - resetting contents of realtime bitmap and summary inodes
> >         - traversing filesystem ...
> > rebuilding directory inode 12955326179
> > Metadata corruption detected at 0x46fa05, inode 0x38983bd88 dinode
>
> Can you run 'gdb xfs_repair' and run the command 'l *0x46fa05' to
> dump the line of code that the error was detected at? You probably
> need the distro debug package for xfsprogs installed to do this.

At first try, doesn't look like gdb is available on Unraid and I think
it would be more trouble than it's worth to get it set up.
---
On the Ubuntu machine, I have
# xfs_repair -V
xfs_repair version 6.1.1

When running gdb on Ubuntu, I get:
$ gdb xfs_repair
(gdb) set args -f /media/sdl1.img
(gdb) run
...
Metadata corruption detected at 0x5555555afd95, inode 0x38983bd88 dinode

fatal error -- couldn't map inode 15192014216, err = 117
...
(gdb) l *0x5555555afd95
0x5555555afd95 is in libxfs_dinode_verify (../libxfs/xfs_inode_buf.c:586).
Downloading source file
/build/xfsprogs-QBD5z8/xfsprogs-6.3.0/repair/../libxfs/xfs_inode_buf.c
581     ../libxfs/xfs_inode_buf.c: No such file or directory.

> > fatal error -- couldn't map inode 15192014216, err = 117
> >
> > The full log is:
> > https://raw.githubusercontent.com/phillipjf/xfs_debug/main/xfs_repair_1.log
>
> That's messy.
>
> > Based on another discussion (https://narkive.com/4dDxIees.10), I've
> > included the specific inode:
> > https://raw.githubusercontent.com/phillipjf/xfs_debug/main/xfs_db_01.log
>
> Nothing obviously wrong with that inode in the image file - it's a
> directory inode in node format that looks to be internally
> consistent.  But that inode has been processed earlier in the repair
> process, so maybe it's bad in memory as a result of trying to fix
> some other problem. Very hard to say given how much other stuff is
> broken and is getting either trashed, re-initialised or repaired up
> to that point....
>
> > I also cannot create a metadump due to the following issue:
> > https://raw.githubusercontent.com/phillipjf/xfs_debug/main/xfs_metadump_01.log.
>
> No surprise, metadump has to traverse the metadata in order to dump
> it, and if the metadata is corrupt then the traversals can fail
> leading to a failed dump. The more badly damaged the filesystem is,
> the more likely a metadump failure is.
>
> -Dave.
> --
> Dave Chinner
> david@fromorbit.com

I only had one extra 12TB drive on hand which currently has the .img
of the bad drive on it. Have some 10TB drives on order which should
give enough space to restore data with UFS Explorer and, if so, will
resolve all these issues. But, possibly, this information is helpful
for development.

Thanks for the help! Please let me know what more info I can provide.
- Phillip Ferentinos

Re: Metadata corruption detected, fatal error -- couldn't map inode, err = 117

[Dave Chinner]

On Wed, Dec 27, 2023 at 02:22:47PM -0600, Phillip Ferentinos wrote:
> On Tue, Dec 26, 2023 at 2:36 PM Dave Chinner <david@fromorbit.com> wrote:
> >
> > On Thu, Dec 21, 2023 at 08:05:43PM -0600, Phillip Ferentinos wrote:
> > > Hello,
> > >
> > > Looking for opinions on recovering a filesystem that does not
> > > successfully repair from xfs_repair.
> >
> > What version of xfs_repair?
> 
> # xfs_repair -V
> xfs_repair version 6.1.1
> 
> > > On the first xfs_repair, I was
> > > prompted to mount the disk to replay the log which I did successfully.
> > > I created an image of the disk with ddrescue and am attempting to
> > > recover the data. Unfortunately, I do not have a recent backup of this
> > > disk.
> >
> > There is lots of random damage all over the filesystem. What caused
> > this damage to occur? I generally only see this sort of widespread
> > damage when RAID devices (hardware or software) go bad...
> >
> > Keep in mind that regardless of whether xfs_repair returns the
> > filesystem to a consistent state, the data in the filesystem is
> > still going to be badly corrupted. If you don't have backups, then
> > there's a high probability of significant data loss here....
> 
> My best guess is a power outage causing an unclean shutdown. I am
> running Unraid:
> # uname -a
> Linux Tower 6.1.64-Unraid #1 SMP PREEMPT_DYNAMIC Wed Nov 29 12:48:16
> PST 2023 x86_64 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz GenuineIntel
> GNU/Linux

I think you are largely on your own, then, in terms of recovering
from this. I'd guess that the proprietary RAID implementation is not
power-fail safe....


> I was able to load the disk img in UFS Explorer on an Ubuntu machine
> and, as far as I can tell, all the data is in the img but it reports a
> handful of bad objects.
> https://github.com/phillipjf/xfs_debug/blob/main/ufs-explorer.png

AFAICT, that doesn't tell you that the data in the files is intact,
just that it can access them.

Keep in mind that whilst UFS-explorer reports ~321000 files in the
filesystem, the superblock state reported by phase 2 of xfs_repair
that there should be ~405000 files and directories in the filesystem
(icount - ifree):

....
sb_icount 407296, counted 272960
sb_ifree 2154, counted 1676
....

So, at minimum the damage to the inode btrees indicates that records
for ~130000 allocated files are missing from the inode btrees, and
that there should be ~80,000 more files present than UFS-explorer
found. IOWs, be careful trusting what UFS explorer tells you is
present - it looks like it may not have found a big chunk of the
data present that xfs_repair was trying to recover (and probably
going to attach to lost+found) when it crashed.

> > > The final output of xfs_repair is:
> > >
> > > Phase 5 - rebuild AG headers and trees...
> > >         - reset superblock...
> > > Phase 6 - check inode connectivity...
> > >         - resetting contents of realtime bitmap and summary inodes
> > >         - traversing filesystem ...
> > > rebuilding directory inode 12955326179
> > > Metadata corruption detected at 0x46fa05, inode 0x38983bd88 dinode
> >
> > Can you run 'gdb xfs_repair' and run the command 'l *0x46fa05' to
> > dump the line of code that the error was detected at? You probably
> > need the distro debug package for xfsprogs installed to do this.
> 
> At first try, doesn't look like gdb is available on Unraid and I think
> it would be more trouble than it's worth to get it set up.
> ---
> On the Ubuntu machine, I have
> # xfs_repair -V
> xfs_repair version 6.1.1
> 
> When running gdb on Ubuntu, I get:
> $ gdb xfs_repair
> (gdb) set args -f /media/sdl1.img
> (gdb) run
> ...
> Metadata corruption detected at 0x5555555afd95, inode 0x38983bd88 dinode
> 
> fatal error -- couldn't map inode 15192014216, err = 117
> ...
> (gdb) l *0x5555555afd95
> 0x5555555afd95 is in libxfs_dinode_verify (../libxfs/xfs_inode_buf.c:586).
> Downloading source file
> /build/xfsprogs-QBD5z8/xfsprogs-6.3.0/repair/../libxfs/xfs_inode_buf.c
> 581     ../libxfs/xfs_inode_buf.c: No such file or directory.


580         /* don't allow reflink/cowextsize if we don't have reflink */
581         if ((flags2 & (XFS_DIFLAG2_REFLINK | XFS_DIFLAG2_COWEXTSIZE)) &&
582              !xfs_has_reflink(mp))
583                 return __this_address;
584
585         /* only regular files get reflink */
586         if ((flags2 & XFS_DIFLAG2_REFLINK) && (mode & S_IFMT) != S_IFREG)
587                 return __this_address;
588

From the inode dump:

v3.flags2 = 0x8

Which is XFS_DIFLAG2_BIGTIME, so neither the REFLINK or COWEXTSIZE
flags are set. That means repair has changed the state of the inode
in memory if we've got the REFLINK flag set.

And there we are. After junking a number of blocks in the directory
because of bad magic numbers in phase 4, repair does this:

setting reflink flag on inode 15192014216

Which implies that there the multiple references to at least one of
the data blocks in the directory. This should not be allowed, so
the problem here is that repair is marking the directory inode as
having shared extents when this is not allowed. Hence we trip over
the inconsistency when rebuilding the directory and abort.

Ok, so that's a bug in the reflink state rebuild code - we should be
checking that inodes that shared extents have been found for are
regular files and if they aren't removing the block from the inode.

I'll look at this in a couple of weeks when I'm back from holidays
if our resident reflink expert (Darrick) doesn't get to it first....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Metadata corruption detected, fatal error -- couldn't map inode, err = 117

[Phillip Ferentinos]

On Wed, Dec 27, 2023 at 4:07 PM Dave Chinner <david@fromorbit.com> wrote:
>
> On Wed, Dec 27, 2023 at 02:22:47PM -0600, Phillip Ferentinos wrote:
> > On Tue, Dec 26, 2023 at 2:36 PM Dave Chinner <david@fromorbit.com> wrote:
> > >
> > > On Thu, Dec 21, 2023 at 08:05:43PM -0600, Phillip Ferentinos wrote:
> > > > Hello,
> > > >
> > > > Looking for opinions on recovering a filesystem that does not
> > > > successfully repair from xfs_repair.
> > >
> > > What version of xfs_repair?
> >
> > # xfs_repair -V
> > xfs_repair version 6.1.1
> >
> > > > On the first xfs_repair, I was
> > > > prompted to mount the disk to replay the log which I did successfully.
> > > > I created an image of the disk with ddrescue and am attempting to
> > > > recover the data. Unfortunately, I do not have a recent backup of this
> > > > disk.
> > >
> > > There is lots of random damage all over the filesystem. What caused
> > > this damage to occur? I generally only see this sort of widespread
> > > damage when RAID devices (hardware or software) go bad...
> > >
> > > Keep in mind that regardless of whether xfs_repair returns the
> > > filesystem to a consistent state, the data in the filesystem is
> > > still going to be badly corrupted. If you don't have backups, then
> > > there's a high probability of significant data loss here....
> >
> > My best guess is a power outage causing an unclean shutdown. I am
> > running Unraid:
> > # uname -a
> > Linux Tower 6.1.64-Unraid #1 SMP PREEMPT_DYNAMIC Wed Nov 29 12:48:16
> > PST 2023 x86_64 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz GenuineIntel
> > GNU/Linux
>
> I think you are largely on your own, then, in terms of recovering
> from this. I'd guess that the proprietary RAID implementation is not
> power-fail safe....
>
>
> > I was able to load the disk img in UFS Explorer on an Ubuntu machine
> > and, as far as I can tell, all the data is in the img but it reports a
> > handful of bad objects.
> > https://github.com/phillipjf/xfs_debug/blob/main/ufs-explorer.png
>
> AFAICT, that doesn't tell you that the data in the files is intact,
> just that it can access them.
>
> Keep in mind that whilst UFS-explorer reports ~321000 files in the
> filesystem, the superblock state reported by phase 2 of xfs_repair
> that there should be ~405000 files and directories in the filesystem
> (icount - ifree):
>
> ....
> sb_icount 407296, counted 272960
> sb_ifree 2154, counted 1676
> ....
>
> So, at minimum the damage to the inode btrees indicates that records
> for ~130000 allocated files are missing from the inode btrees, and
> that there should be ~80,000 more files present than UFS-explorer
> found. IOWs, be careful trusting what UFS explorer tells you is
> present - it looks like it may not have found a big chunk of the
> data present that xfs_repair was trying to recover (and probably
> going to attach to lost+found) when it crashed.
>

Thanks for the heads up! Once the new drives get here, I'll try moving
the data out via UFS explorer and merged back into the shares. Based
on what UFS explorer _is_ showing me, most of the important files are
recoverable. There are a handful of directories I'd expect there to be
thousands of files (specifically like the `previews` folder in the
screenshot) which are not critical.

> > > > The final output of xfs_repair is:
> > > >
> > > > Phase 5 - rebuild AG headers and trees...
> > > >         - reset superblock...
> > > > Phase 6 - check inode connectivity...
> > > >         - resetting contents of realtime bitmap and summary inodes
> > > >         - traversing filesystem ...
> > > > rebuilding directory inode 12955326179
> > > > Metadata corruption detected at 0x46fa05, inode 0x38983bd88 dinode
> > >
> > > Can you run 'gdb xfs_repair' and run the command 'l *0x46fa05' to
> > > dump the line of code that the error was detected at? You probably
> > > need the distro debug package for xfsprogs installed to do this.
> >
> > At first try, doesn't look like gdb is available on Unraid and I think
> > it would be more trouble than it's worth to get it set up.
> > ---
> > On the Ubuntu machine, I have
> > # xfs_repair -V
> > xfs_repair version 6.1.1
> >
> > When running gdb on Ubuntu, I get:
> > $ gdb xfs_repair
> > (gdb) set args -f /media/sdl1.img
> > (gdb) run
> > ...
> > Metadata corruption detected at 0x5555555afd95, inode 0x38983bd88 dinode
> >
> > fatal error -- couldn't map inode 15192014216, err = 117
> > ...
> > (gdb) l *0x5555555afd95
> > 0x5555555afd95 is in libxfs_dinode_verify (../libxfs/xfs_inode_buf.c:586).
> > Downloading source file
> > /build/xfsprogs-QBD5z8/xfsprogs-6.3.0/repair/../libxfs/xfs_inode_buf.c
> > 581     ../libxfs/xfs_inode_buf.c: No such file or directory.
>
>
> 580         /* don't allow reflink/cowextsize if we don't have reflink */
> 581         if ((flags2 & (XFS_DIFLAG2_REFLINK | XFS_DIFLAG2_COWEXTSIZE)) &&
> 582              !xfs_has_reflink(mp))
> 583                 return __this_address;
> 584
> 585         /* only regular files get reflink */
> 586         if ((flags2 & XFS_DIFLAG2_REFLINK) && (mode & S_IFMT) != S_IFREG)
> 587                 return __this_address;
> 588
>
> From the inode dump:
>
> v3.flags2 = 0x8
>
> Which is XFS_DIFLAG2_BIGTIME, so neither the REFLINK or COWEXTSIZE
> flags are set. That means repair has changed the state of the inode
> in memory if we've got the REFLINK flag set.
>
> And there we are. After junking a number of blocks in the directory
> because of bad magic numbers in phase 4, repair does this:
>
> setting reflink flag on inode 15192014216
>
> Which implies that there the multiple references to at least one of
> the data blocks in the directory. This should not be allowed, so
> the problem here is that repair is marking the directory inode as
> having shared extents when this is not allowed. Hence we trip over
> the inconsistency when rebuilding the directory and abort.
>
> Ok, so that's a bug in the reflink state rebuild code - we should be
> checking that inodes that shared extents have been found for are
> regular files and if they aren't removing the block from the inode.
>
> I'll look at this in a couple of weeks when I'm back from holidays
> if our resident reflink expert (Darrick) doesn't get to it first....
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com

Thanks again, especially for the quick response. I plan to keep the
original disk and the disk with the .img out of the array for some
time so when/if you do look back into this, I'm more than happy to
test anything or provide information (I like to think I'm fairly
technical, but I might need a bit more detailed instructions if it
gets complicated...).

Enjoy your holidays!
- Phillip Ferentinos