xfs_repair hangs at "process newly discovered inodes..."

xfs_repair hangs at "process newly discovered inodes..."

[iamdooser]

Hello,

I'm not sure this is the correct forum; if not I'd appreciate guidance.

I have a Unraid machine that experienced an unmountable file system on 
an array disc. Running:

xfs_repair -nv /dev/md3

works, however when running

xfs_repair -v /dev/md3

it stops at "process newly discovered inodes..." and doesn't seem to be 
doing anything.

I've asked in the unraid forum and they've directed me to the xfs 
mailing list.

Appreciate any help.

Re: xfs_repair hangs at "process newly discovered inodes..."

[Eric Sandeen]

On 11/17/22 12:40 PM, iamdooser wrote:
> Hello,
> 
> I'm not sure this is the correct forum; if not I'd appreciate guidance.
> 
> I have a Unraid machine that experienced an unmountable file system on an array disc. Running:
> 
> xfs_repair -nv /dev/md3

Did that find errors?

> works, however when running
> 
> xfs_repair -v /dev/md3
> 
> it stops at "process newly discovered inodes..." and doesn't seem to be doing anything.
> 
> I've asked in the unraid forum and they've directed me to the xfs mailing list.
> 
> Appreciate any help.

Please tell us the version of xfsprogs you're using, and provide the full xfs_repair
output (with and without -n).

If it really looks like a bug, and not simply a slow repair, providing an xfs_metadump
may help us evaluate the problem further.

-Eric

Re: xfs_repair hangs at "process newly discovered inodes..."

[iamdooser]

Thank you for responding.

Yes that found errors, although I'm not accustomed to interpreting the 
output.

xfs_repair version 5.18.0

The output of xfs_repair -nv was quite large, as was the 
xfs_metadump...not sure that's indicative of something, but I've 
uploaded them here:
https://drive.google.com/drive/folders/1OyQOZNsTS1w1Utx1ZfQEH-bS_Cyj8-F2?usp=sharing


There doesn't seem to be much activity once it hangs at "process newly 
discovered inodes..." so it doesn't seem like just a slow repair. 
Desipte there being no sign of activity, I've let it run for 24+ hours 
and saw no changes..


On 11/17/22 13:48, Eric Sandeen wrote:
> On 11/17/22 12:40 PM, iamdooser wrote:
>> Hello,
>>
>> I'm not sure this is the correct forum; if not I'd appreciate guidance.
>>
>> I have a Unraid machine that experienced an unmountable file system on an array disc. Running:
>>
>> xfs_repair -nv /dev/md3
> 
> Did that find errors?
> 
>> works, however when running
>>
>> xfs_repair -v /dev/md3
>>
>> it stops at "process newly discovered inodes..." and doesn't seem to be doing anything.
>>
>> I've asked in the unraid forum and they've directed me to the xfs mailing list.
>>
>> Appreciate any help.
> 
> Please tell us the version of xfsprogs you're using, and provide the full xfs_repair
> output (with and without -n).
> 
> If it really looks like a bug, and not simply a slow repair, providing an xfs_metadump
> may help us evaluate the problem further.
> 
> -Eric
> 

Re: xfs_repair hangs at "process newly discovered inodes..."

[Carlos Maiolino]

Hi.


On Sat, Nov 19, 2022 at 12:24:18PM -0500, iamdooser wrote:
> Thank you for responding.
> 
> Yes that found errors, although I'm not accustomed to interpreting the
> output.
> 
> xfs_repair version 5.18.0
> 
> The output of xfs_repair -nv was quite large, as was the
> xfs_metadump...not sure that's indicative of something, but I've
> uploaded them here:
> https://drive.google.com/drive/folders/1OyQOZNsTS1w1Utx1ZfQEH-bS_Cyj8-F2?usp=sharing
> 
> 
> There doesn't seem to be much activity once it hangs at "process newly
> discovered inodes..." so it doesn't seem like just a slow repair.
> Desipte there being no sign of activity, I've let it run for 24+ hours
> and saw no changes..
> 

Before anything else, could you please try to run the latest xfsprogs from:

https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/log/?h=master

A quick test in my laptop using the metadump you provided, finished the repair
in about 5 mintues:

Maximum metadata LSN (2138201505:-135558109) is ahead of log (96:0).
Format log to cycle 2138201508.

        XFS_REPAIR Summary    Mon Nov 21 17:04:44 2022

Phase		Start		End		Duration
Phase 1:	11/21 16:59:36	11/21 16:59:36	
Phase 2:	11/21 16:59:36	11/21 16:59:37	1 second
Phase 3:	11/21 16:59:37	11/21 17:03:47	4 minutes, 10 seconds
Phase 4:	11/21 17:03:47	11/21 17:04:06	19 seconds
Phase 5:	11/21 17:04:06	11/21 17:04:07	1 second
Phase 6:	11/21 17:04:07	11/21 17:04:38	31 seconds
Phase 7:	11/21 17:04:38	11/21 17:04:38	

Total run time: 5 minutes, 2 seconds
done

Also, feel free to compress any file you need to share with us :)

Cheers.

> 
> On 11/17/22 13:48, Eric Sandeen wrote:
> > On 11/17/22 12:40 PM, iamdooser wrote:
> >> Hello,
> >>
> >> I'm not sure this is the correct forum; if not I'd appreciate guidance.
> >>
> >> I have a Unraid machine that experienced an unmountable file system on an array disc. Running:
> >>
> >> xfs_repair -nv /dev/md3
> >
> > Did that find errors?
> >
> >> works, however when running
> >>
> >> xfs_repair -v /dev/md3
> >>
> >> it stops at "process newly discovered inodes..." and doesn't seem to be doing anything.
> >>
> >> I've asked in the unraid forum and they've directed me to the xfs mailing list.
> >>
> >> Appreciate any help.
> >
> > Please tell us the version of xfsprogs you're using, and provide the full xfs_repair
> > output (with and without -n).
> >
> > If it really looks like a bug, and not simply a slow repair, providing an xfs_metadump
> > may help us evaluate the problem further.
> >
> > -Eric
> >

-- 
Carlos Maiolino

Re: xfs_repair hangs at "process newly discovered inodes..."

[Eric Sandeen]

On 11/21/22 10:14 AM, Carlos Maiolino wrote:
> Hi.
> 
> 
> On Sat, Nov 19, 2022 at 12:24:18PM -0500, iamdooser wrote:
>> Thank you for responding.
>>
>> Yes that found errors, although I'm not accustomed to interpreting the
>> output.
>>
>> xfs_repair version 5.18.0
>>
>> The output of xfs_repair -nv was quite large, as was the
>> xfs_metadump...not sure that's indicative of something, but I've
>> uploaded them here:
>> https://drive.google.com/drive/folders/1OyQOZNsTS1w1Utx1ZfQEH-bS_Cyj8-F2?usp=sharing
>>
>>
>> There doesn't seem to be much activity once it hangs at "process newly
>> discovered inodes..." so it doesn't seem like just a slow repair.
>> Desipte there being no sign of activity, I've let it run for 24+ hours
>> and saw no changes..
>>
> 
> Before anything else, could you please try to run the latest xfsprogs from:
> 
> https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/log/?h=master
> 
> A quick test in my laptop using the metadump you provided, finished the repair
> in about 5 mintues:
> 
> Maximum metadata LSN (2138201505:-135558109) is ahead of log (96:0).
> Format log to cycle 2138201508.
> 
>         XFS_REPAIR Summary    Mon Nov 21 17:04:44 2022
> 
> Phase		Start		End		Duration
> Phase 1:	11/21 16:59:36	11/21 16:59:36	
> Phase 2:	11/21 16:59:36	11/21 16:59:37	1 second
> Phase 3:	11/21 16:59:37	11/21 17:03:47	4 minutes, 10 seconds
> Phase 4:	11/21 17:03:47	11/21 17:04:06	19 seconds
> Phase 5:	11/21 17:04:06	11/21 17:04:07	1 second
> Phase 6:	11/21 17:04:07	11/21 17:04:38	31 seconds
> Phase 7:	11/21 17:04:38	11/21 17:04:38	
> 
> Total run time: 5 minutes, 2 seconds
> done
> 
> Also, feel free to compress any file you need to share with us :)

Yup. All those files should compress quite well.

So, the "-nv" output shows many, many errors.  While xfs_repair should eventually
make the filesystem metadata consistent again, that's not the same thing as data
recovery.

What happened to this filesystem and/or its storage that prompted the need for
repair?

-Eric

Re: xfs_repair hangs at "process newly discovered inodes..."

[Dave Chinner]

On Sat, Nov 19, 2022 at 12:24:18PM -0500, iamdooser wrote:
> Thank you for responding.
> 
> Yes that found errors, although I'm not accustomed to interpreting the
> output.
> 
> xfs_repair version 5.18.0
> 
> The output of xfs_repair -nv was quite large, as was the xfs_metadump...not
> sure that's indicative of something, but I've uploaded them here:
> https://drive.google.com/drive/folders/1OyQOZNsTS1w1Utx1ZfQEH-bS_Cyj8-F2?usp=sharing

Ok....

According to the the "-nv" output, you a clean log and widespread
per-AG btree corruptions and inconsistencies. Free inodes not found
in the finobt, free space only found in on free sapce btree, records
in btrees out of order, multiply-claimed blocks (cross linked files
and cross linked free space!), etc.

Every AG shows the same corruption pattern - I've never seen a
filesystem with a clean log in this state before. This sort of
widespread lack of consistency in btree structures isn't a result of
an isolated storage media or I/O error - something major has
happened here.

The first thing I have to ask: did you zero the log with xfs_repair
because you couldn't repair it and then take these repair output
dumps? This *smells* zeroing the log with xfs_repair and throwing
away all the metadata in the log after removing a bunch of files
and the system crashing immediately afterwards. Log recovery in that
case would have made the btrees and inode states mostly
consistent...

Can you please explain how the filesystem got into this state in the
first place? What storage you have, what kernel you are running,
what distro/appliance this filesystem is hosted on, what operations
were being performed when it all went wrong, etc? We really need to
know how the fs got into this state so that we can determine if
other users are at risk of this sort of thing...

> There doesn't seem to be much activity once it hangs at "process newly
> discovered inodes..." so it doesn't seem like just a slow repair. Desipte
> there being no sign of activity, I've let it run for 24+ hours and saw no
> changes..

Use "-t 300" for xfs_repair to output a progress report every 5
minutes. Likely the operation is slow because it is IO bound moving
one inode at a time to lost+found...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com