Re: Bug#387057: xfsprogs: repeated xfs_repair does not fix the filesystem

Re: Bug#387057: xfsprogs: repeated xfs_repair does not fix the filesystem

[Ferenc Wagner]

Nathan Scott <nathans@debian.org> writes:

> On Tue, Sep 12, 2006 at 12:30:08AM +0200, Ferenc Wagner wrote:
>> Package: xfsprogs
>> Version: 2.8.11-1
>> Severity: normal
>> 
>> I guess my problem is rooted in the 'well known' 2.6.17 error, or maybe
>> not.  Anyway, my experience under a current Sid system is that
>> xfs_repair does not fix my filesystem.  It does something, as the first
>> two runs produced slightly different outputs, but the further runs did
>> not.  I've got similar problems on two filesystems:
>
> Try moving aside the contents of lost+found after the first run,
> and see if the problems persist.

After renaming lost+found to l+f, xfs_repair didn't report any
errors:
=> Phase 1 - find and verify superblock...
=> Phase 2 - using internal log
=>         - zero log...
=>         - scan filesystem freespace and inode maps...
=>         - found root inode chunk
=> Phase 3 - for each AG...
=>         - scan and clear agi unlinked lists...
=>         - process known inodes and perform inode discovery...
=>         - agno = 0
=>         - agno = 1
=>         - agno = 2
=>         - agno = 3
=>         - agno = 4
=>         - agno = 5
=>         - agno = 6
=>         - agno = 7
=>         - process newly discovered inodes...
=> Phase 4 - check for duplicate blocks...
=>         - setting up duplicate extent list...
=>         - clear lost+found (if it exists) ...
=>         - check for inodes claiming duplicate blocks...
=>         - agno = 0
=>         - agno = 1
=>         - agno = 2
=>         - agno = 3
=>         - agno = 4
=>         - agno = 5
=>         - agno = 6
=>         - agno = 7
=> Phase 5 - rebuild AG headers and trees...
=>         - reset superblock...
=> Phase 6 - check inode connectivity...
=>         - resetting contents of realtime bitmap and summary inodes
=>         - ensuring existence of lost+found directory
=>         - traversing filesystem starting at / ... 
=>         - traversal finished ... 
=>         - traversing all unattached subtrees ... 
=>         - traversals finished ... 
=>         - moving disconnected inodes to lost+found ... 
=> Phase 7 - verify and correct link counts...
=> done

Still, xfs_check reported:
=> link count mismatch for inode 400254 (name ?), nlink 0, counted 2
=> link count mismatch for inode 4239409 (name ?), nlink 0, counted 2
=> link count mismatch for inode 8388736 (name ?), nlink 39, counted 38

Further runs of xfs_repair didn't bring any change.  On the root
filesystem the results are much the same, but xfs_check reports:
=> sb_ifree 3042, counted 3041

I read that xfs_check is being obsoleted in the future, but not sure
which program to trust.  Are my filesystems healthy or not?
-- 
Thanks,
Feri.
(Please Cc: me, I'm not subscribed to the xfs mailing list.)

RE: Bug#387057: xfsprogs: repeated xfs_repair does not fix the filesystem

[Barry Naujok]

 

> -----Original Message-----
> From: xfs-bounce@oss.sgi.com [mailto:xfs-bounce@oss.sgi.com] 
> On Behalf Of Ferenc Wagner
> Sent: Wednesday, 13 September 2006 7:46 PM
> To: Nathan Scott
> Cc: 387057@bugs.debian.org; xfs@oss.sgi.com
> Subject: Re: Bug#387057: xfsprogs: repeated xfs_repair does 
> not fix the filesystem
> 
> 
> Still, xfs_check reported:
> => link count mismatch for inode 400254 (name ?), nlink 0, counted 2
> => link count mismatch for inode 4239409 (name ?), nlink 0, counted 2
> => link count mismatch for inode 8388736 (name ?), nlink 39, 
> counted 38
> 
> Further runs of xfs_repair didn't bring any change.  On the root
> filesystem the results are much the same, but xfs_check reports:
> => sb_ifree 3042, counted 3041
> 
> I read that xfs_check is being obsoleted in the future, but not sure
> which program to trust.  Are my filesystems healthy or not?

This has been reported before, can you try running an older xfsprogs before
2.8.10 and see how that goes? I think with the dir2 fixes, the nlink stuff
might be a tad broken. I'll also look into it from my end.

Barry.

Re: Bug#387057: xfsprogs: repeated xfs_repair does not fix the filesystem

[Peter Palfrader]

On Thu, 14 Sep 2006, Barry Naujok wrote:

> > Still, xfs_check reported:
> > => link count mismatch for inode 400254 (name ?), nlink 0, counted 2
> > => link count mismatch for inode 4239409 (name ?), nlink 0, counted 2
> > => link count mismatch for inode 8388736 (name ?), nlink 39, 
> > counted 38
> > 
> > Further runs of xfs_repair didn't bring any change.  On the root
> > filesystem the results are much the same, but xfs_check reports:
> > => sb_ifree 3042, counted 3041
> > 
> > I read that xfs_check is being obsoleted in the future, but not sure
> > which program to trust.  Are my filesystems healthy or not?
> 
> This has been reported before, can you try running an older xfsprogs before
> 2.8.10 and see how that goes? I think with the dir2 fixes, the nlink stuff
> might be a tad broken. I'll also look into it from my end.


I have just had a similar problem.  xfs_repair 2.8.11 did its stuff,
moving a few items into lost+found in the process.  After mounting,
trying to rmdir the directories under lost+found the filesystem was shut
down immediately again:

| [  434.590246] Ending clean XFS mount for filesystem: md0
| [  447.677811] xfs_inotobp: xfs_imap()  returned an error 22 on md0.  Returning error.
| [  447.678090] xfs_iunlink_remove: xfs_inotobp()  returned an error 22 on md0.  Returning error.
| [  447.678383] xfs_inactive:    xfs_ifree() returned an error = 22 on md0
| [  447.678605] xfs_force_shutdown(md0,0x1) called from line 1763 of file fs/xfs/xfs_vnodeops.c.  Return address = 0xffffffff803ca50a
| [  447.678986] Filesystem "md0": I/O Error Detected.  Shutting down filesystem: md0

Downgrading xfsprogs to 2.6.20 solves the issue:

} [...]
} Phase 7 - verify and correct link counts...
} resetting inode 789921 nlinks from 0 to 2
} resetting inode 4290022 nlinks from 0 to 2
} resetting inode 4290023 nlinks from 0 to 2
} resetting inode 4290024 nlinks from 0 to 2
} resetting inode 4290025 nlinks from 0 to 2
} [..]
} resetting inode 59501189 nlinks from 0 to 2
} resetting inode 63698112 nlinks from 0 to 2
} resetting inode 63698118 nlinks from 0 to 2
} done

Now my filesystem appears functional again.  If you need any more info,
I still have an image of the filesystem prior to the treatment with
2.6.20.

Cheers,
Peter
-- 
                           |  .''`.  ** Debian GNU/Linux **
      Peter Palfrader      | : :' :      The  universal
 http://www.palfrader.org/ | `. `'      Operating System
                           |   `-    http://www.debian.org/

Re: Bug#387057: xfsprogs: repeated xfs_repair does not fix the filesystem

[Ferenc Wagner]

"Barry Naujok" <bnaujok@melbourne.sgi.com> writes:

>> -----Original Message-----
>> From: xfs-bounce@oss.sgi.com [mailto:xfs-bounce@oss.sgi.com] 
>> On Behalf Of Ferenc Wagner
>> Sent: Wednesday, 13 September 2006 7:46 PM
>> To: Nathan Scott
>> Cc: 387057@bugs.debian.org; xfs@oss.sgi.com
>> Subject: Re: Bug#387057: xfsprogs: repeated xfs_repair does 
>> not fix the filesystem
>> 
>> 
>> Still, xfs_check reported:
>> => link count mismatch for inode 400254 (name ?), nlink 0, counted 2
>> => link count mismatch for inode 4239409 (name ?), nlink 0, counted 2
>> => link count mismatch for inode 8388736 (name ?), nlink 39, 
>> counted 38
>> 
>> Further runs of xfs_repair didn't bring any change.  On the root
>> filesystem the results are much the same, but xfs_check reports:
>> => sb_ifree 3042, counted 3041
>
> This has been reported before, can you try running an older xfsprogs before
> 2.8.10 and see how that goes? I think with the dir2 fixes, the nlink stuff
> might be a tad broken. I'll also look into it from my end.

Thanks.  I tried xfsprogs 2.8.4 with the following result:

# xfs_repair /dev/main/usr
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - clear lost+found (if it exists) ...
        - clearing existing "lost+found" inode
        - marking entry "lost+found" to be deleted
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - ensuring existence of lost+found directory
        - traversing filesystem starting at / ...
rebuilding directory inode 128
        - traversal finished ...
        - traversing all unattached subtrees ...
        - traversals finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
resetting inode 400254 nlinks from 0 to 2
resetting inode 4239409 nlinks from 0 to 2
resetting inode 8388736 nlinks from 39 to 38
done
# xfs_check /dev/main/usr
#

That is, my /usr partition looks fixed.  On the other hand:

# xfs_repair -d /dev/main/root
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - clear lost+found (if it exists) ...
        - clearing existing "lost+found" inode
        - marking entry "lost+found" to be deleted
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - ensuring existence of lost+found directory
        - traversing filesystem starting at / ... 
rebuilding directory inode 128
        - traversal finished ... 
        - traversing all unattached subtrees ... 
        - traversals finished ... 
        - moving disconnected inodes to lost+found ... 
Phase 7 - verify and correct link counts...
done
# xfs_check /dev/main/root
sb_ifree 3044, counted 3043
#

That is, almost like before, but with incremented numbers (there was
filesystem activity since then).

So, there is some progress, but part of the problem remained.
-- 
Thanks,
Feri.