Re: XFS File system in trouble

From: Eric Sandeen <sandeen@sandeen.net>
To: Roger Willcocks <roger@filmlight.ltd.uk>,
	Leslie Rhorer <lrhorer@mygrande.net>
Cc: Brian Foster <bfoster@redhat.com>,
	Kris Rusocki <kszysiu@braxis.org>,
	"Rhorer, Leslie" <Leslie.Rhorer@level3.com>,
	"xfs@oss.sgi.com" <xfs@oss.sgi.com>
Subject: Re: XFS File system in trouble
Date: Sat, 15 Aug 2015 19:32:54 -0500	[thread overview]
Message-ID: <55CFDA36.1030700@sandeen.net> (raw)
In-Reply-To: <55CF8C4F.1080906@sandeen.net>

On 8/15/15 2:00 PM, Eric Sandeen wrote:
> On 8/15/15 1:48 PM, Eric Sandeen wrote:
>> On 8/15/15 7:28 AM, Roger Willcocks wrote:
>>> xfs_repair 3.2.1 runs cleanly.
>>>
>>> xfs_repair 3.1.1 complains about a load of stuff, including:
>>
>> I wouldn't expect v3.1.1 to work at all, because:
>>
>> # db/xfs_db -V
>> xfs_db version 4.2.0-rc1
>> # db/xfs_db /mnt/test2/leslie/md0.img -c version
>> versionnum [0xbdb4+0x8a] = V4,NLINK,DIRV2,ATTR,ALIGN,DALIGN,LOGV2,EXTFLG,SECTOR,MOREBITS,ATTR2,LAZYSBCOUNT,PROJID32BIT
>>
>> the filesystem has 32-bit project IDs, and:
>>
>> # git log --oneline | grep -i "projid32bit"
>> dd536e1 xfsprogs: Note projid32bit default change in mkfs.xfs manpage
>> 22bc10e xfsprogs: projid32bit handling
>>
>> # git describe --contains 22bc10e
>> v3.1.4~2
>>
>> that feature support didn't show up until v3.1.4.  Were you running a stock v3.1.1?
>>
>> Anyway, in my testing, up to v3.2.0, repair finds a lot of errors (and spends some
>> time looking for a proper superblock)
>>
>> v3.2.1 finds no errors.
>>
>> The errors stopped showing up as of:
>>
>> commit d085fb486f8b33304f2fdf704411313ffc8bcc0c
>> Author: Dave Chinner <dchinner@redhat.com>
>> Date:   Tue Jul 8 10:36:39 2014 +1000
>>
>>     repair: fix quota inode handling in secondary superblocks
> 
> I take that back, a bit:  "errors vs. no errors" was for xfs_repair -n.
> 
> a "full" repair with v3.1.4 shows only:
> 
> # repair/xfs_repair /mnt/test2/leslie/md0.img 
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
> non-null user quota inode field in superblock 5
> reset bad sb for ag 5
> non-null user quota inode field in superblock 10
> reset bad sb for ag 10
> non-null user quota inode field in superblock 18
> reset bad sb for ag 18
> non-null user quota inode field in superblock 23
> reset bad sb for ag 23
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan and clear agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
> ...
>         - agno = 29
> bad version number 0x3 on inode 124656869424
> bad version number 0x3 on inode 124656869424, resetting version number

And that does match the runtime error he was hitting:

It works for a while, because some of the directories already exist, but when it started to try to create directories, it started getting FS errors and I had to wind up rebooting.  After reboot, I get the following when I try to run the command from the directory again:

[  380.556635] XFS (md0): xfs_iread: validation failed for inode 124656869424 failed

FWIW, this inode is unused; when he hit it runtime, we were attempting to allocate it.

I suppose the bad version could be chalked up to a bit-flip from the bad memory that was in the box.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs