Corrupted file system

Corrupted file system

[Diane Trout]

Hi,

I had a raid failure at work that ended up corrupting an xfs filesystem the 
tail of the xfs_repair command looks like the below. I was able to generate a 
metadata dump but is there a point to making it available? 

It does crash repeatedly at the same place

(I'm not subscribed to the list, so could you reply directly as well?)

disconnected inode 15276, moving to lost+found
disconnected inode 15277, moving to lost+found
disconnected inode 15278, moving to lost+found
disconnected inode 15279, moving to lost+found
disconnected inode 15280, moving to lost+found
disconnected inode 15281, moving to lost+found
disconnected inode 15282, moving to lost+found
disconnected inode 15283, moving to lost+found
disconnected inode 15284, moving to lost+found
disconnected inode 15286, moving to lost+found
disconnected inode 15287, moving to lost+found
disconnected inode 15288, moving to lost+found
disconnected inode 15289, moving to lost+found
disconnected inode 15290, moving to lost+found
disconnected inode 15291, moving to lost+found
disconnected inode 15292, moving to lost+found
disconnected inode 15293, moving to lost+found
disconnected inode 15294, moving to lost+found
disconnected inode 15295, moving to lost+found
disconnected inode 15360, moving to lost+found
corrupt dinode 15360, extent total = 1, nblocks = 0.  This is a bug.
Please capture the filesystem metadata with xfs_metadump and
report it to xfs@oss.sgi.com.
cache_node_purge: refcount was 1, not zero (node=0x7f369883e6f0)

fatal error -- 117 - couldn't iget disconnected inode


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Corrupted file system

[Sean Caron]

[-- Attachment #1.1: Type: text/plain, Size: 2361 bytes --]

Hi Diane,

Probably best to reformat and restore from a backup at this point. I'm
notorious for my views on xfs_repair but most folks here will agree that in
the case where there has been a major underlying array failure, there is
not much it can do to help. Better to just mount ro,noreplaylog and get
what you can in these sorts of scenarios, IMO.

"Did you try the latest copy of xfs_repair"? Sometimes it will get further
or not crash, but it likely will still maul whatever's left of your
filesystem.

Best of luck,

Sean


On Tue, Sep 23, 2014 at 6:02 PM, Diane Trout <diane@caltech.edu> wrote:

> Hi,
>
> I had a raid failure at work that ended up corrupting an xfs filesystem the
> tail of the xfs_repair command looks like the below. I was able to
> generate a
> metadata dump but is there a point to making it available?
>
> It does crash repeatedly at the same place
>
> (I'm not subscribed to the list, so could you reply directly as well?)
>
> disconnected inode 15276, moving to lost+found
> disconnected inode 15277, moving to lost+found
> disconnected inode 15278, moving to lost+found
> disconnected inode 15279, moving to lost+found
> disconnected inode 15280, moving to lost+found
> disconnected inode 15281, moving to lost+found
> disconnected inode 15282, moving to lost+found
> disconnected inode 15283, moving to lost+found
> disconnected inode 15284, moving to lost+found
> disconnected inode 15286, moving to lost+found
> disconnected inode 15287, moving to lost+found
> disconnected inode 15288, moving to lost+found
> disconnected inode 15289, moving to lost+found
> disconnected inode 15290, moving to lost+found
> disconnected inode 15291, moving to lost+found
> disconnected inode 15292, moving to lost+found
> disconnected inode 15293, moving to lost+found
> disconnected inode 15294, moving to lost+found
> disconnected inode 15295, moving to lost+found
> disconnected inode 15360, moving to lost+found
> corrupt dinode 15360, extent total = 1, nblocks = 0.  This is a bug.
> Please capture the filesystem metadata with xfs_metadump and
> report it to xfs@oss.sgi.com.
> cache_node_purge: refcount was 1, not zero (node=0x7f369883e6f0)
>
> fatal error -- 117 - couldn't iget disconnected inode
>
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>

[-- Attachment #1.2: Type: text/html, Size: 3063 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Corrupted file system

[Diane Trout]

Hello,

Yes, I had my doubts about that there's anything reasonable one could do after 
part of the file system was randomized. I'll trying updating to xfsprogs 3.2.1 
and see if that does any better.

Thank you for the advice.

Diane

On Tuesday, September 23, 2014 18:12:22 Sean Caron wrote:
> Hi Diane,
> 
> Probably best to reformat and restore from a backup at this point. I'm
> notorious for my views on xfs_repair but most folks here will agree that in
> the case where there has been a major underlying array failure, there is
> not much it can do to help. Better to just mount ro,noreplaylog and get
> what you can in these sorts of scenarios, IMO.
> 
> "Did you try the latest copy of xfs_repair"? Sometimes it will get further
> or not crash, but it likely will still maul whatever's left of your
> filesystem.
> 
> Best of luck,
> 
> Sean
> 
> On Tue, Sep 23, 2014 at 6:02 PM, Diane Trout <diane@caltech.edu> wrote:
> > Hi,
> > 
> > I had a raid failure at work that ended up corrupting an xfs filesystem
> > the
> > tail of the xfs_repair command looks like the below. I was able to
> > generate a
> > metadata dump but is there a point to making it available?
> > 
> > It does crash repeatedly at the same place
> > 
> > (I'm not subscribed to the list, so could you reply directly as well?)
> > 
> > disconnected inode 15276, moving to lost+found
> > disconnected inode 15277, moving to lost+found
> > disconnected inode 15278, moving to lost+found
> > disconnected inode 15279, moving to lost+found
> > disconnected inode 15280, moving to lost+found
> > disconnected inode 15281, moving to lost+found
> > disconnected inode 15282, moving to lost+found
> > disconnected inode 15283, moving to lost+found
> > disconnected inode 15284, moving to lost+found
> > disconnected inode 15286, moving to lost+found
> > disconnected inode 15287, moving to lost+found
> > disconnected inode 15288, moving to lost+found
> > disconnected inode 15289, moving to lost+found
> > disconnected inode 15290, moving to lost+found
> > disconnected inode 15291, moving to lost+found
> > disconnected inode 15292, moving to lost+found
> > disconnected inode 15293, moving to lost+found
> > disconnected inode 15294, moving to lost+found
> > disconnected inode 15295, moving to lost+found
> > disconnected inode 15360, moving to lost+found
> > corrupt dinode 15360, extent total = 1, nblocks = 0.  This is a bug.
> > Please capture the filesystem metadata with xfs_metadump and
> > report it to xfs@oss.sgi.com.
> > cache_node_purge: refcount was 1, not zero (node=0x7f369883e6f0)
> > 
> > fatal error -- 117 - couldn't iget disconnected inode
> > 
> > 
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Corrupted file system

[Eric Sandeen]

On 9/23/14 5:29 PM, Diane Trout wrote:
> Hello,
> 
> Yes, I had my doubts about that there's anything reasonable one could do after 
> part of the file system was randomized. I'll trying updating to xfsprogs 3.2.1 
> and see if that does any better.

Yes - you didn't mention which version you used, but I'd definitely try something
current.  The bug rings a bell, maybe fixed now.

To be safe, you can always make an xfs_metadump image (use -o to not obfuscate
filenames for convenience), xfs_mdrestore that, and point repair at that fs image.
Then you'll see what it *would* do before you *do* do.  ;)

But yeah, depending on the damage done to the storage, you may be in trouble
that repair can't help with.  Depends on the failure, though.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Corrupted file system

[Emmanuel Florac]

Le Tue, 23 Sep 2014 15:02:57 -0700
Diane Trout <diane@caltech.edu> écrivait:

> I had a raid failure at work that ended up corrupting an xfs
> filesystem the tail of the xfs_repair command looks like the below. I
> was able to generate a metadata dump but is there a point to making
> it available? 
> 
> It does crash repeatedly at the same place

Did you try the very latest xfs_repair?

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Corrupted file system

[Diane Trout]

On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote:
> Le Tue, 23 Sep 2014 15:02:57 -0700
> 
> Diane Trout <diane@caltech.edu> écrivait:
> > I had a raid failure at work that ended up corrupting an xfs
> > filesystem the tail of the xfs_repair command looks like the below. I
> > was able to generate a metadata dump but is there a point to making
> > it available?
> > 
> > It does crash repeatedly at the same place
> 
> Did you try the very latest xfs_repair?


I grabbed the one out of debian unstable (3.2.1), verified that it was your 
latest release and used that. I didn't try building the version in git.

It did get much further than version 3.1.7+b1 (debian stable)

Diane

Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it in 
ino 2051.
bad hash table for directory inode 256 (no data entry): rebuilding
rebuilding directory inode 256
7f70c79c8700: Badness in key lookup (length)
bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes)
Metadata corruption detected at block 0x4000070/0x1000
xfs_da_do_buf(2): XFS_CORRUPTION_ERROR

fatal error -- can't read block 8388608 for directory inode 259, error 117


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Corrupted file system

[Eric Sandeen]

If you want to make a metadata image, compress it, and send it my way I'd be glad
to look at it - I've been doing some repair work lately.

-Eric

On 9/24/14 3:38 PM, Diane Trout wrote:
> On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote:
>> Le Tue, 23 Sep 2014 15:02:57 -0700
>>
>> Diane Trout <diane@caltech.edu> écrivait:
>>> I had a raid failure at work that ended up corrupting an xfs
>>> filesystem the tail of the xfs_repair command looks like the below. I
>>> was able to generate a metadata dump but is there a point to making
>>> it available?
>>>
>>> It does crash repeatedly at the same place
>>
>> Did you try the very latest xfs_repair?
> 
> 
> I grabbed the one out of debian unstable (3.2.1), verified that it was your 
> latest release and used that. I didn't try building the version in git.
> 
> It did get much further than version 3.1.7+b1 (debian stable)
> 
> Diane
> 
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
> entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it in 
> ino 2051.
> bad hash table for directory inode 256 (no data entry): rebuilding
> rebuilding directory inode 256
> 7f70c79c8700: Badness in key lookup (length)
> bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes)
> Metadata corruption detected at block 0x4000070/0x1000
> xfs_da_do_buf(2): XFS_CORRUPTION_ERROR
> 
> fatal error -- can't read block 8388608 for directory inode 259, error 117
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Corrupted file system

[Diane Trout]

Huh.

I'd managed to make a metadump using the 3.1.7 version of the tools. Then I 
tried to run xfs_repair on the partition, and now xfs_metadump is segfaulting 
on the partition.

Would the older metadata dump be useful?

I'd need to rebuild xfsprogs before the stack trace would be useful... debian 
likes to strip the symbols and they didn't make a -dbg package.

Diane

On Wednesday, September 24, 2014 15:57:53 Eric Sandeen wrote:
> If you want to make a metadata image, compress it, and send it my way I'd be
> glad to look at it - I've been doing some repair work lately.
> 
> -Eric
> 
> On 9/24/14 3:38 PM, Diane Trout wrote:
> > On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote:
> >> Le Tue, 23 Sep 2014 15:02:57 -0700
> >> 
> >> Diane Trout <diane@caltech.edu> écrivait:
> >>> I had a raid failure at work that ended up corrupting an xfs
> >>> filesystem the tail of the xfs_repair command looks like the below. I
> >>> was able to generate a metadata dump but is there a point to making
> >>> it available?
> >>> 
> >>> It does crash repeatedly at the same place
> >> 
> >> Did you try the very latest xfs_repair?
> > 
> > I grabbed the one out of debian unstable (3.2.1), verified that it was
> > your
> > latest release and used that. I didn't try building the version in git.
> > 
> > It did get much further than version 3.1.7+b1 (debian stable)
> > 
> > Diane
> > 
> > Phase 6 - check inode connectivity...
> > 
> >         - resetting contents of realtime bitmap and summary inodes
> >         - traversing filesystem ...
> > 
> > entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it
> > in ino 2051.
> > bad hash table for directory inode 256 (no data entry): rebuilding
> > rebuilding directory inode 256
> > 7f70c79c8700: Badness in key lookup (length)
> > bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes)
> > Metadata corruption detected at block 0x4000070/0x1000
> > xfs_da_do_buf(2): XFS_CORRUPTION_ERROR
> > 
> > fatal error -- can't read block 8388608 for directory inode 259, error 117
> > 
> > 
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Corrupted file system

[Eric Sandeen]

On 9/24/14 7:25 PM, Diane Trout wrote:
> Huh.
> 
> I'd managed to make a metadump using the 3.1.7 version of the tools. Then I 
> tried to run xfs_repair on the partition, and now xfs_metadump is segfaulting 
> on the partition.
> 
> Would the older metadata dump be useful?

sure.  Compress it & send me a private note with it attached (or with a location
if it's too big to email)

-Eric

> I'd need to rebuild xfsprogs before the stack trace would be useful... debian 
> likes to strip the symbols and they didn't make a -dbg package.
> 
> Diane
> 
> On Wednesday, September 24, 2014 15:57:53 Eric Sandeen wrote:
>> If you want to make a metadata image, compress it, and send it my way I'd be
>> glad to look at it - I've been doing some repair work lately.
>>
>> -Eric
>>
>> On 9/24/14 3:38 PM, Diane Trout wrote:
>>> On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote:
>>>> Le Tue, 23 Sep 2014 15:02:57 -0700
>>>>
>>>> Diane Trout <diane@caltech.edu> écrivait:
>>>>> I had a raid failure at work that ended up corrupting an xfs
>>>>> filesystem the tail of the xfs_repair command looks like the below. I
>>>>> was able to generate a metadata dump but is there a point to making
>>>>> it available?
>>>>>
>>>>> It does crash repeatedly at the same place
>>>>
>>>> Did you try the very latest xfs_repair?
>>>
>>> I grabbed the one out of debian unstable (3.2.1), verified that it was
>>> your
>>> latest release and used that. I didn't try building the version in git.
>>>
>>> It did get much further than version 3.1.7+b1 (debian stable)
>>>
>>> Diane
>>>
>>> Phase 6 - check inode connectivity...
>>>
>>>         - resetting contents of realtime bitmap and summary inodes
>>>         - traversing filesystem ...
>>>
>>> entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it
>>> in ino 2051.
>>> bad hash table for directory inode 256 (no data entry): rebuilding
>>> rebuilding directory inode 256
>>> 7f70c79c8700: Badness in key lookup (length)
>>> bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes)
>>> Metadata corruption detected at block 0x4000070/0x1000
>>> xfs_da_do_buf(2): XFS_CORRUPTION_ERROR
>>>
>>> fatal error -- can't read block 8388608 for directory inode 259, error 117
>>>
>>>
>>> _______________________________________________
>>> xfs mailing list
>>> xfs@oss.sgi.com
>>> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs