* Corrupted file system @ 2014-09-23 22:02 Diane Trout 2014-09-23 22:12 ` Sean Caron 2014-09-24 10:08 ` Emmanuel Florac 0 siblings, 2 replies; 9+ messages in thread From: Diane Trout @ 2014-09-23 22:02 UTC (permalink / raw) To: xfs Hi, I had a raid failure at work that ended up corrupting an xfs filesystem the tail of the xfs_repair command looks like the below. I was able to generate a metadata dump but is there a point to making it available? It does crash repeatedly at the same place (I'm not subscribed to the list, so could you reply directly as well?) disconnected inode 15276, moving to lost+found disconnected inode 15277, moving to lost+found disconnected inode 15278, moving to lost+found disconnected inode 15279, moving to lost+found disconnected inode 15280, moving to lost+found disconnected inode 15281, moving to lost+found disconnected inode 15282, moving to lost+found disconnected inode 15283, moving to lost+found disconnected inode 15284, moving to lost+found disconnected inode 15286, moving to lost+found disconnected inode 15287, moving to lost+found disconnected inode 15288, moving to lost+found disconnected inode 15289, moving to lost+found disconnected inode 15290, moving to lost+found disconnected inode 15291, moving to lost+found disconnected inode 15292, moving to lost+found disconnected inode 15293, moving to lost+found disconnected inode 15294, moving to lost+found disconnected inode 15295, moving to lost+found disconnected inode 15360, moving to lost+found corrupt dinode 15360, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to xfs@oss.sgi.com. cache_node_purge: refcount was 1, not zero (node=0x7f369883e6f0) fatal error -- 117 - couldn't iget disconnected inode _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Corrupted file system 2014-09-23 22:02 Corrupted file system Diane Trout @ 2014-09-23 22:12 ` Sean Caron 2014-09-23 22:29 ` Diane Trout 2014-09-24 10:08 ` Emmanuel Florac 1 sibling, 1 reply; 9+ messages in thread From: Sean Caron @ 2014-09-23 22:12 UTC (permalink / raw) To: Diane Trout, Sean Caron; +Cc: xfs@oss.sgi.com [-- Attachment #1.1: Type: text/plain, Size: 2361 bytes --] Hi Diane, Probably best to reformat and restore from a backup at this point. I'm notorious for my views on xfs_repair but most folks here will agree that in the case where there has been a major underlying array failure, there is not much it can do to help. Better to just mount ro,noreplaylog and get what you can in these sorts of scenarios, IMO. "Did you try the latest copy of xfs_repair"? Sometimes it will get further or not crash, but it likely will still maul whatever's left of your filesystem. Best of luck, Sean On Tue, Sep 23, 2014 at 6:02 PM, Diane Trout <diane@caltech.edu> wrote: > Hi, > > I had a raid failure at work that ended up corrupting an xfs filesystem the > tail of the xfs_repair command looks like the below. I was able to > generate a > metadata dump but is there a point to making it available? > > It does crash repeatedly at the same place > > (I'm not subscribed to the list, so could you reply directly as well?) > > disconnected inode 15276, moving to lost+found > disconnected inode 15277, moving to lost+found > disconnected inode 15278, moving to lost+found > disconnected inode 15279, moving to lost+found > disconnected inode 15280, moving to lost+found > disconnected inode 15281, moving to lost+found > disconnected inode 15282, moving to lost+found > disconnected inode 15283, moving to lost+found > disconnected inode 15284, moving to lost+found > disconnected inode 15286, moving to lost+found > disconnected inode 15287, moving to lost+found > disconnected inode 15288, moving to lost+found > disconnected inode 15289, moving to lost+found > disconnected inode 15290, moving to lost+found > disconnected inode 15291, moving to lost+found > disconnected inode 15292, moving to lost+found > disconnected inode 15293, moving to lost+found > disconnected inode 15294, moving to lost+found > disconnected inode 15295, moving to lost+found > disconnected inode 15360, moving to lost+found > corrupt dinode 15360, extent total = 1, nblocks = 0. This is a bug. > Please capture the filesystem metadata with xfs_metadump and > report it to xfs@oss.sgi.com. > cache_node_purge: refcount was 1, not zero (node=0x7f369883e6f0) > > fatal error -- 117 - couldn't iget disconnected inode > > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs > [-- Attachment #1.2: Type: text/html, Size: 3063 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Corrupted file system 2014-09-23 22:12 ` Sean Caron @ 2014-09-23 22:29 ` Diane Trout 2014-09-23 23:12 ` Eric Sandeen 0 siblings, 1 reply; 9+ messages in thread From: Diane Trout @ 2014-09-23 22:29 UTC (permalink / raw) To: Sean Caron; +Cc: xfs@oss.sgi.com Hello, Yes, I had my doubts about that there's anything reasonable one could do after part of the file system was randomized. I'll trying updating to xfsprogs 3.2.1 and see if that does any better. Thank you for the advice. Diane On Tuesday, September 23, 2014 18:12:22 Sean Caron wrote: > Hi Diane, > > Probably best to reformat and restore from a backup at this point. I'm > notorious for my views on xfs_repair but most folks here will agree that in > the case where there has been a major underlying array failure, there is > not much it can do to help. Better to just mount ro,noreplaylog and get > what you can in these sorts of scenarios, IMO. > > "Did you try the latest copy of xfs_repair"? Sometimes it will get further > or not crash, but it likely will still maul whatever's left of your > filesystem. > > Best of luck, > > Sean > > On Tue, Sep 23, 2014 at 6:02 PM, Diane Trout <diane@caltech.edu> wrote: > > Hi, > > > > I had a raid failure at work that ended up corrupting an xfs filesystem > > the > > tail of the xfs_repair command looks like the below. I was able to > > generate a > > metadata dump but is there a point to making it available? > > > > It does crash repeatedly at the same place > > > > (I'm not subscribed to the list, so could you reply directly as well?) > > > > disconnected inode 15276, moving to lost+found > > disconnected inode 15277, moving to lost+found > > disconnected inode 15278, moving to lost+found > > disconnected inode 15279, moving to lost+found > > disconnected inode 15280, moving to lost+found > > disconnected inode 15281, moving to lost+found > > disconnected inode 15282, moving to lost+found > > disconnected inode 15283, moving to lost+found > > disconnected inode 15284, moving to lost+found > > disconnected inode 15286, moving to lost+found > > disconnected inode 15287, moving to lost+found > > disconnected inode 15288, moving to lost+found > > disconnected inode 15289, moving to lost+found > > disconnected inode 15290, moving to lost+found > > disconnected inode 15291, moving to lost+found > > disconnected inode 15292, moving to lost+found > > disconnected inode 15293, moving to lost+found > > disconnected inode 15294, moving to lost+found > > disconnected inode 15295, moving to lost+found > > disconnected inode 15360, moving to lost+found > > corrupt dinode 15360, extent total = 1, nblocks = 0. This is a bug. > > Please capture the filesystem metadata with xfs_metadump and > > report it to xfs@oss.sgi.com. > > cache_node_purge: refcount was 1, not zero (node=0x7f369883e6f0) > > > > fatal error -- 117 - couldn't iget disconnected inode > > > > > > _______________________________________________ > > xfs mailing list > > xfs@oss.sgi.com > > http://oss.sgi.com/mailman/listinfo/xfs _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Corrupted file system 2014-09-23 22:29 ` Diane Trout @ 2014-09-23 23:12 ` Eric Sandeen 0 siblings, 0 replies; 9+ messages in thread From: Eric Sandeen @ 2014-09-23 23:12 UTC (permalink / raw) To: Diane Trout, Sean Caron; +Cc: xfs@oss.sgi.com On 9/23/14 5:29 PM, Diane Trout wrote: > Hello, > > Yes, I had my doubts about that there's anything reasonable one could do after > part of the file system was randomized. I'll trying updating to xfsprogs 3.2.1 > and see if that does any better. Yes - you didn't mention which version you used, but I'd definitely try something current. The bug rings a bell, maybe fixed now. To be safe, you can always make an xfs_metadump image (use -o to not obfuscate filenames for convenience), xfs_mdrestore that, and point repair at that fs image. Then you'll see what it *would* do before you *do* do. ;) But yeah, depending on the damage done to the storage, you may be in trouble that repair can't help with. Depends on the failure, though. -Eric _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Corrupted file system 2014-09-23 22:02 Corrupted file system Diane Trout 2014-09-23 22:12 ` Sean Caron @ 2014-09-24 10:08 ` Emmanuel Florac 2014-09-24 20:38 ` Diane Trout 1 sibling, 1 reply; 9+ messages in thread From: Emmanuel Florac @ 2014-09-24 10:08 UTC (permalink / raw) To: Diane Trout; +Cc: xfs Le Tue, 23 Sep 2014 15:02:57 -0700 Diane Trout <diane@caltech.edu> écrivait: > I had a raid failure at work that ended up corrupting an xfs > filesystem the tail of the xfs_repair command looks like the below. I > was able to generate a metadata dump but is there a point to making > it available? > > It does crash repeatedly at the same place Did you try the very latest xfs_repair? -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Corrupted file system 2014-09-24 10:08 ` Emmanuel Florac @ 2014-09-24 20:38 ` Diane Trout 2014-09-24 20:57 ` Eric Sandeen 0 siblings, 1 reply; 9+ messages in thread From: Diane Trout @ 2014-09-24 20:38 UTC (permalink / raw) To: Emmanuel Florac; +Cc: xfs On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote: > Le Tue, 23 Sep 2014 15:02:57 -0700 > > Diane Trout <diane@caltech.edu> écrivait: > > I had a raid failure at work that ended up corrupting an xfs > > filesystem the tail of the xfs_repair command looks like the below. I > > was able to generate a metadata dump but is there a point to making > > it available? > > > > It does crash repeatedly at the same place > > Did you try the very latest xfs_repair? I grabbed the one out of debian unstable (3.2.1), verified that it was your latest release and used that. I didn't try building the version in git. It did get much further than version 3.1.7+b1 (debian stable) Diane Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it in ino 2051. bad hash table for directory inode 256 (no data entry): rebuilding rebuilding directory inode 256 7f70c79c8700: Badness in key lookup (length) bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes) Metadata corruption detected at block 0x4000070/0x1000 xfs_da_do_buf(2): XFS_CORRUPTION_ERROR fatal error -- can't read block 8388608 for directory inode 259, error 117 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Corrupted file system 2014-09-24 20:38 ` Diane Trout @ 2014-09-24 20:57 ` Eric Sandeen 2014-09-25 0:25 ` Diane Trout 0 siblings, 1 reply; 9+ messages in thread From: Eric Sandeen @ 2014-09-24 20:57 UTC (permalink / raw) To: Diane Trout; +Cc: xfs-oss If you want to make a metadata image, compress it, and send it my way I'd be glad to look at it - I've been doing some repair work lately. -Eric On 9/24/14 3:38 PM, Diane Trout wrote: > On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote: >> Le Tue, 23 Sep 2014 15:02:57 -0700 >> >> Diane Trout <diane@caltech.edu> écrivait: >>> I had a raid failure at work that ended up corrupting an xfs >>> filesystem the tail of the xfs_repair command looks like the below. I >>> was able to generate a metadata dump but is there a point to making >>> it available? >>> >>> It does crash repeatedly at the same place >> >> Did you try the very latest xfs_repair? > > > I grabbed the one out of debian unstable (3.2.1), verified that it was your > latest release and used that. I didn't try building the version in git. > > It did get much further than version 3.1.7+b1 (debian stable) > > Diane > > Phase 6 - check inode connectivity... > - resetting contents of realtime bitmap and summary inodes > - traversing filesystem ... > entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it in > ino 2051. > bad hash table for directory inode 256 (no data entry): rebuilding > rebuilding directory inode 256 > 7f70c79c8700: Badness in key lookup (length) > bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes) > Metadata corruption detected at block 0x4000070/0x1000 > xfs_da_do_buf(2): XFS_CORRUPTION_ERROR > > fatal error -- can't read block 8388608 for directory inode 259, error 117 > > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Corrupted file system 2014-09-24 20:57 ` Eric Sandeen @ 2014-09-25 0:25 ` Diane Trout 2014-09-25 0:50 ` Eric Sandeen 0 siblings, 1 reply; 9+ messages in thread From: Diane Trout @ 2014-09-25 0:25 UTC (permalink / raw) To: Eric Sandeen; +Cc: xfs-oss Huh. I'd managed to make a metadump using the 3.1.7 version of the tools. Then I tried to run xfs_repair on the partition, and now xfs_metadump is segfaulting on the partition. Would the older metadata dump be useful? I'd need to rebuild xfsprogs before the stack trace would be useful... debian likes to strip the symbols and they didn't make a -dbg package. Diane On Wednesday, September 24, 2014 15:57:53 Eric Sandeen wrote: > If you want to make a metadata image, compress it, and send it my way I'd be > glad to look at it - I've been doing some repair work lately. > > -Eric > > On 9/24/14 3:38 PM, Diane Trout wrote: > > On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote: > >> Le Tue, 23 Sep 2014 15:02:57 -0700 > >> > >> Diane Trout <diane@caltech.edu> écrivait: > >>> I had a raid failure at work that ended up corrupting an xfs > >>> filesystem the tail of the xfs_repair command looks like the below. I > >>> was able to generate a metadata dump but is there a point to making > >>> it available? > >>> > >>> It does crash repeatedly at the same place > >> > >> Did you try the very latest xfs_repair? > > > > I grabbed the one out of debian unstable (3.2.1), verified that it was > > your > > latest release and used that. I didn't try building the version in git. > > > > It did get much further than version 3.1.7+b1 (debian stable) > > > > Diane > > > > Phase 6 - check inode connectivity... > > > > - resetting contents of realtime bitmap and summary inodes > > - traversing filesystem ... > > > > entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it > > in ino 2051. > > bad hash table for directory inode 256 (no data entry): rebuilding > > rebuilding directory inode 256 > > 7f70c79c8700: Badness in key lookup (length) > > bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes) > > Metadata corruption detected at block 0x4000070/0x1000 > > xfs_da_do_buf(2): XFS_CORRUPTION_ERROR > > > > fatal error -- can't read block 8388608 for directory inode 259, error 117 > > > > > > _______________________________________________ > > xfs mailing list > > xfs@oss.sgi.com > > http://oss.sgi.com/mailman/listinfo/xfs _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Corrupted file system 2014-09-25 0:25 ` Diane Trout @ 2014-09-25 0:50 ` Eric Sandeen 0 siblings, 0 replies; 9+ messages in thread From: Eric Sandeen @ 2014-09-25 0:50 UTC (permalink / raw) To: Diane Trout; +Cc: xfs-oss On 9/24/14 7:25 PM, Diane Trout wrote: > Huh. > > I'd managed to make a metadump using the 3.1.7 version of the tools. Then I > tried to run xfs_repair on the partition, and now xfs_metadump is segfaulting > on the partition. > > Would the older metadata dump be useful? sure. Compress it & send me a private note with it attached (or with a location if it's too big to email) -Eric > I'd need to rebuild xfsprogs before the stack trace would be useful... debian > likes to strip the symbols and they didn't make a -dbg package. > > Diane > > On Wednesday, September 24, 2014 15:57:53 Eric Sandeen wrote: >> If you want to make a metadata image, compress it, and send it my way I'd be >> glad to look at it - I've been doing some repair work lately. >> >> -Eric >> >> On 9/24/14 3:38 PM, Diane Trout wrote: >>> On Wednesday, September 24, 2014 12:08:40 Emmanuel Florac wrote: >>>> Le Tue, 23 Sep 2014 15:02:57 -0700 >>>> >>>> Diane Trout <diane@caltech.edu> écrivait: >>>>> I had a raid failure at work that ended up corrupting an xfs >>>>> filesystem the tail of the xfs_repair command looks like the below. I >>>>> was able to generate a metadata dump but is there a point to making >>>>> it available? >>>>> >>>>> It does crash repeatedly at the same place >>>> >>>> Did you try the very latest xfs_repair? >>> >>> I grabbed the one out of debian unstable (3.2.1), verified that it was >>> your >>> latest release and used that. I didn't try building the version in git. >>> >>> It did get much further than version 3.1.7+b1 (debian stable) >>> >>> Diane >>> >>> Phase 6 - check inode connectivity... >>> >>> - resetting contents of realtime bitmap and summary inodes >>> - traversing filesystem ... >>> >>> entry "lymphoblastoid" in dir ino 256 doesn't have a .. entry, will set it >>> in ino 2051. >>> bad hash table for directory inode 256 (no data entry): rebuilding >>> rebuilding directory inode 256 >>> 7f70c79c8700: Badness in key lookup (length) >>> bp=(bno 0x3480, len 4096 bytes) key=(bno 0x3480, len 8192 bytes) >>> Metadata corruption detected at block 0x4000070/0x1000 >>> xfs_da_do_buf(2): XFS_CORRUPTION_ERROR >>> >>> fatal error -- can't read block 8388608 for directory inode 259, error 117 >>> >>> >>> _______________________________________________ >>> xfs mailing list >>> xfs@oss.sgi.com >>> http://oss.sgi.com/mailman/listinfo/xfs > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-09-25 0:50 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-09-23 22:02 Corrupted file system Diane Trout 2014-09-23 22:12 ` Sean Caron 2014-09-23 22:29 ` Diane Trout 2014-09-23 23:12 ` Eric Sandeen 2014-09-24 10:08 ` Emmanuel Florac 2014-09-24 20:38 ` Diane Trout 2014-09-24 20:57 ` Eric Sandeen 2014-09-25 0:25 ` Diane Trout 2014-09-25 0:50 ` Eric Sandeen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox