Broken XFS partition

Broken XFS partition

[Sylwester Gruszka]

Hello,

I have a problem with XFS partition, it has broken down suddenly. I was doing 
no special things, just browsing www and suddenly my PC has restarted, after 
that I could not boot by PLD Th linux, after few usual kernel messages I get 
some matrix of numbers and it stops.

I realised that it is broken partition after I tried to repair system, I just 
could not mount this partition (using sysrescuecd), 'mount' gave me lots of 
weird numbers instead of mounting partition. It only do that way at first try, 
the second try always freezes mount program and nothing is happening.

Here are some screenshots (mount using SystemRescueCd-1.1.0 so the kernel is 
propably 2.6.25.16):
http://img397.imageshack.us/my.php?image=dsc01043ko6.jpg
http://img179.imageshack.us/my.php?image=dsc01044yj4.jpg
http://img158.imageshack.us/my.php?image=dsc01045sr3.jpg
http://img221.imageshack.us/my.php?image=dsc01047js4.jpg

PLD Linux Th booting:
http://img530.imageshack.us/my.php?image=dsc01048mk3.jpg

Suse booting:
http://img87.imageshack.us/my.php?image=dsc01050as4.jpg
Except this warning it boots normal.

Now I'm running Suse and I tried xfs_check this partition but xfs_check is 
freezing too.

Somebody on PLD Linux mailing list told me to mail this problem here.

I'm nod subscribed to this list so please send me CC.

--
greets,
Sylwester Gruszka

Re: Broken XFS partition

[Willy Tarreau]

On Wed, Sep 24, 2008 at 12:36:25PM +0200, Sylwester Gruszka wrote:
> Hello,
> 
> I have a problem with XFS partition, it has broken down suddenly. I was doing 
> no special things, just browsing www and suddenly my PC has restarted, after 
> that I could not boot by PLD Th linux, after few usual kernel messages I get 
> some matrix of numbers and it stops.
> 
> I realised that it is broken partition after I tried to repair system, I just 
> could not mount this partition (using sysrescuecd), 'mount' gave me lots of 
> weird numbers instead of mounting partition. It only do that way at first try, 
> the second try always freezes mount program and nothing is happening.
> 
> Here are some screenshots (mount using SystemRescueCd-1.1.0 so the kernel is 
> propably 2.6.25.16):
> http://img397.imageshack.us/my.php?image=dsc01043ko6.jpg
> http://img179.imageshack.us/my.php?image=dsc01044yj4.jpg
> http://img158.imageshack.us/my.php?image=dsc01045sr3.jpg
> http://img221.imageshack.us/my.php?image=dsc01047js4.jpg
> 
> PLD Linux Th booting:
> http://img530.imageshack.us/my.php?image=dsc01048mk3.jpg
> 
> Suse booting:
> http://img87.imageshack.us/my.php?image=dsc01050as4.jpg
> Except this warning it boots normal.
> 
> Now I'm running Suse and I tried xfs_check this partition but xfs_check is 
> freezing too.
> 
> Somebody on PLD Linux mailing list told me to mail this problem here.
> 
> I'm nod subscribed to this list so please send me CC.

I'm sure there's an XFS mailing list where people are really skilled
and will surely help for such trouble, I just don't have the address
here at hand. You should search the list archives on the net (google
for "linux xfs mailing list" and check the first or second one).

In the mean time, do not modify your FS unless told to do so!

Regards,
Willy

Re: Broken XFS partition

[David Greaves]

cc'ed xfs list

Willy Tarreau wrote:
> On Wed, Sep 24, 2008 at 12:36:25PM +0200, Sylwester Gruszka wrote:
>> Hello,
>>
>> I have a problem with XFS partition, it has broken down suddenly. I was doing 
>> no special things, just browsing www and suddenly my PC has restarted, after 
>> that I could not boot by PLD Th linux, after few usual kernel messages I get 
>> some matrix of numbers and it stops.

First thing is why?
If you could boot then you could check the log to see if there was a disk error.

I suggest rebooting to an up-to-date rescue disk (ie 2008 vintage - mainly for
the latest xfs utils) and using smartctl -a to look at the disk state.

Then run smartctl -t long /dev/sdX or some other disk check utility.

This will take a long time to run but should let you see if there are any bad
blocks.

If so then I suggest using ddrescue to recover the disk to a spare/replacement.

Running xfs_repair on a failing disk will likely cause more trouble.

>> I realised that it is broken partition after I tried to repair system, I just 
>> could not mount this partition (using sysrescuecd), 'mount' gave me lots of 
>> weird numbers instead of mounting partition. It only do that way at first try, 
>> the second try always freezes mount program and nothing is happening.
>>
>> Here are some screenshots (mount using SystemRescueCd-1.1.0 so the kernel is 
>> propably 2.6.25.16):
>> http://img397.imageshack.us/my.php?image=dsc01043ko6.jpg
>> http://img179.imageshack.us/my.php?image=dsc01044yj4.jpg
>> http://img158.imageshack.us/my.php?image=dsc01045sr3.jpg
>> http://img221.imageshack.us/my.php?image=dsc01047js4.jpg
>>
>> PLD Linux Th booting:
>> http://img530.imageshack.us/my.php?image=dsc01048mk3.jpg
>>
>> Suse booting:
>> http://img87.imageshack.us/my.php?image=dsc01050as4.jpg
>> Except this warning it boots normal.
>>
>> Now I'm running Suse and I tried xfs_check this partition but xfs_check is 
>> freezing too.

IIRC xfs_check used to be less robust than xfs_repair.

If the disk looks sound then running xfs_repair should help.
Read the manpage. no-modify (-n) may be a good idea first time.

It *may* tell you to remount to replay the log.
You should try this (mounting) but from what you've said it sounds like it won't
work.

You may then try using the -L option which will discard the log/journal. (If
you're nervous about this then wait for expert help - IMHO it's no big deal)

If this fails then you need expert help.

I have had a fair few disk failures over the years and this process is usually
quick and painless :)

David

"Don't worry, you'll be fine; I saw it work in a cartoon once..."

Re: Broken XFS partition

[Dave Chinner]

On Wed, Sep 24, 2008 at 12:36:25PM +0200, Sylwester Gruszka wrote:
> Hello,
> 
> I have a problem with XFS partition, it has broken down suddenly. I was doing 
> no special things, just browsing www and suddenly my PC has restarted, after 
> that I could not boot by PLD Th linux, after few usual kernel messages I get 
> some matrix of numbers and it stops.
> 
> I realised that it is broken partition after I tried to repair system, I just 
> could not mount this partition (using sysrescuecd), 'mount' gave me lots of 
> weird numbers instead of mounting partition. It only do that way at first try, 
> the second try always freezes mount program and nothing is happening.
> 
> Here are some screenshots (mount using SystemRescueCd-1.1.0 so the kernel is 
> propably 2.6.25.16):
> http://img397.imageshack.us/my.php?image=dsc01043ko6.jpg
> http://img179.imageshack.us/my.php?image=dsc01044yj4.jpg
> http://img158.imageshack.us/my.php?image=dsc01045sr3.jpg
> http://img221.imageshack.us/my.php?image=dsc01047js4.jpg

This indicates a corrupted log record is being incorrectly parsed.
Please try adding the patch I posted yesterrday that fixed a
fs-fuzzer triggered occurrence of the same oops....

http://oss.sgi.com/archives/xfs/2008-09/msg00296.html

This patch will prevent the oops, but you'll still need to manually
repair the damage to the FS.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com