able to mount a fs, unable to repair it

able to mount a fs, unable to repair it

[laurent+xfs]

Hi !

For a reason all my xfs FS got corrupted at the same time.
My setup is: xen host using lvm on top of soft raid6 on top of jbod
3ware connected sata disks.
Both the host and the VM fs got corrupted.

ATM host is using 2.6.32 (debian/squeeze), xfsprogs are 3.1.4

I've had 2 kinds of corruptions:
1) unable to mount the FS, been able to xfs_repair -L
2) able to mount the FS, xfs_check was reporting errors, unable to
xfs_repair the fs

For most of the FS, i've managed to either mkfs.xfs or xfs_repair then
rsync from the backup.

I'm left with my 1.5TB fs for which i also have a full backup, i'm
able to mount it (no kernel error or anything when
mounting it). xfs_check on this unmounted FS gives errors. xfs_repair
(after a long long time) is unable to find a suitable thingie to
repair the FS.

https://gist.github.com/752411 xfs_{db,check,repair,metadump} results.
(haven't checked the memory/disk usage when doing the metadump)

I can't really tell why all this happened in the first place.
Been having power supply issues some weeks ago, been upgrading from
lenny to squeeze, been playing with xen & lxc at the same, been doing
my xmas shopping a bit late ! :-)

I'd be curious to know how to repair the 1.5TB FS, any idea on what i
could do ?

TIA

-- 
Laurent

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: able to mount a fs, unable to repair it

[Eric Sandeen]

On 12/22/10 7:53 PM, laurent+xfs@u-picardie.fr wrote:
> 
> Hi !
> 
> For a reason all my xfs FS got corrupted at the same time.
> My setup is: xen host using lvm on top of soft raid6 on top of jbod
> 3ware connected sata disks.
> Both the host and the VM fs got corrupted.
> 
> ATM host is using 2.6.32 (debian/squeeze), xfsprogs are 3.1.4
> 
> I've had 2 kinds of corruptions:
> 1) unable to mount the FS, been able to xfs_repair -L
> 2) able to mount the FS, xfs_check was reporting errors, unable to
> xfs_repair the fs
> 
> For most of the FS, i've managed to either mkfs.xfs or xfs_repair then
> rsync from the backup.
> 
> I'm left with my 1.5TB fs for which i also have a full backup, i'm
> able to mount it (no kernel error or anything when
> mounting it). xfs_check on this unmounted FS gives errors. xfs_repair
> (after a long long time) is unable to find a suitable thingie to
> repair the FS.
> 
> https://gist.github.com/752411 xfs_{db,check,repair,metadump} results.
> (haven't checked the memory/disk usage when doing the metadump)
> 
> I can't really tell why all this happened in the first place.
> Been having power supply issues some weeks ago, been upgrading from
> lenny to squeeze, been playing with xen & lxc at the same, been doing
> my xmas shopping a bit late ! :-)
> 
> I'd be curious to know how to repair the 1.5TB FS, any idea on what i
> could do ?
> 
> TIA
> 

for the image that segfaults, I'd say make an xfs_metadump and provide it
for analysis, if the latest xfsprogs xfs_repair still segfaults.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: able to mount a fs, unable to repair it

[Emmanuel Florac]

Le Thu, 23 Dec 2010 02:53:08 +0100 vous écriviez:

> For a reason all my xfs FS got corrupted at the same time.
> My setup is: xen host using lvm on top of soft raid6 on top of jbod
> 3ware connected sata disks.
> Both the host and the VM fs got corrupted.

RAID-6 is extremely stable on the 3Ware, while software RAID-6 is
dubbed experimental in the kernel. Software RAID can provide seducely
high performance when running benchmarks, but IMO it doesn't fly when
you need your CPU to do some other jobs, like running NFS daemons or
even worse, VMs. Under high CPU loads concurrently coming from the
software RAID and virtualisation stack, so many things may go wrong...

My advice : software RAID stinks under high CPU load/ high IO. Use
hardware RAID whenever possible because it provides more predictable
performance and remove many weird software interactions from the
picture.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: able to mount a fs, unable to repair it

[laurent+xfs]

Emmanuel Florac <eflorac@intellique.com> writes:

Bonsoir Emmanuel,

> Le Thu, 23 Dec 2010 02:53:08 +0100 vous écriviez:
>
>> For a reason all my xfs FS got corrupted at the same time.
>> My setup is: xen host using lvm on top of soft raid6 on top of jbod
>> 3ware connected sata disks.
>> Both the host and the VM fs got corrupted.
>
> RAID-6 is extremely stable on the 3Ware, while software RAID-6 is
> dubbed experimental in the kernel. Software RAID can provide seducely
> high performance when running benchmarks, but IMO it doesn't fly when
> you need your CPU to do some other jobs, like running NFS daemons or
> even worse, VMs. Under high CPU loads concurrently coming from the
> software RAID and virtualisation stack, so many things may go
> wrong...

I can't agree more, I'm using both nfs AND VMs :)
It is very slow and very cpu consuming.
I'm moving away from xen to lxc on this home server, pretty sure it
will help.

> My advice : software RAID stinks under high CPU load/ high IO. Use
> hardware RAID whenever possible because it provides more predictable
> performance and remove many weird software interactions from the
> picture.

The 3w9550sx doesn't raid6, but I admit I've been thinking of using
its raid5 instead of the slow soft raid6.

Thanks for your advices :)

-- 
Laurent

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: able to mount a fs, unable to repair it

[Emmanuel Florac]

Le Thu, 23 Dec 2010 20:34:41 +0100 vous écriviez:

> > My advice : software RAID stinks under high CPU load/ high IO. Use
> > hardware RAID whenever possible because it provides more predictable
> > performance and remove many weird software interactions from the
> > picture.  
> 
> The 3w9550sx doesn't raid6, but I admit I've been thinking of using
> its raid5 instead of the slow soft raid6.

Ah that's a valid reason :) RAID-5 is reasonable with disks under 1TB
and arrays up to 6 TB. Further it's a bet against the law of physics :)

However, a reliable RAID-5 is probably safer than an experimental
RAID-6 overall :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: able to mount a fs, unable to repair it

[Richard Scobie]

Emmanuel Florac wrote:

 > However, a reliable RAID-5 is probably safer than an experimental
 > RAID-6 overall.

Looking at the 2.6.29.1 kernel running on this machine, the only md RAID 
mode marked as experimental is RAID-10.

I have no experience using Xen, but have used md RAID6 extensively in 
heavily loaded situations for over three years without problems.

Regards,

Richard

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs