Help with XFS

Help with XFS

[Thomas Førde]

[-- Attachment #1.1: Type: text/plain, Size: 2971 bytes --]

during the other day our XFS filesystem had a unexpected shutdown 
according to logs.

un 16 15:48:42 strmng04 kernel: 0x0: 00 00 c0 46 01 00 c0 46 77 02 c0 46 
fe 7d 00 40
Jun 16 15:48:42 strmng04 kernel: Filesystem "sdf1": XFS internal error 
xfs_da_do_buf(2) at line 2276 of file 
/home/buildcentos/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_smp/xfs_da_bt$
Jun 16 15:48:42 strmng04 kernel:
Jun 16 15:48:42 strmng04 kernel: Call 
Trace:<ffffffffa0211762>{:xfs:xfs_corruption_error+236} 
<ffffffffa023c7cc>{:xfs:kmem_zone_alloc+66}
Jun 16 15:48:42 strmng04 kernel: 
<ffffffffa0202526>{:xfs:xfs_da_do_buf+1335} 
<ffffffffa0202653>{:xfs:xfs_da_read_buf+21}
Jun 16 15:48:42 strmng04 kernel: 
<ffffffffa0202653>{:xfs:xfs_da_read_buf+21} 
<ffffffffa0207210>{:xfs:xfs_dir2_block_lookup_int+71}
Jun 16 15:48:42 strmng04 kernel: 
<ffffffffa0207210>{:xfs:xfs_dir2_block_lookup_int+71}
Jun 16 15:48:42 strmng04 kernel: 
<ffffffffa0207359>{:xfs:xfs_dir2_block_lookup+19} 
<ffffffffa0206457>{:xfs:xfs_dir2_lookup+200}
Jun 16 15:48:42 strmng04 kernel:        <ffffffff801347b0>{__wake_up+54} 
<ffffffff801ef3a1>{__up_read+16}
Jun 16 15:48:42 strmng04 kernel: 
<ffffffffa0229d22>{:xfs:xfs_dir_lookup_int+53} 
<ffffffffa022e47d>{:xfs:xfs_lookup+74}
Jun 16 15:48:42 strmng04 kernel: <ffffffffa02371e0>{:xfs:linvfs_lookup+48} 
<ffffffff80189fc5>{__lookup_hash+263}
Jun 16 15:48:42 strmng04 kernel: <ffffffff8018a04f>{lookup_one_len+94} 
<ffffffffa03bd2bc>{:nfsd:nfsd_lookup+936}
Jun 16 15:48:42 strmng04 kernel: 
<ffffffffa03c55fa>{:nfsd:nfsd3_proc_lookup+198} 
<ffffffffa03ba7dc>{:nfsd:nfsd_dispatch+219}
Jun 16 15:48:42 strmng04 kernel: 
<ffffffffa03474ac>{:sunrpc:svc_process+1227} 
<ffffffff80134709>{default_wake_function+0}
Jun 16 15:48:42 strmng04 kernel:        <ffffffffa03ba31b>{:nfsd:nfsd+0} 
<ffffffffa03ba553>{:nfsd:nfsd+568}
Jun 16 15:48:42 strmng04 kernel: <ffffffff80133470>{schedule_tail+55} 
<ffffffff80110fd3>{child_rip+8}
Jun 16 15:48:42 strmng04 kernel:        <ffffffffa03ba31b>{:nfsd:nfsd+0} 
<ffffffffa03ba31b>{:nfsd:nfsd+0}
Jun 16 15:48:42 strmng04 kernel:        <ffffffff80110fcb>{child_rip+0}

many of those. and at the end

Jun 16 15:50:13 strmng04 kernel: xfs_force_shutdown(sdf1,0x8) called from 
line 4092 of file 
/home/buildcentos/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_smp/xfs_bmap.c. 
Return address
 = 0xffffffffa01f7fb4
Jun 16 15:50:13 strmng04 kernel: Filesystem "sdf1": Corruption of 
in-memory data detected.  Shutting down filesystem: sdf1
Jun 16 15:50:13 strmng04 kernel: Please umount the filesystem, and rectify 
the problem(s)


i have run xfs_check without any verbos information given
i have run xfs_repair without any information errors fixed.

when i try to mount the XFS filesystem of 10TB, my 7.8 TB of data is now 
gone.

before:         /dev/sdf1            10527858560 8801101708 1726756852 84% 
/D5
after:          /dev/sdf1            10527858560       528 10527858032 1% 
/D5

help, where did my data go, and how can i get it back.

[-- Attachment #1.2: Type: text/html, Size: 4952 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 715 bytes --]

On Donnerstag, 17. Juni 2010 Thomas Førde wrote:
> when i try to mount the XFS filesystem of 10TB, my 7.8 TB of data is
>  now  gone.
 
Do you have any logs in /var/log/warn about errors before that crash?
How long had that machine been up?
What kernel version?
How many files/dirs have there been?

I'm not a dev, but I guess running xfs_metadump, zipping the outcome and 
putting it on a website would be a good preparation.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Emmanuel Florac]

Le Thu, 17 Jun 2010 15:18:26 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> i have run xfs_check without any verbos information given
> i have run xfs_repair without any information errors fixed.

That's unfortunate that you didn't post here before doing irreparable 
damage. Any activity to the filesystem may delete permanently data, so
you should stop using it completely until done with the rescue
operations and unmount it.

> 
> when i try to mount the XFS filesystem of 10TB, my 7.8 TB of data is
> now gone.
> 
> before:         /dev/sdf1            10527858560 8801101708
> 1726756852 84% /D5
> after:          /dev/sdf1            10527858560       528
> 10527858032 1% /D5
> 

Some old versions of xfs_repair may fail on big FS. You didn't mention
which version you're using, which distribution and which kernel. The
hardware setup may be important, too.

> help, where did my data go, and how can i get it back.

Apparently xfs_repair deleted it because it was corrupt (alas, happen
sometimes with xfs_repair before 2.8). The best thing to do is to get it
back from backups. I suppose that you have no backup (people don't do
any backup until they lose important data, which always occurs).

If you have no backups, your last hope is to use a tool like photorec
to find your data on the raw device; however it won't reconstruct the
metadata (no names, no folders, no hierarchy, no user rights). BTW you
need some other place to store the newly discovered files with at least
as much space (7.8 TB).

My condolences. I once lost 13TB and see, I'm not dead. You'll get over
it somehow.
-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Thomas Førde]

[-- Attachment #1.1: Type: text/plain, Size: 3200 bytes --]

both the xfs_check and xfs_Reapir was done from a unmounted system, the 
system has not been mounted or used since.

its a IBM x3650 with stadnard 2x73GB disk.

[root@strmng04 ~]# cat /etc/issue
CentOS release 4.7 (Final)
Kernel \r on an \m

[root@strmng04 ~]# uname -a
Linux strmng04 2.6.9-78.0.5.ELsmp #1 SMP Wed Oct 8 07:06:30 EDT 2008 
x86_64 x86_64 x86_64 GNU/Linux


there is no backup no, as how to backup 8 TB of data,very expensive tape 
wise.

xfs_irecover ias been mentioned to me, so my fingers are crossed.


Sincerely
-------------------------------------------------------------------
Thomas Førde    IT-Manager 
Mobile +47 90 41 41 90   Fax +47 55 31 67 66
thomas.forde@bergenofs.no     www.bergenofs.no
-------------------------------------------------------------------
Bergen Oilfield Services AS
Address: Nedre Aastveit 12    NO-5106 Ovre Ervik
-------------------------------------------------------------------

Please consider the environment before printing



From:   Emmanuel Florac <eflorac@intellique.com>
To:     Thomas Førde <thomas.forde@bergenofs.no>
Cc:     xfs@oss.sgi.com
Date:   18.06.2010 13:21
Subject:        Re: Help with XFS



Le Thu, 17 Jun 2010 15:18:26 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> i have run xfs_check without any verbos information given
> i have run xfs_repair without any information errors fixed.

That's unfortunate that you didn't post here before doing irreparable 
damage. Any activity to the filesystem may delete permanently data, so
you should stop using it completely until done with the rescue
operations and unmount it.

> 
> when i try to mount the XFS filesystem of 10TB, my 7.8 TB of data is
> now gone.
> 
> before:         /dev/sdf1            10527858560 8801101708
> 1726756852 84% /D5
> after:          /dev/sdf1            10527858560       528
> 10527858032 1% /D5
> 

Some old versions of xfs_repair may fail on big FS. You didn't mention
which version you're using, which distribution and which kernel. The
hardware setup may be important, too.

> help, where did my data go, and how can i get it back.

Apparently xfs_repair deleted it because it was corrupt (alas, happen
sometimes with xfs_repair before 2.8). The best thing to do is to get it
back from backups. I suppose that you have no backup (people don't do
any backup until they lose important data, which always occurs).

If you have no backups, your last hope is to use a tool like photorec
to find your data on the raw device; however it won't reconstruct the
metadata (no names, no folders, no hierarchy, no user rights). BTW you
need some other place to store the newly discovered files with at least
as much space (7.8 TB).

My condolences. I once lost 13TB and see, I'm not dead. You'll get over
it somehow.
-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |            <eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------



[-- Attachment #1.2: Type: text/html, Size: 5093 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Emmanuel Florac]

Le Fri, 18 Jun 2010 13:27:01 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> both the xfs_check and xfs_Reapir was done from a unmounted system,
> the system has not been mounted or used since.

That's good. Did you run xfs_repair -n first ? Did you run 
xfs_repair -L afterwards?

> its a IBM x3650 with stadnard 2x73GB disk.

But what about the RAID array? 
 
> [root@strmng04 ~]# cat /etc/issue
> CentOS release 4.7 (Final)
> Kernel \r on an \m
> 
> [root@strmng04 ~]# uname -a
> Linux strmng04 2.6.9-78.0.5.ELsmp #1 SMP Wed Oct 8 07:06:30 EDT 2008 
> x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> there is no backup no, as how to backup 8 TB of data,very expensive
> tape wise.

You could have a disk backup on cheaper drives. And current tapes are
800 GB (LTO-3) or 1.5 TB (LTO-4), and libraries are cheaper nowadays.

> xfs_irecover ias been mentioned to me, so my fingers are crossed.
> 

If your data is a huge bunch of media files or similar, photorec really
can help you as a very last resort, though you'll have to manually sort
through all this data...

good luck,
-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Thomas Førde]

[-- Attachment #1.1: Type: text/plain, Size: 2360 bytes --]

the raid array is a FC SAN array consisting of DS4700 unit with EXP810 
expansion system

xfs_check first, xfs_repair -n, and xfs_repair

xfs_Repair -L has not been used.

its seismisc processed data, large text files.



Sincerely
-------------------------------------------------------------------
Thomas Førde    IT-Manager 
Mobile +47 90 41 41 90   Fax +47 55 31 67 66
thomas.forde@bergenofs.no     www.bergenofs.no
-------------------------------------------------------------------
Bergen Oilfield Services AS
Address: Nedre Aastveit 12    NO-5106 Ovre Ervik
-------------------------------------------------------------------

Please consider the environment before printing



From:   Emmanuel Florac <eflorac@intellique.com>
To:     Thomas Førde <thomas.forde@bergenofs.no>
Cc:     xfs@oss.sgi.com
Date:   18.06.2010 13:35
Subject:        Re: Help with XFS



Le Fri, 18 Jun 2010 13:27:01 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> both the xfs_check and xfs_Reapir was done from a unmounted system,
> the system has not been mounted or used since.

That's good. Did you run xfs_repair -n first ? Did you run 
xfs_repair -L afterwards?

> its a IBM x3650 with stadnard 2x73GB disk.

But what about the RAID array? 
 
> [root@strmng04 ~]# cat /etc/issue
> CentOS release 4.7 (Final)
> Kernel \r on an \m
> 
> [root@strmng04 ~]# uname -a
> Linux strmng04 2.6.9-78.0.5.ELsmp #1 SMP Wed Oct 8 07:06:30 EDT 2008 
> x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> there is no backup no, as how to backup 8 TB of data,very expensive
> tape wise.

You could have a disk backup on cheaper drives. And current tapes are
800 GB (LTO-3) or 1.5 TB (LTO-4), and libraries are cheaper nowadays.

> xfs_irecover ias been mentioned to me, so my fingers are crossed.
> 

If your data is a huge bunch of media files or similar, photorec really
can help you as a very last resort, though you'll have to manually sort
through all this data...

good luck,
-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |            <eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------



[-- Attachment #1.2: Type: text/html, Size: 3911 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Emmanuel Florac]

Le Fri, 18 Jun 2010 13:43:57 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> the raid array is a FC SAN array consisting of DS4700 unit with
> EXP810 expansion system
> 
> xfs_check first, xfs_repair -n, and xfs_repair
> 
> xfs_Repair -L has not been used.

That's fortunate, maybe something can be saved from that.

> its seismisc processed data, large text files.
I see. Extremely expensive to reproduce... 

If you know the file structure and the metadata isn't much ineresting,
a simple perl or python script can scrap the raw device for the data.
I've done it before. I can send you code snippets if you want.

Did you have any success compiling xfs_irecover?

It's a side note but one of my customers is specialised in seismic data
acquisition, and record every data twice on two different RAID-1
systems and always keep at least two copies of everything, plus an
on-tape copy... A commodity 16TB NAS costs much less (10000 € or less)
than the data it holds and makes a nice backup system. NOTHING replaces
a backup. Your system WILL fail someday (well it has now apparently).

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Thomas Førde]

[-- Attachment #1.1: Type: text/plain, Size: 2252 bytes --]

got the libXH installed and compiled, but get a shared.22 error file does 
not exist trying to run it.



Sincerely
-------------------------------------------------------------------
Thomas Førde    IT-Manager 
Mobile +47 90 41 41 90   Fax +47 55 31 67 66
thomas.forde@bergenofs.no     www.bergenofs.no
-------------------------------------------------------------------
Bergen Oilfield Services AS
Address: Nedre Aastveit 12    NO-5106 Ovre Ervik
-------------------------------------------------------------------

Please consider the environment before printing



From:   Emmanuel Florac <eflorac@intellique.com>
To:     Thomas Førde <thomas.forde@bergenofs.no>
Cc:     xfs@oss.sgi.com
Date:   18.06.2010 14:00
Subject:        Re: Help with XFS



Le Fri, 18 Jun 2010 13:43:57 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> the raid array is a FC SAN array consisting of DS4700 unit with
> EXP810 expansion system
> 
> xfs_check first, xfs_repair -n, and xfs_repair
> 
> xfs_Repair -L has not been used.

That's fortunate, maybe something can be saved from that.

> its seismisc processed data, large text files.
I see. Extremely expensive to reproduce... 

If you know the file structure and the metadata isn't much ineresting,
a simple perl or python script can scrap the raw device for the data.
I've done it before. I can send you code snippets if you want.

Did you have any success compiling xfs_irecover?

It's a side note but one of my customers is specialised in seismic data
acquisition, and record every data twice on two different RAID-1
systems and always keep at least two copies of everything, plus an
on-tape copy... A commodity 16TB NAS costs much less (10000 € or less)
than the data it holds and makes a nice backup system. NOTHING replaces
a backup. Your system WILL fail someday (well it has now apparently).

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |            <eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------




[-- Attachment #1.2: Type: text/html, Size: 3603 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Emmanuel Florac]

Le Fri, 18 Jun 2010 14:17:29 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> got the libXH installed and compiled, but get a shared.22 error file
> does not exist trying to run it.
> 

Trying to run what? Apparently I can't compile hxtools with hxlib v1
nor version 3, only v2 would work apparently...

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Thomas Førde]

[-- Attachment #1.1: Type: text/plain, Size: 1535 bytes --]

testrun of xfs_irecover


error while loading shard libraries: libHX.so.22, cannot open shared 
object file, no such file or directory.

prolly some libs at the wrong place point elsewhere.


Sincerely
-------------------------------------------------------------------
Thomas Førde    IT-Manager 
Mobile +47 90 41 41 90   Fax +47 55 31 67 66
thomas.forde@bergenofs.no     www.bergenofs.no
-------------------------------------------------------------------
Bergen Oilfield Services AS
Address: Nedre Aastveit 12    NO-5106 Ovre Ervik
-------------------------------------------------------------------

Please consider the environment before printing



From:   Emmanuel Florac <eflorac@intellique.com>
To:     Thomas Førde <thomas.forde@bergenofs.no>
Cc:     xfs@oss.sgi.com
Date:   18.06.2010 15:08
Subject:        Re: Help with XFS



Le Fri, 18 Jun 2010 14:17:29 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> got the libXH installed and compiled, but get a shared.22 error file
> does not exist trying to run it.
> 

Trying to run what? Apparently I can't compile hxtools with hxlib v1
nor version 3, only v2 would work apparently...

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |            <eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------



[-- Attachment #1.2: Type: text/html, Size: 2895 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Emmanuel Florac]

Le Fri, 18 Jun 2010 15:35:39 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> testrun of xfs_irecover
> 
> 
> error while loading shard libraries: libHX.so.22, cannot open shared 
> object file, no such file or directory.
> 
> prolly some libs at the wrong place point elsewhere.
> 

What version of libhx did you install and how? I've finally managed to
compile xfs_irecover by using libhx18.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Thomas Førde]

[-- Attachment #1.1: Type: text/plain, Size: 1525 bytes --]

when i compiled it i used libHX 3.4 as thats why the xfs_irecover 
./configure demanded.


Sincerely
-------------------------------------------------------------------
Thomas Førde    IT-Manager 
Mobile +47 90 41 41 90   Fax +47 55 31 67 66
thomas.forde@bergenofs.no     www.bergenofs.no
-------------------------------------------------------------------
Bergen Oilfield Services AS
Address: Nedre Aastveit 12    NO-5106 Ovre Ervik
-------------------------------------------------------------------

Please consider the environment before printing



From:   Emmanuel Florac <eflorac@intellique.com>
To:     Thomas Førde <thomas.forde@bergenofs.no>
Cc:     xfs@oss.sgi.com
Date:   18.06.2010 16:01
Subject:        Re: Help with XFS



Le Fri, 18 Jun 2010 15:35:39 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> testrun of xfs_irecover
> 
> 
> error while loading shard libraries: libHX.so.22, cannot open shared 
> object file, no such file or directory.
> 
> prolly some libs at the wrong place point elsewhere.
> 

What version of libhx did you install and how? I've finally managed to
compile xfs_irecover by using libhx18.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |            <eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------



[-- Attachment #1.2: Type: text/html, Size: 2824 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Emmanuel Florac]

Le Fri, 18 Jun 2010 20:50:49 +0200 vous écriviez:

> when i compiled it i used libHX 3.4 as thats why the xfs_irecover 
> ./configure demanded.

Yes, but did you install libhx? Do you have /usr/lib/libHX.so* ?

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Thomas Førde]

[-- Attachment #1.1: Type: text/plain, Size: 2425 bytes --]

they have all the data in tape, but not the processed data, so that has to 
be redone, will take about  15-30 days work to get updated.

ill see if we can rebuild a different SAN with mirror functionalityt, y 
have 384x300GB disk in my SAN, so it should be possible.






Sincerely
-------------------------------------------------------------------
Thomas Førde    IT-Manager 
Mobile +47 90 41 41 90   Fax +47 55 31 67 66
thomas.forde@bergenofs.no     www.bergenofs.no
-------------------------------------------------------------------
Bergen Oilfield Services AS
Address: Nedre Aastveit 12    NO-5106 Ovre Ervik
-------------------------------------------------------------------

Please consider the environment before printing



From:   Emmanuel Florac <eflorac@intellique.com>
To:     Thomas Førde <thomas.forde@bergenofs.no>
Cc:     xfs@oss.sgi.com
Date:   18.06.2010 14:00
Subject:        Re: Help with XFS



Le Fri, 18 Jun 2010 13:43:57 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> the raid array is a FC SAN array consisting of DS4700 unit with
> EXP810 expansion system
> 
> xfs_check first, xfs_repair -n, and xfs_repair
> 
> xfs_Repair -L has not been used.

That's fortunate, maybe something can be saved from that.

> its seismisc processed data, large text files.
I see. Extremely expensive to reproduce... 

If you know the file structure and the metadata isn't much ineresting,
a simple perl or python script can scrap the raw device for the data.
I've done it before. I can send you code snippets if you want.

Did you have any success compiling xfs_irecover?

It's a side note but one of my customers is specialised in seismic data
acquisition, and record every data twice on two different RAID-1
systems and always keep at least two copies of everything, plus an
on-tape copy... A commodity 16TB NAS costs much less (10000 € or less)
than the data it holds and makes a nice backup system. NOTHING replaces
a backup. Your system WILL fail someday (well it has now apparently).

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |            <eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------




[-- Attachment #1.2: Type: text/html, Size: 3840 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Emmanuel Florac]

Le Fri, 18 Jun 2010 14:30:43 +0200
Thomas Førde <thomas.forde@bergenofs.no> écrivait:

> ill see if we can rebuild a different SAN with mirror functionalityt,
> y have 384x300GB disk in my SAN, so it should be possible.

No, a mirror will corrupt your data as well, a simple copy is better;
unless you break the mirror as soon as the data is on it, to keep a
copy safe. But the copy has another benefit, it's a different
filesystem too.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Robert Brockway]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 843 bytes --]

On Fri, 18 Jun 2010, Emmanuel Florac wrote:

> Le Fri, 18 Jun 2010 14:30:43 +0200
> Thomas Førde <thomas.forde@bergenofs.no> écrivait:
>
>> ill see if we can rebuild a different SAN with mirror functionalityt,
>> y have 384x300GB disk in my SAN, so it should be possible.
>
> No, a mirror will corrupt your data as well, a simple copy is better;

Exactly.  High availability and backups are two different things, although 
people often confuse them.

Here are my notes from my backup talk:

http://www.timetraveller.org/talks/backup_talk.pdf

This approach is honed over many years, having done many disaster 
recoveries in the real world.

Cheers,

Rob

-- 
Email: robert@timetraveller.org		Linux counter ID #16440
IRC: Solver (OFTC & Freenode)
Web: http://www.practicalsysadmin.com
Open Source: The revolution that silently changed the world

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Stan Hoeppner]

Thomas Førde put forth on 6/18/2010 7:30 AM:
> they have all the data in tape, but not the processed data, so that has to 
> be redone, will take about  15-30 days work to get updated.
> 
> ill see if we can rebuild a different SAN with mirror functionalityt, y 
> have 384x300GB disk in my SAN, so it should be possible.

Heheh.  Well, scratch my Nexsan recommendation then.

With that much free space on your SAN arrays, one wonders why you didn't
already have some kind of tar'd or other backup of the processed data...

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Stan Hoeppner]

Emmanuel Florac put forth on 6/18/2010 7:00 AM:
> A commodity 16TB NAS costs much less (10000 € or less)
> than the data it holds and makes a nice backup system. NOTHING replaces
> a backup. Your system WILL fail someday (well it has now apparently).

Since he already has a FC SAN, going with something like a 4Gb FC Nexsan
Sataboy w/16 1TB drives would be a better option.  Pricing should be around
$10-12k USD.  I bought an 8x500GB drive 2Gb FC Satablade back in 2006 for
approximately $9k USD because expanding our IBM FasTt 600 would have cost over
$20K USD for 1/4th the additional storage of thh Nexsan (the FasTt 600 wasn't
compatible with the EXP810, so the only expansion option was adding another FC
tray with 14x73GB 15k FC disks).

As long as the OP has a free SFP LC port on his FC switch (and the ability to
program zones) this is definitely the way to go.  No NFS/CIFS overhead.  And
given the default 1GB cache on the Sataboy, it'll likely have better
performance than the DS4700/EXP810 combo (at half the cost).  The web GUI
admin interface makes setting up and managing the Nexsan units a breeze.  I've
been really happy with ours, and with Nexsan support.  Great products, great
company.

http://www.nexsan.com/sataboy.php

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Emmanuel Florac]

Le Fri, 18 Jun 2010 07:55:14 -0500
Stan Hoeppner <stan@hardwarefreak.com> écrivait:

> Since he already has a FC SAN, going with something like a 4Gb FC
> Nexsan Sataboy w/16 1TB drives would be a better option. 

Why not, actually anything that could hold a backup will do :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS -- use DS4700 Volume Copy feature for backing up the seismic date LUN

[Stan Hoeppner]

Emmanuel Florac put forth on 6/18/2010 9:04 AM:
> Le Fri, 18 Jun 2010 07:55:14 -0500
> Stan Hoeppner <stan@hardwarefreak.com> écrivait:
> 
>> Since he already has a FC SAN, going with something like a 4Gb FC
>> Nexsan Sataboy w/16 1TB drives would be a better option. 
> 
> Why not, actually anything that could hold a backup will do :)

That really depends on the size of one's filesystem(s) to be backed up and the
required backup schedule.

Backup times, optimal configurations:

            10TB FS     20TB FS    40TB FS

4Gb FC      7 hours     14 hours   28 hours
1GbE NFS   28 hours     56 hours  111 hours


With 4Gb FC one could backup 40TB in just over a day.  Using a 1GbE NAS filer
it would take 4.6 days, almost an entire standard business _week_.  Now,
depending on the data, one could always tar the FS with compression and
substantially decrease these wire xfer times with either FC or NAS, which most
OPs would probably do with given the FS sizes involved.

And this is where the really nice high end arrays start to be worth their
price tag.  Even some upper midrange arrays, both SAN and NAS, have PIT
snapshot and LUN mirroring backup capabilities.  Both operations are much
faster than a filesystem level backup over the wire.

The DS4700 offers both a PIT snapshot feature called "FlashCopy" and a LUN
mirroring feature called "Volume Copy".  It would seem in the OP's case that
making use of the DS4700's Volume Copy feature would be of great benefit.  He
would simply make a Volume Copy of the current LUN holding the processed
seismic data using the System Storage DS4000 Storage Manager software.  He
should be able to schedule this operation to occur nightly, every other day,
weekly, biweekly, etc, or he could just run it manually occasionally if the
dataset doesn't change after it's been processed.

Again, given the OP's SAN hardware, this is really the only way to fly in this
case.

-- 
Stan


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Andi Kleen]

Thomas Førde <thomas.forde@bergenofs.no> writes:

> the raid array is a FC SAN array consisting of DS4700 unit with EXP810
> expansion system
>
> xfs_check first, xfs_repair -n, and xfs_repair
>
> xfs_Repair -L has not been used.
>
> its seismisc processed data, large text files.

I've seen people get text files (source files) back with strings
/dev/...

If you're lucky it's mostly continous and large 
and not too hard to sort out.

But definitely backups are a good idea next time. RAID is no replacement
for backups.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 592 bytes --]

On Freitag, 18. Juni 2010 Thomas Førde wrote:
> the raid array is a FC SAN array consisting of DS4700 unit with
>  EXP810  expansion system
 
You have the money for a IBM DS4700 (costs with disk >15.000€), but not 
for 4 cheap 2TB SATA disks to at least copy the data there? That would 
have cost <500€.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Emmanuel Florac]

[-- Attachment #1.1: Type: text/plain, Size: 745 bytes --]

Le Fri, 18 Jun 2010 18:43:01 +0200
Michael Monnerie <michael.monnerie@is.it-management.at> écrivait:

>  
> You have the money for a IBM DS4700 (costs with disk >15.000€), but
> not for 4 cheap 2TB SATA disks to at least copy the data there? That
> would have cost <500€.

This isn't very helpful. Advice to everyone : read the tao of backup
( http://www.taobackup.com ). Old, funny and still right on.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1137 bytes --]

On Freitag, 18. Juni 2010 Emmanuel Florac wrote:
> > You have the money for a IBM DS4700 (costs with disk >15.000€), but
> > not for 4 cheap 2TB SATA disks to at least copy the data there?
> > That would have cost <500€.
> 
> This isn't very helpful. Advice to everyone : read the tao of backup
> ( http://www.taobackup.com ). Old, funny and still right on.
 
I didn't say it's a backup solution. It's the simplest form of keeping 
data, and would have helped here. 

We tend to do backup-to-disk and then disk-to-tape, which until now is 
superior to just backup-to-tape. It speeds up backup and restore time, 
even when "just" using simple 2TB SATA drives (in RAID-5 arrays). The 
following disk-to-tape runs without any start/stop cycles of the tape 
(mostly LTO4 now), leading to much longer tape and tape drive lifetimes 
than before.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Stan Hoeppner]

Michael Monnerie put forth on 6/18/2010 11:43 AM:
> On Freitag, 18. Juni 2010 Thomas Førde wrote:
>> the raid array is a FC SAN array consisting of DS4700 unit with
>>  EXP810  expansion system
>  
> You have the money for a IBM DS4700 (costs with disk >15.000€), but not 
> for 4 cheap 2TB SATA disks to at least copy the data there? That would 
> have cost <500€.

I'm guessing the OP works for a petroleum company, given the seismic data.
Either that or a government agency studying volcanism.  He states their DS4700
system(s) (a single DS4700 is limited to 112 drives) have 384x300GB drives.
That setup would cost at least $250k USD at IBM list price, probably more
given the number of drives.

Money isn't the issue here.  Lack of planning and procedure seems to be the issue.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS

[Emmanuel Florac]

Le Fri, 18 Jun 2010 12:21:13 -0500 vous écriviez:

> Money isn't the issue here.  Lack of planning and procedure seems to
> be the issue.
> 

Yes, we all agree on that, Thomas must be inexperienced and nobody ever
makes backup prior to serious data loss :) Now he'll see how to do
backups, but his current problem is being able to restore the data if
possible...

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Help with XFS...

[Eric Sandeen]

Bernard Li wrote:
> Hi Eric:

You should just send this to the list (cc'd), many more eyes will see it
and might have better suggestions.

In general though if the log won't replay on mount, xfs_repair -L is the
thing.

since you're on devicemapper, you probably got burned by the lack of
barriers / ordered write support in dm, and that led to your
unreplayable log corruption when you lost power.

Going forward, I'd suggest that a UPS would be a good investment.

-Eric

> Sorry for writing you directly, but I figure you're probably the best
> person I could ask, regarding this issue.  So if you have some time, I
> would appreciate some feedback from you, thanks!
> 
> So we had a power outage and upon bringing up a file server, we can no
> longer mount a XFS partition:
> 
> mount: Structure needs cleaning
> 
> and we got the following in /var/log/messages:
> 
> Jul 25 11:48:52 vus-sxmail-v032 kernel: XFS mounting filesystem dm-7
> Jul 25 11:48:52 vus-sxmail-v032 kernel: Starting XFS recovery on
> filesystem: dm-7 (logdev: internal)
> Jul 25 11:48:52 vus-sxmail-v032 kernel: Filesystem "dm-7": XFS
> internal error xlog_valid_rec_header(1) at line 3449 of file
> fs/xfs/xfs_log_recover.c.  Caller 0xf8e833bb
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e80dcd>]
> xlog_valid_rec_header+0xcd/0xd9 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e833bb>]
> xlog_do_recovery_pass+0x21f/0x79d [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e833bb>]
> xlog_do_recovery_pass+0x21f/0x79d [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c012682d>]
> autoremove_wake_function+0xd/0x2d
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c0279d18>] schedule+0x4c2/0x538
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e8396d>]
> xlog_do_log_recovery+0x34/0x94 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e839df>]
> xlog_do_recover+0x12/0xf3 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e83b35>]
> xlog_recover+0x75/0x84 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e7e165>]
> xfs_log_mount+0x96/0xce [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e8559e>]
> xfs_mountfs+0x87f/0xaf9 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e93dc1>]
> xfs_setsize_buftarg_flags+0x2a/0x89 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e9417b>]
> xfs_buf_rele+0x21/0x5f [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e84af6>]
> xfs_readsb+0x17c/0x198 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e8b811>]
> xfs_mount+0x2e1/0x357 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e8b530>] xfs_mount+0x0/0x357 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e9a3df>] vfs_mount+0x17/0x1a [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e9a2bb>]
> xfs_fs_fill_super+0x68/0x175 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c01abdee>] snprintf+0x1c/0x1f
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c017c607>] disk_name+0x1f/0x60
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c015567e>] get_sb_bdev+0xc6/0x10e
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c013bcf1>] __alloc_pages+0x57/0x297
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e99753>]
> xfs_fs_get_sb+0xe/0x11 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<f8e9a253>]
> xfs_fs_fill_super+0x0/0x175 [xfs]
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c0155483>] do_kern_mount+0x8f/0x13d
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c0166a24>] do_mount+0x5fa/0x648
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c0165e2c>] mntput_no_expire+0x11/0x62
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c015c6b5>] link_path_walk+0xaf/0xb9
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c01626f1>] dput+0x22/0x113
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c0165e2c>] mntput_no_expire+0x11/0x62
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c01420d1>]
> __handle_mm_fault+0x7dd/0x7f3
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c013ba6c>]
> get_page_from_freelist+0x6f/0x29d
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c027b7fb>] do_page_fault+0x16c/0x51e
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c0165a75>]
> copy_mount_options+0x26/0x109
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c0166ad6>] sys_mount+0x64/0x97
> Jul 25 11:48:52 vus-sxmail-v032 kernel:  [<c010299b>]
> sysenter_past_esp+0x54/0x79
> Jul 25 11:48:52 vus-sxmail-v032 kernel: XFS: log mount/recovery
> failed: error 117
> Jul 25 11:48:52 vus-sxmail-v032 kernel: XFS: log mount failed
> 
> I think we have two options with xfs_repair:
> 
> 1) Run it with -L and zero the log
> 2) Run it without -L and try to replay the log
> 
> Any insight?  We are running 2.6.16.54-0.2.5-default kernel.
> 
> I guess it might be a good idea to run xfs_dump on the filesystem to
> try to salvage whatever we have before we try xfs_repair...
> 
> Thanks a lot in advance,
> 
> Bernard
>