xfs_repair - segemnation fault | git error

xfs_repair - segemnation fault | git error

[Velimir Galic]

[-- Attachment #1.1: Type: text/plain, Size: 1410 bytes --]

hi Guys,

I hope you are able to help me, because i think that i
tried everything that i know off to repair my raid after a disk failure.

Systemspecs:

   - HP N40L
      - CPU: AMD Turion II @ 1,5GHZ
      - 8GB ECC RAM
      - Disks:
         - 250GB | OS
         - 4x2GB | RAID 5 | Softraid
      - OS: Openmediavault @ 3.0.20 (last version)


Problem:

In my raid a disk was fault so mdadm kicked the disk out of the raid. I put
a replacement in the box and started a resync of the raid and than
is happened, a second disk got lost in the raid. The raidsync wasn't
finished and now i'm fighting 3 weeks or more i don't no any more :-( to
repair the file system.

I tried a lot of different versions of xfs_repair (2.9.4, 2.9.8, 2.10.2,
3.0.4, 3.1.2, 3.1.6, 3.1,8), but every precompiled version got an
"segmentation fault" or hung at phase 3 with 100% cpu load for few day. I
also tried different distros like ubuntu, debian, redhat, gentoo (yes, i'm
a little bit desperate :-) )  Than i tried the git version and at first it
looked good but than i got an another error but i don't understand what the
problem is.

tried options without any luck: xfs_repair -v -P | xfs_repair -v -P -m 6144
| xfs_repair -v -P -L

Metadump of the filesystem isn't possible at this moment, but i could
bootup a live cd and give it a try with it. I'm pretty stuck and don't no
what to do any more.


with kind regards
Veli

[-- Attachment #1.2: Type: text/html, Size: 1789 bytes --]

[-- Attachment #2: md127.metadump.err --]
[-- Type: application/octet-stream, Size: 3536 bytes --]

root@hpbox:~# xfs_metadump /dev/md127 /media/localhdd/md127.metadump
*** glibc detected *** xfs_db: double free or corruption (!prev): 0x0000000001ccb000 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71bd6)[0x7f5e34b55bd6]
/lib/libc.so.6(cfree+0x6c)[0x7f5e34b5a94c]
xfs_db[0x41a8e3]
xfs_db[0x41c3ec]
xfs_db[0x41e2bc]
xfs_db[0x41c152]
xfs_db[0x41dfd4]
xfs_db[0x41c152]
xfs_db[0x41eedb]
xfs_db[0x418d08]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7f5e34b02c8d]
xfs_db[0x4027d9]
======= Memory map: ========
00400000-00475000 r-xp 00000000 08:41 2338                               /usr/sbin/xfs_db
00675000-00676000 rw-p 00075000 08:41 2338                               /usr/sbin/xfs_db
00676000-00689000 rw-p 00000000 00:00 0 
01c83000-01f1d000 rw-p 00000000 00:00 0                                  [heap]
7f5e30000000-7f5e30021000 rw-p 00000000 00:00 0 
7f5e30021000-7f5e34000000 ---p 00000000 00:00 0 
7f5e348c7000-7f5e348dd000 r-xp 00000000 08:41 21105                      /lib/libgcc_s.so.1
7f5e348dd000-7f5e34adc000 ---p 00016000 08:41 21105                      /lib/libgcc_s.so.1
7f5e34adc000-7f5e34add000 rw-p 00015000 08:41 21105                      /lib/libgcc_s.so.1
7f5e34ae4000-7f5e34c3d000 r-xp 00000000 08:41 3222                       /lib/libc-2.11.3.so
7f5e34c3d000-7f5e34e3c000 ---p 00159000 08:41 3222                       /lib/libc-2.11.3.so
7f5e34e3c000-7f5e34e40000 r--p 00158000 08:41 3222                       /lib/libc-2.11.3.so
7f5e34e40000-7f5e34e41000 rw-p 0015c000 08:41 3222                       /lib/libc-2.11.3.so
7f5e34e41000-7f5e34e46000 rw-p 00000000 00:00 0 
7f5e34e46000-7f5e34e5d000 r-xp 00000000 08:41 3217                       /lib/libpthread-2.11.3.so
7f5e34e5d000-7f5e3505c000 ---p 00017000 08:41 3217                       /lib/libpthread-2.11.3.so
7f5e3505c000-7f5e3505d000 r--p 00016000 08:41 3217                       /lib/libpthread-2.11.3.so
7f5e3505d000-7f5e3505e000 rw-p 00017000 08:41 3217                       /lib/libpthread-2.11.3.so
7f5e3505e000-7f5e35062000 rw-p 00000000 00:00 0 
7f5e35062000-7f5e35069000 r-xp 00000000 08:41 3219                       /lib/librt-2.11.3.so
7f5e35069000-7f5e35268000 ---p 00007000 08:41 3219                       /lib/librt-2.11.3.so
7f5e35268000-7f5e35269000 r--p 00006000 08:41 3219                       /lib/librt-2.11.3.so
7f5e35269000-7f5e3526a000 rw-p 00007000 08:41 3219                       /lib/librt-2.11.3.so
7f5e3526a000-7f5e3526e000 r-xp 00000000 08:41 21195                      /lib/libuuid.so.1.3.0
7f5e3526e000-7f5e3546d000 ---p 00004000 08:41 21195                      /lib/libuuid.so.1.3.0
7f5e3546d000-7f5e3546e000 rw-p 00003000 08:41 21195                      /lib/libuuid.so.1.3.0
7f5e3546e000-7f5e3548c000 r-xp 00000000 08:41 3218                       /lib/ld-2.11.3.so
7f5e354c7000-7f5e35509000 rw-p 00000000 00:00 0 
7f5e35509000-7f5e3567e000 r--p 00000000 08:41 2104                       /usr/lib/locale/locale-archive
7f5e3567e000-7f5e35682000 rw-p 00000000 00:00 0 
7f5e35689000-7f5e3568b000 rw-p 00000000 00:00 0 
7f5e3568b000-7f5e3568c000 r--p 0001d000 08:41 3218                       /lib/ld-2.11.3.so
7f5e3568c000-7f5e3568d000 rw-p 0001e000 08:41 3218                       /lib/ld-2.11.3.so
7f5e3568d000-7f5e3568e000 rw-p 00000000 00:00 0 
7fff33b52000-7fff33b73000 rw-p 00000000 00:00 0                          [stack]
7fff33bff000-7fff33c00000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair - segemnation fault | git error

[Dave Chinner]

On Fri, Aug 10, 2012 at 10:23:03AM +0200, Velimir Galic wrote:
> hi Guys,
> 
> I hope you are able to help me, because i think that i
> tried everything that i know off to repair my raid after a disk failure.
> 
> Systemspecs:
> 
>    - HP N40L
>       - CPU: AMD Turion II @ 1,5GHZ
>       - 8GB ECC RAM
>       - Disks:
>          - 250GB | OS
>          - 4x2GB | RAID 5 | Softraid
>       - OS: Openmediavault @ 3.0.20 (last version)
> 
> 
> Problem:
> 
> In my raid a disk was fault so mdadm kicked the disk out of the raid. I put
> a replacement in the box and started a resync of the raid and than
> is happened, a second disk got lost in the raid. The raidsync wasn't

Ok, so you've lost a good chunk of the filesystem then? I don't lik
eto say it, but double disk failures tend to result in unrecoverable
data loss. Given the contents of this email, I know you don't have
backups to restore from, so you might be left with only bits and
pieces even if a repair can be made to run successfully.

> finished and now i'm fighting 3 weeks or more i don't no any more :-( to
> repair the file system.
> 
> I tried a lot of different versions of xfs_repair (2.9.4, 2.9.8, 2.10.2,
> 3.0.4, 3.1.2, 3.1.6, 3.1,8), but every precompiled version got an
> "segmentation fault" or hung at phase 3 with 100% cpu load for few day. I
> also tried different distros like ubuntu, debian, redhat, gentoo (yes, i'm
> a little bit desperate :-) )  Than i tried the git version and at first it
> looked good but than i got an another error but i don't understand what the
> problem is.

Perhaps you should tell us whatthe error is. i.e. attach the output
of xfs_repair when it fails.

> 
> tried options without any luck: xfs_repair -v -P | xfs_repair -v -P -m 6144
> | xfs_repair -v -P -L
> 
> Metadump of the filesystem isn't possible at this moment,

Why not?

> but i could
> bootup a live cd and give it a try with it. I'm pretty stuck and don't no
> what to do any more.

Run your compiled, unstripped xfs-repair binary under gdb and when
it segfaults, dump of the stack trace. That will at least tell use
where it is failing....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair - segemnation fault | git error

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 878 bytes --]

Am Freitag, 10. August 2012, 10:23:03 schrieb Velimir Galic:
> a second disk got lost in the raid. The raidsync wasn't finished and
> now i'm fighting 3 weeks or more i don't no any more :-( to repair
> the file system. 

How far did the recovery go when it broke? If it was on 10% when the 2nd 
disk broke, you can stop to try to get anything. If it was 90% finished, 
you might have a chance...

> 4x2GB | RAID 5 | Softraid

This is a 6GB net RAID 5 device? Which devices?

> I tried a lot of different versions of xfs_repair

I hope you got a copy of all disks before you even tried to repair? 
Otherwise it might be too late now, when repair wrote something on that 
device.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair - segemnation fault | git error

[Velimir Galic]

[-- Attachment #1.1: Type: text/plain, Size: 1256 bytes --]

hi,

Thanks for the fast replay and the effort!

2012/8/11 Michael Monnerie <michael.monnerie@is.it-management.at>

> Am Freitag, 10. August 2012, 10:23:03 schrieb Velimir Galic:
> > a second disk got lost in the raid. The raidsync wasn't finished and
> > now i'm fighting 3 weeks or more i don't no any more :-( to repair
> > the file system.
>
> How far did the recovery go when it broke? If it was on 10% when the 2nd
> disk broke, you can stop to try to get anything. If it was 90% finished,
> you might have a chance...
>

that's what I thought but hoped for that isn't a problem. The sync failed
at approx. 20% or beneath.


> > 4x2GB | RAID 5 | Softraid
>
> This is a 6GB net RAID 5 device? Which devices?
>

WDC WD20EARS-22MVWB0


>
> > I tried a lot of different versions of xfs_repair
>
> I hope you got a copy of all disks before you even tried to repair?
> Otherwise it might be too late now, when repair wrote something on that
> device.
>
>
Nope, I wasn't clever enough to do that, but it isn't hopeless because I
have a
3 months old backup form my important files. So I have to life with the 3
months lost, I hope is not to much.


Thanks for trying to help, but I stop her with trying to recover anything
and start over.

Shit happens! :-)

[-- Attachment #1.2: Type: text/html, Size: 2155 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs