xfs_repair breaks with assertion

Victor K
2013-04-11T05:25:31Z

[-- Attachment #1.1: Type: text/plain, Size: 4380 bytes --]

Hello,

I'm trying to repair an XFS file system on our mdadm raid6 array after
sudden system failure.
Running xfs_repair /dev/md1 the first time resulted in suggestion to
mount/unmount to replay log, but mounting would not work. After running
xfs_repair -v -L -P /dev/md1 this happens:
(lots of output on stderr, moving to Phase 3, then more output - not sure
if it is relevant, the log file is ~170Mb in size), then stops and prints
the only line on stdout:

xfs_repair: dinode.c:768: process_bmbt_reclist_int: Assertion `i <
*numrecs' failed.
Aborted

After inserting a printf before the assert, I get the following:

i = 0, *numrecs = -570425343 for printf( "%d, %d")
or
i= 0, *numrecs = 3724541953 for printf("%ld, %ld) - makes me wonder if
it's signed/unsigned int related

both trips on if(i>*numrecs) conditional

The filesystem size is 10Tb (7x2Tb disks in raid6) and it is about 8Tb full.

xfsprogs version is 3.1.10 compiled from git source this morning.

The system is Ubuntu 12.04.2 with kernel version 3.8.5.

When I try to run xfs_metadump, it crashes:
*** glibc detected *** xfs_db: double free or corruption (!prev):
0x0000000000da800
0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f3d9501cb96]
xfs_db[0x417383]
xfs_db[0x41a941]
xfs_db[0x419030]
xfs_db[0x41a85c]
xfs_db[0x419030]
xfs_db[0x41b89e]
xfs_db[0x4050c0]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f3d94fbf76d]
xfs_db[0x4051c5]
======= Memory map: ========
00400000-0046e000 r-xp 00000000 08:81 1837319
/usr/sbin/
xfs_db
0066d000-0066e000 r--p 0006d000 08:81 1837319
/usr/sbin/
xfs_db
0066e000-0066f000 rw-p 0006e000 08:81 1837319
/usr/sbin/
xfs_db
0066f000-00682000 rw-p 00000000 00:00 0
00d63000-00dc4000 rw-p 00000000 00:00 0
[heap]
7f3d94d88000-7f3d94d9d000 r-xp 00000000 08:81 2363486
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f3d94d9d000-7f3d94f9c000 ---p 00015000 08:81 2363486
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f3d94f9c000-7f3d94f9d000 r--p 00014000 08:81 2363486
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f3d94f9d000-7f3d94f9e000 rw-p 00015000 08:81 2363486
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f3d94f9e000-7f3d95153000 r-xp 00000000 08:81 2423054
/lib/x86_64-linux-gnu/libc-2.15.so
7f3d95153000-7f3d95352000 ---p 001b5000 08:81 2423054
/lib/x86_64-linux-gnu/libc-2.15.so
7f3d95352000-7f3d95356000 r--p 001b4000 08:81 2423054
/lib/x86_64-linux-gnu/libc-2.15.so
7f3d95356000-7f3d95358000 rw-p 001b8000 08:81 2423054
/lib/x86_64-linux-gnu/libc-2.15.so
7f3d95358000-7f3d9535d000 rw-p 00000000 00:00 0
7f3d9535d000-7f3d95375000 r-xp 00000000 08:81 2423056
/lib/x86_64-linux-gnu/libpthread-2.15.so
7f3d95375000-7f3d95574000 ---p 00018000 08:81 2423056
/lib/x86_64-linux-gnu/libpthread-2.15.so
7f3d95574000-7f3d95575000 r--p 00017000 08:81 2423056
/lib/x86_64-linux-gnu/libpthread-2.15.so
7f3d95575000-7f3d95576000 rw-p 00018000 08:81 2423056
/lib/x86_64-linux-gnu/libpthread-2.15.so
7f3d95576000-7f3d9557a000 rw-p 00000000 00:00 0
7f3d9557a000-7f3d9557e000 r-xp 00000000 08:81 2359972
/lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f3d9557e000-7f3d9577d000 ---p 00004000 08:81 2359972
/lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f3d9577d000-7f3d9577e000 r--p 00003000 08:81 2359972
/lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f3d9577e000-7f3d9577f000 rw-p 00004000 08:81 2359972
/lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f3d9577f000-7f3d957a1000 r-xp 00000000 08:81 2423068
/lib/x86_64-linux-gnu/ld-2.15.so
7f3d957ba000-7f3d957fb000 rw-p 00000000 00:00 0
7f3d957fb000-7f3d95985000 r--p 00000000 08:81 1967430
/usr/lib/locale/locale-archive
7f3d95985000-7f3d95989000 rw-p 00000000 00:00 0
7f3d9599d000-7f3d959a1000 rw-p 00000000 00:00 0
7f3d959a1000-7f3d959a2000 r--p 00022000 08:81 2423068
/lib/x86_64-linux-gnu/ld-2.15.so
7f3d959a2000-7f3d959a4000 rw-p 00023000 08:81 2423068
/lib/x86_64-linux-gnu/ld-2.15.so
7fffa80d8000-7fffa80f9000 rw-p 00000000 00:00 0
[stack]
7fffa8170000-7fffa8171000 r-xp 00000000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
Aborted

It produces a file of size 4325376 bytes - not sure if it's right, as I
read about sizes of 80Mb for the dump file.

If I try now (after running xfs_repair -L) to mount the fs read-only, it
mounts but says some directories have structures that need cleaning, so the
dirs are inaccessible.

Any suggestion on how to possibly fix this?

Thanks!
Victor

[-- Attachment #1.2: Type: text/html, Size: 6849 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]