xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

[Arkadiusz Miśkiewicz]

xfs_repair version 3.1.6

disconnected inode 17491441754, moving to lost+found
disconnected inode 17491441755, moving to lost+found
disconnected inode 17491441756, moving to lost+found
disconnected inode 17491441757, moving to lost+found
corrupt dinode 17491441757, extent total = 1, nblocks = 0.  This is a bug.
Please capture the filesystem metadata with xfs_metadump and
report it to xfs@oss.sgi.com.
cache_node_purge: refcount was 1, not zero (node=0x21450c90)

fatal error -- 117 - couldn't iget disconnected inode

30GB metadump image, 6.1GB compressed of ~7TB real partition
http://ixion.pld-linux.org/~arekm/lv_storage1.metadump.xz

You need ~8-12GB of memory for xfs_repair on this.

I can also provide ssh access to the system with this image and all needed 
stuff, so you don't need to download it or waste own resources.

In meantime I'll probably make ugly hack by making "couldn't iget disconnected 
inode" non fatal, so repair will be able to finish.
-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

[Christoph Hellwig]

On Mon, Oct 31, 2011 at 11:56:20AM +0100, Arkadiusz Mi??kiewicz wrote:
> 
> xfs_repair version 3.1.6
> 
> disconnected inode 17491441754, moving to lost+found
> disconnected inode 17491441755, moving to lost+found
> disconnected inode 17491441756, moving to lost+found
> disconnected inode 17491441757, moving to lost+found
> corrupt dinode 17491441757, extent total = 1, nblocks = 0.  This is a bug.
> Please capture the filesystem metadata with xfs_metadump and
> report it to xfs@oss.sgi.com.
> cache_node_purge: refcount was 1, not zero (node=0x21450c90)
> 
> fatal error -- 117 - couldn't iget disconnected inode
> 
> 30GB metadump image, 6.1GB compressed of ~7TB real partition
> http://ixion.pld-linux.org/~arekm/lv_storage1.metadump.xz
> 
> You need ~8-12GB of memory for xfs_repair on this.
> 
> I can also provide ssh access to the system with this image and all needed 
> stuff, so you don't need to download it or waste own resources.

I think I understand the problem - we found a disconnected inode,
which we try to move to lost + found.  For some reason the inode
is found to be incorrect by xfs_iformat, so iget bailds out.

The fix will be to do a pass over the the inodes we want to move
to correct such inconsistencies and/or junk them.  I'll try to prepare
a fix as soon as I get some time, but I'm fairly busy at the moment.

Btw, what did you to to the fs?  Having the total blocks out of sync
with the numbers in the data and attribute forks seems like an extremly
unusal error case.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

[Arkadiusz Miśkiewicz]

On Thursday 03 of November 2011, Christoph Hellwig wrote:
> On Mon, Oct 31, 2011 at 11:56:20AM +0100, Arkadiusz Mi??kiewicz wrote:
> > xfs_repair version 3.1.6
> > 
> > disconnected inode 17491441754, moving to lost+found
> > disconnected inode 17491441755, moving to lost+found
> > disconnected inode 17491441756, moving to lost+found
> > disconnected inode 17491441757, moving to lost+found
> > corrupt dinode 17491441757, extent total = 1, nblocks = 0.  This is a
> > bug. Please capture the filesystem metadata with xfs_metadump and
> > report it to xfs@oss.sgi.com.
> > cache_node_purge: refcount was 1, not zero (node=0x21450c90)
> > 
> > fatal error -- 117 - couldn't iget disconnected inode
> > 
> > 30GB metadump image, 6.1GB compressed of ~7TB real partition
> > http://ixion.pld-linux.org/~arekm/lv_storage1.metadump.xz
> > 
> > You need ~8-12GB of memory for xfs_repair on this.
> > 
> > I can also provide ssh access to the system with this image and all
> > needed stuff, so you don't need to download it or waste own resources.
> 
> I think I understand the problem - we found a disconnected inode,
> which we try to move to lost + found.  For some reason the inode
> is found to be incorrect by xfs_iformat, so iget bailds out.
> 
> The fix will be to do a pass over the the inodes we want to move
> to correct such inconsistencies and/or junk them.  I'll try to prepare
> a fix as soon as I get some time, but I'm fairly busy at the moment.
> 
> Btw, what did you to to the fs?  Having the total blocks out of sync
> with the numbers in the data and attribute forks seems like an extremly
> unusal error case.

Well,

This serwer has 16 various SATA disk connected to art-of-crap controller - 
Promise SuperTrak EX16350.

The system exhibits funny issues with intel_idle driver 
(https://lkml.org/lkml/2011/10/28/270).

It has only 8GB of ram which xfs_repair eats for breakfast causing watchdog to 
reboot machine while xfs_repair was in progress (would be nice if repair could 
estimate needed ram before it is too late).

All these issues combined caused few reboots in which some were in middle of 
xfs_repair. Most likely all that caused such corruption.

-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

[Christoph Hellwig]

On Thu, Nov 03, 2011 at 11:40:46AM +0100, Arkadiusz Mi??kiewicz wrote:
> This serwer has 16 various SATA disk connected to art-of-crap controller - 
> Promise SuperTrak EX16350.
> 
> The system exhibits funny issues with intel_idle driver 
> (https://lkml.org/lkml/2011/10/28/270).
> 
> It has only 8GB of ram which xfs_repair eats for breakfast causing watchdog to 
> reboot machine while xfs_repair was in progress (would be nice if repair could 
> estimate needed ram before it is too late).

Yes.  So far most of the issues are with the internal buffercache and I
suspect we could do better sizing decisions there.  If you can provide
some testing (xfs_repair -n should be enough) I'll happily send you
some RFC patches as soon as I get time for it.

In the meantime is there any chance you could send the output of

	xfs_repair -n -vv -m 1

for this filesystem?

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

[Arkadiusz Miśkiewicz]

On Thursday 03 of November 2011, Christoph Hellwig wrote:
> On Thu, Nov 03, 2011 at 11:40:46AM +0100, Arkadiusz Mi??kiewicz wrote:
> > This serwer has 16 various SATA disk connected to art-of-crap controller
> > - Promise SuperTrak EX16350.
> > 
> > The system exhibits funny issues with intel_idle driver
> > (https://lkml.org/lkml/2011/10/28/270).
> > 
> > It has only 8GB of ram which xfs_repair eats for breakfast causing
> > watchdog to reboot machine while xfs_repair was in progress (would be
> > nice if repair could estimate needed ram before it is too late).
> 
> Yes.  So far most of the issues are with the internal buffercache and I
> suspect we could do better sizing decisions there.  If you can provide
> some testing (xfs_repair -n should be enough) I'll happily send you
> some RFC patches as soon as I get time for it.

I can do some testing (on image though but that shouldn't matter).

> 
> In the meantime is there any chance you could send the output of
> 
> 	xfs_repair -n -vv -m 1
> 
> for this filesystem?

Will such repair done on metadumped & restored image be enough for you?

-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

[Christoph Hellwig]

On Thu, Nov 03, 2011 at 11:57:38AM +0100, Arkadiusz Mi??kiewicz wrote:
> > In the meantime is there any chance you could send the output of
> > 
> > 	xfs_repair -n -vv -m 1
> > 
> > for this filesystem?
> 
> Will such repair done on metadumped & restored image be enough for you?

Yes.

I also looked at the memory heuristics and found some fairly obvious
flaws.  I'll have some test patches for you ASAP.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

[Christoph Hellwig]

[-- Attachment #1: Type: text/plain, Size: 78 bytes --]

Can you give this patch a quick try on the image in that constrained
setup?



[-- Attachment #2: libxfs-estimate-free-memory-better --]
[-- Type: text/plain, Size: 6213 bytes --]

Index: xfsprogs-dev/libxfs/linux.c
===================================================================
--- xfsprogs-dev.orig/libxfs/linux.c	2011-11-03 12:01:14.213689743 +0100
+++ xfsprogs-dev/libxfs/linux.c	2011-11-03 12:46:56.928191650 +0100
@@ -207,8 +207,13 @@ platform_nproc(void)
 	return sysconf(_SC_NPROCESSORS_ONLN);
 }
 
+/*
+ * Return the memory that we can use freely.
+ *
+ * The return value is in kilobytes.
+ */
 unsigned long
-platform_physmem(void)
+platform_freemem(void)
 {
 	struct sysinfo  si;
 
@@ -217,5 +222,14 @@ platform_physmem(void)
 			progname);
 		exit(1);
 	}
-	return (si.totalram >> 10) * si.mem_unit;	/* kilobytes */
+
+	/*
+	 * Assume we can use memory that is marked free.  This is a very
+	 * conservative approximation given that there might be a lot of
+	 * pagecache that is easily reclaimable, but the only way to figure
+	 * out pagecache size is by parsing /proc/meminfo, and the format
+	 * of that file keeps changing.  This approach is still better than
+	 * guessing based on si.totalram which might be highly overestimated.
+	 */
+	return (si.freeram >> 10) * si.mem_unit;
 }
Index: xfsprogs-dev/include/libxfs.h
===================================================================
--- xfsprogs-dev.orig/include/libxfs.h	2011-11-03 12:29:02.001691672 +0100
+++ xfsprogs-dev/include/libxfs.h	2011-11-03 12:31:49.780691838 +0100
@@ -475,7 +475,7 @@ enum ce { CE_DEBUG, CE_CONT, CE_NOTE, CE
 
 #define LIBXFS_BBTOOFF64(bbs)	(((xfs_off_t)(bbs)) << BBSHIFT)
 extern int		libxfs_nproc(void);
-extern unsigned long	libxfs_physmem(void);	/* in kilobytes */
+extern unsigned long	libxfs_freemem(void);	/* in kilobytes */
 
 #include <xfs/xfs_ialloc.h>
 #include <xfs/xfs_rtalloc.h>
Index: xfsprogs-dev/libxfs/darwin.c
===================================================================
--- xfsprogs-dev.orig/libxfs/darwin.c	2011-11-03 12:29:02.017691106 +0100
+++ xfsprogs-dev/libxfs/darwin.c	2011-11-03 12:40:10.720691752 +0100
@@ -128,18 +128,27 @@ platform_nproc(void)
 	return ncpu;
 }
 
+/*
+ * Return the memory that we can use freely.
+ *
+ * The return value is in kilobytes.
+ */
 unsigned long
-platform_physmem(void)
+platform_freemem(void)
 {
-	unsigned long	physmem;
-	size_t		len = sizeof(physmem);
+	unsigned long	freemem;
+	size_t		len = sizeof(freemem);
 	static int	mib[2] = {CTL_HW, HW_PHYSMEM};
 
-	if (sysctl(mib, 2, &physmem, &len, NULL, 0) < 0) {
+	if (sysctl(mib, 2, &freemem, &len, NULL, 0) < 0) {
 		fprintf(stderr, _("%s: can't determine memory size\n"),
 			progname);
 		exit(1);
 	}
-	return physmem >> 10;
+
+ 	/*
+	 * Assume we can use approximately 3/4 of the physical memory.
+	 */
+	return (freemem >> (10 + 2)) * 3;
 }
 
Index: xfsprogs-dev/libxfs/freebsd.c
===================================================================
--- xfsprogs-dev.orig/libxfs/freebsd.c	2011-11-03 12:29:02.037693919 +0100
+++ xfsprogs-dev/libxfs/freebsd.c	2011-11-03 12:40:19.000190942 +0100
@@ -187,17 +187,26 @@ platform_nproc(void)
 	return ncpu;
 }
 
+/*
+ * Return the memory that we can use freely.
+ *
+ * The return value is in kilobytes.
+ */
 unsigned long
-platform_physmem(void)
+platform_freemem(void)
 {
-	unsigned long	physmem;
-	size_t		len = sizeof(physmem);
+	unsigned long	freemem;
+	size_t		len = sizeof(freemem);
 	static int	mib[2] = {CTL_HW, HW_PHYSMEM};
 
-	if (sysctl(mib, 2, &physmem, &len, NULL, 0) < 0) {
+	if (sysctl(mib, 2, &freemem, &len, NULL, 0) < 0) {
 		fprintf(stderr, _("%s: can't determine memory size\n"),
 			progname);
 		exit(1);
 	}
-	return physmem >> 10;
+
+ 	/*
+	 * Assume we can use approximately 3/4 of the physical memory.
+	 */
+	return (freemem >> (10 + 2)) * 3;
 }
Index: xfsprogs-dev/libxfs/init.c
===================================================================
--- xfsprogs-dev.orig/libxfs/init.c	2011-11-03 12:29:02.057690208 +0100
+++ xfsprogs-dev/libxfs/init.c	2011-11-03 12:29:50.044192568 +0100
@@ -862,7 +862,7 @@ libxfs_nproc(void)
 }
 
 unsigned long
-libxfs_physmem(void)
+libxfs_freemem(void)
 {
-	return platform_physmem();
+	return platform_freemem();
 }
Index: xfsprogs-dev/libxfs/init.h
===================================================================
--- xfsprogs-dev.orig/libxfs/init.h	2011-11-03 12:29:02.077690698 +0100
+++ xfsprogs-dev/libxfs/init.h	2011-11-03 12:29:52.296691230 +0100
@@ -32,7 +32,7 @@ extern char *platform_findblockpath (cha
 extern int platform_direct_blockdev (void);
 extern int platform_align_blockdev (void);
 extern int platform_nproc(void);
-extern unsigned long platform_physmem(void);	/* in kilobytes */
+extern unsigned long platform_freemem(void);	/* in kilobytes */
 extern int platform_has_uuid;
 
 #endif	/* LIBXFS_INIT_H */
Index: xfsprogs-dev/libxfs/irix.c
===================================================================
--- xfsprogs-dev.orig/libxfs/irix.c	2011-11-03 12:29:02.097691363 +0100
+++ xfsprogs-dev/libxfs/irix.c	2011-11-03 12:42:59.724691783 +0100
@@ -97,8 +97,13 @@ platform_nproc(void)
 	return sysmp(MP_NPROCS);
 }
 
+/*
+ * Return the memory that we can use freely.
+ *
+ * The return value is in kilobytes.
+ */
 unsigned long
-platform_physmem(void)
+platform_freemem(void)
 {
 	struct rminfo ri;
 
@@ -107,5 +112,9 @@ platform_physmem(void)
 			progname);
 		exit(1);
 	}
-	return (ri.physmem >> 10) * getpagesize();	/* kilobytes */
-}
\ No newline at end of file
+
+	/*
+	 * Assume we can use all free memory.
+	 */
+	return (ri.freemem >> 10) * getpagesize();	/* kilobytes */
+}
Index: xfsprogs-dev/repair/xfs_repair.c
===================================================================
--- xfsprogs-dev.orig/repair/xfs_repair.c	2011-11-03 12:29:02.117690613 +0100
+++ xfsprogs-dev/repair/xfs_repair.c	2011-11-03 12:32:42.348190446 +0100
@@ -631,8 +631,11 @@ main(int argc, char **argv)
 		mem_used = (mp->m_sb.sb_icount >> (10 - 2)) +
 					(mp->m_sb.sb_dblocks >> (10 + 1)) +
 					50000;	/* rough estimate of 50MB overhead */
-		max_mem = max_mem_specified ? max_mem_specified * 1024 :
-						libxfs_physmem() * 3 / 4;
+
+		if (max_mem_specified)
+			max_mem = max_mem_specified * 1024;
+		else
+			max_mem = libxfs_freemem();
 
 		if (getrlimit(RLIMIT_AS, &rlim) != -1 &&
 					rlim.rlim_cur != RLIM_INFINITY) {

[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

[Arkadiusz Miśkiewicz]

On Thursday 03 of November 2011, Christoph Hellwig wrote:
> Can you give this patch a quick try on the image in that constrained
> setup?

Ate up all 8GB of ram and then was killed:

[217802.127030] Out of memory: Kill process 22898:#0 (xfs_repair) score 784 or 
sacrifice child
[217802.127033] Killed process 22898:#0 (xfs_repair) total-vm:7156064kB, anon-
rss:6652784kB, file-rss:620kB

Now the question is if 8GB should be enough for repairing 7TB filesystem?

-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

[Arkadiusz Miśkiewicz]

On Thursday 03 of November 2011, Christoph Hellwig wrote:
> On Thu, Nov 03, 2011 at 11:57:38AM +0100, Arkadiusz Mi??kiewicz wrote:
> > > In the meantime is there any chance you could send the output of
> > > 
> > > 	xfs_repair -n -vv -m 1
> > > 
> > > for this filesystem?
> > 
> > Will such repair done on metadumped & restored image be enough for you?
> 
> Yes.
> 
> I also looked at the memory heuristics and found some fairly obvious
> flaws.  I'll have some test patches for you ASAP.

# LC_ALL=C ./xfs_repair -vv -m 1 /dev/vg_sys/lv_storage1
Phase 1 - find and verify superblock...
        - max_mem = 1024, icount = 97495744, imem = 380842, dblock = 
1884752896, dmem = 920289
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 1319.


-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

[Arkadiusz Miśkiewicz]

On Monday 31 of October 2011, Arkadiusz Miśkiewicz wrote:
> xfs_repair version 3.1.6
> 
> disconnected inode 17491441754, moving to lost+found
> disconnected inode 17491441755, moving to lost+found
> disconnected inode 17491441756, moving to lost+found
> disconnected inode 17491441757, moving to lost+found
> corrupt dinode 17491441757, extent total = 1, nblocks = 0.  This is a bug.
> Please capture the filesystem metadata with xfs_metadump and
> report it to xfs@oss.sgi.com.
> cache_node_purge: refcount was 1, not zero (node=0x21450c90)
> 
> fatal error -- 117 - couldn't iget disconnected inode

> In meantime I'll probably make ugly hack by making "couldn't iget
> disconnected inode" non fatal, so repair will be able to finish.

With this one repair finished and then while repairing for second time
(this time without hack) I got glibc catching invalid free:

name create failed in ino 17873999459 (117), filesystem may be out of space
bad hash table for directory inode 17875023137 (brak wpisu danych): przebudowano
rebuilding directory inode 17875023137
*** glibc detected *** /sbin/xfs_repair: free(): invalid next size (normal): 0x00007f9ffcdc4c00 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x758b6)[0x7fa146a6b8b6]
/lib64/libxfs.so.0(libxfs_getbufr+0x81)[0x7fa14738de81]
/lib64/libxfs.so.0(cache_node_get+0xca)[0x7fa14738a39a]
/lib64/libxfs.so.0(libxfs_getbuf+0x29)[0x7fa14738dfc9]
/sbin/xfs_repair[0x425cd0]
/sbin/xfs_repair[0x426aa6]
/lib64/libpthread.so.0(+0x7ed5)[0x7fa146d8ced5]
/lib64/libc.so.6(clone+0x6d)[0x7fa146acfe5d]
======= Memory map: ========
00400000-0043e000 r-xp 00000000 08:12 125854083                          /sbin/xfs_repair
0043f000-00440000 r--p 0003e000 08:12 125854083                          /sbin/xfs_repair
00440000-00441000 rw-p 0003f000 08:12 125854083                          /sbin/xfs_repair
00441000-2aa06000 rw-p 00000000 00:00 0                                  [heap]
7f9f00000000-7f9f01123000 rw-p 00000000 00:00 0
7f9f01123000-7f9f04000000 ---p 00000000 00:00 0
7f9f077ff000-7f9f07800000 ---p 00000000 00:00 0
7f9f07800000-7f9f08000000 rw-p 00000000 00:00 0
[...]

-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs