filesystem corruption in linus tree

filesystem corruption in linus tree

[Marc Dietrich]

Hi,

somewhere after the release of 2.6.24 my xfs filesystem got  
corrupted. Initialy I thought it was only related to the readdir bug.  
(http://oss.sgi.com/archives/xfs/2008-02/msg00027.html) So I waited  
for the fix to go into mainline. Yesterday I tried again, but got  
this error during boot:

Feb 23 22:27:00 g3 kernel: Filesystem "sda5": XFS internal error  
xfs_bmapi at line 4757 of file fs/xfs/xfs_bmap.c.  Caller 0xc01199a8
Feb 23 22:27:00 g3 kernel: Call Trace:
Feb 23 22:27:00 g3 kernel: [df84fa20] [c0008260] show_stack 
+0x48/0x19c (unreliable)
Feb 23 22:27:00 g3 kernel: [df84fa60] [c0109c80] xfs_error_report 
+0x58/0x5c
Feb 23 22:27:00 g3 kernel: [df84fa70] [c00f6b04] xfs_bmapi+0x10a4/0x1128
Feb 23 22:27:00 g3 kernel: [df84fbd0] [c01199a8] xfs_iomap+0x13c/0x33c
Feb 23 22:27:00 g3 kernel: [df84fc50] [c01372c4] xfs_map_blocks 
+0x34/0xac
Feb 23 22:27:00 g3 kernel: [df84fc80] [c01384a0]  
xfs_page_state_convert+0x560/0x74c
Feb 23 22:27:00 g3 kernel: [df84fd40] [c01389e0] xfs_vm_writepage 
+0x70/0x11c
Feb 23 22:27:00 g3 kernel: [df84fd70] [c0063e7c] __writepage+0x24/0x8c
Feb 23 22:27:00 g3 kernel: [df84fd80] [c0064c48] write_cache_pages 
+0x284/0x360
Feb 23 22:27:00 g3 kernel: [df84fe20] [c0138930] xfs_vm_writepages 
+0x64/0x78
Feb 23 22:27:00 g3 kernel: [df84fe30] [c0064d98] do_writepages+0x4c/0x84
Feb 23 22:27:00 g3 kernel: [df84fe40] [c00aac48]  
__writeback_single_inode+0xa0/0x3f8
Feb 23 22:27:00 g3 kernel: [df84fea0] [c00ab53c] sync_sb_inodes 
+0x2b4/0x410
Feb 23 22:27:00 g3 kernel: [df84fee0] [c00ab9b4] writeback_inodes 
+0xb8/0x190
Feb 23 22:27:00 g3 kernel: [df84ff00] [c006547c] background_writeout 
+0xc4/0x104
Feb 23 22:27:00 g3 kernel: [df84ff70] [c0065ad8] pdflush+0x178/0x270
Feb 23 22:27:00 g3 kernel: [df84ffd0] [c0042ad8] kthread+0x50/0x88
Feb 23 22:27:00 g3 kernel: [df84fff0] [c00104b8] kernel_thread+0x44/0x60

I seems that the log is corrupted because I can't mount the  
filesystem anymore (blocks while replaying). Mounting with norecovery  
option works. I tried to repair the filesystem, but xfs_repair also  
hangs in "Phase 3 - scan and clear agi unlinks lists". Before, Phase  
2 reports some "ir_freecount/free.mismatch" lines.

This happens on a newly created filesystem with xfsprogs 2.6.9 (I  
also tested 2.5.22). I have a 32bit powermac, so it could be an  
endian issue.

Thanks

Marc

Re: filesystem corruption in linus tree

[Barry Naujok]

On Sun, 24 Feb 2008 20:58:26 +1100, Marc Dietrich  
<Marc.Dietrich@ap.physik.uni-giessen.de> wrote:

> Hi,
>
> somewhere after the release of 2.6.24 my xfs filesystem got corrupted.  
> Initialy I thought it was only related to the readdir bug.  
> (http://oss.sgi.com/archives/xfs/2008-02/msg00027.html) So I waited for  
> the fix to go into mainline. Yesterday I tried again, but got this error  
> during boot:

[stack trace snipped]

> I seems that the log is corrupted because I can't mount the filesystem  
> anymore (blocks while replaying). Mounting with norecovery option works.  
> I tried to repair the filesystem, but xfs_repair also hangs in "Phase 3  
> - scan and clear agi unlinks lists". Before, Phase 2 reports some  
> "ir_freecount/free.mismatch" lines.

Might have to add this to the FAQ as this is the second time someone has
reported Phase 3 hanging.

Try running xfs_repair -P to try and get past this problem if you are
running xfsprogs 2.9.x.

> This happens on a newly created filesystem with xfsprogs 2.6.9 (I also  
> tested 2.5.22). I have a 32bit powermac, so it could be an endian issue.
>
> Thanks
>
> Marc
>
>
>

Re: filesystem corruption in linus tree

[Marc Dietrich]

Hi,

On Monday 25 February 2008 01:36:28 Barry Naujok wrote:
> On Sun, 24 Feb 2008 20:58:26 +1100, Marc Dietrich
>
> <Marc.Dietrich@ap.physik.uni-giessen.de> wrote:
> > Hi,
> >
> > somewhere after the release of 2.6.24 my xfs filesystem got corrupted.
> > Initialy I thought it was only related to the readdir bug.
> > (http://oss.sgi.com/archives/xfs/2008-02/msg00027.html) So I waited for
> > the fix to go into mainline. Yesterday I tried again, but got this error
> > during boot:
>
> [stack trace snipped]
>
> > I seems that the log is corrupted because I can't mount the filesystem
> > anymore (blocks while replaying). Mounting with norecovery option works.
> > I tried to repair the filesystem, but xfs_repair also hangs in "Phase 3
> > - scan and clear agi unlinks lists". Before, Phase 2 reports some
> > "ir_freecount/free.mismatch" lines.
>
> Might have to add this to the FAQ as this is the second time someone has
> reported Phase 3 hanging.
>
> Try running xfs_repair -P to try and get past this problem if you are
> running xfsprogs 2.9.x.

I tried this, but didn't work either and is also not my main problem. The 
problem is, as the subject says, that every time I boot linux the filesystem 
gets corrupted (and unrepairable). 

I will setup a testing partition, do some additional tests (could be a 
miscompilation, I'm using gcc-4.3) and report back. 

> > This happens on a newly created filesystem with xfsprogs 2.6.9 (I also
> > tested 2.5.22). I have a 32bit powermac, so it could be an endian issue.

Thanks

Marc

Re: filesystem corruption in linus tree

[Lachlan McIlroy]

[-- Attachment #1: Type: text/plain, Size: 1819 bytes --]

Marc Dietrich wrote:
> Hi,
> 
> On Monday 25 February 2008 01:36:28 Barry Naujok wrote:
>> On Sun, 24 Feb 2008 20:58:26 +1100, Marc Dietrich
>>
>> <Marc.Dietrich@ap.physik.uni-giessen.de> wrote:
>>> Hi,
>>>
>>> somewhere after the release of 2.6.24 my xfs filesystem got corrupted.
>>> Initialy I thought it was only related to the readdir bug.
>>> (http://oss.sgi.com/archives/xfs/2008-02/msg00027.html) So I waited for
>>> the fix to go into mainline. Yesterday I tried again, but got this error
>>> during boot:
>> [stack trace snipped]
>>
>>> I seems that the log is corrupted because I can't mount the filesystem
>>> anymore (blocks while replaying). Mounting with norecovery option works.
>>> I tried to repair the filesystem, but xfs_repair also hangs in "Phase 3
>>> - scan and clear agi unlinks lists". Before, Phase 2 reports some
>>> "ir_freecount/free.mismatch" lines.
>> Might have to add this to the FAQ as this is the second time someone has
>> reported Phase 3 hanging.
>>
>> Try running xfs_repair -P to try and get past this problem if you are
>> running xfsprogs 2.9.x.
> 
> I tried this, but didn't work either and is also not my main problem. The 
> problem is, as the subject says, that every time I boot linux the filesystem 
> gets corrupted (and unrepairable). 
> 
> I will setup a testing partition, do some additional tests (could be a 
> miscompilation, I'm using gcc-4.3) and report back. 
> 
>>> This happens on a newly created filesystem with xfsprogs 2.6.9 (I also
>>> tested 2.5.22). I have a 32bit powermac, so it could be an endian issue.
> 

We've had a few problems reported with XFS on 32-bit powermacs and the culprit
appears to be some changes to bit manipulation routines.  Could you please try
reverse applying the attached patches and see if the problem is resolved?

Lachlan

[-- Attachment #2: patch.1 --]
[-- Type: text/plain, Size: 1209 bytes --]

commit edd319dc527733e61eec5bdc9ce20c94634b6482
Author: David Chinner <dgc@sgi.com>
Date:   Fri Dec 7 14:08:48 2007 +1100

    [XFS] Fix xfs_lowbit64
    
    xfs_lowbit64 was broken on 32 bit platforms in a recent cleanup of the xfs
    bitops. Fix it back up again.
    
    SGI-PV: 974005
    SGI-Modid: xfs-linux-melb:xfs-kern:30202a
    
    Signed-off-by: David Chinner <dgc@sgi.com>
    Signed-off-by: Lachlan McIlroy <lachlan@sgi.com>

diff --git a/fs/xfs/xfs_bit.h b/fs/xfs/xfs_bit.h
index 0f9fc9a..325a007 100644
--- a/fs/xfs/xfs_bit.h
+++ b/fs/xfs/xfs_bit.h
@@ -61,15 +61,15 @@ static inline int xfs_highbit64(__uint64
 /* Get low bit set out of 32-bit argument, -1 if none set */
 static inline int xfs_lowbit32(__uint32_t v)
 {
-	unsigned long	t = v;
-	return (v) ? find_first_bit(&t, 32) : -1;
+	__uint32_t t = v;
+	return (t) ? find_first_bit((unsigned long *)&t, 32) : -1;
 }
 
 /* Get low bit set out of 64-bit argument, -1 if none set */
 static inline int xfs_lowbit64(__uint64_t v)
 {
-	unsigned long	t = v;
-	return (v) ? find_first_bit(&t, 64) : -1;
+	__uint64_t t = v;
+	return (t) ? find_first_bit((unsigned long *)&t, 64) : -1;
 }
 
 /* Return whether bitmap is empty (1 == empty) */

[-- Attachment #3: patch.2 --]
[-- Type: text/plain, Size: 6115 bytes --]

commit a69b176df246d59626e6a9c640b44c0921fa4566
Author: David Chinner <dgc@sgi.com>
Date:   Fri Nov 23 16:27:59 2007 +1100

    [XFS] Use the generic bitops rather than implementing them ourselves.
    Patch inspired by Andi Kleen.
    
    SGI-PV: 971186
    SGI-Modid: xfs-linux-melb:xfs-kern:30000a
    
    Signed-off-by: David Chinner <dgc@sgi.com>
    Signed-off-by: Christoph Hellwig <hch@infradead.org>
    Signed-off-by: Lachlan McIlroy <lachlan@sgi.com>

diff --git a/fs/xfs/xfs_bit.c b/fs/xfs/xfs_bit.c
index fab0b6d..4822884 100644
--- a/fs/xfs/xfs_bit.c
+++ b/fs/xfs/xfs_bit.c
@@ -25,109 +25,6 @@
  * XFS bit manipulation routines, used in non-realtime code.
  */
 
-#ifndef HAVE_ARCH_HIGHBIT
-/*
- * Index of high bit number in byte, -1 for none set, 0..7 otherwise.
- */
-static const char xfs_highbit[256] = {
-       -1, 0, 1, 1, 2, 2, 2, 2,			/* 00 .. 07 */
-	3, 3, 3, 3, 3, 3, 3, 3,			/* 08 .. 0f */
-	4, 4, 4, 4, 4, 4, 4, 4,			/* 10 .. 17 */
-	4, 4, 4, 4, 4, 4, 4, 4,			/* 18 .. 1f */
-	5, 5, 5, 5, 5, 5, 5, 5,			/* 20 .. 27 */
-	5, 5, 5, 5, 5, 5, 5, 5,			/* 28 .. 2f */
-	5, 5, 5, 5, 5, 5, 5, 5,			/* 30 .. 37 */
-	5, 5, 5, 5, 5, 5, 5, 5,			/* 38 .. 3f */
-	6, 6, 6, 6, 6, 6, 6, 6,			/* 40 .. 47 */
-	6, 6, 6, 6, 6, 6, 6, 6,			/* 48 .. 4f */
-	6, 6, 6, 6, 6, 6, 6, 6,			/* 50 .. 57 */
-	6, 6, 6, 6, 6, 6, 6, 6,			/* 58 .. 5f */
-	6, 6, 6, 6, 6, 6, 6, 6,			/* 60 .. 67 */
-	6, 6, 6, 6, 6, 6, 6, 6,			/* 68 .. 6f */
-	6, 6, 6, 6, 6, 6, 6, 6,			/* 70 .. 77 */
-	6, 6, 6, 6, 6, 6, 6, 6,			/* 78 .. 7f */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* 80 .. 87 */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* 88 .. 8f */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* 90 .. 97 */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* 98 .. 9f */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* a0 .. a7 */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* a8 .. af */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* b0 .. b7 */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* b8 .. bf */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* c0 .. c7 */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* c8 .. cf */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* d0 .. d7 */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* d8 .. df */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* e0 .. e7 */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* e8 .. ef */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* f0 .. f7 */
-	7, 7, 7, 7, 7, 7, 7, 7,			/* f8 .. ff */
-};
-#endif
-
-/*
- * xfs_highbit32: get high bit set out of 32-bit argument, -1 if none set.
- */
-inline int
-xfs_highbit32(
-	__uint32_t	v)
-{
-#ifdef HAVE_ARCH_HIGHBIT
-	return highbit32(v);
-#else
-	int		i;
-
-	if (v & 0xffff0000)
-		if (v & 0xff000000)
-			i = 24;
-		else
-			i = 16;
-	else if (v & 0x0000ffff)
-		if (v & 0x0000ff00)
-			i = 8;
-		else
-			i = 0;
-	else
-		return -1;
-	return i + xfs_highbit[(v >> i) & 0xff];
-#endif
-}
-
-/*
- * xfs_lowbit64: get low bit set out of 64-bit argument, -1 if none set.
- */
-int
-xfs_lowbit64(
-	__uint64_t	v)
-{
-	__uint32_t	w = (__uint32_t)v;
-	int		n = 0;
-
-	if (w) {	/* lower bits */
-		n = ffs(w);
-	} else {	/* upper bits */
-		w = (__uint32_t)(v >> 32);
-		if (w && (n = ffs(w)))
-			n += 32;
-	}
-	return n - 1;
-}
-
-/*
- * xfs_highbit64: get high bit set out of 64-bit argument, -1 if none set.
- */
-int
-xfs_highbit64(
-	__uint64_t	v)
-{
-	__uint32_t	h = (__uint32_t)(v >> 32);
-
-	if (h)
-		return xfs_highbit32(h) + 32;
-	return xfs_highbit32((__uint32_t)v);
-}
-
-
 /*
  * Return whether bitmap is empty.
  * Size is number of words in the bitmap, which is padded to word boundary
diff --git a/fs/xfs/xfs_bit.h b/fs/xfs/xfs_bit.h
index 082641a..0f9fc9a 100644
--- a/fs/xfs/xfs_bit.h
+++ b/fs/xfs/xfs_bit.h
@@ -47,13 +47,30 @@ static inline __uint64_t xfs_mask64lo(in
 }
 
 /* Get high bit set out of 32-bit argument, -1 if none set */
-extern int xfs_highbit32(__uint32_t v);
-
-/* Get low bit set out of 64-bit argument, -1 if none set */
-extern int xfs_lowbit64(__uint64_t v);
+static inline int xfs_highbit32(__uint32_t v)
+{
+	return fls(v) - 1;
+}
 
 /* Get high bit set out of 64-bit argument, -1 if none set */
-extern int xfs_highbit64(__uint64_t);
+static inline int xfs_highbit64(__uint64_t v)
+{
+	return fls64(v) - 1;
+}
+
+/* Get low bit set out of 32-bit argument, -1 if none set */
+static inline int xfs_lowbit32(__uint32_t v)
+{
+	unsigned long	t = v;
+	return (v) ? find_first_bit(&t, 32) : -1;
+}
+
+/* Get low bit set out of 64-bit argument, -1 if none set */
+static inline int xfs_lowbit64(__uint64_t v)
+{
+	unsigned long	t = v;
+	return (v) ? find_first_bit(&t, 64) : -1;
+}
 
 /* Return whether bitmap is empty (1 == empty) */
 extern int xfs_bitmap_empty(uint *map, uint size);
diff --git a/fs/xfs/xfs_rtalloc.c b/fs/xfs/xfs_rtalloc.c
index 47082c0..ca83ddf 100644
--- a/fs/xfs/xfs_rtalloc.c
+++ b/fs/xfs/xfs_rtalloc.c
@@ -73,18 +73,6 @@ STATIC int xfs_rtmodify_summary(xfs_moun
  */
 
 /*
- * xfs_lowbit32: get low bit set out of 32-bit argument, -1 if none set.
- */
-STATIC int
-xfs_lowbit32(
-	__uint32_t	v)
-{
-	if (v)
-		return ffs(v) - 1;
-	return -1;
-}
-
-/*
  * Allocate space to the bitmap or summary file, and zero it, for growfs.
  */
 STATIC int				/* error */
@@ -444,6 +432,7 @@ xfs_rtallocate_extent_near(
 	}
 	bbno = XFS_BITTOBLOCK(mp, bno);
 	i = 0;
+	ASSERT(minlen != 0);
 	log2len = xfs_highbit32(minlen);
 	/*
 	 * Loop over all bitmap blocks (bbno + i is current block).
@@ -612,6 +601,8 @@ xfs_rtallocate_extent_size(
 	xfs_suminfo_t	sum;		/* summary information for extents */
 
 	ASSERT(minlen % prod == 0 && maxlen % prod == 0);
+	ASSERT(maxlen != 0);
+
 	/*
 	 * Loop over all the levels starting with maxlen.
 	 * At each level, look at all the bitmap blocks, to see if there
@@ -669,6 +660,9 @@ xfs_rtallocate_extent_size(
 		*rtblock = NULLRTBLOCK;
 		return 0;
 	}
+	ASSERT(minlen != 0);
+	ASSERT(maxlen != 0);
+
 	/*
 	 * Loop over sizes, from maxlen down to minlen.
 	 * This time, when we do the allocations, allow smaller ones
@@ -1954,6 +1948,7 @@ xfs_growfs_rt(
 				  nsbp->sb_blocksize * nsbp->sb_rextsize);
 		nsbp->sb_rextents = nsbp->sb_rblocks;
 		do_div(nsbp->sb_rextents, nsbp->sb_rextsize);
+		ASSERT(nsbp->sb_rextents != 0);
 		nsbp->sb_rextslog = xfs_highbit32(nsbp->sb_rextents);
 		nrsumlevels = nmp->m_rsumlevels = nsbp->sb_rextslog + 1;
 		nrsumsize =

Re: filesystem corruption in linus tree

[Marc Dietrich]

Hi again,

On Tuesday 26 February 2008 08:38:50 Lachlan McIlroy wrote:
> Marc Dietrich wrote:
> > Hi,
> >
> > On Monday 25 February 2008 01:36:28 Barry Naujok wrote:
> >> On Sun, 24 Feb 2008 20:58:26 +1100, Marc Dietrich
> >>
> >> <Marc.Dietrich@ap.physik.uni-giessen.de> wrote:
> >>> Hi,
> >>>
> >>> somewhere after the release of 2.6.24 my xfs filesystem got corrupted.
> >>> Initialy I thought it was only related to the readdir bug.
> >>> (http://oss.sgi.com/archives/xfs/2008-02/msg00027.html) So I waited for
> >>> the fix to go into mainline. Yesterday I tried again, but got this
> >>> error during boot:

<sniped>

> We've had a few problems reported with XFS on 32-bit powermacs and the
> culprit appears to be some changes to bit manipulation routines.  Could you
> please try reverse applying the attached patches and see if the problem is
> resolved?

I saw, that you already pushed it into mainline - for a good reason ;-) Works 
as expeted.

Please also don't forget 2.6.24-stable !

Thanks

Marc

Re: filesystem corruption in linus tree

[Lachlan McIlroy]

Marc Dietrich wrote:
> Hi again,
> 
> On Tuesday 26 February 2008 08:38:50 Lachlan McIlroy wrote:
>> Marc Dietrich wrote:
>>> Hi,
>>>
>>> On Monday 25 February 2008 01:36:28 Barry Naujok wrote:
>>>> On Sun, 24 Feb 2008 20:58:26 +1100, Marc Dietrich
>>>>
>>>> <Marc.Dietrich@ap.physik.uni-giessen.de> wrote:
>>>>> Hi,
>>>>>
>>>>> somewhere after the release of 2.6.24 my xfs filesystem got corrupted.
>>>>> Initialy I thought it was only related to the readdir bug.
>>>>> (http://oss.sgi.com/archives/xfs/2008-02/msg00027.html) So I waited for
>>>>> the fix to go into mainline. Yesterday I tried again, but got this
>>>>> error during boot:
> 
> <sniped>
> 
>> We've had a few problems reported with XFS on 32-bit powermacs and the
>> culprit appears to be some changes to bit manipulation routines.  Could you
>> please try reverse applying the attached patches and see if the problem is
>> resolved?
> 
> I saw, that you already pushed it into mainline - for a good reason ;-) Works 
> as expeted.
> 
> Please also don't forget 2.6.24-stable !

The changes that caused this regression went into 2.6.25-rc1 so no need for a
2.6.24 stable fix.

Re: filesystem corruption in linus tree

[Marc]

Hi,

Am Montag 03 März 2008 05:23:46 schrieb Lachlan McIlroy:
> Marc Dietrich wrote:
> > Hi again,
> >
> > On Tuesday 26 February 2008 08:38:50 Lachlan McIlroy wrote:
> >> Marc Dietrich wrote:
> >>> Hi,
> >>>
> >>> On Monday 25 February 2008 01:36:28 Barry Naujok wrote:
> >>>> On Sun, 24 Feb 2008 20:58:26 +1100, Marc Dietrich
> >>>>
> >>>> <Marc.Dietrich@ap.physik.uni-giessen.de> wrote:
> >>>>> Hi,
> >>>>>
> >>>>> somewhere after the release of 2.6.24 my xfs filesystem got
> >>>>> corrupted. Initialy I thought it was only related to the readdir bug.
> >>>>> (http://oss.sgi.com/archives/xfs/2008-02/msg00027.html) So I waited
> >>>>> for the fix to go into mainline. Yesterday I tried again, but got
> >>>>> this error during boot:
> >
> > <sniped>
> >
> >> We've had a few problems reported with XFS on 32-bit powermacs and the
> >> culprit appears to be some changes to bit manipulation routines.  Could
> >> you please try reverse applying the attached patches and see if the
> >> problem is resolved?
> >
> > I saw, that you already pushed it into mainline - for a good reason ;-)
> > Works as expeted.
> >
> > Please also don't forget 2.6.24-stable !
>
> The changes that caused this regression went into 2.6.25-rc1 so no need for
> a 2.6.24 stable fix.

yes - and I could have sworn, that 2.6.24 is also affected. My fault.

Thanks 

Marc


-- 
	"Those who question our statements are traitors", 
Lord Arthur Ponsonby, "Falsehood in Wartime: Propaganda Lies of the First 
World War", 1928