Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

[Nathan Scott]

On Fri, Aug 04, 2006 at 10:22:21AM +0200, Jesper Juhl wrote:
> I just hit a BUG that looks XFS related.
> 
> The machine is running 2.6.18-rc3-git3
> 
> (more info below the BUG messages)
> 

Thanks for reporting, Jesper - is it reproducible?  Could you try this
patch for me?  We had a couple of other reports of this, but the earlier
reporters have vanished ... could you let me know if this helps?

cheers.

-- 
Nathan

--- fs/xfs/xfs_alloc.c.orig	2006-08-04 20:00:34.333456250 +1000
+++ fs/xfs/xfs_alloc.c	2006-08-04 20:00:50.586472000 +1000
@@ -1949,14 +1949,8 @@ xfs_alloc_fix_freelist(
 		 * the restrictions correctly.  Can happen for free calls
 		 * on a completely full ag.
 		 */
-		if (targs.agbno == NULLAGBLOCK) {
-			if (!(flags & XFS_ALLOC_FLAG_FREEING)) {
-				xfs_trans_brelse(tp, agflbp);
-				args->agbp = NULL;
-				return 0;
-			}
+		if (targs.agbno == NULLAGBLOCK)
 			break;
-		}
 		/*
 		 * Put each allocated block on the list.
 		 */

Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

[Jesper Juhl]

On 04/08/06, Nathan Scott <nathans@sgi.com> wrote:
> On Fri, Aug 04, 2006 at 10:22:21AM +0200, Jesper Juhl wrote:
> > I just hit a BUG that looks XFS related.
> >
> > The machine is running 2.6.18-rc3-git3
> >
> > (more info below the BUG messages)
> >
>
> Thanks for reporting, Jesper - is it reproducible?

I don't know. I only tried that kernel once and when it broke on me
went back to the previous 2.6.11 kernel it was running before.


>  Could you try this
> patch for me?

Sure.
The machine is semi-production, so there are limits to how much and
when I can test on it.
roughly wednesday and thursday each week I should be able to run
experimental kernels on it, the rest of the week the box needs to be
stable.

>  We had a couple of other reports of this, but the earlier
> reporters have vanished ... could you let me know if this helps?
>

What I'll do is apply that patch to the 2.6.18-rc3-git3 kernel that
BUG'ed on me, then wednesday next week I'll boot the machine with the
patched kernel and it should be able to run for ~24hrs, then I can
report back to you if it crashed or not.

Or is there some other way you'd rather have me do it (subject to the
constraint that I can only do experiments one and a half to two days a
week) ?


-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

[Nathan Scott]

On Fri, Aug 11, 2006 at 12:25:03PM +0200, Jesper Juhl wrote:
> I didn't capture all of the xfs_repair output, but I did get this :
> ...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - clear lost+found (if it exists) ...
>         - clearing existing "lost+found" inode
>         - deleting existing "lost+found" entry
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
> LEAFN node level is 1 inode 412035424 bno = 8388608

Ooh.  Can you describe this test case you're using?  Something with
a bunch of renames in it, obviously, but I'd also like to be able to
reproduce locally with the exact data set (file names in particular),
if at all possible.

thanks!

-- 
Nathan

Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

[Jesper Juhl]

On 14/08/06, Nathan Scott <nathans@sgi.com> wrote:
> On Fri, Aug 11, 2006 at 12:25:03PM +0200, Jesper Juhl wrote:
> > I didn't capture all of the xfs_repair output, but I did get this :
> > ...
> > Phase 4 - check for duplicate blocks...
> >         - setting up duplicate extent list...
> >         - clear lost+found (if it exists) ...
> >         - clearing existing "lost+found" inode
> >         - deleting existing "lost+found" entry
> >         - check for inodes claiming duplicate blocks...
> >         - agno = 0
> >         - agno = 1
> >         - agno = 2
> >         - agno = 3
> >         - agno = 4
> >         - agno = 5
> >         - agno = 6
> > LEAFN node level is 1 inode 412035424 bno = 8388608
>
> Ooh.  Can you describe this test case you're using?

Sure.

The server has a bunch of XFS filesystems :

# mount
/dev/md1 on / type xfs (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/md0 on /boot type ext3 (rw)
/dev/mapper/Archive-Backup on /mnt/backup type xfs (rw)
/dev/mapper/Mirror-ws1 on /mnt/rsync/ws1 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws1_log)
/dev/mapper/Mirror-ws2 on /mnt/rsync/ws2 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws2_log)
/dev/mapper/Mirror-ws3 on /mnt/rsync/ws3 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws3_log)
/dev/mapper/Mirror-ws4 on /mnt/rsync/ws4 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws4_log)
/dev/mapper/Mirror-ws5 on /mnt/rsync/ws5 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws5_log)
/dev/mapper/Mirror-ws6 on /mnt/rsync/ws6 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws6_log)
/dev/mapper/Mirror-ws7 on /mnt/rsync/ws7 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws7_log)
/dev/mapper/Mirror-ws8 on /mnt/rsync/ws8 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws8_log)
/dev/mapper/Mirror-ws9 on /mnt/rsync/ws9 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws9_log)
/dev/mapper/Mirror-ws10 on /mnt/rsync/ws10 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws10_log)
/dev/mapper/Mirror-ws11 on /mnt/rsync/ws11 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws11_log)
/dev/mapper/Mirror-ws12 on /mnt/rsync/ws12 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws12_log)
/dev/mapper/Mirror-ws13 on /mnt/rsync/ws13 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws13_log)
/dev/mapper/Mirror-ws14 on /mnt/rsync/ws14 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws14_log)
/dev/mapper/Mirror-ws15 on /mnt/rsync/ws15 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws15_log)
/dev/mapper/Mirror-ws16 on /mnt/rsync/ws16 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws16_log)
/dev/mapper/Mirror-ws17 on /mnt/rsync/ws17 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws17_log)
/dev/mapper/Mirror-ws18 on /mnt/rsync/ws18 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws18_log)
/dev/mapper/Mirror-ws19 on /mnt/rsync/ws19 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws19_log)
/dev/mapper/Mirror-ws20 on /mnt/rsync/ws20 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws20_log)
/dev/mapper/Mirror-ws21 on /mnt/rsync/ws21 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/ws21_log)
/dev/mapper/Mirror-wsb1 on /mnt/rsync/wsb1 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/wsb1_log)
/dev/mapper/Mirror-wsb2 on /mnt/rsync/wsb2 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/wsb2_log)
/dev/mapper/Mirror-wsb3 on /mnt/rsync/wsb3 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/wsb3_log)
/dev/mapper/Mirror-wsb4 on /mnt/rsync/wsb4 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/wsb4_log)
/dev/mapper/Mirror-wsp1 on /mnt/rsync/wsp1 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/wsp1_log)
/dev/mapper/Mirror-wsp2 on /mnt/rsync/wsp2 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/wsp2_log)
/dev/mapper/Mirror-wsp3 on /mnt/rsync/wsp3 type xfs
(rw,noatime,ihashsize=64433,logdev=/dev/Log/wsp3_log)
/dev/mapper/Mirror-obr1 on /mnt/ob/obr1 type xfs (rw,noatime,ihashsize=64433)
tmpfs on /dev type tmpfs (rw,size=10M,mode=0755)

These filesystems vary in size from 50G to 3.5T

The XFS filesystems contain rsync copies of filesystems on as many servers.
The workload that triggers the problem is when all the servers start
updating their copy via rsync - Then within a few hours the problem
triggers.

So to recreate the same scenario you'll want 28 servers doing rsync of
filesystems of various sizes between 50G & 3.5T to a central server
running 2.6.18-rc4 with 28 XFS filesystems.
The XFS filesystems are on LVM and each physical volume is made up of
two disks in a RAID1.


>  Something with
> a bunch of renames in it, obviously, but I'd also like to be able to
> reproduce locally with the exact data set (file names in particular),
> if at all possible.
>
There are millions of files. The data the server recievs is copies of
websites. Each server that sends data to the server with the 28 XFS
filesystems hosts between 1800 and 2600 websites, so there are lots of
files and every concievable strange filename.

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

[Nathan Scott]

On Mon, Aug 14, 2006 at 09:49:10AM +0200, Jesper Juhl wrote:
> On 14/08/06, Nathan Scott <nathans@sgi.com> wrote:
> > > LEAFN node level is 1 inode 412035424 bno = 8388608
> > 
> > Ooh.  Can you describe this test case you're using?
> ...
> These filesystems vary in size from 50G to 3.5T
> 
> The XFS filesystems contain rsync copies of filesystems on as many servers.
> The workload that triggers the problem is when all the servers start
> updating their copy via rsync - Then within a few hours the problem
> triggers.
> 
> So to recreate the same scenario you'll want 28 servers doing rsync of
> filesystems of various sizes between 50G & 3.5T to a central server
> running 2.6.18-rc4 with 28 XFS filesystems.
> ...
> There are millions of files. The data the server recievs is copies of
> websites. Each server that sends data to the server with the 28 XFS
> filesystems hosts between 1800 and 2600 websites, so there are lots of
> files and every concievable strange filename.

Wow, a special kind of hell for a filesystem developer...! ;-)

Its not clear to me where the rename operation happens in all of
this - does rsync create a local, temporary copy of the file and
then rename it?  Is it that central server going down or one of
those 28 other server machines?  (I assume it is, I can't see an
opportunity for renaming out there...).

When you hit it again, could you grab the contents of the inode
(you'll get that from xfs_repair -n, e.g. 412035424 above) with
xfs_db (see last entry in the XFS FAQ which describes how to do
that), then mail that to me please?  If you can get the source
and target names in the rename that'll help alot too... I can
explain how to use KDB to get that, but maybe you have another
debugger handy already?

thanks!

-- 
Nathan

Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

[Jesper Juhl]

On 15/08/06, Nathan Scott <nathans@sgi.com> wrote:
> On Mon, Aug 14, 2006 at 09:49:10AM +0200, Jesper Juhl wrote:
> > On 14/08/06, Nathan Scott <nathans@sgi.com> wrote:
> > > > LEAFN node level is 1 inode 412035424 bno = 8388608
> > >
> > > Ooh.  Can you describe this test case you're using?
> > ...
> > These filesystems vary in size from 50G to 3.5T
> >
> > The XFS filesystems contain rsync copies of filesystems on as many servers.
> > The workload that triggers the problem is when all the servers start
> > updating their copy via rsync - Then within a few hours the problem
> > triggers.
> >
> > So to recreate the same scenario you'll want 28 servers doing rsync of
> > filesystems of various sizes between 50G & 3.5T to a central server
> > running 2.6.18-rc4 with 28 XFS filesystems.
> > ...
> > There are millions of files. The data the server recievs is copies of
> > websites. Each server that sends data to the server with the 28 XFS
> > filesystems hosts between 1800 and 2600 websites, so there are lots of
> > files and every concievable strange filename.
>
> Wow, a special kind of hell for a filesystem developer...! ;-)
>
> Its not clear to me where the rename operation happens in all of
> this - does rsync create a local, temporary copy of the file and
> then rename it?

I'm not sure. I'll investigate and see if I can work out what exactely
rsync does.

> Is it that central server going down or one of
> those 28 other server machines?  (I assume it is, I can't see an
> opportunity for renaming out there...).
>
It's the central server with all the xfs filesystems that dies. The
one recieving all the rsync data.


> When you hit it again, could you grab the contents of the inode
> (you'll get that from xfs_repair -n, e.g. 412035424 above) with
> xfs_db (see last entry in the XFS FAQ which describes how to do
> that), then mail that to me please?

Sure, I'll read up on that and make sure to grab that info next time.


> If you can get the source
> and target names in the rename that'll help alot too... I can
> explain how to use KDB to get that, but maybe you have another
> debugger handy already?
>
An explanation of how exactely to do that would be greatly appreciated.

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

[Chris Wedgwood]

On Tue, Aug 15, 2006 at 07:03:43PM +1000, Nathan Scott wrote:

> Its not clear to me where the rename operation happens in all of
> this - does rsync create a local, temporary copy of the file and
> then rename it?

Yes, this is normally how rsync does it.

Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

[Jan Engelhardt]

>> Its not clear to me where the rename operation happens in all of
>> this - does rsync create a local, temporary copy of the file and
>> then rename it?
>
>Yes, this is normally how rsync does it.

If file already exists {
    foreach block {
        copy block either from disk or from the 
        source operand, whichever is never, to temp file
    }
}

When rsync catches a signal {
    move the tempfile to the original name
}





Jan Engelhardt
-- 

Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

[Nathan Scott]

On Tue, Aug 15, 2006 at 01:42:27PM +0200, Jesper Juhl wrote:
> On 15/08/06, Nathan Scott <nathans@sgi.com> wrote:
> > If you can get the source
> > and target names in the rename that'll help alot too... I can
> > explain how to use KDB to get that, but maybe you have another
> > debugger handy already?
> >
> An explanation of how exactely to do that would be greatly appreciated.

- patch in KDB
- echo 127 > /proc/sys/fs/xfs/panic_mask
[ filesystem shutdown now == panic ]
- kdb> bt
[ pick out parameters to rename from the backtrace ]
- kdb> md 0xXXX
[ gives a memory dump of the pointers to pathnames ]

cheers.

-- 
Nathan

Re: 2.6.18-rc3-git3 - XFS - BUG: unable to handle kernel NULL pointer dereference at virtual address 00000078

[Jesper Juhl]

On 16/08/06, Nathan Scott <nathans@sgi.com> wrote:
> On Tue, Aug 15, 2006 at 01:42:27PM +0200, Jesper Juhl wrote:
> > On 15/08/06, Nathan Scott <nathans@sgi.com> wrote:
> > > If you can get the source
> > > and target names in the rename that'll help alot too... I can
> > > explain how to use KDB to get that, but maybe you have another
> > > debugger handy already?
> > >
> > An explanation of how exactely to do that would be greatly appreciated.
>
> - patch in KDB
> - echo 127 > /proc/sys/fs/xfs/panic_mask
> [ filesystem shutdown now == panic ]
> - kdb> bt
> [ pick out parameters to rename from the backtrace ]
> - kdb> md 0xXXX
> [ gives a memory dump of the pointers to pathnames ]
>

Thanks a lot for the explanation.

Unfortunately I didn't get a chance to run new tests on the server
this week (always the big problem when it's a production machine).
I'm also going on a short vacation, so I won't have the oppotunity to
try and recreate a simpler test case at home for the next few days.

When I get back (in some 4 days time) I'll try to build a more simple
test case and in about a week or so I hopefully will get a new chance
to run new tests on the server that has so far shown the problem.
If there are additional tests you want me to run or data you want me
to collect, then let me know and I'll do so the first chance I get.

I'll be back in touch in ~1 weeks time.

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html