NFSD crashes with oops

NFSD crashes with oops

[Bas van der Vlies]

At our side we have an 4 node GFS cluster. The nodes act as an NFS- 
server to +/- 640 nodes.

GFS -nodes :
  * Debian sarge
  * 2.6.14.7
  * nfs-common 1:1.0.6-3.1
  * nfs-kernel-server 1:1.0.6-3.1
  * 128 NFSD-threads

clients:
  * Debian sarge
  * 2.6.15.6
  * nfs-common 1:1.0.6-3.1


After a while a nfsd-daemon will crash with the following error  
message. Sometimes it will survive a crash an go on and have
one thread less. But after a while all nfsd-daemons freezes. GFS and  
the machine is still working, but the server will not response to NFS- 
requests anymore. We have to reboot to fix the problem.

Has anybody seen this before or have an solution?

---- kernel Oops ----
Unable to handle kernel NULL pointer dereference at virtual address  
00000038
printing eip:
f89bf999
*pde = 37bff001
*pte = 00000000
Oops: 0000 [#1]
SMP
Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg  
ide_floppy ide_cd cdrom qla2300 qla2xxx_conf qla2xxx firmware_class  
siimage piix e1000 gfs lock_harness dm_mod
CPU:    0
EIP:    0060:[<f89bf999>]    Tainted: GF     VLI
EFLAGS: 00010246   (2.6.14.7-sara1)
EIP is at gfs_create+0xa9/0x1e0 [gfs]
eax: ffffffef   ebx: ffffffef   ecx: 00000001   edx: 00000000
esi: f296e24c   edi: ebf01e18   ebp: ebf01e84   esp: ebf01df8
ds: 007b   es: 007b   ss: 0068
Process nfsd (pid: 16924, threadinfo=ebf00000 task=ebe84540)
Stack: ebf01e48 f296e24c 00000001 00008180 ebf01e18 00000001 f8cb9000  
dd042254
        ebf01e18 ebf01e18 00000000 ebe84540 00000001 00000120  
00000000 000000c2
        00000000 00000001 ebf01e40 ebf01e40 ebf01e48 ebf01e48  
df0bd858 ebe84540
Call Trace:
[<c0103e5f>] show_stack+0x7f/0xa0
[<c0104012>] show_registers+0x162/0x1d0
[<c0104224>] die+0xf4/0x180
[<c035f697>] do_page_fault+0x2e7/0x6b2
[<c0103b03>] error_code+0x4f/0x54
[<c016b663>] vfs_create+0x83/0xf0
[<c01b81ce>] nfsd_create_v3+0x40e/0x550
[<c01bed2d>] nfsd3_proc_create+0x11d/0x180
[<c01b2f87>] nfsd_dispatch+0xd7/0x200
[<c0353a96>] svc_process+0x536/0x670
[<c01b2d1d>] nfsd+0x1bd/0x350
[<c010127d>] kernel_thread_helper+0x5/0x18
Code: 24 08 8d 45 c4 89 54 24 0c 89 74 24 04 89 04 24 e8 1d c3 fe ff  
85 c0 89 c3 0f 84 2e 01 00 00 83 f8 ef 0f 85 13 01 00 00 8b 55 14  
<80> 7a 38 00 0f 88 06 01 00 00 89 7c 24 0c 31 c0 8d 55 c4 89 44

--
Bas van der Vlies
basv@sara.nl





-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFSD crashes with oops

[Bas van der Vlies]

Bas van der Vlies wrote:
> At our side we have an 4 node GFS cluster. The nodes act as an 
> NFS-server to +/- 640 nodes.
>

Just upgraded to 2.6.16-rc5 and still nfsd crashes. With more debug 
output. So some code is fixed in this release. Nfsd crashes instead
of 128 we now have 127, nfsd still runing now.

== Crash Output ==
Unable to handle kernel NULL pointer dereference at virtual address 00000038
  printing eip:
f89a4be3
*pde = 37809001
*pte = 00000000
Oops: 0000 [#1]
SMP
Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg 
ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness dm_mod
CPU:    0
EIP:    0060:[<f89a4be3>]    Tainted: GF     VLI
EFLAGS: 00010246   (2.6.16-rc5-sara3 #1)
EIP is at gfs_create+0x6f/0x153 [gfs]
eax: 00000000   ebx: ffffffef   ecx: f27d0d98   edx: ffffffef
esi: f2f84690   edi: f8b93000   ebp: f34a5e98   esp: f34a5e20
ds: 007b   es: 007b   ss: 0068
Process nfsd (pid: 8973, threadinfo=f34a4000 task=f3462a70)
Stack: <0>f092a530 00000001 f34a5e48 00000000 f34a5e84 f89a6628 f34a5e48 
ee1fc324 00000003 00000000 f34a5e48 f34a5e48 00000000 f3462a70 00000003 
f34a5e5c f34a5e5c f27d0d98 f3462a70 00000001 00000020 00000000 000000c2 
00000000
Call Trace:
  [<c0103599>] show_stack_log_lvl+0xad/0xb5
  [<c01036db>] show_registers+0x10d/0x176
  [<c01038ad>] die+0xf2/0x16d
  [<c010f668>] do_page_fault+0x3dd/0x57a
  [<c010322f>] error_code+0x4f/0x54
  [<c01585f2>] vfs_create+0x6a/0xa7
  [<c0195e1c>] nfsd_create_v3+0x2b1/0x48a
  [<c019af2f>] nfsd3_proc_create+0x116/0x123
  [<c019229f>] nfsd_dispatch+0xbe/0x17f
  [<c02e0a52>] svc_process+0x381/0x5c7
  [<c019208c>] nfsd+0x18d/0x2e2
  [<c0100ed9>] kernel_thread_helper+0x5/0xb
Code: 94 50 8b 45 0c ff 75 10 83 c0 1c 6a 01 89 45 88 50 8d 45 c4 50 e8 
70 08 ff ff 83 c4 14 89 c3 85 c0 74 4883 f8 ef 75 33 8b 45 14 <80> 78 38 
00 78 2a 8d 45 94 50 8d 45 c4 6a 00 ff 75 88 50 e8 3c
  BUG: nfsd/8973, lock held at task exit time!
  [ee1fc398] {inode_init_once}
.. held by:              nfsd: 8973 [f3462a70, 115]
... acquired at:               nfsd_create_v3+0x127/0x48a


-- 
--
********************************************************************
*                                                                  *
*  Bas van der Vlies                     e-mail: basv@sara.nl      *
*  SARA - Academic Computing Services    phone:  +31 20 592 8012   *
*  Kruislaan 415                         fax:    +31 20 6683167    *
*  1098 SJ Amsterdam                                               *
*                                                                  *
********************************************************************


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFSD crashes with oops

[Bas van der Vlies]

Our FS-nodes are SMP systems with 2 CPU's

Bas van der Vlies wrote:
> 
> Bas van der Vlies wrote:
>> At our side we have an 4 node GFS cluster. The nodes act as an 
>> NFS-server to +/- 640 nodes.
>>
> 
> Just upgraded to 2.6.16-rc5 and still nfsd crashes. With more debug 
> output. So some code is fixed in this release. Nfsd crashes instead
> of 128 we now have 127, nfsd still runing now.
> 
> == Crash Output ==
> Unable to handle kernel NULL pointer dereference at virtual address 
> 00000038
>  printing eip:
> f89a4be3
> *pde = 37809001
> *pte = 00000000
> Oops: 0000 [#1]
> SMP
> Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg 
> ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness dm_mod
> CPU:    0
> EIP:    0060:[<f89a4be3>]    Tainted: GF     VLI
> EFLAGS: 00010246   (2.6.16-rc5-sara3 #1)
> EIP is at gfs_create+0x6f/0x153 [gfs]
> eax: 00000000   ebx: ffffffef   ecx: f27d0d98   edx: ffffffef
> esi: f2f84690   edi: f8b93000   ebp: f34a5e98   esp: f34a5e20
> ds: 007b   es: 007b   ss: 0068
> Process nfsd (pid: 8973, threadinfo=f34a4000 task=f3462a70)
> Stack: <0>f092a530 00000001 f34a5e48 00000000 f34a5e84 f89a6628 f34a5e48 
> ee1fc324 00000003 00000000 f34a5e48 f34a5e48 00000000 f3462a70 00000003 
> f34a5e5c f34a5e5c f27d0d98 f3462a70 00000001 00000020 00000000 000000c2 
> 00000000
> Call Trace:
>  [<c0103599>] show_stack_log_lvl+0xad/0xb5
>  [<c01036db>] show_registers+0x10d/0x176
>  [<c01038ad>] die+0xf2/0x16d
>  [<c010f668>] do_page_fault+0x3dd/0x57a
>  [<c010322f>] error_code+0x4f/0x54
>  [<c01585f2>] vfs_create+0x6a/0xa7
>  [<c0195e1c>] nfsd_create_v3+0x2b1/0x48a
>  [<c019af2f>] nfsd3_proc_create+0x116/0x123
>  [<c019229f>] nfsd_dispatch+0xbe/0x17f
>  [<c02e0a52>] svc_process+0x381/0x5c7
>  [<c019208c>] nfsd+0x18d/0x2e2
>  [<c0100ed9>] kernel_thread_helper+0x5/0xb
> Code: 94 50 8b 45 0c ff 75 10 83 c0 1c 6a 01 89 45 88 50 8d 45 c4 50 e8 
> 70 08 ff ff 83 c4 14 89 c3 85 c0 74 4883 f8 ef 75 33 8b 45 14 <80> 78 38 
> 00 78 2a 8d 45 94 50 8d 45 c4 6a 00 ff 75 88 50 e8 3c
>  BUG: nfsd/8973, lock held at task exit time!
>  [ee1fc398] {inode_init_once}
> .. held by:              nfsd: 8973 [f3462a70, 115]
> ... acquired at:               nfsd_create_v3+0x127/0x48a
> 
> 


-- 
--
********************************************************************
*                                                                  *
*  Bas van der Vlies                     e-mail: basv@sara.nl      *
*  SARA - Academic Computing Services    phone:  +31 20 592 8012   *
*  Kruislaan 415                         fax:    +31 20 6683167    *
*  1098 SJ Amsterdam                                               *
*                                                                  *
********************************************************************


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFSD crashes with oops

[Trond Myklebust]

On Wed, 2006-03-08 at 15:08 +0100, Bas van der Vlies wrote:
> Bas van der Vlies wrote:
> > At our side we have an 4 node GFS cluster. The nodes act as an 
> > NFS-server to +/- 640 nodes.
> >
> 
> Just upgraded to 2.6.16-rc5 and still nfsd crashes. With more debug 
> output. So some code is fixed in this release. Nfsd crashes instead
> of 128 we now have 127, nfsd still runing now.
> 
> == Crash Output ==
> Unable to handle kernel NULL pointer dereference at virtual address 00000038
>   printing eip:
> f89a4be3
> *pde = 37809001
> *pte = 00000000
> Oops: 0000 [#1]
> SMP
> Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg 
> ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness dm_mod
> CPU:    0
> EIP:    0060:[<f89a4be3>]    Tainted: GF     VLI
> EFLAGS: 00010246   (2.6.16-rc5-sara3 #1)
> EIP is at gfs_create+0x6f/0x153 [gfs]

This is Oopsing in the GFS code, not nfsd.

Cheers,
  Trond

> eax: 00000000   ebx: ffffffef   ecx: f27d0d98   edx: ffffffef
> esi: f2f84690   edi: f8b93000   ebp: f34a5e98   esp: f34a5e20
> ds: 007b   es: 007b   ss: 0068
> Process nfsd (pid: 8973, threadinfo=f34a4000 task=f3462a70)
> Stack: <0>f092a530 00000001 f34a5e48 00000000 f34a5e84 f89a6628 f34a5e48 
> ee1fc324 00000003 00000000 f34a5e48 f34a5e48 00000000 f3462a70 00000003 
> f34a5e5c f34a5e5c f27d0d98 f3462a70 00000001 00000020 00000000 000000c2 
> 00000000
> Call Trace:
>   [<c0103599>] show_stack_log_lvl+0xad/0xb5
>   [<c01036db>] show_registers+0x10d/0x176
>   [<c01038ad>] die+0xf2/0x16d
>   [<c010f668>] do_page_fault+0x3dd/0x57a
>   [<c010322f>] error_code+0x4f/0x54
>   [<c01585f2>] vfs_create+0x6a/0xa7
>   [<c0195e1c>] nfsd_create_v3+0x2b1/0x48a
>   [<c019af2f>] nfsd3_proc_create+0x116/0x123
>   [<c019229f>] nfsd_dispatch+0xbe/0x17f
>   [<c02e0a52>] svc_process+0x381/0x5c7
>   [<c019208c>] nfsd+0x18d/0x2e2
>   [<c0100ed9>] kernel_thread_helper+0x5/0xb
> Code: 94 50 8b 45 0c ff 75 10 83 c0 1c 6a 01 89 45 88 50 8d 45 c4 50 e8 
> 70 08 ff ff 83 c4 14 89 c3 85 c0 74 4883 f8 ef 75 33 8b 45 14 <80> 78 38 
> 00 78 2a 8d 45 94 50 8d 45 c4 6a 00 ff 75 88 50 e8 3c
>   BUG: nfsd/8973, lock held at task exit time!
>   [ee1fc398] {inode_init_once}
> .. held by:              nfsd: 8973 [f3462a70, 115]
> ... acquired at:               nfsd_create_v3+0x127/0x48a
> 
> 



-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFSD crashes with oops

[Bas van der Vlies]

Trond Myklebust wrote:
> On Wed, 2006-03-08 at 15:08 +0100, Bas van der Vlies wrote:
>> Bas van der Vlies wrote:
>>> At our side we have an 4 node GFS cluster. The nodes act as an 
>>> NFS-server to +/- 640 nodes.
>>>
>> Just upgraded to 2.6.16-rc5 and still nfsd crashes. With more debug 
>> output. So some code is fixed in this release. Nfsd crashes instead
>> of 128 we now have 127, nfsd still runing now.
>>
>> == Crash Output ==
>> Unable to handle kernel NULL pointer dereference at virtual address 00000038
>>   printing eip:
>> f89a4be3
>> *pde = 37809001
>> *pte = 00000000
>> Oops: 0000 [#1]
>> SMP
>> Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg 
>> ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness dm_mod
>> CPU:    0
>> EIP:    0060:[<f89a4be3>]    Tainted: GF     VLI
>> EFLAGS: 00010246   (2.6.16-rc5-sara3 #1)
>> EIP is at gfs_create+0x6f/0x153 [gfs]
> 
> This is Oopsing in the GFS code, not nfsd.
> 
Thanks for the info. Just for my knowledge. The nfsd crashes because 
there is an error in the gfs_create code?

The problem is that gfs-filesystems are still accesible and if we are
lucky we have on thread less else the load on the gfs-fileserver is sky 
high and it does not respond to the nfs-requests anymore. So all nfsd 
processes are locked.

Regards


-- 
--
********************************************************************
*                                                                  *
*  Bas van der Vlies                     e-mail: basv@sara.nl      *
*  SARA - Academic Computing Services    phone:  +31 20 592 8012   *
*  Kruislaan 415                         fax:    +31 20 6683167    *
*  1098 SJ Amsterdam                                               *
*                                                                  *
********************************************************************


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFSD crashes with oops

[Trond Myklebust]

On Wed, 2006-03-08 at 16:23 +0100, Bas van der Vlies wrote:
> Trond Myklebust wrote:
> > On Wed, 2006-03-08 at 15:08 +0100, Bas van der Vlies wrote:
> >> Bas van der Vlies wrote:
> >>> At our side we have an 4 node GFS cluster. The nodes act as an 
> >>> NFS-server to +/- 640 nodes.
> >>>
> >> Just upgraded to 2.6.16-rc5 and still nfsd crashes. With more debug 
> >> output. So some code is fixed in this release. Nfsd crashes instead
> >> of 128 we now have 127, nfsd still runing now.
> >>
> >> == Crash Output ==
> >> Unable to handle kernel NULL pointer dereference at virtual address 00000038
> >>   printing eip:
> >> f89a4be3
> >> *pde = 37809001
> >> *pte = 00000000
> >> Oops: 0000 [#1]
> >> SMP
> >> Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg 
> >> ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness dm_mod
> >> CPU:    0
> >> EIP:    0060:[<f89a4be3>]    Tainted: GF     VLI
> >> EFLAGS: 00010246   (2.6.16-rc5-sara3 #1)
> >> EIP is at gfs_create+0x6f/0x153 [gfs]
> > 
> > This is Oopsing in the GFS code, not nfsd.
> > 
> Thanks for the info. Just for my knowledge. The nfsd crashes because 
> there is an error in the gfs_create code?
> 
> The problem is that gfs-filesystems are still accesible and if we are
> lucky we have on thread less else the load on the gfs-fileserver is sky 
> high and it does not respond to the nfs-requests anymore. So all nfsd 
> processes are locked.

It is hard to tell which is at fault here since GFS is externally
developed, and is not part of the mainline kernel. I therefore suggest
you pursue this issue with the GFS developers.

Cheers,
  Trond



-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFSD/gfs crashes with oops

[Bas van der Vlies]

Just for the recorde. The GFS guys have made a patch for the crash in 
CVS stable:

CVSROOT:	/cvs/cluster
Module name:	cluster
Branch: 	STABLE
Changes by:	bmarzins@sourceware.org	2006-03-08 20:47:09

Modified files:
	gfs-kernel/src/gfs: ops_inode.c

Log message:
  Really gross hack!!!
  This is a workaround for one of the bugs the got lumped into 166701. It
  breaks POSIX behavior in a corner case to avoid crashing... It's icky.
  when NFS opens a file with O_CREAT, the kernel nfs daemon checks to see
  if the file exists. If it does, nfsd does the *right thing* (either
  opens the file, or if the file was opened with O_EXCL, returns an
  error).  If the file doesn't exist, it passes the request down to the
  underlying file system. Unfortunately, since nfs *knows* that the file
  doesn't exist, it doesn't  bother to pass a nameidata structure, which
  would include the intent information. However since gfs is a cluster
  file system, the file could have been created on another node after nfs
  checks for it. If this is the case, gfs needs the intent information to
  do the *right thing*.  It panics when it finds a NULL pointer, instead
  of the nameidata. Now, instead of panicing, if gfs finds a NULL
  nameidata pointer. It assumes that the file was not created with _EXCL.

  This assumption could be wrong, with the result that an application
  could thing that it has created a new file, when in fact, it has opened
  an existing one.

-- 
--
********************************************************************
*                                                                  *
*  Bas van der Vlies                     e-mail: basv@sara.nl      *
*  SARA - Academic Computing Services    phone:  +31 20 592 8012   *
*  Kruislaan 415                         fax:    +31 20 6683167    *
*  1098 SJ Amsterdam                                               *
*                                                                  *
********************************************************************


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs