nfs client oops, all 2.4 kernels

nfs client oops, all 2.4 kernels

[Erik DeBill]

I've been running into a repeatable oops in the NFS client code,
apparently related to file locking.

I've verified this with UP, SMP(2,4 procs), small (128M) and large
(4G) memory under RedHat 7.1, SuSe 7.1 and SuSe 7.2.  All on x86
hardware.

The problem occurs while running DB2 EEE.  DB2 EEE is a clustered
database, which uses NFS to share the database home directory between
nodes in the cluster.  This is primarily to synchronize some config
files, but by default it will also store all data and logs in that
same nfs mounted directory.  Oopses occur on the machine (I've only
been testing on a 2 node cluster) which does the NFS client mount of
the home directory.  

If you let it keep all it's data in the nfs mounted directory you can
generate an oops very quickly once multiple clients start connecting
to the db.  The more clients, the faster the oops.  The more thorough
the job you do of removing things from NFS, the harder to reproduce.

If you run it without NFS (completely unsupported) it runs fine and I
haven't been able to reproduce the oops. 

It occurs with all 2.4 series kernels, at least up to 2.4.9-ac7 (which
had some nfs client fixes)

I've attached the ksymoops trace from 2 different incidents.  The
first occurred under 2.4.9-ac7, and is pretty typical.  Death in lock
related code as part of sys_close().  The second is from 2.4.9 +
linux-2.4.9-NFS_ALL.dif.  It's a little unusual in that several oopses
happened all at once.

Please let me know if there's anything I can do to help track these
guys down.  I can give you access to several machines with different
hardware, and distros which all exhibit this bug.


Thanks,
Erik DeBill



ksymoops 2.4.0 on i686 2.4.9-ac7.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.9-ac7/ (default)
     -m /boot/System.map-2.4.9-ac7 (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Warning (compare_maps): mismatch on symbol md_size  , md says c8a0e380, /lib/modules/2.4.9-ac7/kernel/drivers/md/md.o says c8a0e1a0.  Ignoring /lib/modules/2.4.9-ac7/kernel/drivers/md/md.o entry
Warning (compare_maps): mismatch on symbol mddev_map  , md says c8a0db80, /lib/modules/2.4.9-ac7/kernel/drivers/md/md.o says c8a0d9a0.  Ignoring /lib/modules/2.4.9-ac7/kernel/drivers/md/md.o entry
Sep  8 22:00:06 jerry kernel: WARNING: MP table in the EBDA can be UNSAFE, contact linux-smp@vger.kernel.org if you experience SMP problems!
Sep  8 22:00:07 jerry kernel: cpu: 0, clocks: 1329124, slice: 664562
Sep  8 22:04:15 jerry kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep  8 22:04:15 jerry kernel: c0145c2b
Sep  8 22:04:15 jerry kernel: *pde = 00000000
Sep  8 22:04:15 jerry kernel: Oops: 0000
Sep  8 22:04:15 jerry kernel: CPU:    0
Sep  8 22:04:15 jerry kernel: EIP:    0010:[<c0145c2b>]
Using defaults from ksymoops -t elf32-i386 -a i386
Sep  8 22:04:15 jerry kernel: EFLAGS: 00010286
Sep  8 22:04:15 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c72cc4e0   edx: c79155c8
Sep  8 22:04:15 jerry kernel: esi: c240b440   edi: 00000000   ebp: bfffba10   esp: c21a3f60
Sep  8 22:04:15 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep  8 22:04:15 jerry kernel: Process db2sysc (pid: 867, stackpage=c21a3000)
Sep  8 22:04:15 jerry kernel: Stack: c79153fc c79155c8 c240b440 00000000 c79155c8 c240b440 c0147756 c79155c8 
Sep  8 22:04:15 jerry kernel:        00000000 c21a2000 c27cc500 c0135a96 c27cc500 c240b440 00000000 c27cc500 
Sep  8 22:04:15 jerry kernel:        00000000 c27cc500 00000030 00000000 c0135b0b c27cc500 c240b440 c21a2000 
Sep  8 22:04:15 jerry kernel: Call Trace: [<c0147756>] [<c0135a96>] [<c0135b0b>] [<c0106efb>] 
Sep  8 22:04:15 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00 00 00 8b 56 04 89 

>>EIP; c0145c2b <locks_delete_lock+b/d0>   <=====
Trace; c0147756 <locks_remove_posix+86/c0>
Trace; c0135a96 <filp_close+86/a0>
Trace; c0135b0b <sys_close+5b/70>
Trace; c0106efb <system_call+33/38>
Code;  c0145c2b <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code;  c0145c2b <locks_delete_lock+b/d0>   <=====
   0:   8b 03                     mov    (%ebx),%eax   <=====
Code;  c0145c2d <locks_delete_lock+d/d0>
   2:   8d 73 04                  lea    0x4(%ebx),%esi
Code;  c0145c30 <locks_delete_lock+10/d0>
   5:   89 02                     mov    %eax,(%edx)
Code;  c0145c32 <locks_delete_lock+12/d0>
   7:   8b 43 04                  mov    0x4(%ebx),%eax
Code;  c0145c35 <locks_delete_lock+15/d0>
   a:   c7 03 00 00 00 00         movl   $0x0,(%ebx)
Code;  c0145c3b <locks_delete_lock+1b/d0>
  10:   8b 56 04                  mov    0x4(%esi),%edx
Code;  c0145c3e <locks_delete_lock+1e/d0>
  13:   89 00                     mov    %eax,(%eax)


3 warnings issued.  Results may not be reliable.



---------------------------------------------------------------------------
these next are from a single run under 2.4.9 + the all encompassing NFS patch.

DB2 gets shut down as soon as an oops occurs, these must be from the
death throes.
---------------------------------------------------------------------------

ksymoops 2.4.0 on i686 2.4.9nfs.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.9nfs/ (default)
     -m /boot/System.map-2.4.9nfs (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Warning (compare_maps): mismatch on symbol md_size  , md says c8a0e560, /lib/modules/2.4.9nfs/kernel/drivers/md/md.o says c8a0e380.  Ignoring /lib/modules/2.4.9nfs/kernel/drivers/md/md.o entry
Warning (compare_maps): mismatch on symbol mddev_map  , md says c8a0dd60, /lib/modules/2.4.9nfs/kernel/drivers/md/md.o says c8a0db80.  Ignoring /lib/modules/2.4.9nfs/kernel/drivers/md/md.o entry
Sep 10 08:47:05 jerry kernel: WARNING: MP table in the EBDA can be UNSAFE, contact linux-smp@vger.kernel.org if you experience SMP problems!
Sep 10 08:47:08 jerry kernel: cpu: 0, clocks: 1329118, slice: 664559
Sep 10 08:49:09 jerry kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 10 08:49:09 jerry kernel: c0146cab
Sep 10 08:49:09 jerry kernel: *pde = 00000000
Sep 10 08:49:09 jerry kernel: Oops: 0000
Sep 10 08:49:09 jerry kernel: CPU:    0
Sep 10 08:49:09 jerry kernel: EIP:    0010:[fcntl_setlease+171/640]
Sep 10 08:49:09 jerry kernel: EIP:    0010:[<c0146cab>]
Using defaults from ksymoops -t elf32-i386 -a i386
Sep 10 08:49:09 jerry kernel: EFLAGS: 00010286
Sep 10 08:49:09 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c7302b60   edx: c7926d54
Sep 10 08:49:09 jerry kernel: esi: c6b76a40   edi: 00000000   ebp: bfffba10   esp: c6a4ff60
Sep 10 08:49:09 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep 10 08:49:09 jerry kernel: Process db2sysc (pid: 769, stackpage=c6a4f000)
Sep 10 08:49:09 jerry kernel: Stack: c7926e0c c7926d54 c6b76a40 00000000 c7926d54 c6b76a40 c01488f6 c7926d54 
Sep 10 08:49:09 jerry kernel:        00000000 c6a4e000 c6a23d40 c013512d c6a23d40 c6b76a40 00000000 c6a23d40 
Sep 10 08:49:09 jerry kernel:        00000000 c6a23d40 00000030 00000000 c013519b c6a23d40 c6b76a40 c6a4e000 
Sep 10 08:49:09 jerry kernel: Call Trace: [d_validate+54/224] [sys_chdir+253/256] [sys_fchdir+107/208] [system_call+19/56] 
Sep 10 08:49:09 jerry kernel: Call Trace: [<c01488f6>] [<c013512d>] [<c013519b>] [<c0106edb>] 
Sep 10 08:49:09 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00 00 00 8b 56 04 89 

>>EIP; c0146cab <locks_delete_lock+b/d0>   <=====
Trace; c01488f6 <locks_remove_posix+86/e0>
Trace; c013512d <filp_close+9d/b0>
Trace; c013519b <sys_close+5b/70>
Trace; c0106edb <system_call+33/38>
Code;  c0146cab <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code;  c0146cab <locks_delete_lock+b/d0>   <=====
   0:   8b 03                     mov    (%ebx),%eax   <=====
Code;  c0146cad <locks_delete_lock+d/d0>
   2:   8d 73 04                  lea    0x4(%ebx),%esi
Code;  c0146cb0 <locks_delete_lock+10/d0>
   5:   89 02                     mov    %eax,(%edx)
Code;  c0146cb2 <locks_delete_lock+12/d0>
   7:   8b 43 04                  mov    0x4(%ebx),%eax
Code;  c0146cb5 <locks_delete_lock+15/d0>
   a:   c7 03 00 00 00 00         movl   $0x0,(%ebx)
Code;  c0146cbb <locks_delete_lock+1b/d0>
  10:   8b 56 04                  mov    0x4(%esi),%edx
Code;  c0146cbe <locks_delete_lock+1e/d0>
  13:   89 00                     mov    %eax,(%eax)

Sep 10 08:49:09 jerry kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 10 08:49:09 jerry kernel: c0146cab
Sep 10 08:49:09 jerry kernel: *pde = 00000000
Sep 10 08:49:09 jerry kernel: Oops: 0000
Sep 10 08:49:09 jerry kernel: CPU:    0
Sep 10 08:49:09 jerry kernel: EIP:    0010:[fcntl_setlease+171/640]
Sep 10 08:49:10 jerry kernel: EIP:    0010:[<c0146cab>]
Sep 10 08:49:10 jerry kernel: EFLAGS: 00010286
Sep 10 08:49:10 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c7302b60   edx: c7926be4
Sep 10 08:49:10 jerry kernel: esi: c6b76700   edi: 00000000   ebp: bfffba10   esp: c5d15f60
Sep 10 08:49:10 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep 10 08:49:10 jerry kernel: Process db2sysc (pid: 770, stackpage=c5d15000)
Sep 10 08:49:10 jerry kernel: Stack: c7926f20 c7926be4 c6b76700 00000000 c7926be4 c6b76700 c01488f6 c7926be4 
Sep 10 08:49:10 jerry kernel:        00000000 c5d14000 c6a23f20 c013512d c6a23f20 c6b76700 00000000 c6a23f20 
Sep 10 08:49:10 jerry kernel:        00000000 c6a23f20 00000030 00000000 c013519b c6a23f20 c6b76700 c5d14000 
Sep 10 08:49:10 jerry kernel: Call Trace: [d_validate+54/224] [sys_chdir+253/256] [sys_fchdir+107/208] [system_call+19/56] 
Sep 10 08:49:10 jerry kernel: Call Trace: [<c01488f6>] [<c013512d>] [<c013519b>] [<c0106edb>] 
Sep 10 08:49:10 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00 00 00 8b 56 04 89 

>>EIP; c0146cab <locks_delete_lock+b/d0>   <=====
Trace; c01488f6 <locks_remove_posix+86/e0>
Trace; c013512d <filp_close+9d/b0>
Trace; c013519b <sys_close+5b/70>
Trace; c0106edb <system_call+33/38>
Code;  c0146cab <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code;  c0146cab <locks_delete_lock+b/d0>   <=====
   0:   8b 03                     mov    (%ebx),%eax   <=====
Code;  c0146cad <locks_delete_lock+d/d0>
   2:   8d 73 04                  lea    0x4(%ebx),%esi
Code;  c0146cb0 <locks_delete_lock+10/d0>
   5:   89 02                     mov    %eax,(%edx)
Code;  c0146cb2 <locks_delete_lock+12/d0>
   7:   8b 43 04                  mov    0x4(%ebx),%eax
Code;  c0146cb5 <locks_delete_lock+15/d0>
   a:   c7 03 00 00 00 00         movl   $0x0,(%ebx)
Code;  c0146cbb <locks_delete_lock+1b/d0>
  10:   8b 56 04                  mov    0x4(%esi),%edx
Code;  c0146cbe <locks_delete_lock+1e/d0>
  13:   89 00                     mov    %eax,(%eax)

Sep 10 08:49:10 jerry kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 10 08:49:10 jerry kernel: c0146cab
Sep 10 08:49:10 jerry kernel: *pde = 00000000
Sep 10 08:49:10 jerry kernel: Oops: 0000
Sep 10 08:49:10 jerry kernel: CPU:    0
Sep 10 08:49:10 jerry kernel: EIP:    0010:[fcntl_setlease+171/640]
Sep 10 08:49:10 jerry kernel: EIP:    0010:[<c0146cab>]
Sep 10 08:49:10 jerry kernel: EFLAGS: 00010286
Sep 10 08:49:10 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c7302b60   edx: c7926c40
Sep 10 08:49:10 jerry kernel: esi: c6439dc0   edi: 00000000   ebp: bfffba10   esp: c643bf60
Sep 10 08:49:10 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep 10 08:49:10 jerry kernel: Process db2sysc (pid: 776, stackpage=c643b000)
Sep 10 08:49:10 jerry kernel: Stack: c7926db0 c7926c40 c6439dc0 00000000 c7926c40 c6439dc0 c01488f6 c7926c40 
Sep 10 08:49:10 jerry kernel:        00000000 c643a000 c6a23da0 c013512d c6a23da0 c6439dc0 00000000 c6a23da0 
Sep 10 08:49:10 jerry kernel:        00000000 c6a23da0 00000030 00000000 c013519b c6a23da0 c6439dc0 c643a000 
Sep 10 08:49:10 jerry kernel: Call Trace: [d_validate+54/224] [sys_chdir+253/256] [sys_fchdir+107/208] [system_call+19/56] 
Sep 10 08:49:10 jerry kernel: Call Trace: [<c01488f6>] [<c013512d>] [<c013519b>] [<c0106edb>] 
Sep 10 08:49:10 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00 00 00 8b 56 04 89 

>>EIP; c0146cab <locks_delete_lock+b/d0>   <=====
Trace; c01488f6 <locks_remove_posix+86/e0>
Trace; c013512d <filp_close+9d/b0>
Trace; c013519b <sys_close+5b/70>
Trace; c0106edb <system_call+33/38>
Code;  c0146cab <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code;  c0146cab <locks_delete_lock+b/d0>   <=====
   0:   8b 03                     mov    (%ebx),%eax   <=====
Code;  c0146cad <locks_delete_lock+d/d0>
   2:   8d 73 04                  lea    0x4(%ebx),%esi
Code;  c0146cb0 <locks_delete_lock+10/d0>
   5:   89 02                     mov    %eax,(%edx)
Code;  c0146cb2 <locks_delete_lock+12/d0>
   7:   8b 43 04                  mov    0x4(%ebx),%eax
Code;  c0146cb5 <locks_delete_lock+15/d0>
   a:   c7 03 00 00 00 00         movl   $0x0,(%ebx)
Code;  c0146cbb <locks_delete_lock+1b/d0>
  10:   8b 56 04                  mov    0x4(%esi),%edx
Code;  c0146cbe <locks_delete_lock+1e/d0>
  13:   89 00                     mov    %eax,(%eax)

Sep 10 08:49:10 jerry kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 10 08:49:10 jerry kernel: c0146cab
Sep 10 08:49:10 jerry kernel: *pde = 00000000
Sep 10 08:49:10 jerry kernel: Oops: 0000
Sep 10 08:49:10 jerry kernel: CPU:    0
Sep 10 08:49:10 jerry kernel: EIP:    0010:[fcntl_setlease+171/640]
Sep 10 08:49:10 jerry kernel: EIP:    0010:[<c0146cab>]
Sep 10 08:49:10 jerry kernel: EFLAGS: 00010286
Sep 10 08:49:10 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c7302b60   edx: c7926b88
Sep 10 08:49:10 jerry kernel: esi: c4275240   edi: 00000000   ebp: bfffba10   esp: c5d8df60
Sep 10 08:49:10 jerry kernel: ds: 0018   es: 0018   ss: 0018
Sep 10 08:49:10 jerry kernel: Process db2sysc (pid: 774, stackpage=c5d8d000)
Sep 10 08:49:10 jerry kernel: Stack: c7926b2c c7926b88 c4275240 00000000 c7926b88 c4275240 c01488f6 c7926b88 
Sep 10 08:49:11 jerry kernel:        00000000 c5d8c000 c71e30c0 c013512d c71e30c0 c4275240 00000000 c71e30c0 
Sep 10 08:49:11 jerry kernel:        00000000 c71e30c0 00000030 00000000 c013519b c71e30c0 c4275240 c5d8c000 
Sep 10 08:49:11 jerry kernel: Call Trace: [d_validate+54/224] [sys_chdir+253/256] [sys_fchdir+107/208] [system_call+19/56] 
Sep 10 08:49:11 jerry kernel: Call Trace: [<c01488f6>] [<c013512d>] [<c013519b>] [<c0106edb>] 
Sep 10 08:49:11 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00 00 00 8b 56 04 89 

>>EIP; c0146cab <locks_delete_lock+b/d0>   <=====
Trace; c01488f6 <locks_remove_posix+86/e0>
Trace; c013512d <filp_close+9d/b0>
Trace; c013519b <sys_close+5b/70>
Trace; c0106edb <system_call+33/38>
Code;  c0146cab <locks_delete_lock+b/d0>
00000000 <_EIP>:
Code;  c0146cab <locks_delete_lock+b/d0>   <=====
   0:   8b 03                     mov    (%ebx),%eax   <=====
Code;  c0146cad <locks_delete_lock+d/d0>
   2:   8d 73 04                  lea    0x4(%ebx),%esi
Code;  c0146cb0 <locks_delete_lock+10/d0>
   5:   89 02                     mov    %eax,(%edx)
Code;  c0146cb2 <locks_delete_lock+12/d0>
   7:   8b 43 04                  mov    0x4(%ebx),%eax
Code;  c0146cb5 <locks_delete_lock+15/d0>
   a:   c7 03 00 00 00 00         movl   $0x0,(%ebx)
Code;  c0146cbb <locks_delete_lock+1b/d0>
  10:   8b 56 04                  mov    0x4(%esi),%edx
Code;  c0146cbe <locks_delete_lock+1e/d0>
  13:   89 00                     mov    %eax,(%eax)


3 warnings issued.  Results may not be reliable.

Re: nfs client oops, all 2.4 kernels

[Stephan von Krawczynski]

On Mon, 10 Sep 2001 10:02:02 -0500 Erik DeBill <erik@www.creditminders.com>
wrote:

> I've been running into a repeatable oops in the NFS client code,
> apparently related to file locking.
> Sep 10 08:49:10 jerry kernel: Unable to handle kernel NULL pointer
dereference at virtual address 00000000
> Sep 10 08:49:10 jerry kernel: c0146cab
> Sep 10 08:49:10 jerry kernel: *pde = 00000000
> Sep 10 08:49:10 jerry kernel: Oops: 0000
> Sep 10 08:49:10 jerry kernel: CPU:    0
> Sep 10 08:49:10 jerry kernel: EIP:    0010:[fcntl_setlease+171/640]
> Sep 10 08:49:10 jerry kernel: EIP:    0010:[<c0146cab>]
> Sep 10 08:49:10 jerry kernel: EFLAGS: 00010286
> Sep 10 08:49:10 jerry kernel: eax: 00000000   ebx: 00000000   ecx: c7302b60  
edx: c7926b88
> Sep 10 08:49:10 jerry kernel: esi: c4275240   edi: 00000000   ebp: bfffba10  
esp: c5d8df60
> Sep 10 08:49:10 jerry kernel: ds: 0018   es: 0018   ss: 0018
> Sep 10 08:49:10 jerry kernel: Process db2sysc (pid: 774, stackpage=c5d8d000)
> Sep 10 08:49:10 jerry kernel: Stack: c7926b2c c7926b88 c4275240 00000000
c7926b88 c4275240 c01488f6 c7926b88 
> Sep 10 08:49:11 jerry kernel:        00000000 c5d8c000 c71e30c0 c013512d
c71e30c0 c4275240 00000000 c71e30c0 
> Sep 10 08:49:11 jerry kernel:        00000000 c71e30c0 00000030 00000000
c013519b c71e30c0 c4275240 c5d8c000 
> Sep 10 08:49:11 jerry kernel: Call Trace: [d_validate+54/224]
[sys_chdir+253/256] [sys_fchdir+107/208] [system_call+19/56] 
> Sep 10 08:49:11 jerry kernel: Call Trace: [<c01488f6>] [<c013512d>]
[<c013519b>] [<c0106edb>] 
> Sep 10 08:49:11 jerry kernel: Code: 8b 03 8d 73 04 89 02 8b 43 04 c7 03 00 00
00 00 8b 56 04 89 
> 
> >>EIP; c0146cab <locks_delete_lock+b/d0>   <=====
> Trace; c01488f6 <locks_remove_posix+86/e0>
> Trace; c013512d <filp_close+9d/b0>
> Trace; c013519b <sys_close+5b/70>
> Trace; c0106edb <system_call+33/38>
> Code;  c0146cab <locks_delete_lock+b/d0>
> 00000000 <_EIP>:
> Code;  c0146cab <locks_delete_lock+b/d0>   <=====
>    0:   8b 03                     mov    (%ebx),%eax   <=====
> Code;  c0146cad <locks_delete_lock+d/d0>
>    2:   8d 73 04                  lea    0x4(%ebx),%esi
> Code;  c0146cb0 <locks_delete_lock+10/d0>
>    5:   89 02                     mov    %eax,(%edx)
> Code;  c0146cb2 <locks_delete_lock+12/d0>
>    7:   8b 43 04                  mov    0x4(%ebx),%eax
> Code;  c0146cb5 <locks_delete_lock+15/d0>
>    a:   c7 03 00 00 00 00         movl   $0x0,(%ebx)
> Code;  c0146cbb <locks_delete_lock+1b/d0>
>   10:   8b 56 04                  mov    0x4(%esi),%edx
> Code;  c0146cbe <locks_delete_lock+1e/d0>
>   13:   89 00                     mov    %eax,(%eax)
> 

Hm, I am sure no nfs or fs guru, but looking at

static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
{
        struct file_lock *fl = *thisfl_p;

        *thisfl_p = fl->fl_next;
        fl->fl_next = NULL;

        list_del(&fl->fl_link);
        INIT_LIST_HEAD(&fl->fl_link);

        fasync_helper(0, fl->fl_file, 0, &fl->fl_fasync);
        if (fl->fl_fasync != NULL){
                printk(KERN_ERR "locks_delete_lock: fasync == %p\n",
fl->fl_fasync);
                fl->fl_fasync = NULL;
        }

        if (fl->fl_remove)
                fl->fl_remove(fl);

        locks_wake_up_blocks(fl, wait);
        locks_free_lock(fl);
}

in linux/fs/locks.c I would say it fails either because thisfl_p is NULL or
*thisfl_p is NULL. Try securing it via:


static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
{
        struct file_lock *fl;
	
	if (thisfl_p == NULL || *thisfl_p == NULL)
		return;

	fl = *thisfl_p;

        *thisfl_p = fl->fl_next;
        fl->fl_next = NULL;

...
}

This is for sure not the cure, but may help your setup.

Regards,
Stephan

Re: nfs client oops, all 2.4 kernels

[Olivier Molteni]

Stephan von Krawczynski wrote:

> On Mon, 10 Sep 2001 10:02:02 -0500 Erik DeBill <erik@www.creditminders.com>
> wrote:
>
> > I've been running into a repeatable oops in the NFS client code,
> > apparently related to file locking.
>

>
> in linux/fs/locks.c I would say it fails either because thisfl_p is NULL or
> *thisfl_p is NULL. Try securing it via:
>
> static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
> {
>         struct file_lock *fl;
>
>         if (thisfl_p == NULL || *thisfl_p == NULL)
>                 return;
>
>         fl = *thisfl_p;
>
>         *thisfl_p = fl->fl_next;
>         fl->fl_next = NULL;
>
> ...
> }
>
> This is for sure not the cure, but may help your setup.

Hi, see my post and related answers [ Oops NFS Locking in 2.4.x] I have the same
Problem.
Returning on *thisfl_p == NULL don't fix the trouble... Kernel no more Oops, but
process stay in wait state on IO (D).

See the answers from Trond Myklebust, I think he is right...

Regards,
Olivier.

Re: nfs client oops, all 2.4 kernels

[edebill]

On Mon, Sep 10, 2001 at 09:52:10PM +0200, Olivier Molteni wrote:
> 
> Hi, see my post and related answers [ Oops NFS Locking in 2.4.x] I have the same
> Problem.
> Returning on *thisfl_p == NULL don't fix the trouble... Kernel no more Oops, but
> process stay in wait state on IO (D).
> 
> See the answers from Trond Myklebust, I think he is right...

That's what I get for not reading LKML over the weekend :-)

Sounds like he has a handle on the problem - I'll gladly test patches
if one of you comes up with a (proposed) fix.


Erik

nfs client oops, all 2.4 kernels

[Trond Myklebust]

>>>>> " " == Erik DeBill <erik@www.creditminders.com> writes:

     > I've been running into a repeatable oops in the NFS client
     > code, apparently related to file locking.

This is the same Oops as was sent in by Olivier Molteni. It's due to
an incorrect assumption in the locking code. To repeat:

2 processes are calling close(), which under POSIX, tries to free all
locks held by a program. The problem occurs when you `dup()' a file.

In that case, both processes can call close() at the same time, thus
triggering a call to filp_close().

The race boils down to:

   -  locks_unlock_delete() assumes that the BKL (kernel_lock()) is
      sufficient to protect against *thisfl_p from disappearing
      beneath it due to some second process.
BUT
   -  The call to lock() in locks_unlock_delete() sleeps when the
      underlying filesystem is NFS, hence 2 processes can race despite
      the BKL assumption.

IMHO the correct thing to do is to detach the lock from the global
linked list *before* one makes the call to f_op->lock() down to the
NFS code.

This would probably entail splitting locks_delete_lock() into 2
procedures: one that does the unlinking from the global list:

        *thisfl_p = fl->fl_next;
        fl->fl_next = NULL;

        list_del(&fl->fl_link);
        INIT_LIST_HEAD(&fl->fl_link);

the other that does the rest.

Could you check if the appended patch works?

Cheers,
   Trond

--- linux-2.4.9/fs/locks.c.orig	Thu Jul  5 00:39:28 2001
+++ linux-2.4.9/fs/locks.c	Tue Sep 11 10:41:53 2001
@@ -473,21 +473,26 @@
 		fl->fl_insert(fl);
 }
 
-/* Delete a lock and then free it.
- * Remove our lock from the lock lists, wake up processes that are blocked
- * waiting for this lock, notify the FS that the lock has been cleared and
- * finally free the lock.
+/*
+ * Remove lock from the lock lists
  */
-static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
+static inline void _unhash_lock(struct file_lock **thisfl_p)
 {
 	struct file_lock *fl = *thisfl_p;
 
 	*thisfl_p = fl->fl_next;
 	fl->fl_next = NULL;
 
-	list_del(&fl->fl_link);
-	INIT_LIST_HEAD(&fl->fl_link);
+	list_del_init(&fl->fl_link);
+}
 
+/*
+ * Wake up processes that are blocked waiting for this lock,
+ * notify the FS that the lock has been cleared and
+ * finally free the lock.
+ */
+static inline void _delete_lock(struct file_lock *fl, unsigned int wait)
+{
 	fasync_helper(0, fl->fl_file, 0, &fl->fl_fasync);
 	if (fl->fl_fasync != NULL){
 		printk(KERN_ERR "locks_delete_lock: fasync == %p\n", fl->fl_fasync);
@@ -502,6 +507,15 @@
 }
 
 /*
+ * Delete a lock and then free it.
+ */
+static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
+{
+	_unhash_lock(thisfl_p);
+	_delete_lock(*thisfl_p, wait);
+}
+
+/*
  * Call back client filesystem in order to get it to unregister a lock,
  * then delete lock. Essentially useful only in locks_remove_*().
  * Note: this must be called with the semaphore already held!
@@ -511,12 +525,13 @@
 	struct file_lock *fl = *thisfl_p;
 	int (*lock)(struct file *, int, struct file_lock *);
 
+	_unhash_lock(thisfl_p);
 	if (fl->fl_file->f_op &&
 	    (lock = fl->fl_file->f_op->lock) != NULL) {
 		fl->fl_type = F_UNLCK;
 		lock(fl->fl_file, F_SETLK, fl);
 	}
-	locks_delete_lock(thisfl_p, 0);
+	_delete_lock(fl, 0);
 }
 
 /* Determine if lock sys_fl blocks lock caller_fl. Common functionality
@@ -1661,6 +1676,7 @@
 	while ((fl = *before) != NULL) {
 		if ((fl->fl_flags & FL_POSIX) && fl->fl_owner == owner) {
 			locks_unlock_delete(before);
+			before = &inode->i_flock;
 			continue;
 		}
 		before = &fl->fl_next;

Re: nfs client oops, all 2.4 kernels

[Trond Myklebust]

>>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes:

     > Could you check if the appended patch works?

The previous patch had a bug in the rewrite of locks_delete_lock()
which will Oops/corrupt the inode->i_flock list. Please consign to
/dev/null, and try the following instead.

(Note to self: Never attempt to write any patches *before* lunch...)

Cheers,
   Trond

--- linux-2.4.9/fs/locks.c.orig	Thu Jul  5 00:39:28 2001
+++ linux-2.4.9/fs/locks.c	Tue Sep 11 15:09:13 2001
@@ -473,21 +473,26 @@
 		fl->fl_insert(fl);
 }
 
-/* Delete a lock and then free it.
- * Remove our lock from the lock lists, wake up processes that are blocked
- * waiting for this lock, notify the FS that the lock has been cleared and
- * finally free the lock.
+/*
+ * Remove lock from the lock lists
  */
-static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
+static inline void _unhash_lock(struct file_lock **thisfl_p)
 {
 	struct file_lock *fl = *thisfl_p;
 
 	*thisfl_p = fl->fl_next;
 	fl->fl_next = NULL;
 
-	list_del(&fl->fl_link);
-	INIT_LIST_HEAD(&fl->fl_link);
+	list_del_init(&fl->fl_link);
+}
 
+/*
+ * Wake up processes that are blocked waiting for this lock,
+ * notify the FS that the lock has been cleared and
+ * finally free the lock.
+ */
+static inline void _delete_lock(struct file_lock *fl, unsigned int wait)
+{
 	fasync_helper(0, fl->fl_file, 0, &fl->fl_fasync);
 	if (fl->fl_fasync != NULL){
 		printk(KERN_ERR "locks_delete_lock: fasync == %p\n", fl->fl_fasync);
@@ -502,6 +507,17 @@
 }
 
 /*
+ * Delete a lock and then free it.
+ */
+static void locks_delete_lock(struct file_lock **thisfl_p, unsigned int wait)
+{
+	struct file_lock *fl = *thisfl_p;
+
+	_unhash_lock(thisfl_p);
+	_delete_lock(fl, wait);
+}
+
+/*
  * Call back client filesystem in order to get it to unregister a lock,
  * then delete lock. Essentially useful only in locks_remove_*().
  * Note: this must be called with the semaphore already held!
@@ -511,12 +527,13 @@
 	struct file_lock *fl = *thisfl_p;
 	int (*lock)(struct file *, int, struct file_lock *);
 
+	_unhash_lock(thisfl_p);
 	if (fl->fl_file->f_op &&
 	    (lock = fl->fl_file->f_op->lock) != NULL) {
 		fl->fl_type = F_UNLCK;
 		lock(fl->fl_file, F_SETLK, fl);
 	}
-	locks_delete_lock(thisfl_p, 0);
+	_delete_lock(fl, 0);
 }
 
 /* Determine if lock sys_fl blocks lock caller_fl. Common functionality
@@ -1661,6 +1678,7 @@
 	while ((fl = *before) != NULL) {
 		if ((fl->fl_flags & FL_POSIX) && fl->fl_owner == owner) {
 			locks_unlock_delete(before);
+			before = &inode->i_flock;
 			continue;
 		}
 		before = &fl->fl_next;

Re: nfs client oops, all 2.4 kernels

[Erik DeBill]

On Tue, Sep 11, 2001 at 03:15:34PM +0200, Trond Myklebust wrote:
> >>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes:
> 
>      > Could you check if the appended patch works?
> 
> The previous patch had a bug in the rewrite of locks_delete_lock()
> which will Oops/corrupt the inode->i_flock list. Please consign to
> /dev/null, and try the following instead.
> 
> (Note to self: Never attempt to write any patches *before* lunch...)



This patch seems to work.  I'll keep testing, but it's been running
well for an hour or so already.


Thanks,
Erik

Re: nfs client oops, all 2.4 kernels

[Olivier Molteni]

Trond Myklebust wrote:

> >>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes:
>
>      > Could you check if the appended patch works?

Hi,

First, thank you very much for posting this patch !!

Because I'm awake and at work for too long, I have no had time to test it...
Now, I MUST to sleep if I want to survive ;)) But, I promise I will test your patch
ASAP and I will give you a feed back on it.

Cheers
Olivier.

Re: nfs client oops, all 2.4 kernels

[Olivier Molteni]

Trond Myklebust wrote:

> >>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes:
>
>      > Could you check if the appended patch works?
>

Hi,

I have just finished to test with 4 PCs doing intensive lock/unlock on several NFS
files for 3 hours... it seems to work !
I will now test with the initial mail application, but if I will post again only if the
test fails...

Thank's a lot again !!

Cheers,
Olivier.