kswapd oops in 2.4.20 SMP+NFS

kswapd oops in 2.4.20 SMP+NFS

[Soeren Sonnenburg]

Hi.

I got the following oops recently. The machine is still up and running
and was working stably for a year now...

Linux 2.4.20 #1 SMP Tue Dec 10 11:16:20 CET 2002 i686 unknown
2 x AMD K7-MP 1200MHz PCI(5-64)	TYAN Thunder K7 S2462 Mainboard 1G ECC Memory

[...]
nfsd-fh: found a name that I didn't expect: sys/oz
nfsd-fh: found a name that I didn't expect: sys/oz
nfsd-fh: found a name that I didn't expect: bin/x86
nfsd-fh: found a name that I didn't expect: bin/x86
nfsd-fh: found a name that I didn't expect: etc/bla
VFS: Busy inodes after unmount. Self-destruct in 5 seconds.  Have a nice day...
Unable to handle kernel paging request at virtual address 00268eb7
 printing eip:
c014f5a3
*pde = 00000000
Oops: 0000
CPU:    1
EIP:    0010:[<c014f5a3>]    Not tainted
EFLAGS: 00010202
eax: 00000000   ebx: e34b05c0   ecx: e34b05d0   edx: d1da1c40
esi: d6c03e00   edi: 00268ea7   ebp: 0000318e   esp: f7edff34
ds: 0018   es: 0018   ss: 0018
Process kswapd (pid: 5, stackpage=f7edf000)
Stack: d1da1c58 e34b05c0 e34b05c0 c017a75f e34b05c0 d1da1c58 d1da1c40 c014cf20 
       d1da1c40 e34b05c0 00000006 000001d0 00000020 00000006 c014d2ab 000108c9 
       c0132046 00000006 000001d0 00000006 00000020 000001d0 c02d94b4 c02d94b4 
Call Trace:    [<c017a75f>] [<c014cf20>] [<c014d2ab>] [<c0132046>] [<c013209c>]
  [<c01321a1>] [<c0132206>] [<c013232d>] [<c0105708>]

Code: 8b 47 10 85 c0 74 06 53 ff d0 83 c4 04 68 f0 a3 2d c0 8d 43 

Re: kswapd oops in 2.4.20 SMP+NFS

[Oleg Drokin]

Hello!

On Thu, Mar 20, 2003 at 03:23:24PM +0100, Soeren Sonnenburg wrote:

> I got the following oops recently. The machine is still up and running
> and was working stably for a year now...
> Linux 2.4.20 #1 SMP Tue Dec 10 11:16:20 CET 2002 i686 unknown
> 2 x AMD K7-MP 1200MHz PCI(5-64)	TYAN Thunder K7 S2462 Mainboard 1G ECC Memory
> [...]
> nfsd-fh: found a name that I didn't expect: sys/oz
> nfsd-fh: found a name that I didn't expect: sys/oz
> nfsd-fh: found a name that I didn't expect: bin/x86
> nfsd-fh: found a name that I didn't expect: bin/x86
> nfsd-fh: found a name that I didn't expect: etc/bla
> VFS: Busy inodes after unmount. Self-destruct in 5 seconds.  Have a nice day...

Hm, what is the underlying host filesystem?

Also feed below oops to ksymoops please.

> Unable to handle kernel paging request at virtual address 00268eb7
>  printing eip:
> c014f5a3
> *pde = 00000000
> Oops: 0000
> CPU:    1
> EIP:    0010:[<c014f5a3>]    Not tainted
> EFLAGS: 00010202
> eax: 00000000   ebx: e34b05c0   ecx: e34b05d0   edx: d1da1c40
> esi: d6c03e00   edi: 00268ea7   ebp: 0000318e   esp: f7edff34
> ds: 0018   es: 0018   ss: 0018
> Process kswapd (pid: 5, stackpage=f7edf000)
> Stack: d1da1c58 e34b05c0 e34b05c0 c017a75f e34b05c0 d1da1c58 d1da1c40 c014cf20 
>        d1da1c40 e34b05c0 00000006 000001d0 00000020 00000006 c014d2ab 000108c9 
>        c0132046 00000006 000001d0 00000006 00000020 000001d0 c02d94b4 c02d94b4 
> Call Trace:    [<c017a75f>] [<c014cf20>] [<c014d2ab>] [<c0132046>] [<c013209c>]
>   [<c01321a1>] [<c0132206>] [<c013232d>] [<c0105708>]
> 
> Code: 8b 47 10 85 c0 74 06 53 ff d0 83 c4 04 68 f0 a3 2d c0 8d 43 

Thank you.

Bye,
    Oleg

Re: kswapd oops in 2.4.20 SMP+NFS

[Soeren Sonnenburg]

On Fri, 2003-03-21 at 09:28, Oleg Drokin wrote:
> Hello!
> > nfsd-fh: found a name that I didn't expect: sys/oz
> > nfsd-fh: found a name that I didn't expect: bin/x86
> > nfsd-fh: found a name that I didn't expect: bin/x86
> > nfsd-fh: found a name that I didn't expect: etc/bla
> > VFS: Busy inodes after unmount. Self-destruct in 5 seconds.  Have a nice day...
> 
> Hm, what is the underlying host filesystem?

reiserfs of course ;-)
> Thank you.

Here we go:

Unable to handle kernel paging request at virtual address 00268eb7
c014f5a3
*pde = 00000000
Oops: 0000
CPU:    1
EIP:    0010:[<c014f5a3>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010202
eax: 00000000   ebx: e34b05c0   ecx: e34b05d0   edx: d1da1c40
esi: d6c03e00   edi: 00268ea7   ebp: 0000318e   esp: f7edff34
ds: 0018   es: 0018   ss: 0018
Process kswapd (pid: 5, stackpage=f7edf000)
Stack: d1da1c58 e34b05c0 e34b05c0 c017a75f e34b05c0 d1da1c58 d1da1c40 c014cf20 
       d1da1c40 e34b05c0 00000006 000001d0 00000020 00000006 c014d2ab 000108c9 
       c0132046 00000006 000001d0 00000006 00000020 000001d0 c02d94b4 c02d94b4 
Call Trace:    [<c017a75f>] [<c014cf20>] [<c014d2ab>] [<c0132046>] [<c013209c>]
  [<c01321a1>] [<c0132206>] [<c013232d>] [<c0105708>]
Code: 8b 47 10 85 c0 74 06 53 ff d0 83 c4 04 68 f0 a3 2d c0 8d 43 

>>EIP; c014f5a2 <iput+32/1f0>   <=====
Trace; c017a75e <journal_blocks_per_page+b92e/25f20>
Trace; c014cf20 <prune_dcache+e0/180>
Trace; c014d2aa <shrink_dcache_parent+3a/60>
Trace; c0132046 <kmem_find_general_cachep+1696/2150>
Trace; c013209c <kmem_find_general_cachep+16ec/2150>
Trace; c01321a0 <kmem_find_general_cachep+17f0/2150>
Trace; c0132206 <kmem_find_general_cachep+1856/2150>
Trace; c013232c <kmem_find_general_cachep+197c/2150>
Trace; c0105708 <kernel_thread+28/1f0>
Code;  c014f5a2 <iput+32/1f0>
00000000 <_EIP>:
Code;  c014f5a2 <iput+32/1f0>   <=====
   0:   8b 47 10                  mov    0x10(%edi),%eax   <=====
Code;  c014f5a4 <iput+34/1f0>
   3:   85 c0                     test   %eax,%eax
Code;  c014f5a6 <iput+36/1f0>
   5:   74 06                     je     d <_EIP+0xd> c014f5ae <iput+3e/1f0>
Code;  c014f5a8 <iput+38/1f0>
   7:   53                        push   %ebx
Code;  c014f5aa <iput+3a/1f0>
   8:   ff d0                     call   *%eax
Code;  c014f5ac <iput+3c/1f0>
   a:   83 c4 04                  add    $0x4,%esp
Code;  c014f5ae <iput+3e/1f0>
   d:   68 f0 a3 2d c0            push   $0xc02da3f0
Code;  c014f5b4 <iput+44/1f0>
  12:   8d 43 00                  lea    0x0(%ebx),%eax

Soeren.

Re: kswapd oops in 2.4.20 SMP+NFS

[Soeren Sonnenburg]

On Fri, 2003-03-21 at 09:28, Oleg Drokin wrote:
> Hello!
> 
> On Thu, Mar 20, 2003 at 03:23:24PM +0100, Soeren Sonnenburg wrote:
> 
> > I got the following oops recently. The machine is still up and running
> > and was working stably for a year now...
> > Linux 2.4.20 #1 SMP Tue Dec 10 11:16:20 CET 2002 i686 unknown
> > 2 x AMD K7-MP 1200MHz PCI(5-64)	TYAN Thunder K7 S2462 Mainboard 1G ECC Memory
> > [...]
> > nfsd-fh: found a name that I didn't expect: sys/oz
> > nfsd-fh: found a name that I didn't expect: sys/oz
> > nfsd-fh: found a name that I didn't expect: bin/x86
> > nfsd-fh: found a name that I didn't expect: bin/x86
> > nfsd-fh: found a name that I didn't expect: etc/bla
> > VFS: Busy inodes after unmount. Self-destruct in 5 seconds.  Have a nice day...
> 
> Hm, what is the underlying host filesystem?

oops sorry, it is running ext2 on the smaller disks... and reiserfs
everywhere else but the above files were on a reiserfs partition which
is rather young (i.e. has not seen anything else than kernel 2.4.20)...

Soeren

Re: kswapd oops in 2.4.20 SMP+NFS

[Oleg Drokin]

Hello!

On Fri, Mar 21, 2003 at 10:50:47AM +0100, Soeren Sonnenburg wrote:
> > > I got the following oops recently. The machine is still up and running
> > > and was working stably for a year now...
> > > Linux 2.4.20 #1 SMP Tue Dec 10 11:16:20 CET 2002 i686 unknown
> > > 2 x AMD K7-MP 1200MHz PCI(5-64)	TYAN Thunder K7 S2462 Mainboard 1G ECC Memory
> > > [...]
> > > nfsd-fh: found a name that I didn't expect: sys/oz
> > > nfsd-fh: found a name that I didn't expect: sys/oz
> > > nfsd-fh: found a name that I didn't expect: bin/x86
> > > nfsd-fh: found a name that I didn't expect: bin/x86
> > > nfsd-fh: found a name that I didn't expect: etc/bla
> > > VFS: Busy inodes after unmount. Self-destruct in 5 seconds.  Have a nice day...
> > Hm, what is the underlying host filesystem?
> oops sorry, it is running ext2 on the smaller disks... and reiserfs
> everywhere else but the above files were on a reiserfs partition which
> is rather young (i.e. has not seen anything else than kernel 2.4.20)...

Do you have any idea of what filesystem was unmounted? (the one with busy inodes)

Bye,
    Oleg

Re: kswapd oops in 2.4.20 SMP+NFS

[Soeren Sonnenburg]

On Fri, 2003-03-21 at 11:58, Oleg Drokin wrote:
> Hello!
> 
> On Fri, Mar 21, 2003 at 11:41:39AM +0100, Soeren Sonnenburg wrote:
> > > > > > VFS: Busy inodes after unmount. Self-destruct in 5 seconds.  Have a nice day...
> > > > > Hm, what is the underlying host filesystem?
> > > > oops sorry, it is running ext2 on the smaller disks... and reiserfs
> > > > everywhere else but the above files were on a reiserfs partition which
> > > > is rather young (i.e. has not seen anything else than kernel 2.4.20)...
> > > Do you have any idea of what filesystem was unmounted? (the one with busy inodes)
> > I *think* it was either reiserfs or some nfs share (on that machine
> > there is only reiserfs... its again a 2.4.20 SMP machine ... the most
> > stable one we have).
> 
> Wait-wait, I am confused. First of all you just said you also have ext2 on smaller disks,
> now you are saying there is only reiserfs.
> Do you routinely unmount (and mount back ?) some filesystems on that NFS server box? What are those
> filesystems? How many filesystems do you have at all on that box?
> Given the fact that you usually cannot unmount nfs-exported fs, I presume nfs server was stopped
> at the time of umount, or was that non-exported filesystem, that was unmounted?
> I am trying to understand from what FS that 'busy inodes' thing came.
> (as it seems we already verified that another message from nfs-fh is from reiserfs (And I am working
> on that)).

Ok, sorry for the confusion. I try to clear that up:

There is machine1 (the one with ksymoops trace). It mounts 6 ext2
partitions, 4 reiserfs partitions, a number of nfs partitions through
automounter (like maybe 2-10) and 1 nfs partitions to backup to (that
one is regularly mounted and umounted after the backup).

the machine it backs up to say machine2 is running reiserfs only but
also mounts stuff using automounter...

both machines{1,2} are smp, kernel 2.4.20, both are nfs-servers.

I just checked the logs... It seems that the oops occurs 24 seconds
after the backup script was started (which is run hourly). 
This script first mounts the nfs share, then looks for modified files
and tar's them over, then umounts the share.

So it is probably the umount of that nfs partion.

HTH,
Soeren.

Re: kswapd oops in 2.4.20 SMP+NFS

[Oleg Drokin]

Hello!

On Fri, Mar 21, 2003 at 12:20:14PM +0100, Soeren Sonnenburg wrote:


> I just checked the logs... It seems that the oops occurs 24 seconds
> after the backup script was started (which is run hourly). 
> This script first mounts the nfs share, then looks for modified files
> and tar's them over, then umounts the share.
> So it is probably the umount of that nfs partion.

Well, then I think NFS people may be interested in that oops.
Resend it to them please and state that this is most likely from NFS.

Thank you.

Bye,
    Oleg