Oops in lockd (2.6.20.1)

Oops in lockd (2.6.20.1)

[Jan Rekorajski]

Hi,
I have NFS server, serving /, /home and /usr to some 100 workstations,
and the following oops bit me today in the morning.

yellow11 is a workstation (probably someone rebooted it with a power
switch).
Server is 4xDualCore Opteron with 32GB RAM and 500GB of exported
filesystems .

This kernel is heavily patched, but nothing AFAIK touches NFS or
anything near it.

lockd: server yellow11 not responding, timed out
lockd: couldn't create RPC handle for yellow11
lockd: server yellow11 not responding, timed out
lockd: couldn't create RPC handle for yellow11
general protection fault: 0000 [1] SMP 
CPU 0 
Modules linked in: vroot nfsd exportfs lockd nfs_acl sunrpc tg3 ext2 mbcache dm_mod 8021q
Pid: 4114:#0, comm: lockd Not tainted 2.6.20.1 #3
RIP: 0010:[<ffffffff8807e058>]  [<ffffffff8807e058>] :lockd:nlm_release_host+0x28/0x110
RSP: 0018:ffff8107c5075e40  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0008000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0008000000000000
RBP: ffff81077ce4d080 R08: 0000000000000000 R09: 0000000000000000
R10: 2222222222222222 R11: 2222222222222222 R12: 00000000ffffffdb
R13: 000000000000000a R14: ffff8107c5075ea0 R15: ffffffff880849c0
FS:  00002b5d76afeb00(0000) GS:ffffffff805e0000(0000) knlGS:00000000f7b5a6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000f7ee0ba9 CR3: 00000005d50db000 CR4: 00000000000006e0
Process lockd (pid: 4114[#0], threadinfo ffff8107c5074000, task ffff8107d8e1c100)
Stack:  ffff8107b75ab000 ffffffff8807cc1d ffff8107b75ab000 ffffffff8807cfca
0000000000000000 ffff8107b75ab038 ffff8107da3132c0 ffff8107b75ab000
ffff8106c24bd4c0 ffffffff8807e7a0 0000000000000000 ffffffff8807d032
Call Trace:
[<ffffffff8807cc1d>] :lockd:nlm_release_call+0xd/0x20
[<ffffffff8807cfca>] :lockd:__nlm_async_call+0x9a/0xc0
[<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
[<ffffffff8807d032>] :lockd:nlm_async_call+0x42/0x50
[<ffffffff8807fbae>] :lockd:nlmsvc_grant_blocked+0x12e/0x170
[<ffffffff8807fdd3>] :lockd:nlmsvc_retry_blocked+0x73/0xa0
[<ffffffff8807e8d7>] :lockd:lockd+0x137/0x280
[<ffffffff80253df8>] child_rip+0xa/0x12
[<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
[<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
[<ffffffff80253dee>] child_rip+0x0/0x12


Code: 8b 73 74 85 f6 79 25 48 c7 c1 60 47 08 88 ba 24 01 00 00 48 
RIP  [<ffffffff8807e058>] :lockd:nlm_release_host+0x28/0x110
RSP <ffff8107c5075e40>

-- 
Jan Rekorajski            |  ALL SUSPECTS ARE GUILTY. PERIOD!
baggins<at>mimuw.edu.pl   |  OTHERWISE THEY WOULDN'T BE SUSPECTS, WOULD THEY?
BOFH, MANIAC              |                   -- TROOPS by Kevin Rubio

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Oops in lockd (2.6.20.1)

[Olaf Kirch]

On Monday 05 March 2007 14:47, Jan Rekorajski wrote:
> [<ffffffff8807cc1d>] :lockd:nlm_release_call+0xd/0x20
> [<ffffffff8807cfca>] :lockd:__nlm_async_call+0x9a/0xc0
> [<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
> [<ffffffff8807d032>] :lockd:nlm_async_call+0x42/0x50
> [<ffffffff8807fbae>] :lockd:nlmsvc_grant_blocked+0x12e/0x170
> [<ffffffff8807fdd3>] :lockd:nlmsvc_retry_blocked+0x73/0xa0
> [<ffffffff8807e8d7>] :lockd:lockd+0x137/0x280
> [<ffffffff80253df8>] child_rip+0xa/0x12
> [<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
> [<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
> [<ffffffff80253dee>] child_rip+0x0/0x12
>
>
> Code: 8b 73 74 85 f6 79 25 48 c7 c1 60 47 08 88 ba 24 01 00 00 48
> RIP  [<ffffffff8807e058>] :lockd:nlm_release_host+0x28/0x110
> RSP <ffff8107c5075e40>

It seems it's dying on a bogus nlm_host pointer. In __nlm_async_call we have:

        status = rpc_call_async(clnt, msg, RPC_TASK_ASYNC, tk_ops, req);
        if (status == 0)
                return 0;
out_err:
        nlm_release_call(req);
        return status;
}

So we ended up in nlm_release_call because rpc_call_async returned an
error. However, when rpc_call_async fails, it calls the rpc_call_done handler.
In this case, it's nlmsvc_release_block, which ends up freeing the nlm_block
object, which does nlm_release_call(block->b_call).

So it appears to me that it's a double free.

IMHO, when rpc_call_async fails, we should just return status right away.

A similar argument applies to nlmsvc_grant_blocked, where we try to
release the nlm_block object if nlm_async_call() returns an error. When
we get there, the block object will have been freed already.

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir@lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Oops in lockd (2.6.20.1)

[Trond Myklebust]

On Mon, 2007-03-05 at 16:36 +0100, Olaf Kirch wrote:
> On Monday 05 March 2007 14:47, Jan Rekorajski wrote:
> > [<ffffffff8807cc1d>] :lockd:nlm_release_call+0xd/0x20
> > [<ffffffff8807cfca>] :lockd:__nlm_async_call+0x9a/0xc0
> > [<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
> > [<ffffffff8807d032>] :lockd:nlm_async_call+0x42/0x50
> > [<ffffffff8807fbae>] :lockd:nlmsvc_grant_blocked+0x12e/0x170
> > [<ffffffff8807fdd3>] :lockd:nlmsvc_retry_blocked+0x73/0xa0
> > [<ffffffff8807e8d7>] :lockd:lockd+0x137/0x280
> > [<ffffffff80253df8>] child_rip+0xa/0x12
> > [<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
> > [<ffffffff8807e7a0>] :lockd:lockd+0x0/0x280
> > [<ffffffff80253dee>] child_rip+0x0/0x12
> >
> >
> > Code: 8b 73 74 85 f6 79 25 48 c7 c1 60 47 08 88 ba 24 01 00 00 48
> > RIP  [<ffffffff8807e058>] :lockd:nlm_release_host+0x28/0x110
> > RSP <ffff8107c5075e40>
> 
> It seems it's dying on a bogus nlm_host pointer. In __nlm_async_call we have:
> 
>         status = rpc_call_async(clnt, msg, RPC_TASK_ASYNC, tk_ops, req);
>         if (status == 0)
>                 return 0;
> out_err:
>         nlm_release_call(req);
>         return status;
> }
> 
> So we ended up in nlm_release_call because rpc_call_async returned an
> error. However, when rpc_call_async fails, it calls the rpc_call_done handler.
> In this case, it's nlmsvc_release_block, which ends up freeing the nlm_block
> object, which does nlm_release_call(block->b_call).
> 
> So it appears to me that it's a double free.
> 
> IMHO, when rpc_call_async fails, we should just return status right away.
> 
> A similar argument applies to nlmsvc_grant_blocked, where we try to
> release the nlm_block object if nlm_async_call() returns an error. When
> we get there, the block object will have been freed already.
> 
> Olaf

Already fixed in 2.6.21-rc1 mainline. See

http://client.linux-nfs.org/Linux-2.6.x/2.6.20/linux-2.6.20-001-fix_nlm_async_call.dif

Cheers,
  Trond


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Oops in lockd (2.6.20.1)

[Olaf Kirch]

On Monday 05 March 2007 17:18, Trond Myklebust wrote:
> Already fixed in 2.6.21-rc1 mainline.

Nice :)

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir@lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Oops in lockd (2.6.20.1)

[Jan Rekorajski]

On Mon, 05 Mar 2007, Trond Myklebust wrote:

> Already fixed in 2.6.21-rc1 mainline. See
> 
> http://client.linux-nfs.org/Linux-2.6.x/2.6.20/linux-2.6.20-001-fix_nlm_async_call.dif

Thank you :)
I'll test it tomorrow and I really hope my problems with
unresponsive/dying lockd will go away at last.

Jan
-- 
Jan Rekorajski            |  ALL SUSPECTS ARE GUILTY. PERIOD!
baggins<at>mimuw.edu.pl   |  OTHERWISE THEY WOULDN'T BE SUSPECTS, WOULD THEY?
BOFH, MANIAC              |                   -- TROOPS by Kevin Rubio

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs