task->tk_timeout = 0

task->tk_timeout = 0

[Olaf Kirch]

Hi all,

I've been debugging mysterious hangs in the NFS clients for a while
now. The symptoms were always the same: NFS server goes away, comes back
after a long time, but the mount point remains stuck.

When inspecting the list of RPC tasks, I often found that one process
was on the pending queue (i.e. waiting for a reply) but task->tk_timeout
was 0.

I added a couple of diagnostic printks, and one of them triggered
yesterday, pointing to this piece of code in xprt_transmit:

if (!xprt->nocong) {
	int timer = task->tk_msg.rpc_proc->p_timer;
	timeout = rpc_calc_rto(clnt->cl_rtt, timer);
	timeout <<= rpc_ntimeo(clnt->cl_rtt, timer);
	timeout <<= clnt->cl_timeout.to_retries
		- req->rq_timeout.to_retries;
	if (timeout > req->rq_timeout.to_maxval)
		timeout = req->rq_timeout.to_maxval;
	else if (timeout == 0) {
		printk(KERN_ERR "RPC task timeout == 0, please tell okir\n");
		timeout = req->rq_timeout.to_maxval;
	}
	...
}

So apparently one of the shift operations above overflows.  I suspect
rpc_ntimeo because it can be arbitrarily large. But the ntimeouts value
isn't updated until we've received a reply (is this intentional, Trond?)

So what would have to happen for this bug to trigger is

	send request
	server hangs
	retransmit request 1000 times
	server comes back
	retransmit request
	receive reply, set ntimeout = 1000

	send another request. timeout overflows and becomes 0
	request gets lost
	task waits indefinitely.

I think the most robust way to fix this is simply this:

		- req->rq_timeout.to_retries;
	if (timeout > req->rq_timeout.to_maxval)
		timeout = req->rq_timeout.to_maxval;
-	if (timeout > req->rq_timeout.to_maxval)
+	if (!timeout || timeout > req->rq_timeout.to_maxval)
		timeout = req->rq_timeout.to_maxval;

It may also make sense to clamp the ntimeouts value to something
reasonable (e.g. 8)

Olaf
-- 
Olaf Kirch     |  Stop wasting entropy - start using predictable
okir@suse.de   |  tempfile names today!
---------------+ 


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: task->tk_timeout = 0

[trond.myklebust]

På to , 26/02/2004 klokka 03:08, skreiv Olaf Kirch:
> So apparently one of the shift operations above overflows.  I suspect
rpc_ntimeo because it can be arbitrarily large. But the ntimeouts value
isn't updated until we've received a reply (is this intentional, Trond?)
>
> So what would have to happen for this bug to trigger is
>
> 	send request
> 	server hangs
> 	retransmit request 1000 times
> 	server comes back
> 	retransmit request
> 	receive reply, set ntimeout = 1000
>
> 	send another request. timeout overflows and becomes 0
> 	request gets lost
> 	task waits indefinitely.
>
> I think the most robust way to fix this is simply this:
>
> 		- req->rq_timeout.to_retries;
> 	if (timeout > req->rq_timeout.to_maxval)
> 		timeout = req->rq_timeout.to_maxval;
> -	if (timeout > req->rq_timeout.to_maxval)
> +	if (!timeout || timeout > req->rq_timeout.to_maxval)
> 		timeout = req->rq_timeout.to_maxval;
>
> It may also make sense to clamp the ntimeouts value to something
reasonable (e.g. 8)

Agreed, and in fact, I believe at some point it was clamped to 8.
Thanks, and well spotted...

Cheers,
  Trond





-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs