[PATCH] SUNRPC: Fix backchannel reply, again

[PATCH] SUNRPC: Fix backchannel reply, again

[cel]

From: Chuck Lever <chuck.lever@oracle.com>

I still see "RPC: Could not send backchannel reply error: -110"
quite often, along with slow-running tests. Debugging shows that the
backchannel is still stumbling when it has to queue a callback reply
on a busy transport.

Note that every one of these timeouts causes a connection loss by
virtue of the xprt_conditional_disconnect() call in that arm of
call_cb_transmit_status().

I found that setting to_maxval is necessary to get the RPC timeout
logic to behave whenever to_exponential is not set.

Fixes: 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts for backchannel")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 net/sunrpc/svc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c
index 965a27806bfd..f4ddb2961042 100644
--- a/net/sunrpc/svc.c
+++ b/net/sunrpc/svc.c
@@ -1643,6 +1643,7 @@ void svc_process_bc(struct rpc_rqst *req, struct svc_rqst *rqstp)
 		timeout.to_initval = req->rq_xprt->timeout->to_initval;
 		timeout.to_retries = req->rq_xprt->timeout->to_retries;
 	}
+	timeout.to_maxval = timeout.to_initval;
 	memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req->rq_snd_buf));
 	task = rpc_run_bc_task(req, &timeout);
 
-- 
2.45.1

Re: [PATCH] SUNRPC: Fix backchannel reply, again

[Chuck Lever]

On Fri, Jun 14, 2024 at 10:18:52AM -0400, cel@kernel.org wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
> 
> I still see "RPC: Could not send backchannel reply error: -110"
> quite often, along with slow-running tests. Debugging shows that the
> backchannel is still stumbling when it has to queue a callback reply
> on a busy transport.
> 
> Note that every one of these timeouts causes a connection loss by
> virtue of the xprt_conditional_disconnect() call in that arm of
> call_cb_transmit_status().
> 
> I found that setting to_maxval is necessary to get the RPC timeout
> logic to behave whenever to_exponential is not set.
> 
> Fixes: 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts for backchannel")
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>  net/sunrpc/svc.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c
> index 965a27806bfd..f4ddb2961042 100644
> --- a/net/sunrpc/svc.c
> +++ b/net/sunrpc/svc.c
> @@ -1643,6 +1643,7 @@ void svc_process_bc(struct rpc_rqst *req, struct svc_rqst *rqstp)
>  		timeout.to_initval = req->rq_xprt->timeout->to_initval;
>  		timeout.to_retries = req->rq_xprt->timeout->to_retries;
>  	}
> +	timeout.to_maxval = timeout.to_initval;
>  	memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req->rq_snd_buf));
>  	task = rpc_run_bc_task(req, &timeout);
>  
> -- 
> 2.45.1
> 

Hi - would love to see this in 6.10-rc. Is there a chance that
could happen?

-- 
Chuck Lever

Re: [PATCH] SUNRPC: Fix backchannel reply, again

[Jeff Layton]

On Fri, 2024-06-14 at 10:18 -0400, cel@kernel.org wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
> 
> I still see "RPC: Could not send backchannel reply error: -110"
> quite often, along with slow-running tests. Debugging shows that the
> backchannel is still stumbling when it has to queue a callback reply
> on a busy transport.
> 
> Note that every one of these timeouts causes a connection loss by
> virtue of the xprt_conditional_disconnect() call in that arm of
> call_cb_transmit_status().
> 
> I found that setting to_maxval is necessary to get the RPC timeout
> logic to behave whenever to_exponential is not set.
> 
> Fixes: 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts for
> backchannel")
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>  net/sunrpc/svc.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c
> index 965a27806bfd..f4ddb2961042 100644
> --- a/net/sunrpc/svc.c
> +++ b/net/sunrpc/svc.c
> @@ -1643,6 +1643,7 @@ void svc_process_bc(struct rpc_rqst *req,
> struct svc_rqst *rqstp)
>  		timeout.to_initval = req->rq_xprt->timeout-
> >to_initval;
>  		timeout.to_retries = req->rq_xprt->timeout-
> >to_retries;
>  	}
> +	timeout.to_maxval = timeout.to_initval;
>  	memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req-
> >rq_snd_buf));
>  	task = rpc_run_bc_task(req, &timeout);
>  

Reviewed-by: Jeff Layton <jlayton@kernel.org>

Re: [PATCH] SUNRPC: Fix backchannel reply, again

[Trond Myklebust]

On Tue, 2024-06-18 at 16:31 -0400, Chuck Lever wrote:
> On Fri, Jun 14, 2024 at 10:18:52AM -0400, cel@kernel.org wrote:
> > From: Chuck Lever <chuck.lever@oracle.com>
> > 
> > I still see "RPC: Could not send backchannel reply error: -110"
> > quite often, along with slow-running tests. Debugging shows that
> > the
> > backchannel is still stumbling when it has to queue a callback
> > reply
> > on a busy transport.
> > 
> > Note that every one of these timeouts causes a connection loss by
> > virtue of the xprt_conditional_disconnect() call in that arm of
> > call_cb_transmit_status().
> > 
> > I found that setting to_maxval is necessary to get the RPC timeout
> > logic to behave whenever to_exponential is not set.
> > 
> > Fixes: 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts
> > for backchannel")
> > Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> > ---
> >  net/sunrpc/svc.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c
> > index 965a27806bfd..f4ddb2961042 100644
> > --- a/net/sunrpc/svc.c
> > +++ b/net/sunrpc/svc.c
> > @@ -1643,6 +1643,7 @@ void svc_process_bc(struct rpc_rqst *req,
> > struct svc_rqst *rqstp)
> >  		timeout.to_initval = req->rq_xprt->timeout-
> > >to_initval;
> >  		timeout.to_retries = req->rq_xprt->timeout-
> > >to_retries;
> >  	}
> > +	timeout.to_maxval = timeout.to_initval;
> >  	memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req-
> > >rq_snd_buf));
> >  	task = rpc_run_bc_task(req, &timeout);
> >  
> > -- 
> > 2.45.1
> > 
> 
> Hi - would love to see this in 6.10-rc. Is there a chance that
> could happen?

Hmm... Can we please also set the remaining fields in timeout to 0?
Otherwise, we're still playing roulette with what actually ends up
happening in xprt_calc_majortimeo(). If to_increment happens to be
large enough, we could overflow and end up with a silly small timeout
value on a retry.


-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com