Re: Desired RPC client behaviour on socket errors?

[Jamie Bainbridge <jbainbri@redhat.com>]

----- Original Message -----
> From: "Jamie Bainbridge" <jbainbri@redhat.com>
> To: "Jeff Layton" <jeff.layton@primarydata.com>
> Cc: linux-nfs@vger.kernel.org, harshula@redhat.com
> Sent: Monday, 4 May, 2015 4:23:06 PM
> Subject: Re: Desired RPC client behaviour on socket errors?
> 
> ----- Original Message -----
> > From: "Jeff Layton" <jeff.layton@primarydata.com>
> > To: "Jamie Bainbridge" <jbainbri@redhat.com>
> > Cc: linux-nfs@vger.kernel.org, harshula@redhat.com
> > Sent: Friday, 1 May, 2015 11:39:49 PM
> > Subject: Re: Desired RPC client behaviour on socket errors?
> > 
> > On Fri, 1 May 2015 01:22:35 -0400 (EDT)
> > Jamie Bainbridge <jbainbri@redhat.com> wrote:
> > 
> > > Commit 3ed5e2a introduced a change to the RPC client's handling of socket
> > > return on connect.
> > > 
> > > Prior to this commit, any error return was considered instantly fatal and
> > > rpc_exit(task,-EIO) was called.
> > > 
> > > After this commit, socket returns ECONNREFUSED ECONNRESET ECONNABORTED
> > > ENETUNREACH EHOSTUNREACH are passed back to the caller. This is a good
> > > idea and works well.
> > > 
> > > However, this commit also causes those returns to call
> > > rpc_delay(task,3*HZ)
> > > and the RPC connect to retry until the RPC times out. The timeout can be
> > > modified with soft/timeo/retrans but defaults to 3 minutes.
> > > 
> > > In practice this means if a client tries to mount and there is a
> > > permanent
> > > network error outside the client, a TCP Reset or an ICMP error might get
> > > returned, bu the mount will hang and the client will keep trying to
> > > connect many times until the RPC times out. Previously a mount would fail
> > > almost straight away.
> > > 
> > > It seems 3ed5e2a solves a problem for transient network errors but
> > > creates
> > > a problem for permanent network errors.
> > > 
> > > I agree it's probably desirable for a client application (RPC in this
> > > instance) to keep trying to connect until a timeout, and it's good the
> > > timeout is configurable, but it's bad that the timeout must be tied to
> > > all
> > > RPC operations. Someone wanting a quick mount timeout must also suffer a
> > > quick NFS operation timeout, not to mention the data corruption risk that
> > > goes along with soft.
> > > 
> > > Should the RPC client call rpc_exit() on an xprt connect which returns
> > > ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH because
> > > those returns imply a "more permanent" network issue?
> > > 
> > > Disclosure: We came across this because a customer is (ab)using NFSv4
> > > Migrations in a strange way. One server in fs_locations is firewalled
> > > behind a TCP Reset and one is not. Depending on which security zone a
> > > client is in, it can connect to one server but not the other. This
> > > enables
> > > clients in both security zones to use the same NFS mount configuration.
> > > 
> > > Cheers,
> > > Jamie
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> > I'd say no...
> > 
> > One thing to consider is that it's quite common for servers (Linux or
> > otherwise) to return those sorts of errors as they are coming up after
> > a reboot. Network interfaces are often brought online before the nfs
> > server is ready to accept connections.
> > 
> > If we were to change that then you'd likely see RPCs failing in those
> > situations, which is almost certainly not what you want.
> > 
> > AFAICT, mount requests should end up trying to do a rpc_ping first,
> > which should have RPC_TASK_SOFTCONN set. Is that not working for some
> > reason?
> > 
> > --
> > Jeff Layton <jeff.layton@primarydata.com>
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> Thanks for the confirmation of connect behaviour.
> 
> Debug shows a NULL procedure being generated, so it seems rpc_ping is called,
> but the RPC task ends up with a status of -EAGAIN and not -ECONNRESET, so it
> doesn't break out of the switch:case to call rpc_exit().
> 
> It seems socket status is not being passed back to become task status. I'll
> look into why that is.

Turns out this is working fine in 4.0.1 but not in EL6.

The EL6 kernel is currently missing xs_error_report() which sets sock->sk_err to task->status. Without that function, the default state of -EAGAIN is set in xprt_disconnect_done().

I'll handle this further via Red Hat Bugzilla.

Thanks for the help Jeff!

Cheers,
Jamie