Desired RPC client behaviour on socket errors?

Desired RPC client behaviour on socket errors?

[Jamie Bainbridge]

Commit 3ed5e2a introduced a change to the RPC client's handling of socket return on connect.

Prior to this commit, any error return was considered instantly fatal and rpc_exit(task,-EIO) was called.

After this commit, socket returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH are passed back to the caller. This is a good idea and works well.

However, this commit also causes those returns to call rpc_delay(task,3*HZ) and the RPC connect to retry until the RPC times out. The timeout can be modified with soft/timeo/retrans but defaults to 3 minutes.

In practice this means if a client tries to mount and there is a permanent network error outside the client, a TCP Reset or an ICMP error might get returned, bu the mount will hang and the client will keep trying to connect many times until the RPC times out. Previously a mount would fail almost straight away.

It seems 3ed5e2a solves a problem for transient network errors but creates a problem for permanent network errors.

I agree it's probably desirable for a client application (RPC in this instance) to keep trying to connect until a timeout, and it's good the timeout is configurable, but it's bad that the timeout must be tied to all RPC operations. Someone wanting a quick mount timeout must also suffer a quick NFS operation timeout, not to mention the data corruption risk that goes along with soft.

Should the RPC client call rpc_exit() on an xprt connect which returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH because those returns imply a "more permanent" network issue?

Disclosure: We came across this because a customer is (ab)using NFSv4 Migrations in a strange way. One server in fs_locations is firewalled behind a TCP Reset and one is not. Depending on which security zone a client is in, it can connect to one server but not the other. This enables clients in both security zones to use the same NFS mount configuration.

Cheers,
Jamie

Re: Desired RPC client behaviour on socket errors?

[Jeff Layton]

On Fri, 1 May 2015 01:22:35 -0400 (EDT)
Jamie Bainbridge <jbainbri@redhat.com> wrote:

> Commit 3ed5e2a introduced a change to the RPC client's handling of socket return on connect.
> 
> Prior to this commit, any error return was considered instantly fatal and rpc_exit(task,-EIO) was called.
> 
> After this commit, socket returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH are passed back to the caller. This is a good idea and works well.
> 
> However, this commit also causes those returns to call rpc_delay(task,3*HZ) and the RPC connect to retry until the RPC times out. The timeout can be modified with soft/timeo/retrans but defaults to 3 minutes.
> 
> In practice this means if a client tries to mount and there is a permanent network error outside the client, a TCP Reset or an ICMP error might get returned, bu the mount will hang and the client will keep trying to connect many times until the RPC times out. Previously a mount would fail almost straight away.
> 
> It seems 3ed5e2a solves a problem for transient network errors but creates a problem for permanent network errors.
> 
> I agree it's probably desirable for a client application (RPC in this instance) to keep trying to connect until a timeout, and it's good the timeout is configurable, but it's bad that the timeout must be tied to all RPC operations. Someone wanting a quick mount timeout must also suffer a quick NFS operation timeout, not to mention the data corruption risk that goes along with soft.
> 
> Should the RPC client call rpc_exit() on an xprt connect which returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH because those returns imply a "more permanent" network issue?
> 
> Disclosure: We came across this because a customer is (ab)using NFSv4 Migrations in a strange way. One server in fs_locations is firewalled behind a TCP Reset and one is not. Depending on which security zone a client is in, it can connect to one server but not the other. This enables clients in both security zones to use the same NFS mount configuration.
> 
> Cheers,
> Jamie
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


I'd say no...

One thing to consider is that it's quite common for servers (Linux or
otherwise) to return those sorts of errors as they are coming up after
a reboot. Network interfaces are often brought online before the nfs
server is ready to accept connections.

If we were to change that then you'd likely see RPCs failing in those
situations, which is almost certainly not what you want.

AFAICT, mount requests should end up trying to do a rpc_ping first,
which should have RPC_TASK_SOFTCONN set. Is that not working for some
reason?

-- 
Jeff Layton <jeff.layton@primarydata.com>

Re: Desired RPC client behaviour on socket errors?

[Jamie Bainbridge]

----- Original Message -----
> From: "Jeff Layton" <jeff.layton@primarydata.com>
> To: "Jamie Bainbridge" <jbainbri@redhat.com>
> Cc: linux-nfs@vger.kernel.org, harshula@redhat.com
> Sent: Friday, 1 May, 2015 11:39:49 PM
> Subject: Re: Desired RPC client behaviour on socket errors?
> 
> On Fri, 1 May 2015 01:22:35 -0400 (EDT)
> Jamie Bainbridge <jbainbri@redhat.com> wrote:
> 
> > Commit 3ed5e2a introduced a change to the RPC client's handling of socket
> > return on connect.
> > 
> > Prior to this commit, any error return was considered instantly fatal and
> > rpc_exit(task,-EIO) was called.
> > 
> > After this commit, socket returns ECONNREFUSED ECONNRESET ECONNABORTED
> > ENETUNREACH EHOSTUNREACH are passed back to the caller. This is a good
> > idea and works well.
> > 
> > However, this commit also causes those returns to call rpc_delay(task,3*HZ)
> > and the RPC connect to retry until the RPC times out. The timeout can be
> > modified with soft/timeo/retrans but defaults to 3 minutes.
> > 
> > In practice this means if a client tries to mount and there is a permanent
> > network error outside the client, a TCP Reset or an ICMP error might get
> > returned, bu the mount will hang and the client will keep trying to
> > connect many times until the RPC times out. Previously a mount would fail
> > almost straight away.
> > 
> > It seems 3ed5e2a solves a problem for transient network errors but creates
> > a problem for permanent network errors.
> > 
> > I agree it's probably desirable for a client application (RPC in this
> > instance) to keep trying to connect until a timeout, and it's good the
> > timeout is configurable, but it's bad that the timeout must be tied to all
> > RPC operations. Someone wanting a quick mount timeout must also suffer a
> > quick NFS operation timeout, not to mention the data corruption risk that
> > goes along with soft.
> > 
> > Should the RPC client call rpc_exit() on an xprt connect which returns
> > ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH because
> > those returns imply a "more permanent" network issue?
> > 
> > Disclosure: We came across this because a customer is (ab)using NFSv4
> > Migrations in a strange way. One server in fs_locations is firewalled
> > behind a TCP Reset and one is not. Depending on which security zone a
> > client is in, it can connect to one server but not the other. This enables
> > clients in both security zones to use the same NFS mount configuration.
> > 
> > Cheers,
> > Jamie
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> I'd say no...
> 
> One thing to consider is that it's quite common for servers (Linux or
> otherwise) to return those sorts of errors as they are coming up after
> a reboot. Network interfaces are often brought online before the nfs
> server is ready to accept connections.
> 
> If we were to change that then you'd likely see RPCs failing in those
> situations, which is almost certainly not what you want.
> 
> AFAICT, mount requests should end up trying to do a rpc_ping first,
> which should have RPC_TASK_SOFTCONN set. Is that not working for some
> reason?
> 
> --
> Jeff Layton <jeff.layton@primarydata.com>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Thanks for the confirmation of connect behaviour.

Debug shows a NULL procedure being generated, so it seems rpc_ping is called, but the RPC task ends up with a status of -EAGAIN and not -ECONNRESET, so it doesn't break out of the switch:case to call rpc_exit().

It seems socket status is not being passed back to become task status. I'll look into why that is.

Cheers,
Jamie

Re: Desired RPC client behaviour on socket errors?

[Jamie Bainbridge]

----- Original Message -----
> From: "Jamie Bainbridge" <jbainbri@redhat.com>
> To: "Jeff Layton" <jeff.layton@primarydata.com>
> Cc: linux-nfs@vger.kernel.org, harshula@redhat.com
> Sent: Monday, 4 May, 2015 4:23:06 PM
> Subject: Re: Desired RPC client behaviour on socket errors?
> 
> ----- Original Message -----
> > From: "Jeff Layton" <jeff.layton@primarydata.com>
> > To: "Jamie Bainbridge" <jbainbri@redhat.com>
> > Cc: linux-nfs@vger.kernel.org, harshula@redhat.com
> > Sent: Friday, 1 May, 2015 11:39:49 PM
> > Subject: Re: Desired RPC client behaviour on socket errors?
> > 
> > On Fri, 1 May 2015 01:22:35 -0400 (EDT)
> > Jamie Bainbridge <jbainbri@redhat.com> wrote:
> > 
> > > Commit 3ed5e2a introduced a change to the RPC client's handling of socket
> > > return on connect.
> > > 
> > > Prior to this commit, any error return was considered instantly fatal and
> > > rpc_exit(task,-EIO) was called.
> > > 
> > > After this commit, socket returns ECONNREFUSED ECONNRESET ECONNABORTED
> > > ENETUNREACH EHOSTUNREACH are passed back to the caller. This is a good
> > > idea and works well.
> > > 
> > > However, this commit also causes those returns to call
> > > rpc_delay(task,3*HZ)
> > > and the RPC connect to retry until the RPC times out. The timeout can be
> > > modified with soft/timeo/retrans but defaults to 3 minutes.
> > > 
> > > In practice this means if a client tries to mount and there is a
> > > permanent
> > > network error outside the client, a TCP Reset or an ICMP error might get
> > > returned, bu the mount will hang and the client will keep trying to
> > > connect many times until the RPC times out. Previously a mount would fail
> > > almost straight away.
> > > 
> > > It seems 3ed5e2a solves a problem for transient network errors but
> > > creates
> > > a problem for permanent network errors.
> > > 
> > > I agree it's probably desirable for a client application (RPC in this
> > > instance) to keep trying to connect until a timeout, and it's good the
> > > timeout is configurable, but it's bad that the timeout must be tied to
> > > all
> > > RPC operations. Someone wanting a quick mount timeout must also suffer a
> > > quick NFS operation timeout, not to mention the data corruption risk that
> > > goes along with soft.
> > > 
> > > Should the RPC client call rpc_exit() on an xprt connect which returns
> > > ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH because
> > > those returns imply a "more permanent" network issue?
> > > 
> > > Disclosure: We came across this because a customer is (ab)using NFSv4
> > > Migrations in a strange way. One server in fs_locations is firewalled
> > > behind a TCP Reset and one is not. Depending on which security zone a
> > > client is in, it can connect to one server but not the other. This
> > > enables
> > > clients in both security zones to use the same NFS mount configuration.
> > > 
> > > Cheers,
> > > Jamie
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> > I'd say no...
> > 
> > One thing to consider is that it's quite common for servers (Linux or
> > otherwise) to return those sorts of errors as they are coming up after
> > a reboot. Network interfaces are often brought online before the nfs
> > server is ready to accept connections.
> > 
> > If we were to change that then you'd likely see RPCs failing in those
> > situations, which is almost certainly not what you want.
> > 
> > AFAICT, mount requests should end up trying to do a rpc_ping first,
> > which should have RPC_TASK_SOFTCONN set. Is that not working for some
> > reason?
> > 
> > --
> > Jeff Layton <jeff.layton@primarydata.com>
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> Thanks for the confirmation of connect behaviour.
> 
> Debug shows a NULL procedure being generated, so it seems rpc_ping is called,
> but the RPC task ends up with a status of -EAGAIN and not -ECONNRESET, so it
> doesn't break out of the switch:case to call rpc_exit().
> 
> It seems socket status is not being passed back to become task status. I'll
> look into why that is.

Turns out this is working fine in 4.0.1 but not in EL6.

The EL6 kernel is currently missing xs_error_report() which sets sock->sk_err to task->status. Without that function, the default state of -EAGAIN is set in xprt_disconnect_done().

I'll handle this further via Red Hat Bugzilla.

Thanks for the help Jeff!

Cheers,
Jamie