[PATCH] mount.nfs: try the next address after mount fails with ETIMEDOUT

[PATCH] mount.nfs: try the next address after mount fails with ETIMEDOUT

[Jeff Layton]

If a NFS mount attempt fails with an ETIMEDOUT error, the mount.nfs code
doesn't currently attempt the next address in the list. For a NFSv4 mount
the initial mount() call almost always ends up going over
NFS_DEF_FG_TIMEOUT_MINUTES and the mount is never retried.

For a v3 mount, it ends up continually retrying against the same IPv6
address, and never tries the IPv4 address. Eventually it gives up once
it hits the NFS_DEF_FG_TIMEOUT_MINUTES timeout.

It's possible that a server is just unreachable via IPv6 (due to a
routing misconfiguration for instance), or is dropping IPv6 frames on
the floor. In that situation, it might still be reachable via IPv4 and
trying the next address could have allowed the mount to succeed.

Fix this by treating ETIMEDOUT in a similar fashion to ECONNREFUSED.
Have the client try the next address in the list before giving up and
returning an error.

Our QA folks noticed this after a routing problem in one of our test
labs. I was able to reproduce it by having the server drop incoming
IPv6 frames from the client's address.

With this patch, the mount eventually succeeds over IPv4 instead of
returning an error.

Cc: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 utils/mount/stropts.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/utils/mount/stropts.c b/utils/mount/stropts.c
index e09aa7c..0aa9a75 100644
--- a/utils/mount/stropts.c
+++ b/utils/mount/stropts.c
@@ -665,6 +665,7 @@ static int nfs_try_mount_v3v2(struct nfsmount_info *mi)
 		case ECONNREFUSED:
 		case EOPNOTSUPP:
 		case EHOSTUNREACH:
+		case ETIMEDOUT:
 			continue;
 		default:
 			goto out;
@@ -752,6 +753,7 @@ static int nfs_try_mount_v4(struct nfsmount_info *mi)
 		switch (errno) {
 		case ECONNREFUSED:
 		case EHOSTUNREACH:
+		case ETIMEDOUT:
 			continue;
 		default:
 			goto out;
-- 
1.7.1

Re: [PATCH] mount.nfs: try the next address after mount fails with ETIMEDOUT

[Chuck Lever]

On May 31, 2012, at 9:03 AM, Jeff Layton wrote:

> If a NFS mount attempt fails with an ETIMEDOUT error, the mount.nfs code
> doesn't currently attempt the next address in the list. For a NFSv4 mount
> the initial mount() call almost always ends up going over
> NFS_DEF_FG_TIMEOUT_MINUTES and the mount is never retried.
> 
> For a v3 mount, it ends up continually retrying against the same IPv6
> address, and never tries the IPv4 address. Eventually it gives up once
> it hits the NFS_DEF_FG_TIMEOUT_MINUTES timeout.
> 
> It's possible that a server is just unreachable via IPv6 (due to a
> routing misconfiguration for instance), or is dropping IPv6 frames on
> the floor. In that situation, it might still be reachable via IPv4 and
> trying the next address could have allowed the mount to succeed.
> 
> Fix this by treating ETIMEDOUT in a similar fashion to ECONNREFUSED.
> Have the client try the next address in the list before giving up and
> returning an error.

Bzzzzt.  AF fallback is only legal to do if the admin did not specify a netid.

Can you demonstrate that your patch works correctly if an IPv6 netid is specified?  "Correctly" here means that no fallback occurs.

> Our QA folks noticed this after a routing problem in one of our test
> labs. I was able to reproduce it by having the server drop incoming
> IPv6 frames from the client's address.
> 
> With this patch, the mount eventually succeeds over IPv4 instead of
> returning an error.
> 
> Cc: Chuck Lever <chuck.lever@oracle.com>
> Signed-off-by: Jeff Layton <jlayton@redhat.com>
> ---
> utils/mount/stropts.c |    2 ++
> 1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/utils/mount/stropts.c b/utils/mount/stropts.c
> index e09aa7c..0aa9a75 100644
> --- a/utils/mount/stropts.c
> +++ b/utils/mount/stropts.c
> @@ -665,6 +665,7 @@ static int nfs_try_mount_v3v2(struct nfsmount_info *mi)
> 		case ECONNREFUSED:
> 		case EOPNOTSUPP:
> 		case EHOSTUNREACH:
> +		case ETIMEDOUT:
> 			continue;
> 		default:
> 			goto out;
> @@ -752,6 +753,7 @@ static int nfs_try_mount_v4(struct nfsmount_info *mi)
> 		switch (errno) {
> 		case ECONNREFUSED:
> 		case EHOSTUNREACH:
> +		case ETIMEDOUT:
> 			continue;
> 		default:
> 			goto out;
> -- 
> 1.7.1
> 

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH] mount.nfs: try the next address after mount fails with ETIMEDOUT

[Jeff Layton]

On Thu, 31 May 2012 10:28:49 -0400
Chuck Lever <chuck.lever@oracle.com> wrote:

> 
> On May 31, 2012, at 9:03 AM, Jeff Layton wrote:
> 
> > If a NFS mount attempt fails with an ETIMEDOUT error, the mount.nfs code
> > doesn't currently attempt the next address in the list. For a NFSv4 mount
> > the initial mount() call almost always ends up going over
> > NFS_DEF_FG_TIMEOUT_MINUTES and the mount is never retried.
> > 
> > For a v3 mount, it ends up continually retrying against the same IPv6
> > address, and never tries the IPv4 address. Eventually it gives up once
> > it hits the NFS_DEF_FG_TIMEOUT_MINUTES timeout.
> > 
> > It's possible that a server is just unreachable via IPv6 (due to a
> > routing misconfiguration for instance), or is dropping IPv6 frames on
> > the floor. In that situation, it might still be reachable via IPv4 and
> > trying the next address could have allowed the mount to succeed.
> > 
> > Fix this by treating ETIMEDOUT in a similar fashion to ECONNREFUSED.
> > Have the client try the next address in the list before giving up and
> > returning an error.
> 
> Bzzzzt.  AF fallback is only legal to do if the admin did not specify a netid.
> 
> Can you demonstrate that your patch works correctly if an IPv6 netid is specified?  "Correctly" here means that no fallback occurs.
> 

If someone specifies an IPv6 netid (via proto= option) then we would
set AF_INET6 in the addrinfo "hints" and there will be no IPv4
addresses in the mi->address list. So, this does still work correctly
when someone specifies an ipv6 netid.

Chuck also pointed out on IRC that the description was a little
unclear, since it was specific to my test rig where the server had a
single IPv6 and IPv4 address in DNS.

The loop that retries against different addresses doesn't care
about whether the addrs are v4 or v6. What this really tells mount.nfs
to do is to keep trying different addresses in the list if there are
any instead of giving up on an ETIMEDOUT error.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [PATCH] mount.nfs: try the next address after mount fails with ETIMEDOUT

[Chuck Lever]

On May 31, 2012, at 11:40 AM, Jeff Layton wrote:

> On Thu, 31 May 2012 10:28:49 -0400
> Chuck Lever <chuck.lever@oracle.com> wrote:
> 
>> 
>> On May 31, 2012, at 9:03 AM, Jeff Layton wrote:
>> 
>>> If a NFS mount attempt fails with an ETIMEDOUT error, the mount.nfs code
>>> doesn't currently attempt the next address in the list. For a NFSv4 mount
>>> the initial mount() call almost always ends up going over
>>> NFS_DEF_FG_TIMEOUT_MINUTES and the mount is never retried.
>>> 
>>> For a v3 mount, it ends up continually retrying against the same IPv6
>>> address, and never tries the IPv4 address. Eventually it gives up once
>>> it hits the NFS_DEF_FG_TIMEOUT_MINUTES timeout.
>>> 
>>> It's possible that a server is just unreachable via IPv6 (due to a
>>> routing misconfiguration for instance), or is dropping IPv6 frames on
>>> the floor. In that situation, it might still be reachable via IPv4 and
>>> trying the next address could have allowed the mount to succeed.
>>> 
>>> Fix this by treating ETIMEDOUT in a similar fashion to ECONNREFUSED.
>>> Have the client try the next address in the list before giving up and
>>> returning an error.
>> 
>> Bzzzzt.  AF fallback is only legal to do if the admin did not specify a netid.
>> 
>> Can you demonstrate that your patch works correctly if an IPv6 netid is specified?  "Correctly" here means that no fallback occurs.
>> 
> 
> If someone specifies an IPv6 netid (via proto= option) then we would
> set AF_INET6 in the addrinfo "hints" and there will be no IPv4
> addresses in the mi->address list. So, this does still work correctly
> when someone specifies an ipv6 netid.
> 
> Chuck also pointed out on IRC that the description was a little
> unclear, since it was specific to my test rig where the server had a
> single IPv6 and IPv4 address in DNS.
> 
> The loop that retries against different addresses doesn't care
> about whether the addrs are v4 or v6.

Correct.

Moreover, we do get the right behavior because the initial DNS lookup populates the list of addresses to try, based on a specified netid (proto=).  Thus, if no netid is specified, the list contains INET and INET6; if a netid is specified, the list contains only one address family.  That properly limits the addresses that can be used when mount.nfs retries.

> What this really tells mount.nfs
> to do is to keep trying different addresses in the list if there are
> any instead of giving up on an ETIMEDOUT error.
> 
> -- 
> Jeff Layton <jlayton@redhat.com>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH] mount.nfs: try the next address after mount fails with ETIMEDOUT

[Steve Dickson]

On 05/31/2012 09:03 AM, Jeff Layton wrote:
> If a NFS mount attempt fails with an ETIMEDOUT error, the mount.nfs code
> doesn't currently attempt the next address in the list. For a NFSv4 mount
> the initial mount() call almost always ends up going over
> NFS_DEF_FG_TIMEOUT_MINUTES and the mount is never retried.
> 
> For a v3 mount, it ends up continually retrying against the same IPv6
> address, and never tries the IPv4 address. Eventually it gives up once
> it hits the NFS_DEF_FG_TIMEOUT_MINUTES timeout.
> 
> It's possible that a server is just unreachable via IPv6 (due to a
> routing misconfiguration for instance), or is dropping IPv6 frames on
> the floor. In that situation, it might still be reachable via IPv4 and
> trying the next address could have allowed the mount to succeed.
> 
> Fix this by treating ETIMEDOUT in a similar fashion to ECONNREFUSED.
> Have the client try the next address in the list before giving up and
> returning an error.
> 
> Our QA folks noticed this after a routing problem in one of our test
> labs. I was able to reproduce it by having the server drop incoming
> IPv6 frames from the client's address.
> 
> With this patch, the mount eventually succeeds over IPv4 instead of
> returning an error.
> 
> Cc: Chuck Lever <chuck.lever@oracle.com>
> Signed-off-by: Jeff Layton <jlayton@redhat.com>
Committed...

steved.

> ---
>  utils/mount/stropts.c |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/utils/mount/stropts.c b/utils/mount/stropts.c
> index e09aa7c..0aa9a75 100644
> --- a/utils/mount/stropts.c
> +++ b/utils/mount/stropts.c
> @@ -665,6 +665,7 @@ static int nfs_try_mount_v3v2(struct nfsmount_info *mi)
>  		case ECONNREFUSED:
>  		case EOPNOTSUPP:
>  		case EHOSTUNREACH:
> +		case ETIMEDOUT:
>  			continue;
>  		default:
>  			goto out;
> @@ -752,6 +753,7 @@ static int nfs_try_mount_v4(struct nfsmount_info *mi)
>  		switch (errno) {
>  		case ECONNREFUSED:
>  		case EHOSTUNREACH:
> +		case ETIMEDOUT:
>  			continue;
>  		default:
>  			goto out;