mounting via udp from legacy multi-homed servers...

mounting via udp from legacy multi-homed servers...

[NeilBrown]

This problem doesn't seem to go away, does it :-(


While modern Linux always replies to UDP requests from the same address that
the request was sent to, some legacy systems do not, and some customers use
these legacy systems and cannot upgrade.

And as they are legacy systems, it is often the case that UDP is more
performant than TCP.

nfs-utils always uses a connected UDP socket to talk to servers (which I think
is correct so we get errors when reported by ICMP-unreachable messages) and so
fails to talk to these legacy systems (when UDP is requested).

I had such a case recently and managed to make it work by using:

   proto=udp,mountproto=tcp,port=2049

By setting mountproto=tcp, nfs-utils uses tcp for almost all communication
with the server.
The one little problem is that mount does a portmap lookup to check the
port number to talk to the NFS server and seeing we requested UDP for NFS,
it talks to portmap via UDP and never sees the reply.
Setting "port=2049" avoids this lookup so the mount succeeds.

We could just decide that that is the way it is, document that this is how to
talk to systems over UDP when they have a broken portmaper and move on.

But it would be nice if we could find a way to make a solution less ugly and
easier to find.

So some thoughts:

1/ We could use tcp for all rpcbind lookups if tcp was requested for either
   MOUNT or NFS.

2/ If '-v' was given and a UDP lookup to portmap timedout, we could retry
   without connecting (and with a shorter timeout), and print an explanatory 
   message if that succeeded.

3/ We could have a mount option "bindproto=tcp" or maybe "bind=noconnect"
   which could be recommended by the above explanatory message.


Does anyone else have any good ideas, or any deep aversions to any of the
above.
I'm tempted to go for '1' and possibly add '2', but not bother with '3'.

??

Thanks,
NeilBrown

Re: mounting via udp from legacy multi-homed servers...

[Jim Rees]

NeilBrown wrote:

  1/ We could use tcp for all rpcbind lookups if tcp was requested for either
     MOUNT or NFS.

If I requested tcp for mount and then also got tcp for rpcbind, I would find
that natural and useful.

  2/ If '-v' was given and a UDP lookup to portmap timedout, we could retry
     without connecting (and with a shorter timeout), and print an explanatory 
     message if that succeeded.

If '-v' actually changed the behavior of the mount command rather than just
increasing the verbosity I would find that surprising.

Re: mounting via udp from legacy multi-homed servers...

[Steve Dickson]

On 07/13/2011 03:34 AM, NeilBrown wrote:
> 
> 
> This problem doesn't seem to go away, does it :-(
> 
> 
> While modern Linux always replies to UDP requests from the same address that
> the request was sent to, some legacy systems do not, and some customers use
> these legacy systems and cannot upgrade.
> 
> And as they are legacy systems, it is often the case that UDP is more
> performant than TCP.
> 
> nfs-utils always uses a connected UDP socket to talk to servers (which I think
> is correct so we get errors when reported by ICMP-unreachable messages) and so
> fails to talk to these legacy systems (when UDP is requested).
> 
> I had such a case recently and managed to make it work by using:
> 
>    proto=udp,mountproto=tcp,port=2049
> 
> By setting mountproto=tcp, nfs-utils uses tcp for almost all communication
> with the server.
> The one little problem is that mount does a portmap lookup to check the
> port number to talk to the NFS server and seeing we requested UDP for NFS,
> it talks to portmap via UDP and never sees the reply.
> Setting "port=2049" avoids this lookup so the mount succeeds.
> 
> We could just decide that that is the way it is, document that this is how to
> talk to systems over UDP when they have a broken portmaper and move on.
> 
> But it would be nice if we could find a way to make a solution less ugly and
> easier to find.
> 
> So some thoughts:
> 
> 1/ We could use tcp for all rpcbind lookups if tcp was requested for either
>    MOUNT or NFS.
Doing this would burn reserver ports, by putting it into TIME_WAIT which
means they would not be available for actual mounts during mount storms
via automount....

> 
> 2/ If '-v' was given and a UDP lookup to portmap timedout, we could retry
>    without connecting (and with a shorter timeout), and print an explanatory 
>    message if that succeeded.
I agree with Jim. Making '-v' change behavior would be unexpected...
 
> 
> 3/ We could have a mount option "bindproto=tcp" or maybe "bind=noconnect"
>    which could be recommended by the above explanatory message.
I would like to avoid adding something like this... 

> 
> 
> Does anyone else have any good ideas, or any deep aversions to any of the
> above.
> I'm tempted to go for '1' and possibly add '2', but not bother with '3'.
> 
> ??
Personally, explicitly setting the protocols and port is probably
the best way to handle this... Thats what the arguments are there 
for. Plus these are legacy machines so those setting will only be
needed for a small subset machines...  

steved.

Re: mounting via udp from legacy multi-homed servers...

[Chuck Lever]

On Jul 13, 2011, at 8:34 AM, NeilBrown wrote:

> 
> 
> This problem doesn't seem to go away, does it :-(
> 
> 
> While modern Linux always replies to UDP requests from the same address that
> the request was sent to, some legacy systems do not, and some customers use
> these legacy systems and cannot upgrade.
> 
> And as they are legacy systems, it is often the case that UDP is more
> performant than TCP.
> 
> nfs-utils always uses a connected UDP socket to talk to servers (which I think
> is correct so we get errors when reported by ICMP-unreachable messages) and so
> fails to talk to these legacy systems (when UDP is requested).
> 
> I had such a case recently and managed to make it work by using:
> 
>   proto=udp,mountproto=tcp,port=2049
> 
> By setting mountproto=tcp, nfs-utils uses tcp for almost all communication
> with the server.
> The one little problem is that mount does a portmap lookup to check the
> port number to talk to the NFS server and seeing we requested UDP for NFS,
> it talks to portmap via UDP and never sees the reply.
> Setting "port=2049" avoids this lookup so the mount succeeds.
> 
> We could just decide that that is the way it is, document that this is how to
> talk to systems over UDP when they have a broken portmaper and move on.

libtirpc will always perform the rpcbind query on the same transport as the original protocol wants to use, for example.  The libtirpc v2 portmap service I believe performs all queries via UDP.  I don't have the code in front of me to confirm that.

> But it would be nice if we could find a way to make a solution less ugly and
> easier to find.
> 
> So some thoughts:
> 
> 1/ We could use tcp for all rpcbind lookups if tcp was requested for either
>   MOUNT or NFS.

This will probably break things that work today.

> 2/ If '-v' was given and a UDP lookup to portmap timedout, we could retry
>   without connecting (and with a shorter timeout), and print an explanatory 
>   message if that succeeded.

"-v" is already in use, as far as I know, and as has been pointed out.

> 3/ We could have a mount option "bindproto=tcp" or maybe "bind=noconnect"
>   which could be recommended by the above explanatory message.

And that would require some kernel mods too, since the kernel may also perform an rpcbind query in some cases.

4.  nfs(5) could explain more clearly that the transport protocol used for the rpcbind goes with the "proto=" and "mountproto=" options.  "Doctor, it hurts when I do that..."  But you might provide your workaround, above, in the EXAMPLES section.

> Does anyone else have any good ideas, or any deep aversions to any of the
> above.
> I'm tempted to go for '1' and possibly add '2', but not bother with '3'.
> 
> ??
> 
> Thanks,
> NeilBrown

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com