* Re: Status of mount.nfs
2007-07-24 17:24 ` Steinar H. Gunderson
@ 2007-07-24 17:50 ` Trond Myklebust
2007-07-24 17:55 ` Steinar H. Gunderson
2007-07-24 20:46 ` Chuck Lever
` (2 subsequent siblings)
3 siblings, 1 reply; 49+ messages in thread
From: Trond Myklebust @ 2007-07-24 17:50 UTC (permalink / raw)
To: Steinar H. Gunderson; +Cc: nfs
On Tue, 2007-07-24 at 19:24 +0200, Steinar H. Gunderson wrote:
> On Mon, Jul 23, 2007 at 06:13:42PM -0400, Chuck Lever wrote:
> > It would help if we could take a look at a clean network trace of the bad
> > and the good mount operations.
>
> It was quite simple to test this myself. I started the kernel server on a
> machine, then shut down portmap. First I did:
>
> fugl:~> sudo mount -t nfs -o port=2049,mountport=901,nfsvers=3 192.168.0.101:/ /mnt
> mount: mount to NFS server '192.168.0.101' failed: System Error: Connection refused.
>
> The dump is attached as "default.dump". Then I did
>
> fugl:~> sudo mount -t nfs -o port=2049,mountport=901,nfsvers=3,udp 192.168.0.101:/ /mnt
>
> which is attached as "udp.dump".
>
> Note that in default.dump, UDP is simply never tried at all. I believe that
> to be a bug.
Nope. Nowhere in the documentation will you find a promise to fall back
to UDP.
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: Status of mount.nfs
2007-07-24 17:50 ` Trond Myklebust
@ 2007-07-24 17:55 ` Steinar H. Gunderson
0 siblings, 0 replies; 49+ messages in thread
From: Steinar H. Gunderson @ 2007-07-24 17:55 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs
On Tue, Jul 24, 2007 at 01:50:05PM -0400, Trond Myklebust wrote:
>> Note that in default.dump, UDP is simply never tried at all. I believe that
>> to be a bug.
> Nope. Nowhere in the documentation will you find a promise to fall back
> to UDP.
OK, but in that case the regression should still be documented, as it seems
this worked before.
/* Steinar */
--
Homepage: http://www.sesse.net/
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-24 17:24 ` Steinar H. Gunderson
2007-07-24 17:50 ` Trond Myklebust
@ 2007-07-24 20:46 ` Chuck Lever
2007-07-24 21:10 ` Trond Myklebust
2007-07-25 2:08 ` rpcbind behavior on Fedora 7 Chuck Lever
2007-07-25 19:35 ` Status of mount.nfs Chuck Lever
3 siblings, 1 reply; 49+ messages in thread
From: Chuck Lever @ 2007-07-24 20:46 UTC (permalink / raw)
To: Steinar H. Gunderson; +Cc: nfs
[-- Attachment #1: Type: text/plain, Size: 1193 bytes --]
Steinar H. Gunderson wrote:
> On Mon, Jul 23, 2007 at 06:13:42PM -0400, Chuck Lever wrote:
>> It would help if we could take a look at a clean network trace of the bad
>> and the good mount operations.
>
> It was quite simple to test this myself. I started the kernel server on a
> machine, then shut down portmap. First I did:
>
> fugl:~> sudo mount -t nfs -o port=2049,mountport=901,nfsvers=3 192.168.0.101:/ /mnt
> mount: mount to NFS server '192.168.0.101' failed: System Error: Connection refused.
>
> The dump is attached as "default.dump". Then I did
>
> fugl:~> sudo mount -t nfs -o port=2049,mountport=901,nfsvers=3,udp 192.168.0.101:/ /mnt
>
> which is attached as "udp.dump".
>
> Note that in default.dump, UDP is simply never tried at all. I believe that
> to be a bug.
It looks like, in the UDP dump you sent, there is no attempt to contact
the portmapper at all. The mount request is the first thing in the
dump, and the request goes right to the port specified on the command line.
The TCP case fails because mount.nfs is using the portmapper even though
the user has specified the ports on the command line. Could that be the
root cause of the failure?
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 259 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-24 20:46 ` Chuck Lever
@ 2007-07-24 21:10 ` Trond Myklebust
2007-07-24 21:18 ` Chuck Lever
0 siblings, 1 reply; 49+ messages in thread
From: Trond Myklebust @ 2007-07-24 21:10 UTC (permalink / raw)
To: chuck.lever; +Cc: nfs
On Tue, 2007-07-24 at 16:46 -0400, Chuck Lever wrote:
> The TCP case fails because mount.nfs is using the portmapper even though
> the user has specified the ports on the command line. Could that be the
> root cause of the failure?
If the user specifies a port, then there is no real good reason to use
the portmapper:
* For the case of the mount protocol, we should just try an RPC
call and then look at the returned RPC error values to figure
out which versions of the protocol are supported if it fails (or
alternatively, fall back to UDP if the TCP connection fails).
* If the mount call has succeeded and we have a port for the NFS
server, we should probably just try the mount call for a
sufficiently recent kernel, then look at the returned error
codes. For older kernels (pre 2.6.13?) which don't return decent
error values, then ping first in userland and look at the RPC
return values. Retry with UDP if TCP doesn't work...
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: Status of mount.nfs
2007-07-24 21:10 ` Trond Myklebust
@ 2007-07-24 21:18 ` Chuck Lever
0 siblings, 0 replies; 49+ messages in thread
From: Chuck Lever @ 2007-07-24 21:18 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs
[-- Attachment #1: Type: text/plain, Size: 1309 bytes --]
Trond Myklebust wrote:
> On Tue, 2007-07-24 at 16:46 -0400, Chuck Lever wrote:
>
>> The TCP case fails because mount.nfs is using the portmapper even though
>> the user has specified the ports on the command line. Could that be the
>> root cause of the failure?
>
> If the user specifies a port, then there is no real good reason to use
> the portmapper:
Exactly.
> * For the case of the mount protocol, we should just try an RPC
> call and then look at the returned RPC error values to figure
> out which versions of the protocol are supported if it fails (or
> alternatively, fall back to UDP if the TCP connection fails).
> * If the mount call has succeeded and we have a port for the NFS
> server, we should probably just try the mount call for a
> sufficiently recent kernel, then look at the returned error
> codes. For older kernels (pre 2.6.13?) which don't return decent
> error values, then ping first in userland and look at the RPC
> return values. Retry with UDP if TCP doesn't work...
I'm wondering how much of this is already implemented in mount.nfs, but
is just broken. I'm reviewing the code now.
I cleaned this up a little with recent patches so it should be easier to
figure out what's going wrong.
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 259 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* rpcbind behavior on Fedora 7
2007-07-24 17:24 ` Steinar H. Gunderson
2007-07-24 17:50 ` Trond Myklebust
2007-07-24 20:46 ` Chuck Lever
@ 2007-07-25 2:08 ` Chuck Lever
2007-07-25 19:35 ` Status of mount.nfs Chuck Lever
3 siblings, 0 replies; 49+ messages in thread
From: Chuck Lever @ 2007-07-25 2:08 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
[-- Attachment #1: Type: text/plain, Size: 4100 bytes --]
Hi Steve-
I was trying out the mount.nfs test case for another bug (see below).
The test case didn't work against a Fedora 7 server. Trying to mount
with UDP against a specific port just hangs. So I tried an rpcinfo
against it to see what the current rocbind configuration was.
> [root@picasso ~]# rpcinfo ingres
> program version netid address service owner
> 100000 4 tcp6 ::.0.111 portmapper superuser
> 100000 3 tcp6 ::.0.111 portmapper superuser
> 100000 2 tcp6 ::.0.111 portmapper superuser
> 100000 4 udp6 ::.0.111 portmapper superuser
> 100000 3 udp6 ::.0.111 portmapper superuser
> 100000 2 udp6 ::.0.111 portmapper superuser
> 100000 4 local /v portmapper superuser
> 100000 3 local /v portmapper superuser
> 100000 2 local /v portmapper superuser
> 100024 1 udp6 ::.2.222 status unknown
> 100024 1 tcp6 ::.2.225 status unknown
> 100021 1 tcp6 ::.170.233 nlockmgr unknown
> 100021 3 tcp6 ::.170.233 nlockmgr unknown
> 100021 4 tcp6 ::.170.233 nlockmgr unknown
> 100011 1 udp6 ::.3.158 rquotad unknown
> 100011 2 udp6 ::.3.158 rquotad unknown
> 100011 1 tcp6 ::.3.161 rquotad unknown
> 100011 2 tcp6 ::.3.161 rquotad unknown
> 100021 1 udp6 ::.128.0 nlockmgr unknown
> 100021 3 udp6 ::.128.0 nlockmgr unknown
> 100021 4 udp6 ::.128.0 nlockmgr unknown
> 100003 2 udp6 ::.8.1 nfs unknown
> 100003 3 udp6 ::.8.1 nfs unknown
> 100003 4 udp6 ::.8.1 nfs unknown
> 100003 2 tcp6 ::.8.1 nfs unknown
> 100003 3 tcp6 ::.8.1 nfs unknown
> 100003 4 tcp6 ::.8.1 nfs unknown
> 100005 1 udp6 ::.2.135 mountd unknown
> 100005 1 tcp6 ::.2.138 mountd unknown
> 100005 2 udp6 ::.2.135 mountd unknown
> 100005 2 tcp6 ::.2.138 mountd unknown
> 100005 3 udp6 ::.2.135 mountd unknown
> 100005 3 tcp6 ::.2.138 mountd unknown
> [root@picasso ~]#
Um. Ok, where are the IPv4 entries?
I've now completely shut off IPv6 initialization and autoconfiguration
on the only network interface on the system and rebooted several times
(I wasn't using the IPv6 networking stuff yet anyway). I still get *no*
udp4 or tcp4 entries in the rpcbind database. The NFS service on this
system is all IPv4 (it's a Linux NFS server).
How are these getting registered?
Steinar H. Gunderson wrote:
> On Mon, Jul 23, 2007 at 06:13:42PM -0400, Chuck Lever wrote:
>> It would help if we could take a look at a clean network trace of the bad
>> and the good mount operations.
>
> It was quite simple to test this myself. I started the kernel server on a
> machine, then shut down portmap. First I did:
>
> fugl:~> sudo mount -t nfs -o port=2049,mountport=901,nfsvers=3 192.168.0.101:/ /mnt
> mount: mount to NFS server '192.168.0.101' failed: System Error: Connection refused.
>
> The dump is attached as "default.dump". Then I did
>
> fugl:~> sudo mount -t nfs -o port=2049,mountport=901,nfsvers=3,udp 192.168.0.101:/ /mnt
>
> which is attached as "udp.dump".
>
> Note that in default.dump, UDP is simply never tried at all. I believe that
> to be a bug.
>
> /* Steinar */
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 259 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-24 17:24 ` Steinar H. Gunderson
` (2 preceding siblings ...)
2007-07-25 2:08 ` rpcbind behavior on Fedora 7 Chuck Lever
@ 2007-07-25 19:35 ` Chuck Lever
2007-07-26 12:47 ` Steve Dickson
3 siblings, 1 reply; 49+ messages in thread
From: Chuck Lever @ 2007-07-25 19:35 UTC (permalink / raw)
To: Steinar H. Gunderson; +Cc: nfs
[-- Attachment #1: Type: text/plain, Size: 1984 bytes --]
Hi Steinar-
Steinar H. Gunderson wrote:
> On Mon, Jul 23, 2007 at 06:13:42PM -0400, Chuck Lever wrote:
>> It would help if we could take a look at a clean network trace of the bad
>> and the good mount operations.
>
> It was quite simple to test this myself. I started the kernel server on a
> machine, then shut down portmap. First I did:
>
> fugl:~> sudo mount -t nfs -o port=2049,mountport=901,nfsvers=3 192.168.0.101:/ /mnt
> mount: mount to NFS server '192.168.0.101' failed: System Error: Connection refused.
>
> The dump is attached as "default.dump". Then I did
>
> fugl:~> sudo mount -t nfs -o port=2049,mountport=901,nfsvers=3,udp 192.168.0.101:/ /mnt
>
> which is attached as "udp.dump".
>
> Note that in default.dump, UDP is simply never tried at all. I believe that
> to be a bug.
I'm looking at probe_nfsport() and probe_mntport() and I see that the
portmapper call is avoided iff the protocol version, transport protocol,
and the port number are all specified in advance.
So if you specify:
-o mountport=650,port=2049
mount will still contact the server's portmapper to determine which
transport protocols are available (and this breaks through a firewall
that doesn't pass portmapper requests). Only if you specify:
-o tcp,mountport=650,port=2049
or
-o udp,mountport=650,port=2049
then the portmapper calls are avoided entirely. For some reason
probe_bothports() sets the default NFS version to 3 but does not set a
default transport protocol.
The transport protocols are probed differently for NFS and MNT: for MNT,
UDP is probed first then TCP; for NFS, the opposite is true. The mount
command is supposed to try both transport protocol types both for NFS
and MNT, but it appears that it is failing to try the other type if the
first fails... I see this is also problematic for umount.nfs.
So yes, I think the logic is there, but it's broken. I recall that the
transport protocol probing logic was added recently.
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 259 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: Status of mount.nfs
2007-07-25 19:35 ` Status of mount.nfs Chuck Lever
@ 2007-07-26 12:47 ` Steve Dickson
2007-07-27 3:02 ` Chuck Lever
0 siblings, 1 reply; 49+ messages in thread
From: Steve Dickson @ 2007-07-26 12:47 UTC (permalink / raw)
To: chuck.lever; +Cc: nfs
Chuck Lever wrote:
>
> I'm looking at probe_nfsport() and probe_mntport() and I see that the
> portmapper call is avoided iff the protocol version, transport protocol,
> and the port number are all specified in advance.
>
> So if you specify:
>
> -o mountport=650,port=2049
>
> mount will still contact the server's portmapper to determine which
> transport protocols are available (and this breaks through a firewall
> that doesn't pass portmapper requests). Only if you specify:
>
> -o tcp,mountport=650,port=2049
>
> or
>
> -o udp,mountport=650,port=2049
>
> then the portmapper calls are avoided entirely. For some reason
> probe_bothports() sets the default NFS version to 3 but does not set a
> default transport protocol.
The protocol is set by probe_port().
>
> The transport protocols are probed differently for NFS and MNT: for MNT,
> UDP is probed first then TCP; for NFS, the opposite is true. The mount
> command is supposed to try both transport protocol types both for NFS
> and MNT, but it appears that it is failing to try the other type if the
> first fails... I see this is also problematic for umount.nfs.
The I idea here was to using UDP to probing for both rpc.mountd and the
NFS server so as not to put (tcp) ports in TIMEWAIT, basically making
them unavailable for the actual mount. The allowed many more tcp
mounts to happen during autofs mount storms.
Note: if the protocol was explicitly specified (i.e. proto=tcp) only
that protocol was used during the probing and mounting. I have
been asked to change that as well. Meaning if when proto=tcp is
specified, they still want the udp probing to occur.
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-26 12:47 ` Steve Dickson
@ 2007-07-27 3:02 ` Chuck Lever
2007-07-27 15:00 ` Steve Dickson
0 siblings, 1 reply; 49+ messages in thread
From: Chuck Lever @ 2007-07-27 3:02 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
[-- Attachment #1: Type: text/plain, Size: 2797 bytes --]
Steve Dickson wrote:
>> The transport protocols are probed differently for NFS and MNT: for
>> MNT, UDP is probed first then TCP; for NFS, the opposite is true. The
>> mount command is supposed to try both transport protocol types both
>> for NFS and MNT, but it appears that it is failing to try the other
>> type if the first fails... I see this is also problematic for umount.nfs.
>
> The idea here was to use UDP to probe for both rpc.mountd and the
> NFS server so as not to put (tcp) ports in TIMEWAIT, basically making
> them unavailable for the actual mount. That allowed many more tcp
> mounts to happen during autofs mount storms.
Right. I don't see GETPORT using UDP as often as perhaps it should,
though. I need to go back and check it.
And umount.nfs always uses TCP for the mountd request. I have a patch
that fixes that to behave more like mount.nfs does, which I will forward
in the next day or two.
I notice some problems if a share is mounted with TCP, but the server
later disables TCP -- umount.nfs hiccups on that when it tries to umount
using the same protocol as listed in /etc/mtab. Perhaps relying on
/etc/mtab for setting the umount protocol is unnecessary.
> Note: if the protocol was explicitly specified (i.e. proto=tcp) only
> that protocol was used during the probing and mounting. I have
> been asked to change that as well. Meaning if when proto=tcp is
> specified, they still want the udp probing to occur.
I would like to spell out exactly what behavior we want to solder into
the community nfs-utils here, at least so I can approximate it with the
string-ified mount implementation, but also because it seems to be quite
heuristic and totally undocumented. If we tune for a particular use
case, it will break others, so we need to agree on a default that works
for most cases.
(As part of this exercise it would also be lovely to assemble some nice
regression test cases, but that may be too much to hope for ;-)
We have three requests that need to be made:
1. GETPORT -- I think this should UDP all the time unless proto=tcp is
explicitly specified;
2. MNT -- likewise, UDP unless proto=tcp is specified or GETPORT says
UDP is not supported;
3. NFS -- this should be TCP all the time unless proto=udp is specified
or GETPORT says TCP is not supported.
Even better would be to use RPCB_DUMP instead of RPCB_GETPORT. That way
we only need a single rpcbind call for both protocols, and can get
transport protocol information as well, and make an "informed" choice.
Also, can we get rid of the clnt_ping()? If not, can we document why it
is there? It adds two extra round trips to the whole process. If error
reporting is the problem, maybe we can try the pings only if the kernel
part of the mount process fails?
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 315 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
email;internet:chuck dot lever at nospam oracle dot com
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-27 3:02 ` Chuck Lever
@ 2007-07-27 15:00 ` Steve Dickson
2007-07-27 15:56 ` Trond Myklebust
2007-07-27 19:37 ` Chuck Lever
0 siblings, 2 replies; 49+ messages in thread
From: Steve Dickson @ 2007-07-27 15:00 UTC (permalink / raw)
To: chuck.lever; +Cc: nfs
Chuck Lever wrote:
>
> And umount.nfs always uses TCP for the mountd request. I have a patch
> that fixes that to behave more like mount.nfs does, which I will forward
> in the next day or two.
thats a bug... umount should use the protocol the mount did...
I thought I had fixed that... :-\
>
> I notice some problems if a share is mounted with TCP, but the server
> later disables TCP -- umount.nfs hiccups on that when it tries to umount
> using the same protocol as listed in /etc/mtab. Perhaps relying on
> /etc/mtab for setting the umount protocol is unnecessary.
I think I was using /proc/mounts...
>
> We have three requests that need to be made:
>
> 1. GETPORT -- I think this should UDP all the time unless proto=tcp is
> explicitly specified;
Some people have asked that we first try UDP all the time... which
I have resisted but it might make sense...
>
> 2. MNT -- likewise, UDP unless proto=tcp is specified or GETPORT says
> UDP is not supported;
>
> 3. NFS -- this should be TCP all the time unless proto=udp is specified
> or GETPORT says TCP is not supported.
What about rollbacks... meaning if tcp is not supported do we try udp?
if v4 is not supported to we try v3 and the v2 or just fail the mount?
>
> Even better would be to use RPCB_DUMP instead of RPCB_GETPORT. That way
> we only need a single rpcbind call for both protocols, and can get
> transport protocol information as well, and make an "informed" choice.
Good point... but note, a while back I got a request to use GETPORT
instead of DUMP because some Cisco router actually use the GETPORTs
to punch wholes in their firewalls.
>
> Also, can we get rid of the clnt_ping()? If not, can we document why it
> is there? It adds two extra round trips to the whole process. If error
> reporting is the problem, maybe we can try the pings only if the kernel
> part of the mount process fails?
How do we avoid hang down deep in RPC land (governed by
uncontrollable timeout) when either mountd or nfsd are not up?
That was the main reason for the ping. Since neither portmapper or
rpcbind ping their services before they hand out the ports, there
is really no way of telling where the server is up? So to avoid
the hang, we ping them... Sure its costly network wise, but
hanging during a boot because a server is not responding is
a bit more costly... imho...
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-27 15:00 ` Steve Dickson
@ 2007-07-27 15:56 ` Trond Myklebust
2007-07-27 16:16 ` Steve Dickson
2007-07-27 19:37 ` Chuck Lever
1 sibling, 1 reply; 49+ messages in thread
From: Trond Myklebust @ 2007-07-27 15:56 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
On Fri, 2007-07-27 at 11:00 -0400, Steve Dickson wrote:
> That was the main reason for the ping. Since neither portmapper or
> rpcbind ping their services before they hand out the ports, there
> is really no way of telling where the server is up? So to avoid
> the hang, we ping them... Sure its costly network wise, but
> hanging during a boot because a server is not responding is
> a bit more costly... imho...
Right, but recent kernels both can and will ping the NFS service for
you.
Cheers
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-27 15:56 ` Trond Myklebust
@ 2007-07-27 16:16 ` Steve Dickson
2007-07-27 16:27 ` Trond Myklebust
0 siblings, 1 reply; 49+ messages in thread
From: Steve Dickson @ 2007-07-27 16:16 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs
Trond Myklebust wrote:
> On Fri, 2007-07-27 at 11:00 -0400, Steve Dickson wrote:
>
>> That was the main reason for the ping. Since neither portmapper or
>> rpcbind ping their services before they hand out the ports, there
>> is really no way of telling where the server is up? So to avoid
>> the hang, we ping them... Sure its costly network wise, but
>> hanging during a boot because a server is not responding is
>> a bit more costly... imho...
>
> Right, but recent kernels both can and will ping the NFS service for
> you.
Good point... but that's just as costly (wrt network traffic) as
the mount command doing the pinging... plus the status of the
remote mountd is also needed.
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-27 16:16 ` Steve Dickson
@ 2007-07-27 16:27 ` Trond Myklebust
2007-07-27 17:07 ` Steve Dickson
0 siblings, 1 reply; 49+ messages in thread
From: Trond Myklebust @ 2007-07-27 16:27 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
On Fri, 2007-07-27 at 12:16 -0400, Steve Dickson wrote:
>
> Trond Myklebust wrote:
> > On Fri, 2007-07-27 at 11:00 -0400, Steve Dickson wrote:
> >
> >> That was the main reason for the ping. Since neither portmapper or
> >> rpcbind ping their services before they hand out the ports, there
> >> is really no way of telling where the server is up? So to avoid
> >> the hang, we ping them... Sure its costly network wise, but
> >> hanging during a boot because a server is not responding is
> >> a bit more costly... imho...
> >
> > Right, but recent kernels both can and will ping the NFS service for
> > you.
> Good point... but that's just as costly (wrt network traffic) as
> the mount command doing the pinging... plus the status of the
> remote mountd is also needed.
The kernel ping is sent on the same connection to the server than the
NFS client will use, so we don't waste any extra TCP ports.
I agree that you need to figure out the mountd parameters, but the right
thing to do there is simply to try the command. The RPC return values
will tell you if the service or version you tried isn't supported.
Cheers
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-27 16:27 ` Trond Myklebust
@ 2007-07-27 17:07 ` Steve Dickson
2007-07-27 17:13 ` Trond Myklebust
0 siblings, 1 reply; 49+ messages in thread
From: Steve Dickson @ 2007-07-27 17:07 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs
Trond Myklebust wrote:
> On Fri, 2007-07-27 at 12:16 -0400, Steve Dickson wrote:
>> Trond Myklebust wrote:
>>> On Fri, 2007-07-27 at 11:00 -0400, Steve Dickson wrote:
>>>
>>>> That was the main reason for the ping. Since neither portmapper or
>>>> rpcbind ping their services before they hand out the ports, there
>>>> is really no way of telling where the server is up? So to avoid
>>>> the hang, we ping them... Sure its costly network wise, but
>>>> hanging during a boot because a server is not responding is
>>>> a bit more costly... imho...
>>> Right, but recent kernels both can and will ping the NFS service for
>>> you.
>> Good point... but that's just as costly (wrt network traffic) as
>> the mount command doing the pinging... plus the status of the
>> remote mountd is also needed.
>
> The kernel ping is sent on the same connection to the server than the
> NFS client will use, so we don't waste any extra TCP ports.
>
> I agree that you need to figure out the mountd parameters, but the right
> thing to do there is simply to try the command. The RPC return values
> will tell you if the service or version you tried isn't supported.
After how long of a wait? If all the timeouts are controllable then
I agrees, but if we have to wait undefined-able amount of time for
every RPC retransmit, then I think we should do a ping...
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-27 17:07 ` Steve Dickson
@ 2007-07-27 17:13 ` Trond Myklebust
2007-07-27 21:38 ` Chuck Lever
2007-07-28 12:51 ` Steve Dickson
0 siblings, 2 replies; 49+ messages in thread
From: Trond Myklebust @ 2007-07-27 17:13 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
On Fri, 2007-07-27 at 13:07 -0400, Steve Dickson wrote:
> After how long of a wait? If all the timeouts are controllable then
> I agrees, but if we have to wait undefined-able amount of time for
> every RPC retransmit, then I think we should do a ping...
You should be able to set the timeout either on a per-RPC call basis by
using clnt_call(), or by changing the default timeout on the CLIENT
object using clnt_control() (man 3 rpc).
Cheers
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-27 17:13 ` Trond Myklebust
@ 2007-07-27 21:38 ` Chuck Lever
2007-07-28 12:51 ` Steve Dickson
1 sibling, 0 replies; 49+ messages in thread
From: Chuck Lever @ 2007-07-27 21:38 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs, Steve Dickson
[-- Attachment #1: Type: text/plain, Size: 745 bytes --]
Trond Myklebust wrote:
> On Fri, 2007-07-27 at 13:07 -0400, Steve Dickson wrote:
>
>> After how long of a wait? If all the timeouts are controllable then
>> I agrees, but if we have to wait undefined-able amount of time for
>> every RPC retransmit, then I think we should do a ping...
>
> You should be able to set the timeout either on a per-RPC call basis by
> using clnt_call(), or by changing the default timeout on the CLIENT
> object using clnt_control() (man 3 rpc).
The mount command uses a "standard" timeout structure to set a 25 second
timeout for both UDP and TCP, but I've noticed that TCP connections for
most RPC requests from the mount command to unavailable servers appear
to hang indefinitely. I'll take a look at this.
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 259 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-27 17:13 ` Trond Myklebust
2007-07-27 21:38 ` Chuck Lever
@ 2007-07-28 12:51 ` Steve Dickson
2007-07-31 18:30 ` Trond Myklebust
1 sibling, 1 reply; 49+ messages in thread
From: Steve Dickson @ 2007-07-28 12:51 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs
Trond Myklebust wrote:
> On Fri, 2007-07-27 at 13:07 -0400, Steve Dickson wrote:
>
>> After how long of a wait? If all the timeouts are controllable then
>> I agrees, but if we have to wait undefined-able amount of time for
>> every RPC retransmit, then I think we should do a ping...
>
> You should be able to set the timeout either on a per-RPC call basis by
> using clnt_call(), or by changing the default timeout on the CLIENT
> object using clnt_control() (man 3 rpc).
True... but I was thinking of when clnt_cnt() call pmap_getport()
which does not take a timeout value... In that case you are
stuck with a 60 sec hard coded timeout, regardless of the time out
you pass in...
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-28 12:51 ` Steve Dickson
@ 2007-07-31 18:30 ` Trond Myklebust
2007-07-31 21:28 ` Chuck Lever
0 siblings, 1 reply; 49+ messages in thread
From: Trond Myklebust @ 2007-07-31 18:30 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
On Sat, 2007-07-28 at 08:51 -0400, Steve Dickson wrote:
> Trond Myklebust wrote:
> > On Fri, 2007-07-27 at 13:07 -0400, Steve Dickson wrote:
> >
> >> After how long of a wait? If all the timeouts are controllable then
> >> I agrees, but if we have to wait undefined-able amount of time for
> >> every RPC retransmit, then I think we should do a ping...
> >
> > You should be able to set the timeout either on a per-RPC call basis by
> > using clnt_call(), or by changing the default timeout on the CLIENT
> > object using clnt_control() (man 3 rpc).
> True... but I was thinking of when clnt_cnt() call pmap_getport()
> which does not take a timeout value... In that case you are
> stuck with a 60 sec hard coded timeout, regardless of the time out
> you pass in...
In the version of mount.nfs from the linux-nfs.org git tree, we call our
own private implementation of pmap_getport() instead of the one from
glibc.
Cheers
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-31 18:30 ` Trond Myklebust
@ 2007-07-31 21:28 ` Chuck Lever
2007-08-01 10:58 ` Steve Dickson
0 siblings, 1 reply; 49+ messages in thread
From: Chuck Lever @ 2007-07-31 21:28 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs, Steve Dickson
[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]
Trond Myklebust wrote:
> On Sat, 2007-07-28 at 08:51 -0400, Steve Dickson wrote:
>> Trond Myklebust wrote:
>>> On Fri, 2007-07-27 at 13:07 -0400, Steve Dickson wrote:
>>>
>>>> After how long of a wait? If all the timeouts are controllable then
>>>> I agrees, but if we have to wait undefined-able amount of time for
>>>> every RPC retransmit, then I think we should do a ping...
>>> You should be able to set the timeout either on a per-RPC call basis by
>>> using clnt_call(), or by changing the default timeout on the CLIENT
>>> object using clnt_control() (man 3 rpc).
>> True... but I was thinking of when clnt_cnt() call pmap_getport()
>> which does not take a timeout value... In that case you are
>> stuck with a 60 sec hard coded timeout, regardless of the time out
>> you pass in...
>
> In the version of mount.nfs from the linux-nfs.org git tree, we call our
> own private implementation of pmap_getport() instead of the one from
> glibc.
I was looking at this yesterday. The stock timeout for TCP connects on
Linux is 75 seconds. The version of getport() used in the mount command
might control the TCP connect timeout by using a non-blocking connect()
with a select(). The select() then times out if the connection doesn't
complete.
But I'm wondering if we really want to continue using TCP for GETPORT
calls. Solaris mount appears to use only UDP for GETPORT, for example.
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 290 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-31 21:28 ` Chuck Lever
@ 2007-08-01 10:58 ` Steve Dickson
2007-08-01 20:02 ` Chuck Lever
0 siblings, 1 reply; 49+ messages in thread
From: Steve Dickson @ 2007-08-01 10:58 UTC (permalink / raw)
To: chuck.lever; +Cc: nfs
Chuck Lever wrote:
> I was looking at this yesterday. The stock timeout for TCP connects on
> Linux is 75 seconds. The version of getport() used in the mount command
> might control the TCP connect timeout by using a non-blocking connect()
> with a select(). The select() then times out if the connection doesn't
> complete.
>
> But I'm wondering if we really want to continue using TCP for GETPORT
> calls. Solaris mount appears to use only UDP for GETPORT, for example.
As as long as the GETPORTs don't use privilege ports I don't think its
a problem... plus I don't think one size fixes all.. meaning due to
different firewalls requirements both udp and tcp GETPORTS will be
needed... imho...
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-08-01 10:58 ` Steve Dickson
@ 2007-08-01 20:02 ` Chuck Lever
2007-08-01 21:12 ` Steve Dickson
0 siblings, 1 reply; 49+ messages in thread
From: Chuck Lever @ 2007-08-01 20:02 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
[-- Attachment #1: Type: text/plain, Size: 2127 bytes --]
Steve Dickson wrote:
> Chuck Lever wrote:
>> I was looking at this yesterday. The stock timeout for TCP connects
>> on Linux is 75 seconds. The version of getport() used in the mount
>> command might control the TCP connect timeout by using a non-blocking
>> connect() with a select(). The select() then times out if the
>> connection doesn't complete.
>>
>> But I'm wondering if we really want to continue using TCP for GETPORT
>> calls. Solaris mount appears to use only UDP for GETPORT, for example.
> As as long as the GETPORTs don't use privilege ports I don't think its
> a problem...
Not sure what you mean. Yesterday you said the TCP connect timeout
*was* a problem. I've recommended two ways to address it.
The ephemeral port space is limited too, don't forget. It's simply a
somewhat larger space than the privileged port space. If a large
network application (say, a web server) is running on the system, that
space can shrink fairly rapidly, and we're in nearly the same boat as
with privileged ports. Using a TCP connection from an ephemeral port
only mitigates the port space problem, it doesn't really correct it
entirely.
> plus I don't think one size fixes all.. meaning due to
> different firewalls requirements both udp and tcp GETPORTS will be
> needed... imho...
We say "firewall!" a lot, but I would like to see typical use cases for
mounting through a firewall so I understand what kind of implementation
we're aiming for (and maybe even what kind of test cases to build!). Do
our users really expect to mount NFS shares through any firewall with
"-o defaults" ?
I'd like to hear from the distributors what you consider are the use
cases that absolutely must be supported. Otherwise we will end up
standing on our left big toenail to support stuff that isn't worth the
pain or is never used.
And can anyone shed some light on why Solaris uses only UDP for GETPORT
requests at mount time? Does it ever fall back to try GETPORT on a TCP
connection? How do Solaris users mount through a firewall? Come on all
you Solaris NFS guys out there, I know you're lurking.
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 290 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-08-01 20:02 ` Chuck Lever
@ 2007-08-01 21:12 ` Steve Dickson
2007-08-02 16:20 ` Chuck Lever
0 siblings, 1 reply; 49+ messages in thread
From: Steve Dickson @ 2007-08-01 21:12 UTC (permalink / raw)
To: chuck.lever; +Cc: nfs
Chuck Lever wrote:
> Steve Dickson wrote:
>> Chuck Lever wrote:
>>> I was looking at this yesterday. The stock timeout for TCP connects
>>> on Linux is 75 seconds. The version of getport() used in the mount
>>> command might control the TCP connect timeout by using a non-blocking
>>> connect() with a select(). The select() then times out if the
>>> connection doesn't complete.
>>>
>>> But I'm wondering if we really want to continue using TCP for GETPORT
>>> calls. Solaris mount appears to use only UDP for GETPORT, for example.
>
>> As as long as the GETPORTs don't use privilege ports I don't think its
>> a problem...
>
> Not sure what you mean. Yesterday you said the TCP connect timeout
> *was* a problem. I've recommended two ways to address it.
TCP timeouts are a problem if you can't control them... But
point taken... UPD is probably the best way to query a
portmapper or rpcbinder to get the needed info...
>
> The ephemeral port space is limited too, don't forget. It's simply a
> somewhat larger space than the privileged port space. If a large
> network application (say, a web server) is running on the system, that
> space can shrink fairly rapidly, and we're in nearly the same boat as
> with privileged ports. Using a TCP connection from an ephemeral port
> only mitigates the port space problem, it doesn't really correct it
> entirely.
Only mitigates the problem for a short time and you'll always run
out of privileged port before running out of non-privileged but
again... point taken... eliminating the problem is probably
the answer...
>
>> plus I don't think one size fixes all.. meaning due to
>> different firewalls requirements both udp and tcp GETPORTS will be
>> needed... imho...
>
> We say "firewall!" a lot, but I would like to see typical use cases for
> mounting through a firewall so I understand what kind of implementation
> we're aiming for (and maybe even what kind of test cases to build!). Do
> our users really expect to mount NFS shares through any firewall with
> "-o defaults" ?
Yes! Mostly on the server side... meaning people wanted to set the
port the daemons listen on (via the initscripts) so clients can
access the server through a firewall... Is this a common setup?
No. But there are people that want a firewall between the
server and client.. Also I can only assume the reason for the
'mountport=" option was to work better with firewalls...
but that is only speculation...
>
> I'd like to hear from the distributors what you consider are the use
> cases that absolutely must be supported. Otherwise we will end up
> standing on our left big toenail to support stuff that isn't worth the
> pain or is never used.
In the end, I think we need to be able to control the ports and
protocol mounts uses, allowing people to punch holes in firewalls.
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-08-01 21:12 ` Steve Dickson
@ 2007-08-02 16:20 ` Chuck Lever
2007-08-02 18:42 ` Trond Myklebust
2007-08-02 20:46 ` Steve Dickson
0 siblings, 2 replies; 49+ messages in thread
From: Chuck Lever @ 2007-08-02 16:20 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
[-- Attachment #1: Type: text/plain, Size: 3244 bytes --]
Steve Dickson wrote:
> Chuck Lever wrote:
>> Steve Dickson wrote:
>>> Chuck Lever wrote:
>>>> I was looking at this yesterday. The stock timeout for TCP connects
>>>> on Linux is 75 seconds. The version of getport() used in the mount
>>>> command might control the TCP connect timeout by using a
>>>> non-blocking connect() with a select(). The select() then times out
>>>> if the connection doesn't complete.
>>>>
>>>> But I'm wondering if we really want to continue using TCP for
>>>> GETPORT calls. Solaris mount appears to use only UDP for GETPORT,
>>>> for example.
>>
>>> As as long as the GETPORTs don't use privilege ports I don't think its
>>> a problem...
>>
>> Not sure what you mean. Yesterday you said the TCP connect timeout
>> *was* a problem. I've recommended two ways to address it.
> TCP timeouts are a problem if you can't control them... But
> point taken... UPD is probably the best way to query a
> portmapper or rpcbinder to get the needed info...
OK, I have a patch that shortens the TCP connect timeout for mount.nfs.
Will post a follow-up; please take a look.
>> The ephemeral port space is limited too, don't forget. It's simply a
>> somewhat larger space than the privileged port space. If a large
>> network application (say, a web server) is running on the system, that
>> space can shrink fairly rapidly, and we're in nearly the same boat as
>> with privileged ports. Using a TCP connection from an ephemeral port
>> only mitigates the port space problem, it doesn't really correct it
>> entirely.
> Only mitigates the problem for a short time and you'll always run
> out of privileged port before running out of non-privileged but
> again... point taken... eliminating the problem is probably
> the answer...
Yes, and you've suggested a mount connection cache to help with this...
that might be something reasonable to try in the kernel mount
implementation at some point.
>> We say "firewall!" a lot, but I would like to see typical use cases
>> for mounting through a firewall so I understand what kind of
>> implementation we're aiming for (and maybe even what kind of test
>> cases to build!). Do our users really expect to mount NFS shares
>> through any firewall with "-o defaults" ?
> Yes! Mostly on the server side... meaning people wanted to set the
> port the daemons listen on (via the initscripts) so clients can
> access the server through a firewall... Is this a common setup?
> No. But there are people that want a firewall between the
> server and client..
I'm not suggesting that we don't support mounting through a firewall.
I'm wondering, though, how people expect it to work. Is it acceptable
to require a few extra mount options on clients to mount successfully
through a firewall, or should a mount with no options whatsoever always
work in this case?
And, does anyone have real and precise test cases to make sure we don't
break mounting through a firewall when changes are made to the mount
infrastructure?
> Also I can only assume the reason for the
> 'mountport=" option was to work better with firewalls...
> but that is only speculation...
I agree that the mount{prog,vers,port}= options are very likely for
mounting through firewalls.
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 290 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: Status of mount.nfs
2007-08-02 16:20 ` Chuck Lever
@ 2007-08-02 18:42 ` Trond Myklebust
2007-08-02 21:43 ` Chuck Lever
2007-08-02 20:46 ` Steve Dickson
1 sibling, 1 reply; 49+ messages in thread
From: Trond Myklebust @ 2007-08-02 18:42 UTC (permalink / raw)
To: chuck.lever; +Cc: nfs, Steve Dickson
On Thu, 2007-08-02 at 12:20 -0400, Chuck Lever wrote:
> I'm not suggesting that we don't support mounting through a firewall.
> I'm wondering, though, how people expect it to work. Is it acceptable
> to require a few extra mount options on clients to mount successfully
> through a firewall, or should a mount with no options whatsoever always
> work in this case?
The strategy should be to make the _default_ behaviour safe. If you want
to add optimisations that need switching on/off then those may take
extra mount options.
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-08-02 18:42 ` Trond Myklebust
@ 2007-08-02 21:43 ` Chuck Lever
2007-08-03 13:02 ` Trond Myklebust
0 siblings, 1 reply; 49+ messages in thread
From: Chuck Lever @ 2007-08-02 21:43 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs, Steve Dickson
[-- Attachment #1: Type: text/plain, Size: 721 bytes --]
Trond Myklebust wrote:
> On Thu, 2007-08-02 at 12:20 -0400, Chuck Lever wrote:
>
>> I'm not suggesting that we don't support mounting through a firewall.
>> I'm wondering, though, how people expect it to work. Is it acceptable
>> to require a few extra mount options on clients to mount successfully
>> through a firewall, or should a mount with no options whatsoever always
>> work in this case?
>
> The strategy should be to make the _default_ behaviour safe. If you want
> to add optimisations that need switching on/off then those may take
> extra mount options.
Exactly: is mounting through a firewall an optimization, or is it
something that users expect, by default, to work without using extra
options?
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 290 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-08-02 21:43 ` Chuck Lever
@ 2007-08-03 13:02 ` Trond Myklebust
0 siblings, 0 replies; 49+ messages in thread
From: Trond Myklebust @ 2007-08-03 13:02 UTC (permalink / raw)
To: chuck.lever; +Cc: nfs, Steve Dickson
On Thu, 2007-08-02 at 17:43 -0400, Chuck Lever wrote:
> Trond Myklebust wrote:
> > On Thu, 2007-08-02 at 12:20 -0400, Chuck Lever wrote:
> >
> >> I'm not suggesting that we don't support mounting through a firewall.
> >> I'm wondering, though, how people expect it to work. Is it acceptable
> >> to require a few extra mount options on clients to mount successfully
> >> through a firewall, or should a mount with no options whatsoever always
> >> work in this case?
> >
> > The strategy should be to make the _default_ behaviour safe. If you want
> > to add optimisations that need switching on/off then those may take
> > extra mount options.
>
> Exactly: is mounting through a firewall an optimization, or is it
> something that users expect, by default, to work without using extra
> options?
If the user has opened the firewall for outgoing TCP connections, then
that is one of the things that I think should just work.
It doesn't need to work with maximum efficiency (so if you want to try
sending UDP portmap requests before falling back to TCP, then fine) but
opening UDP ports in the firewall or specifying extra mount options
should not be a requirement.
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-08-02 16:20 ` Chuck Lever
2007-08-02 18:42 ` Trond Myklebust
@ 2007-08-02 20:46 ` Steve Dickson
1 sibling, 0 replies; 49+ messages in thread
From: Steve Dickson @ 2007-08-02 20:46 UTC (permalink / raw)
To: chuck.lever; +Cc: nfs
Chuck Lever wrote:
> I'm not suggesting that we don't support mounting through a firewall.
> I'm wondering, though, how people expect it to work. Is it acceptable
> to require a few extra mount options on clients to mount successfully
> through a firewall, or should a mount with no options whatsoever always
> work in this case?
I think I agree with Trond, although I I'm not sure what he means
by safe mount behavior... but I am of the opinion that mounts
which need to go through a firewall will require extra options
will be needed...
>
> And, does anyone have real and precise test cases to make sure we don't
> break mounting through a firewall when changes are made to the mount
> infrastructure?
We don't have explicit test for this... but I can guarantee if we
break it we'll hear about very quickly... ;-)
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-27 15:00 ` Steve Dickson
2007-07-27 15:56 ` Trond Myklebust
@ 2007-07-27 19:37 ` Chuck Lever
2007-07-28 13:20 ` Steve Dickson
1 sibling, 1 reply; 49+ messages in thread
From: Chuck Lever @ 2007-07-27 19:37 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
[-- Attachment #1: Type: text/plain, Size: 4691 bytes --]
Sorry about all the questions... and thanks for providing the history.
The good news is that, now that [u]mount.nfs resides in nfs-utils, we
can easily make this work a whole lot better.
Steve Dickson wrote:
> Chuck Lever wrote:
>>
>> And umount.nfs always uses TCP for the mountd request. I have a patch
>> that fixes that to behave more like mount.nfs does, which I will
>> forward in the next day or two.
> thats a bug... umount should use the protocol the mount did...
> I thought I had fixed that... :-\
Nope... umount.nfs sets the transport protocol to TCP explicitly before
doing the umount call. Check out utils/mount/nfsumount.c:_nfsumount() .
>> I notice some problems if a share is mounted with TCP, but the server
>> later disables TCP -- umount.nfs hiccups on that when it tries to
>> umount using the same protocol as listed in /etc/mtab. Perhaps
>> relying on /etc/mtab for setting the umount protocol is unnecessary.
> I think I was using /proc/mounts...
umount.nfs uses getmntdirbackward(), which probes /etc/mtab, as far as I
can tell. One problem with this is that often the effective transport
protocol isn't listed in /etc/mtab at all, if, say, the user requests
TCP and the server supports only UDP.
I can't see why we need to refer back to either file to determine the
transport protocol for a umount request. Whatever transport mountd is
advertising at the moment is what should be used, right?
[ Steve, since you have a different recollection of how all this mount
stuff works, I wonder if Amit took an older version of mount when he
split out the new mount.nfs helper... Can you verify this? Maybe there
are some fixes you made that need to be ported over. ]
>> We have three requests that need to be made:
>>
>> 1. GETPORT -- I think this should UDP all the time unless proto=tcp
>> is explicitly specified;
> Some people have asked that we first try UDP all the time... which
> I have resisted but it might make sense...
So far I've only found one reason to allow GETPORT over TCP, and that's
to go through firewalls. That's why I proposed allowing proto=tcp to
override the default... I'm very interested to hear other use cases that
might fail with UDP.
>> 2. MNT -- likewise, UDP unless proto=tcp is specified or GETPORT says
>> UDP is not supported;
>>
>> 3. NFS -- this should be TCP all the time unless proto=udp is
>> specified or GETPORT says TCP is not supported.
> What about rollbacks... meaning if tcp is not supported do we try udp?
> if v4 is not supported to we try v3 and the v2 or just fail the mount?
I think breaking back can be supported by grabbing all data about the
interesting services from portmapper at the start of a mount request.
That way mount.nfs can build the correct request based on the list of
services advertised on the server, and the list of options from the
mount command line, then make a single set of requests. No retry logic
is needed except for handling "bg".
>> Even better would be to use RPCB_DUMP instead of RPCB_GETPORT. That
>> way we only need a single rpcbind call for both protocols, and can get
>> transport protocol information as well, and make an "informed" choice.
> Good point... but note, a while back I got a request to use GETPORT
> instead of DUMP because some Cisco router actually use the GETPORTs
> to punch holes in their firewalls.
Sigh. ;-)
>> Also, can we get rid of the clnt_ping()? If not, can we document why
>> it is there? It adds two extra round trips to the whole process. If
>> error reporting is the problem, maybe we can try the pings only if the
>> kernel part of the mount process fails?
> How do we avoid hang down deep in RPC land (governed by
> uncontrollable timeout) when either mountd or nfsd are not up?
I guess I don't see how a NULL RPC is different than sending a real
request, when we're talking about a single MNT request from a user space
application. If the service is down, it fails either way.
> That was the main reason for the ping. Since neither portmapper or
> rpcbind ping their services before they hand out the ports, there
> is really no way of telling where the server is up? So to avoid
> the hang, we ping them... Sure its costly network wise, but
> hanging during a boot because a server is not responding is
> a bit more costly... imho...
My feeling is we should then fix the kernel to behave more reasonably.
I recently changed the kernel's rpcbind client to use "intr" instead of
"nointr" for its requests, for example. Is it practical to track down
the hangs and fix them? Is it just the long time waiting for a failure,
or do the mount processes actually get totally stuck?
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 259 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-27 19:37 ` Chuck Lever
@ 2007-07-28 13:20 ` Steve Dickson
2007-07-28 21:00 ` Chuck Lever
0 siblings, 1 reply; 49+ messages in thread
From: Steve Dickson @ 2007-07-28 13:20 UTC (permalink / raw)
To: chuck.lever; +Cc: nfs
Chuck Lever wrote:
> Sorry about all the questions... and thanks for providing the history.
> The good news is that, now that [u]mount.nfs resides in nfs-utils, we
> can easily make this work a whole lot better.
>
> Steve Dickson wrote:
>> Chuck Lever wrote:
>>>
>>> And umount.nfs always uses TCP for the mountd request. I have a
>>> patch that fixes that to behave more like mount.nfs does, which I
>>> will forward in the next day or two.
>> thats a bug... umount should use the protocol the mount did...
>> I thought I had fixed that... :-\
>
> Nope... umount.nfs sets the transport protocol to TCP explicitly before
> doing the umount call. Check out utils/mount/nfsumount.c:_nfsumount() .
>
>>> I notice some problems if a share is mounted with TCP, but the server
>>> later disables TCP -- umount.nfs hiccups on that when it tries to
>>> umount using the same protocol as listed in /etc/mtab. Perhaps
>>> relying on /etc/mtab for setting the umount protocol is unnecessary.
>> I think I was using /proc/mounts...
>
> umount.nfs uses getmntdirbackward(), which probes /etc/mtab, as far as I
> can tell. One problem with this is that often the effective transport
> protocol isn't listed in /etc/mtab at all, if, say, the user requests
> TCP and the server supports only UDP.
This got lost in the translation... In older mount code (i.e. the one
in utils-linux) /proc/mounts is used which is a much simpler way
of dealing with this... imho..
>
> I can't see why we need to refer back to either file to determine the
> transport protocol for a umount request. Whatever transport mountd is
> advertising at the moment is what should be used, right?
Well for firewall reasons you generally want to use the protocol
that the mount used...
>
> [ Steve, since you have a different recollection of how all this mount
> stuff works, I wonder if Amit took an older version of mount when he
> split out the new mount.nfs helper... Can you verify this? Maybe there
> are some fixes you made that need to be ported over. ]
No... I pretty sure I had Amit use the latest and greatest...
I just think there was some decisions made or liberties taken
without a complete understand of what the ramification were...
>
>>> 2. MNT -- likewise, UDP unless proto=tcp is specified or GETPORT
>>> says UDP is not supported;
>>>
>>> 3. NFS -- this should be TCP all the time unless proto=udp is
>>> specified or GETPORT says TCP is not supported.
>> What about rollbacks... meaning if tcp is not supported do we try udp?
>> if v4 is not supported to we try v3 and the v2 or just fail the mount?
>
> I think breaking back can be supported by grabbing all data about the
> interesting services from portmapper at the start of a mount request.
> That way mount.nfs can build the correct request based on the list of
> services advertised on the server, and the list of options from the
> mount command line, then make a single set of requests. No retry logic
> is needed except for handling "bg".
right...
>>> Also, can we get rid of the clnt_ping()? If not, can we document why
>>> it is there? It adds two extra round trips to the whole process. If
>>> error reporting is the problem, maybe we can try the pings only if
>>> the kernel part of the mount process fails?
>> How do we avoid hang down deep in RPC land (governed by
>> uncontrollable timeout) when either mountd or nfsd are not up?
>
> I guess I don't see how a NULL RPC is different than sending a real
> request, when we're talking about a single MNT request from a user space
> application. If the service is down, it fails either way.
As long as the request does not get caught up in some unreasonably
long timeout in the RPC code... there is no difference... Waiting
60sec for each retry or to find out some service is down would
not be a good thing when a machine is coming up...
>
>> That was the main reason for the ping. Since neither portmapper or
>> rpcbind ping their services before they hand out the ports, there
>> is really no way of telling where the server is up? So to avoid
>> the hang, we ping them... Sure its costly network wise, but
>> hanging during a boot because a server is not responding is
>> a bit more costly... imho...
>
> My feeling is we should then fix the kernel to behave more reasonably. I
> recently changed the kernel's rpcbind client to use "intr" instead of
> "nointr" for its requests, for example. Is it practical to track down
> the hangs and fix them?
In the kernel yes, in glibc no because that code will not
change, period!
> Is it just the long time waiting for a failure,
> or do the mount processes actually get totally stuck?
Its a long time wait that can not be controlled...
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-28 13:20 ` Steve Dickson
@ 2007-07-28 21:00 ` Chuck Lever
2007-07-29 19:24 ` Steve Dickson
0 siblings, 1 reply; 49+ messages in thread
From: Chuck Lever @ 2007-07-28 21:00 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
[-- Attachment #1: Type: text/plain, Size: 4551 bytes --]
Steve Dickson wrote:
> Chuck Lever wrote:
>> Steve Dickson wrote:
>>> Chuck Lever wrote:
>>>>
>>>> And umount.nfs always uses TCP for the mountd request. I have a
>>>> patch that fixes that to behave more like mount.nfs does, which I
>>>> will forward in the next day or two.
>>> thats a bug... umount should use the protocol the mount did...
>>> I thought I had fixed that... :-\
>>
>> Nope... umount.nfs sets the transport protocol to TCP explicitly
>> before doing the umount call. Check out
>> utils/mount/nfsumount.c:_nfsumount() .
>>
>>>> I notice some problems if a share is mounted with TCP, but the
>>>> server later disables TCP -- umount.nfs hiccups on that when it
>>>> tries to umount using the same protocol as listed in /etc/mtab.
>>>> Perhaps relying on /etc/mtab for setting the umount protocol is
>>>> unnecessary.
>>> I think I was using /proc/mounts...
>>
>> umount.nfs uses getmntdirbackward(), which probes /etc/mtab, as far as
>> I can tell. One problem with this is that often the effective
>> transport protocol isn't listed in /etc/mtab at all, if, say, the user
>> requests TCP and the server supports only UDP.
> This got lost in the translation... In older mount code (i.e. the one
> in utils-linux) /proc/mounts is used which is a much simpler way
> of dealing with this... imho..
Miklos seems intent on eliminating /etc/mtab anyway...
>> I can't see why we need to refer back to either file to determine the
>> transport protocol for a umount request. Whatever transport mountd is
>> advertising at the moment is what should be used, right?
> Well for firewall reasons you generally want to use the protocol
> that the mount used...
That could have been a very long time ago, even months, and the server
settings may have changed. Thus sending what mount used seems
inherently unreliable. The race window is enormous!
>> [ Steve, since you have a different recollection of how all this mount
>> stuff works, I wonder if Amit took an older version of mount when he
>> split out the new mount.nfs helper... Can you verify this? Maybe
>> there are some fixes you made that need to be ported over. ]
> No... I pretty sure I had Amit use the latest and greatest...
> I just think there was some decisions made or liberties taken
> without a complete understand of what the ramification were...
Thanks for checking on this. I worried we may have missed some
important bug fixes.
>>>> Also, can we get rid of the clnt_ping()? If not, can we document
>>>> why it is there? It adds two extra round trips to the whole
>>>> process. If error reporting is the problem, maybe we can try the
>>>> pings only if the kernel part of the mount process fails?
>>> How do we avoid hang down deep in RPC land (governed by
>>> uncontrollable timeout) when either mountd or nfsd are not up?
>>
>> I guess I don't see how a NULL RPC is different than sending a real
>> request, when we're talking about a single MNT request from a user
>> space application. If the service is down, it fails either way.
> As long as the request does not get caught up in some unreasonably
> long timeout in the RPC code... there is no difference... Waiting
> 60sec for each retry or to find out some service is down would
> not be a good thing when a machine is coming up...
>
>>
>>> That was the main reason for the ping. Since neither portmapper or
>>> rpcbind ping their services before they hand out the ports, there
>>> is really no way of telling where the server is up? So to avoid
>>> the hang, we ping them... Sure its costly network wise, but
>>> hanging during a boot because a server is not responding is
>>> a bit more costly... imho...
>>
>> My feeling is we should then fix the kernel to behave more reasonably.
>> I recently changed the kernel's rpcbind client to use "intr" instead
>> of "nointr" for its requests, for example. Is it practical to track
>> down the hangs and fix them?
> In the kernel yes, in glibc no because that code will not
> change, period!
Well, if libtirpc is added to nfs-utils, the mount command could use
that instead. We'd be able to fix any bugs in libtirpc quite easily.
That seems like an excellent way to address every problem with glibc's
RPC implementation, and immediately have a "simple" use case for testing
libtirpc (or whatever we have to replace the RPC functionality in glibc).
>> Is it just the long time waiting for a failure, or do the mount
>> processes actually get totally stuck?
> Its a long time wait that can not be controlled...
Ok.
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 259 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: Status of mount.nfs
2007-07-28 21:00 ` Chuck Lever
@ 2007-07-29 19:24 ` Steve Dickson
2007-07-30 4:14 ` Chuck Lever
0 siblings, 1 reply; 49+ messages in thread
From: Steve Dickson @ 2007-07-29 19:24 UTC (permalink / raw)
To: chuck.lever; +Cc: nfs
Chuck Lever wrote:
>>> umount.nfs uses getmntdirbackward(), which probes /etc/mtab, as far
>>> as I can tell. One problem with this is that often the effective
>>> transport protocol isn't listed in /etc/mtab at all, if, say, the
>>> user requests TCP and the server supports only UDP.
>> This got lost in the translation... In older mount code (i.e. the one
>> in utils-linux) /proc/mounts is used which is a much simpler way
>> of dealing with this... imho..
>
> Miklos seems intent on eliminating /etc/mtab anyway...
Good...
>
>>> I can't see why we need to refer back to either file to determine the
>>> transport protocol for a umount request. Whatever transport mountd
>>> is advertising at the moment is what should be used, right?
>> Well for firewall reasons you generally want to use the protocol
>> that the mount used...
>
> That could have been a very long time ago, even months, and the server
> settings may have changed. Thus sending what mount used seems
> inherently unreliable. The race window is enormous!
hmm... I must be missing something... Why is umount-ing with the
same network protocol that mount used unreliable and racy?
>
>>> [ Steve, since you have a different recollection of how all this
>>> mount stuff works, I wonder if Amit took an older version of mount
>>> when he split out the new mount.nfs helper... Can you verify this?
>>> Maybe there are some fixes you made that need to be ported over. ]
>> No... I pretty sure I had Amit use the latest and greatest...
>> I just think there was some decisions made or liberties taken
>> without a complete understand of what the ramification were...
>
> Thanks for checking on this. I worried we may have missed some
> important bug fixes.
A while back I did a patch dump of all the bugs we found when
we added the new code to Fedora... Neil's tree has all the patch
we have..
> Well, if libtirpc is added to nfs-utils, the mount command could use
> that instead. We'd be able to fix any bugs in libtirpc quite easily.
> That seems like an excellent way to address every problem with glibc's
> RPC implementation, and immediately have a "simple" use case for testing
> libtirpc (or whatever we have to replace the RPC functionality in glibc).
I can't agree with you more... At this point both rpcbind and libtirpc
are now fully supported by both Bull and yours truly... Both
tarballs are available on sourceforge:
http://sourceforge.net/projects/libtirpc/
http://sourceforge.net/projects/rpcbind/
Git trees are at:
http://git.infradead.org/?p=users/steved/libtirpc.git
http://git.infradead.org/?p=users/steved/rpcbind.git
And of course the rpms are available from Fedora mirrors
At this point everything is not quite synced up but that
will change very shortly... and new release will be coming
because the code is being used and bugs are being fixed...
The next major step would be to port nfs-utils to use libtirpc
which is on my todo list along with a ton of other things... :-\
In the end, I think it would be a very good move for our community
to own the entire stack... including the RPC library code...
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: Status of mount.nfs
2007-07-29 19:24 ` Steve Dickson
@ 2007-07-30 4:14 ` Chuck Lever
0 siblings, 0 replies; 49+ messages in thread
From: Chuck Lever @ 2007-07-30 4:14 UTC (permalink / raw)
To: Steve Dickson; +Cc: nfs
[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]
Steve Dickson wrote:
>>>> I can't see why we need to refer back to either file to determine
>>>> the transport protocol for a umount request. Whatever transport
>>>> mountd is advertising at the moment is what should be used, right?
>>> Well for firewall reasons you generally want to use the protocol
>>> that the mount used...
>>
>> That could have been a very long time ago, even months, and the server
>> settings may have changed. Thus sending what mount used seems
>> inherently unreliable. The race window is enormous!
> hmm... I must be missing something... Why is umount-ing with the
> same network protocol that mount used unreliable and racy?
Well, I exaggerated a bit, but technically speaking it is a race window
because the client is caching information about the server. When a
mount succeeds, the client remembers the transport protocol it used for
the mount in /etc/mtab.
A long time later, it uses that cached information to set the transport
protocol for the umount request.
During the time the share is mounted (ie the window), some system
administrator can change the server's settings -- for example, by
disabling the TCP mountd service, or by changing the iptables on the
server. At that point, the information in the client's /etc/mtab
becomes stale, and, in fact, useless as a hint.
I'm simply arguing that /etc/mtab should be ignored by umount entirely
when it comes to determining what transport protocol to use because that
cached information in not reliable. Just do a fresh GETPORT before the
umount, and you will be all set.
[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 315 bytes --]
begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
email;internet:chuck dot lever at nospam oracle dot com
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 315 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 49+ messages in thread