Text-based mount interface breaking non-UDP mounts

Text-based mount interface breaking non-UDP mounts

[Steinar H. Gunderson]

Hi,

I've gotten a few bug reports lately that if UDP is blocked at the server,
NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is
the change "Get rid of the "-i" option for mount.nfs[4] and always use the
text-based mount(2) system call for kernel version 2.6.23 and later." -- in
other words, the bug is specific to the text interface. Does anyone know why
this would be the case?

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: Text-based mount interface breaking non-UDP mounts

[Chuck Lever]

Hi Steinar-

On Apr 23, 2008, at 5:49 PM, Steinar H. Gunderson wrote:
> I've gotten a few bug reports lately that if UDP is blocked at the  
> server,
> NFS mounts (even with -o tcp) fail. git bisect shows that the  
> culprit is
> the change "Get rid of the "-i" option for mount.nfs[4] and always  
> use the
> text-based mount(2) system call for kernel version 2.6.23 and  
> later." -- in
> other words, the bug is specific to the text interface. Does anyone  
> know why
> this would be the case?

I specifically tested that when developing the text-based mount  
interface, so if it doesn't work, it's a bug or a missed corner case.

Start with "mount.nfs -v" to see if the mount command tells you  
anything interesting.  Then do a raw network trace to see what's  
failing during the mount version/protocol negotiation.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Text-based mount interface breaking non-UDP mounts

[Steinar H. Gunderson]

On Thu, Apr 24, 2008 at 10:12:58AM -0400, Chuck Lever wrote:
> Start with "mount.nfs -v" to see if the mount command tells you anything 
> interesting.  Then do a raw network trace to see what's failing during 
> the mount version/protocol negotiation.

There's nothing in -v except that it adds an addr= field. As for the
packet dump, that's the most interesting part -- for my part, the server
doesn't have iptables, so I'm blocking on the client. In that case, exactly
zero packets enter the write before mount gives up...

The other submitters have iptables blocking on the server, though, and it
doesn't work for them either (one is using NFSv3, the other NFSv4). Is this
really working for you?

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: Text-based mount interface breaking non-UDP mounts

[Chuck Lever]

On Apr 25, 2008, at 4:01 AM, Steinar H. Gunderson wrote:
> On Thu, Apr 24, 2008 at 10:12:58AM -0400, Chuck Lever wrote:
>> Start with "mount.nfs -v" to see if the mount command tells you  
>> anything
>> interesting.  Then do a raw network trace to see what's failing  
>> during
>> the mount version/protocol negotiation.
>
> There's nothing in -v except that it adds an addr= field. As for the
> packet dump, that's the most interesting part -- for my part, the  
> server
> doesn't have iptables, so I'm blocking on the client. In that case,  
> exactly zero packets enter the write before mount gives up...

The kernel may be returning an incorrect return code to the mount  
command.

Set the NFSDBG_MOUNT flag before you try your mount, then look in /var/ 
log/messages.  That will show you the kernel part of the mount  
process.  And/or use strace with the mount command.

> The other submitters have iptables blocking on the server, though,  
> and it
> doesn't work for them either (one is using NFSv3, the other NFSv4).  
> Is this really working for you?

I can't say until you post a complete description of a specific test  
case.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Text-based mount interface breaking non-UDP mounts

[Steinar H. Gunderson]

On Fri, Apr 25, 2008 at 10:19:54AM -0400, Chuck Lever wrote:
>> The other submitters have iptables blocking on the server, though, and it
>> doesn't work for them either (one is using NFSv3, the other NFSv4). Is
>> this really working for you?
> I can't say until you post a complete description of a specific test  
> case.

On the client:

  iptables -A OUTPUT -d 10.0.0.10 -p udp -j DROP
  mount -t nfs 10.0.0.10:/foo /bar

Substitute 10.0.0.10 with the server, of course.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: Text-based mount interface breaking non-UDP mounts

[Chuck Lever]

On Apr 25, 2008, at 10:27 AM, Steinar H. Gunderson wrote:
> On Fri, Apr 25, 2008 at 10:19:54AM -0400, Chuck Lever wrote:
>>> The other submitters have iptables blocking on the server, though,  
>>> and it
>>> doesn't work for them either (one is using NFSv3, the other  
>>> NFSv4). Is
>>> this really working for you?
>> I can't say until you post a complete description of a specific test
>> case.
>
> On the client:
>
>  iptables -A OUTPUT -d 10.0.0.10 -p udp -j DROP
>  mount -t nfs 10.0.0.10:/foo /bar
>
> Substitute 10.0.0.10 with the server, of course.

I just tried this specific use case.

Because of the local packet filtering on the client, the kernel's RPC  
client is getting -EPERM when trying to send the initial rpcbind  
request.

As far as I can see, nothing in the RPC client knows how to deal  
specifically with that error code, so it punts the request, and the  
mount fails.

When I originally tested mount protocol/version negotiation, I used  
only server-side filtering.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Text-based mount interface breaking non-UDP mounts

[Steinar H. Gunderson]

On Fri, Apr 25, 2008 at 06:13:03PM -0400, Chuck Lever wrote:
> I just tried this specific use case.
>
> Because of the local packet filtering on the client, the kernel's RPC  
> client is getting -EPERM when trying to send the initial rpcbind  
> request.
>
> As far as I can see, nothing in the RPC client knows how to deal  
> specifically with that error code, so it punts the request, and the  
> mount fails.
>
> When I originally tested mount protocol/version negotiation, I used only 
> server-side filtering.

I checked the original bug report, and indeed it seems that the blocking was
on the client there as well. I'll ask the user to check what happens when the
filtering is done server-side instead.

There's also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=476128, but I
believe that's a different bug.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: Text-based mount interface breaking non-UDP mounts

[Steinar H. Gunderson]

On Sat, Apr 26, 2008 at 01:55:46AM +0200, Steinar H. Gunderson wrote:
> I checked the original bug report, and indeed it seems that the blocking was
> on the client there as well. I'll ask the user to check what happens when the
> filtering is done server-side instead.

Hm, no, there's also reports of blocking on a firewall in the middle; see

  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473171

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: Text-based mount interface breaking non-UDP mounts

[Chuck Lever]

On Apr 25, 2008, at 7:57 PM, Steinar H. Gunderson wrote:
> On Sat, Apr 26, 2008 at 01:55:46AM +0200, Steinar H. Gunderson wrote:
>> I checked the original bug report, and indeed it seems that the  
>> blocking was
>> on the client there as well. I'll ask the user to check what  
>> happens when the
>> filtering is done server-side instead.
>
> Hm, no, there's also reports of blocking on a firewall in the  
> middle; see
>
>  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473171

The user is correct that the mount process for NFSv4 should not  
attempt to use the portmapper unless the port=0 option is specified.   
I will look into the problem.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Text-based mount interface breaking non-UDP mounts

[Chuck Lever]

On Apr 28, 2008, at 11:28 AM, Chuck Lever wrote:
> On Apr 25, 2008, at 7:57 PM, Steinar H. Gunderson wrote:
>> On Sat, Apr 26, 2008 at 01:55:46AM +0200, Steinar H. Gunderson wrote:
>>> I checked the original bug report, and indeed it seems that the  
>>> blocking was
>>> on the client there as well. I'll ask the user to check what  
>>> happens when the
>>> filtering is done server-side instead.
>>
>> Hm, no, there's also reports of blocking on a firewall in the  
>> middle; see
>>
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473171
>
> The user is correct that the mount process for NFSv4 should not  
> attempt to use the portmapper unless the port=0 option is  
> specified.  I will look into the problem.


This issue may be addressed as a side-effect of:

commit f22d6d79fe227245363a8849ea8c85fe6c6598c3
Author: Chuck Lever <chuck.lever@oracle.com>
Date:   Fri Mar 14 14:10:22 2008 -0400

     NFS: Save the value of the "port=" mount option

     During a remount based on the mount options displayed in
     /proc/mounts, we want to preserve the original behavior of
     the mount request.  Let's save the original setting of the
     "port=" mount option in the mount's nfs_server structure.


Please give 2.6.25 final a try to see if it prevents the noted  
extraneous rpcbind activity during an NFSv4 mount.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Text-based mount interface breaking non-UDP mounts

[Chuck Lever]

On Apr 25, 2008, at 7:55 PM, Steinar H. Gunderson wrote:
> On Fri, Apr 25, 2008 at 06:13:03PM -0400, Chuck Lever wrote:
>> I just tried this specific use case.
>>
>> Because of the local packet filtering on the client, the kernel's RPC
>> client is getting -EPERM when trying to send the initial rpcbind
>> request.
>>
>> As far as I can see, nothing in the RPC client knows how to deal
>> specifically with that error code, so it punts the request, and the
>> mount fails.
>>
>> When I originally tested mount protocol/version negotiation, I used  
>> only
>> server-side filtering.
>
> I checked the original bug report, and indeed it seems that the  
> blocking was
> on the client there as well. I'll ask the user to check what happens  
> when the filtering is done server-side instead.
>
> There's also http://bugs.debian.org/cgi-bin/bugreport.cgi? 
> bug=476128, but I believe that's a different bug.


FYI: Steve Dickson (nfs-utils maintainer) is already looking at  
removing EACCES from the list of temporary errors, so yes, we're aware  
of that issue.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Text-based mount interface breaking non-UDP mounts

[Steve Dickson]

Chuck Lever wrote:
> On Apr 25, 2008, at 7:55 PM, Steinar H. Gunderson wrote:
>> On Fri, Apr 25, 2008 at 06:13:03PM -0400, Chuck Lever wrote:
>>> I just tried this specific use case.
>>>
>>> Because of the local packet filtering on the client, the kernel's RPC
>>> client is getting -EPERM when trying to send the initial rpcbind
>>> request.
>>>
>>> As far as I can see, nothing in the RPC client knows how to deal
>>> specifically with that error code, so it punts the request, and the
>>> mount fails.
>>>
>>> When I originally tested mount protocol/version negotiation, I used only
>>> server-side filtering.
>>
>> I checked the original bug report, and indeed it seems that the
>> blocking was
>> on the client there as well. I'll ask the user to check what happens
>> when the filtering is done server-side instead.
>>
>> There's also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=476128,
>> but I believe that's a different bug.
> 
> 
> FYI: Steve Dickson (nfs-utils maintainer) is already looking at removing
> EACCES from the list of temporary errors, so yes, we're aware of that
> issue.
Actually it has been removed already...

steved.

Re: Text-based mount interface breaking non-UDP mounts

[Frank A. Kingswood]

Steinar H. Gunderson wrote:
> Hi,
> 
> I've gotten a few bug reports lately that if UDP is blocked at the server,
> NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is
> the change "Get rid of the "-i" option for mount.nfs[4] and always use the
> text-based mount(2) system call for kernel version 2.6.23 and later." -- in
> other words, the bug is specific to the text interface. Does anyone know why
> this would be the case?
This also fails for me. My setup is through an ssh tunnel
    ssh -L2049:srv:2049 -L2050:srv:2050 -L2051:srv:2051ser@gateway

and I follow that with
    strace mount localhost:/export /mountpoint \
              -o nfsvers=3,port=2049,mountport=2050,...

This hangs in this kernel call
   mount("localhost:/export", "/mountpoint", "nfs", 
MS_NOSUID|MS_NOATIME, 
"tcp,intr,fg,soft,nfsvers=3,port=2049,mountport=2050,nolock,addr=127.0.0.1"

This hack makes it work for me again:

diff -ru nfs-utils-1.1.2/utils/mount/mount.c 
nfs-utils-1.1.2-changed/utils/mount/mount.c
--- nfs-utils-1.1.2/utils/mount/mount.c	2008-03-14 15:46:29 +0000
+++ nfs-utils-hack/utils/mount/mount.c	2008-04-26 15:49:55 +0100
@@ -191,7 +191,8 @@
  		nfs_mount_data_version = NFS_MOUNT_VERSION;
  	else
  		if (kernel_version > MAKE_VERSION(2, 6, 22))
-			string++;
+			string=0;
  }

  static void print_one(char *spec, char *node, char *type, char *opts)

Frank

Re: Text-based mount interface breaking non-UDP mounts

[Frank A. Kingswood]

Frank A. Kingswood wrote:
>    ssh -L2049:srv:2049 -L2050:srv:2050 -L2051:srv:2051ser@gateway

sorry, duff mailer and message wrapping got that. it should say

ssh -L2049:srv:2049 -L2050:srv:2050 -L2051:srv:2051 user@gateway

Re: Text-based mount interface breaking non-UDP mounts

[Frank A. Kingswood]

Steinar H. Gunderson wrote:
> I've gotten a few bug reports lately that if UDP is blocked at the server,
> NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is
> the change "Get rid of the "-i" option for mount.nfs[4] and always use the
> text-based mount(2) system call for kernel version 2.6.23 and later." -- in
> other words, the bug is specific to the text interface. Does anyone know why
> this would be the case?

The NFS client code (nfs-utils-1.1.2) tests for
   if (kernel_version > MAKE_VERSION(2, 6, 22))

yet the kernel patch for NFS did not go un until 2.6.25-rc2

> From: Linus Torvalds <torvalds@...>
> Subject: Linux 2.6.25-rc2
> Date: Feb 15, 5:23 pm 2008
> 
> Ok,
>  this kernel is a winner.
> 
> [...]
> 
> Chuck Lever (1):
>       NFS: Allow text-based mounts via compat_sys_mount
> 

So the test should at least be > MAKE_VERSION(2, 6, 24).

Frank

Re: Text-based mount interface breaking non-UDP mounts

[Chuck Lever]

On Apr 26, 2008, at 11:38 AM, Frank A. Kingswood wrote:
> Steinar H. Gunderson wrote:
>> I've gotten a few bug reports lately that if UDP is blocked at the  
>> server,
>> NFS mounts (even with -o tcp) fail. git bisect shows that the  
>> culprit is
>> the change "Get rid of the "-i" option for mount.nfs[4] and always  
>> use the
>> text-based mount(2) system call for kernel version 2.6.23 and  
>> later." -- in
>> other words, the bug is specific to the text interface. Does anyone  
>> know why
>> this would be the case?
>
> The NFS client code (nfs-utils-1.1.2) tests for
>  if (kernel_version > MAKE_VERSION(2, 6, 22))
>
> yet the kernel patch for NFS did not go un until 2.6.25-rc2
>
>> From: Linus Torvalds <torvalds@...>
>> Subject: Linux 2.6.25-rc2
>> Date: Feb 15, 5:23 pm 2008
>> Ok,
>> this kernel is a winner.
>> [...]
>> Chuck Lever (1):
>>      NFS: Allow text-based mounts via compat_sys_mount
>
> So the test should at least be > MAKE_VERSION(2, 6, 24).

Frank, the patch you refer to fixes a bug with the compat_sys_mount  
interface, which is separate from the sys_mount interface, and used  
only on a handful of hardware platforms (SPARC?).

The original NFS text-based mount support was added in 2.6.23.

I don't have a problem with adjusting MAKE_VERSION in the master nfs- 
utils repo as you describe, since that bug does break text-based  
mounts on those few hardware platforms that use compat_sys_mount  
instead of sys_mount.  It might affect some early adopters of NFS over  
RDMA or FS cache, however.

On distributions that don't support NFS over RDMA or the FS cache  
facility, it should be harmless for now to make the mount command use  
the legacy mount API even on newer kernels.  Simply adjust the  
MAKE_VERSION() macro as needed, or disable it entirely as you did with  
your "string = 0" hack.

New features will be added only to the text-based mount interface,  
however, so eventually everyone will need to use the text-based mount  
interface on new kernels.

Because we have little documented history and only a handful of use  
cases and unit tests for the mount command, it's important for  
everyone to test the new API and report problems here so we can  
address them.  NFS mount is complex and has many subtle and hidden  
historical behaviors.

Even if 70-80% of the common use cases are working properly, there are  
going to be outliers that will experience issues until we can get the  
corner cases fixed.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Text-based mount interface breaking non-UDP mounts

[Frank A. Kingswood]

Chuck Lever wrote:
>>
>> The NFS client code (nfs-utils-1.1.2) tests for
>>  if (kernel_version > MAKE_VERSION(2, 6, 22))
>>
>> yet the kernel patch for NFS did not go un until 2.6.25-rc2
>> So the test should at least be > MAKE_VERSION(2, 6, 24).
> The original NFS text-based mount support was added in 2.6.23.
OK.
> New features will be added only to the text-based mount interface, 
> however, so eventually everyone will need to use the text-based mount 
> interface on new kernels.
That sounds like a good idea.
> Because we have little documented history and only a handful of use 
> cases and unit tests for the mount command, it's important for 
> everyone to test the new API and report problems here so we can 
> address them.  NFS mount is complex and has many subtle and hidden 
> historical behaviors.
My system fails with 2.6.23 and 2.6.25 kernels. As I described in my 
original mail, I only have TCP access to the server, and only through an 
ssh tunnel.

Would you like me to try and produce some debug traces, or do you think 
you can find the problem with the information you have already? I've 
seen quite a few reports of this now.

Frank

-- 
------------------------------------------------------------------------
Frank A. Kingswood                      frank-YzJE6wl5KmBjuoFY8sQqqY5Lr7LD32fptUK59QYPAWc@public.gmane.org
Cambridge, United Kingdom                               +44-7943-217 571

Re: Text-based mount interface breaking non-UDP mounts

[Chuck Lever]

On Apr 28, 2008, at 2:24 PM, Frank A. Kingswood wrote:
> Chuck Lever wrote:
>>>
>>> The NFS client code (nfs-utils-1.1.2) tests for
>>> if (kernel_version > MAKE_VERSION(2, 6, 22))
>>>
>>> yet the kernel patch for NFS did not go un until 2.6.25-rc2
>>> So the test should at least be > MAKE_VERSION(2, 6, 24).
>> The original NFS text-based mount support was added in 2.6.23.
> OK.
>> New features will be added only to the text-based mount interface,  
>> however, so eventually everyone will need to use the text-based  
>> mount interface on new kernels.
> That sounds like a good idea.
>> Because we have little documented history and only a handful of use  
>> cases and unit tests for the mount command, it's important for  
>> everyone to test the new API and report problems here so we can  
>> address them.  NFS mount is complex and has many subtle and hidden  
>> historical behaviors.
> My system fails with 2.6.23 and 2.6.25 kernels. As I described in my  
> original mail, I only have TCP access to the server, and only  
> through an ssh tunnel.
>
> Would you like me to try and produce some debug traces, or do you  
> think you can find the problem with the information you have  
> already? I've seen quite a few reports of this now.

Thanks for the offer.  I tested against a server configuration that  
dropped UDP packets, but I never tested through an ssh tunnel.

Network traces might be a little hard with a tunnel, but give it a  
shot.  The best information comes from a full-packet raw trace.  Use  
the "-s0" option on tcpdump.

You can also trace NFS mount processing with:

# sysctl -w sunrpc.nfs_debug = 128

Or trace all RPC activity (including rpcbind and RPC network transport  
errors) with:

# sysctl -w sunrpc.rpc_debug = 32767

You can set both of these at the same time.  Trace output will appear  
in /var/log/messages.  To disable tracing when you're done, set these  
sysctls to zero.

You can post these to me privately.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Text-based mount interface breaking non-UDP mounts

[vincent]

> Chuck Lever wrote:
> Thanks for the offer.  I tested against a server configuration that  
> dropped UDP packets, but I never tested through an ssh tunnel.

In the ssh tunnel case, what can be seen in network captures for
localhost on the client is that mounting requests are sent over UDP, and
since the ports are those specified for the ssh tunnel, they lead to
nothing on localhost UDP, and an ICMP dest unreach (port) is sent back.
This is made twice before mount.nfs exits with an internal error, 2
packets each time.

I have the same problem than Frank, using NFS through a ssh tunnel. It
was working with 2.6.20 and I've just upgraded to 2.6.25 and it does not
works anymore.


Vincent

Re: Text-based mount interface breaking non-UDP mounts

[Frank A. Kingswood]

vincent wrote:
> I have the same problem than Frank, using NFS through a ssh tunnel. It
> was working with 2.6.20 and I've just upgraded to 2.6.25 and it does not
> works anymore.

Chuck sent me a workaround. The kernel understands an option that is not 
present in the userland mount code, mountproto. Set that along with the 
other options, so -o tcp,mountproto=tcp,... and it will force the mount 
call to go over tcp too.

Alternatively, you can grab the source for mount.nfs and disable 
text-based mounts by setting string=0, or by changing the kernel version 
it tests for to >2.6.25.

I think a fix for the kernel is in the works, so that should appear by 
2.6.26 and possibly for a 2.6.25.y too.

Frank

Re: Text-based mount interface breaking non-UDP mounts

[Chuck Lever]

On May 4, 2008, at 6:39 PM, vincent wrote:
>> Chuck Lever wrote:
>> Thanks for the offer.  I tested against a server configuration that
>> dropped UDP packets, but I never tested through an ssh tunnel.
>
> In the ssh tunnel case, what can be seen in network captures for
> localhost on the client is that mounting requests are sent over UDP,  
> and
> since the ports are those specified for the ssh tunnel, they lead to
> nothing on localhost UDP, and an ICMP dest unreach (port) is sent  
> back.
> This is made twice before mount.nfs exits with an internal error, 2
> packets each time.
>
> I have the same problem than Frank, using NFS through a ssh tunnel. It
> was working with 2.6.20 and I've just upgraded to 2.6.25 and it does  
> not
> works anymore.

After 2.6.23, mount options are passed to the kernel as a C string  
instead of binary data structure.  The mount command formerly  
contacted the server via the mountd protocol, and passed mount  
information to the kernel.  Now the kernel has its own mountd client  
and it uses that instead.

The new kernel mount option parser behaves a little differently than  
the user-space mount command with regard to the "proto=" option.  If  
you specify "proto=tcp" the legacy mount command contacts the server's  
mountd via TCP.  However, the kernel continues to use the default  
setting (UDP) for the mount client's transport protocol.

This is a bug, and I have a kernel patch that makes the kernel mount  
option parser behave like the legacy mount command.

Until this bug is addressed, you can use "mountproto=" to specify the  
transport protocol explicitly for the kernel's mount client.  See  
nfs(5) for details.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com