Problems mounting via UDP from a netapp with multiple interfaces

Problems mounting via UDP from a netapp with multiple interfaces

[Gregory Boyce]

Folks,

I've been encountering a problem with NFS clients attempting to mount
from a netapp via UDP where the netapp is responding on the wrong
interface.  On some of our older systems, this mount worked properly,
while on newer systems nfs-utils ends up failing the mount.  Mounting
via TCP works fine.

It appears that there has been various related discussions over the
years, and a relevant Redhat bug opened back in 2006:

http://article.gmane.org/gmane.linux.nfs/22778/match=connect+udp
https://bugzilla.redhat.com/show_bug.cgi?id=208244

Is there a general recommendation for people in this sort of
situation?  I'm assuming the code is currently using connected UDP
sockets (I'm more of a sysadmin than a developer).  Is there an option
I'm missing to disable this?  Otherwise does anyone know of a patch to
change the behavior?

-- 
Greg

Re: Problems mounting via UDP from a netapp with multiple interfaces

[Ben Greear]

On 04/09/2015 12:34 PM, Gregory Boyce wrote:
> Folks,
> 
> I've been encountering a problem with NFS clients attempting to mount
> from a netapp via UDP where the netapp is responding on the wrong
> interface.  On some of our older systems, this mount worked properly,
> while on newer systems nfs-utils ends up failing the mount.  Mounting
> via TCP works fine.
> 
> It appears that there has been various related discussions over the
> years, and a relevant Redhat bug opened back in 2006:
> 
> http://article.gmane.org/gmane.linux.nfs/22778/match=connect+udp
> https://bugzilla.redhat.com/show_bug.cgi?id=208244
> 
> Is there a general recommendation for people in this sort of
> situation?  I'm assuming the code is currently using connected UDP
> sockets (I'm more of a sysadmin than a developer).  Is there an option
> I'm missing to disable this?  Otherwise does anyone know of a patch to
> change the behavior?

I have some patches that allow binding an NFS client to a particular
local IP.  You need modified mount.nfs tools as well.  These patches
might fix your problem, but I am not certain about that.

My kernel trees have various other patches as well...you can pick out just
the NFS stuff if you care to.  Otherwise, the kernel should generally build,
install, and work the same as upstream kernels.

https://github.com/greearb/nfs-utils-ct

# This one has cleaner patch set, but not much testing.
http://dmz2.candelatech.com/git/gitweb.cgi?p=linux-3.19.dev.y/.git;a=summary

# This is lots of extraneous wifi patches, but has had good testing,
# including the nfs bind-to-local-IP feature.

http://dmz2.candelatech.com/git/gitweb.cgi?p=linux-3.17.dev.y/.git;a=summary


Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Problems mounting via UDP from a netapp with multiple interfaces

[Gregory Boyce]

On Thu, Apr 9, 2015 at 5:08 PM, Ben Greear <greearb@candelatech.com> wrote:
> On 04/09/2015 12:34 PM, Gregory Boyce wrote:
>> Folks,
>>
>> I've been encountering a problem with NFS clients attempting to mount
>> from a netapp via UDP where the netapp is responding on the wrong
>> interface.  On some of our older systems, this mount worked properly,
>> while on newer systems nfs-utils ends up failing the mount.  Mounting
>> via TCP works fine.
>>
>> It appears that there has been various related discussions over the
>> years, and a relevant Redhat bug opened back in 2006:
>>
>> http://article.gmane.org/gmane.linux.nfs/22778/match=connect+udp
>> https://bugzilla.redhat.com/show_bug.cgi?id=208244
>>
>> Is there a general recommendation for people in this sort of
>> situation?  I'm assuming the code is currently using connected UDP
>> sockets (I'm more of a sysadmin than a developer).  Is there an option
>> I'm missing to disable this?  Otherwise does anyone know of a patch to
>> change the behavior?
>
> I have some patches that allow binding an NFS client to a particular
> local IP.  You need modified mount.nfs tools as well.  These patches
> might fix your problem, but I am not certain about that.

Re-reading your e-mail, I'm not sure this will help me.   The problem
I'm having is that the server sends responses from a different IP
address than I attempted to mount.  Your description there seems to be
talking about selecting a local IP address to do the mounting with
instead.

For what it's worth, nfs-utils 1.1.2 was the version that successfully
mounts while 1.2.5 is the one I'm currently struggling with.

-- 
Greg

Re: Problems mounting via UDP from a netapp with multiple interfaces

[Ben Greear]

On 04/10/2015 11:22 AM, Gregory Boyce wrote:
> On Thu, Apr 9, 2015 at 5:08 PM, Ben Greear <greearb@candelatech.com> wrote:
>> On 04/09/2015 12:34 PM, Gregory Boyce wrote:
>>> Folks,
>>>
>>> I've been encountering a problem with NFS clients attempting to mount
>>> from a netapp via UDP where the netapp is responding on the wrong
>>> interface.  On some of our older systems, this mount worked properly,
>>> while on newer systems nfs-utils ends up failing the mount.  Mounting
>>> via TCP works fine.
>>>
>>> It appears that there has been various related discussions over the
>>> years, and a relevant Redhat bug opened back in 2006:
>>>
>>> http://article.gmane.org/gmane.linux.nfs/22778/match=connect+udp
>>> https://bugzilla.redhat.com/show_bug.cgi?id=208244
>>>
>>> Is there a general recommendation for people in this sort of
>>> situation?  I'm assuming the code is currently using connected UDP
>>> sockets (I'm more of a sysadmin than a developer).  Is there an option
>>> I'm missing to disable this?  Otherwise does anyone know of a patch to
>>> change the behavior?
>>
>> I have some patches that allow binding an NFS client to a particular
>> local IP.  You need modified mount.nfs tools as well.  These patches
>> might fix your problem, but I am not certain about that.
> 
> Re-reading your e-mail, I'm not sure this will help me.   The problem
> I'm having is that the server sends responses from a different IP
> address than I attempted to mount.  Your description there seems to be
> talking about selecting a local IP address to do the mounting with
> instead.
> 
> For what it's worth, nfs-utils 1.1.2 was the version that successfully
> mounts while 1.2.5 is the one I'm currently struggling with.


Ok, I thought maybe you were using two interfaces on your client and the
request was coming down the wrong interface due to the client selecting
the wrong source address when sending the original request.

But if it is purely server-side issue, then yes, my patches are unlikely
to help anything.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Problems mounting via UDP from a netapp with multiple interfaces

[Trond Myklebust]

On Thu, Apr 9, 2015 at 3:34 PM, Gregory Boyce <gregory.boyce@gmail.com> wrote:
> Folks,
>
> I've been encountering a problem with NFS clients attempting to mount
> from a netapp via UDP where the netapp is responding on the wrong
> interface.  On some of our older systems, this mount worked properly,
> while on newer systems nfs-utils ends up failing the mount.  Mounting
> via TCP works fine.
>
> It appears that there has been various related discussions over the
> years, and a relevant Redhat bug opened back in 2006:
>
> http://article.gmane.org/gmane.linux.nfs/22778/match=connect+udp
> https://bugzilla.redhat.com/show_bug.cgi?id=208244
>
> Is there a general recommendation for people in this sort of
> situation?  I'm assuming the code is currently using connected UDP
> sockets (I'm more of a sysadmin than a developer).  Is there an option
> I'm missing to disable this?  Otherwise does anyone know of a patch to
> change the behavior?
>

This is a server bug.. Those are not fixable on the client.

Trond

Re: Problems mounting via UDP from a netapp with multiple interfaces

[Trond Myklebust]

On Thu, Apr 9, 2015 at 9:20 PM, Gregory Boyce <gregory.boyce@gmail.com> wrote:
> On Thu, Apr 9, 2015, 8:23 PM Trond Myklebust
> <trond.myklebust@primarydata.com> wrote:
>
> On Thu, Apr 9, 2015 at 3:34 PM, Gregory Boyce <gregory.boyce@gmail.com>
> wrote:
>> Folks,
>>
>> I've been encountering a problem with NFS clients attempting to mount
>> from a netapp via UDP where the netapp is responding on the wrong
>> interface.  On some of our older systems, this mount worked properly,
>> while on newer systems nfs-utils ends up failing the mount.  Mounting
>> via TCP works fine.
>
>
> This is a server bug.. Those are not fixable on the client.
>
> Trond
>
>
>
> Since the clients are successfully mounting the filers right now with much
> older client software, it seems to me that it can at least be worked around
> on the client side.
>

No. You are not supposed to be able to work around security issues,
and it is indeed a security issue when a client gets a reply from an
IP address that it does not recognise as being the same as the one it
sent an RPC to.

NetApp is aware of this bug, and has had burts open for it for at
least a decade now. Have you tried contacting them for a fix?

Trond

Re: Problems mounting via UDP from a netapp with multiple interfaces

[Gregory Boyce]

On Fri, Apr 10, 2015 at 2:45 PM, Trond Myklebust
<trond.myklebust@primarydata.com> wrote:

> No. You are not supposed to be able to work around security issues,
> and it is indeed a security issue when a client gets a reply from an
> IP address that it does not recognise as being the same as the one it
> sent an RPC to.

"Working around" security issues is a rather common and accepted
practice when there are mitigating controls in place.  It's never a
black and white world.

> NetApp is aware of this bug, and has had burts open for it for at
> least a decade now. Have you tried contacting them for a fix?

My team is entirely involved in the client side.  I'll see what
options the team responsible for the filer have there.

-- 
Greg

Re: Problems mounting via UDP from a netapp with multiple interfaces

[Gregory Boyce]

[-- Attachment #1: Type: text/plain, Size: 1035 bytes --]

On Tue, Apr 14, 2015 at 3:37 PM, Gregory Boyce <gregory.boyce@gmail.com> wrote:
> On Fri, Apr 10, 2015 at 3:04 PM Gregory Boyce <gregory.boyce@gmail.com>
> wrote:
>>
>> On Fri, Apr 10, 2015 at 2:45 PM, Trond Myklebust
>> <trond.myklebust@primarydata.com> wrote:
>>
>> > No. You are not supposed to be able to work around security issues,
>> > and it is indeed a security issue when a client gets a reply from an
>> > IP address that it does not recognise as being the same as the one it
>> > sent an RPC to.
>>
>> "Working around" security issues is a rather common and accepted
>> practice when there are mitigating controls in place.  It's never a
>> black and white world.
>>
>
>
> The attached patch was able to work around the issue for us until we can get
> the filers working in a more expected manner.  I'm sending it along in case
> anyone else can find a use for it, or if you want to apply it in order to
> give people an option for cases like this.

Re-sending since Google Inbox likes to default to HTML e-mail.

-- 
Greg

[-- Attachment #2: nfs-utils_norewriteopts.diff --]
[-- Type: text/plain, Size: 596 bytes --]

diff -ru nfs-utils-1.2.5.orig/utils/mount/stropts.c nfs-utils-1.2.5/utils/mount/stropts.c
--- nfs-utils-1.2.5.orig/utils/mount/stropts.c	2015-04-13 22:43:20.000000000 +0000
+++ nfs-utils-1.2.5/utils/mount/stropts.c	2015-04-13 22:47:30.000000000 +0000
@@ -497,6 +497,14 @@
 	struct pmap mnt_pmap;
 
 	/*
+	 * "norewriteopts" option bypasses the options rewriting
+	 */
+	if (po_contains(options, "norewriteopts") == PO_FOUND) {
+		po_remove_all(options, "norewriteopts");
+		return 1;
+	}
+
+	/*
 	 * Version and transport negotiation is not required
 	 * and does not work for RDMA mounts.
 	 */

Re: Problems mounting via UDP from a netapp with multiple interfaces

[Steve Dickson]

On 04/14/2015 03:39 PM, Gregory Boyce wrote:
> On Tue, Apr 14, 2015 at 3:37 PM, Gregory Boyce <gregory.boyce@gmail.com> wrote:
>> On Fri, Apr 10, 2015 at 3:04 PM Gregory Boyce <gregory.boyce@gmail.com>
>> wrote:
>>>
>>> On Fri, Apr 10, 2015 at 2:45 PM, Trond Myklebust
>>> <trond.myklebust@primarydata.com> wrote:
>>>
>>>> No. You are not supposed to be able to work around security issues,
>>>> and it is indeed a security issue when a client gets a reply from an
>>>> IP address that it does not recognise as being the same as the one it
>>>> sent an RPC to.
>>>
>>> "Working around" security issues is a rather common and accepted
>>> practice when there are mitigating controls in place.  It's never a
>>> black and white world.
>>>
>>
>>
>> The attached patch was able to work around the issue for us until we can get
>> the filers working in a more expected manner.  I'm sending it along in case
>> anyone else can find a use for it, or if you want to apply it in order to
>> give people an option for cases like this.
> 
> Re-sending since Google Inbox likes to default to HTML e-mail.
> 
Could you please resend this patch using the proper Sign-off-by,
subject and description formats as describe in 
   https://www.kernel.org/doc/Documentation/SubmittingPatches

steved.

Re: Problems mounting via UDP from a netapp with multiple interfaces

[Malahal Naineni]

Gregory Boyce [gregory.boyce@gmail.com] wrote:
> Folks,
> 
> I've been encountering a problem with NFS clients attempting to mount
> from a netapp via UDP where the netapp is responding on the wrong
> interface.  On some of our older systems, this mount worked properly,
> while on newer systems nfs-utils ends up failing the mount.  Mounting
> via TCP works fine.
> 
> It appears that there has been various related discussions over the
> years, and a relevant Redhat bug opened back in 2006:
> 
> http://article.gmane.org/gmane.linux.nfs/22778/match=connect+udp
> https://bugzilla.redhat.com/show_bug.cgi?id=208244
> 
> Is there a general recommendation for people in this sort of
> situation?  I'm assuming the code is currently using connected UDP
> sockets (I'm more of a sysadmin than a developer).  Is there an option
> I'm missing to disable this?  Otherwise does anyone know of a patch to
> change the behavior?

Just an FYI, I encountered this issue on user space ganesha NFS server.
Kernel NFS server uses PKTINFO to account for this. I fixed ganesha NFS
server as well to do the same!

Regards, Malahal.