possible better error handling for nfsv4 and localhost maps

possible better error handling for nfsv4 and localhost maps

[Dave Sullivan]

Hi All,

Recently we had a customer with some legacy maps using localhost run 
into a problem with NFSv4.

They are not quite ready to get rid of those localhost maps.

There is a fairly straight forward workaround/solution to get around 
this issue by modifying /etc/nfsmount.conf with

[ Server "localhost" ]
    Nfsvers=3

What we are asking for is better error handling so users can figure out 
what the problem is without having to debug as much.

i.e. Have the code not retry and have the code send a notification to 
syslog saying something like "nfsv4 does not support localhost maps, man 
nfsmount.conf, use [ Server "localhost" ] with Nfsvers=3 mount option to 
restrict localhost maps to NFSv3"

Here's an example of effort required to debug

Description of problem:

The RHEL automounter does try to do what’s inferred in the above 
notation, ie, make a bind mount.  What happens next, when the local 
filesystem doesn’t exist, is what’s causing us a problem.  It next 
spawns a mount request to an NFS service on localhost for the mount, 
either via IPV6, or via IPV4;  end-user hosts don’t serve NFS, so the 
problem arises due to how this is handled.

On RHEL5, the mount request contacts the portmapper, which reports there 
is no program registered for NFS service, and the mount fails straight away.

On RHEL6, if /etc/hosts has an entry for the IPV6 localhost address, an 
NFS mount from IPV6 localhost gets tried first, which fails immediately 
as expected, as IPV6 is not enabled, and no NFS service is registered on 
the host.  Automount reports the mount failure straight away.
Hi All,

Recently we had a customer with some legacy maps using localhost run 
into a problem with NFSv4.

They are not quite ready to get rid of those localhost maps.

There is a fairly straight forward workaround/solution to get around 
this issue by modifying /etc/nfsmount.conf with

[ Server "localhost" ]
    Nfsvers=3

What we are asking for is better error handling so users can figure out 
what the problem is without having to debug as much.

i.e. Have the code not retry and have the code send a notification to 
syslog saying something like "nfsv4 does not support localhost maps, man 
nfsmount.conf, use [ Server "localhost" ] with Nfsvers=3 mount option to 
restrict localhost maps to NFSv3"

Here's an example of effort required to debug

Description of problem:

The RHEL automounter does try to do what’s inferred in the above 
notation, ie, make a bind mount.  What happens next, when the local 
filesystem doesn’t exist, is what’s causing us a problem.  It next 
spawns a mount request to an NFS service on localhost for the mount, 
either via IPV6, or via IPV4;  end-user hosts don’t serve NFS, so the 
problem arises due to how this is handled.

On RHEL5, the mount request contacts the portmapper, which reports there 
is no program registered for NFS service, and the mount fails straight away.

On RHEL6, if /etc/hosts has an entry for the IPV6 localhost address, an 
NFS mount from IPV6 localhost gets tried first, which fails immediately 
as expected, as IPV6 is not enabled, and no NFS service is registered on 
the host.  Automount reports the mount failure straight away.

On RHEL6 however, if /etc/hosts does not have an entry for the IPV6 
localhost address, automount tries the IPV4 localhost address; when the 
mount is requested using NFSv4, the RPC request seem to get localhost 
port 0 returned for NFS service, instead of being told there is no 
program registered for NFS service (if I’m reading this strace correctly):

1657  1416223283.572722 socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 3
1657  1416223283.572764 bind(3, {sa_family=AF_INET, sin_port=htons(0), 
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
1657  1416223283.572811 connect(3, {sa_family=AF_INET, 
sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
1657  1416223283.572849 getsockname(3, {sa_family=AF_INET, 
sin_port=htons(40477), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
1657  1416223283.572896 mount("localhost:/local/0/blah", "/mnt/test", 
"nfs", 0, "vers=4,addr=127.0.0.1,clientaddr"...) = -1 ECONNREFUSED 
(Connection refused)

The mount continues to pause and retry, until it reaches the retry 
limit, and then eventually fails.  Here’s what the bare mount command shows:

[root@lonlx8001b13 bevaja]# mount -vvv -t nfs -o nfsvers=4 
localhost:/local/0/blah /mnt/test
mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: spec:  "localhost:/local/0/blah"
mount: node:  "/mnt/test"
mount: types: "nfs"
mount: opts:  "nfsvers=4"
final mount options: 'nfsvers=4'
mount: external mount: argv[0] = "/sbin/mount.nfs"
mount: external mount: argv[1] = "localhost:/local/0/blah"
mount: external mount: argv[2] = "/mnt/test"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,nfsvers=4"
mount.nfs: timeout set for Mon Nov 17 14:42:17 2014
mount.nfs: trying text-based options 
'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 
'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 
'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 
'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 
'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
mount.nfs: mount(2): Connection refused


Thoughts?

Thanks,

Dave
-- 

===================================================================
Dave Sullivan RHCE 			Email: dsulliva@redhat.com
Sr. Technical Account Manager 	

+1 312 660 3525 (Office)
+1 989 750 8385 (Cell)
===================================================================
Red Hat, Inc. | 100 East Davie St | Raleigh, NC | 27601

---
Learn. Network. Experience open source.
http://www.redhat.com
http://access.redhat.com
http://www.opensource.com

Re: possible better error handling for nfsv4 and localhost maps

[Steve Dickson]

On 12/05/2014 10:20 AM, Dave Sullivan wrote:
> Hi All,
> 
> Recently we had a customer with some legacy maps using localhost run into a problem with NFSv4.
> 
> They are not quite ready to get rid of those localhost maps.
> 
> There is a fairly straight forward workaround/solution to get around this issue by modifying /etc/nfsmount.conf with
> 
> [ Server "localhost" ]
>    Nfsvers=3
> 
> What we are asking for is better error handling so users can figure out what the problem is without having to debug as much.
> 
> i.e. Have the code not retry and have the code send a notification to syslog saying something like "nfsv4 does not support localhost maps, man nfsmount.conf, use [ Server "localhost" ] with Nfsvers=3 mount option to restrict localhost maps to NFSv3"
> 
> Here's an example of effort required to debug
> 
> Description of problem:
> 
> The RHEL automounter does try to do what’s inferred in the above notation, ie, make a bind mount.  What happens next, when the local filesystem doesn’t exist, is what’s causing us a problem.  It next spawns a mount request to an NFS service on localhost for the mount, either via IPV6, or via IPV4;  end-user hosts don’t serve NFS, so the problem arises due to how this is handled.
> 
> On RHEL5, the mount request contacts the portmapper, which reports there is no program registered for NFS service, and the mount fails straight away.
> 
> On RHEL6, if /etc/hosts has an entry for the IPV6 localhost address, an NFS mount from IPV6 localhost gets tried first, which fails immediately as expected, as IPV6 is not enabled, and no NFS service is registered on the host.  Automount reports the mount failure straight away.
> Hi All,
> 
> Recently we had a customer with some legacy maps using localhost run into a problem with NFSv4.
> 
> They are not quite ready to get rid of those localhost maps.
> 
> There is a fairly straight forward workaround/solution to get around this issue by modifying /etc/nfsmount.conf with
> 
> [ Server "localhost" ]
>    Nfsvers=3
> 
> What we are asking for is better error handling so users can figure out what the problem is without having to debug as much.
> 
> i.e. Have the code not retry and have the code send a notification to syslog saying something like "nfsv4 does not support localhost maps, man nfsmount.conf, use [ Server "localhost" ] with Nfsvers=3 mount option to restrict localhost maps to NFSv3"
> 
> Here's an example of effort required to debug
> 
> Description of problem:
> 
> The RHEL automounter does try to do what’s inferred in the above notation, ie, make a bind mount.  What happens next, when the local filesystem doesn’t exist, is what’s causing us a problem.  It next spawns a mount request to an NFS service on localhost for the mount, either via IPV6, or via IPV4;  end-user hosts don’t serve NFS, so the problem arises due to how this is handled.
> 
> On RHEL5, the mount request contacts the portmapper, which reports there is no program registered for NFS service, and the mount fails straight away.
> 
> On RHEL6, if /etc/hosts has an entry for the IPV6 localhost address, an NFS mount from IPV6 localhost gets tried first, which fails immediately as expected, as IPV6 is not enabled, and no NFS service is registered on the host.  Automount reports the mount failure straight away.
> 
> On RHEL6 however, if /etc/hosts does not have an entry for the IPV6 localhost address, automount tries the IPV4 localhost address; when the mount is requested using NFSv4, the RPC request seem to get localhost port 0 returned for NFS service, instead of being told there is no program registered for NFS service (if I’m reading this strace correctly):
> 
> 1657  1416223283.572722 socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 3
> 1657  1416223283.572764 bind(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> 1657  1416223283.572811 connect(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
> 1657  1416223283.572849 getsockname(3, {sa_family=AF_INET, sin_port=htons(40477), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
> 1657  1416223283.572896 mount("localhost:/local/0/blah", "/mnt/test", "nfs", 0, "vers=4,addr=127.0.0.1,clientaddr"...) = -1 ECONNREFUSED (Connection refused)
> 
> The mount continues to pause and retry, until it reaches the retry limit, and then eventually fails.  Here’s what the bare mount command shows:
> 
> [root@lonlx8001b13 bevaja]# mount -vvv -t nfs -o nfsvers=4 localhost:/local/0/blah /mnt/test
> mount: fstab path: "/etc/fstab"
> mount: mtab path:  "/etc/mtab"
> mount: lock path:  "/etc/mtab~"
> mount: temp path:  "/etc/mtab.tmp"
> mount: UID:        0
> mount: eUID:       0
> mount: spec:  "localhost:/local/0/blah"
> mount: node:  "/mnt/test"
> mount: types: "nfs"
> mount: opts:  "nfsvers=4"
> final mount options: 'nfsvers=4'
> mount: external mount: argv[0] = "/sbin/mount.nfs"
> mount: external mount: argv[1] = "localhost:/local/0/blah"
> mount: external mount: argv[2] = "/mnt/test"
> mount: external mount: argv[3] = "-v"
> mount: external mount: argv[4] = "-o"
> mount: external mount: argv[5] = "rw,nfsvers=4"
> mount.nfs: timeout set for Mon Nov 17 14:42:17 2014
> mount.nfs: trying text-based options 'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
> mount.nfs: mount(2): Connection refused
> mount.nfs: trying text-based options 'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
> mount.nfs: mount(2): Connection refused
> mount.nfs: trying text-based options 'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
> mount.nfs: mount(2): Connection refused
> mount.nfs: trying text-based options 'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
> mount.nfs: mount(2): Connection refused
> mount.nfs: trying text-based options 'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
> mount.nfs: mount(2): Connection refused
> 
> 
> Thoughts?
Have you tried using the -o timeo=1,retry=0 mount options? They might help... 

steved.

Re: possible better error handling for nfsv4 and localhost maps

[Dave Sullivan]

On 12/06/2014 11:38 AM, Steve Dickson wrote:
>
>
> On 12/05/2014 10:20 AM, Dave Sullivan wrote:
>> Hi All,
>>
>> Recently we had a customer with some legacy maps using localhost run into a problem with NFSv4.
>>
>> They are not quite ready to get rid of those localhost maps.
>>
>> There is a fairly straight forward workaround/solution to get around this issue by modifying /etc/nfsmount.conf with
>>
>> [ Server "localhost" ]
>>     Nfsvers=3
>>
>> What we are asking for is better error handling so users can figure out what the problem is without having to debug as much.
>>
>> i.e. Have the code not retry and have the code send a notification to syslog saying something like "nfsv4 does not support localhost maps, man nfsmount.conf, use [ Server "localhost" ] with Nfsvers=3 mount option to restrict localhost maps to NFSv3"
>>
>> Here's an example of effort required to debug
>>
>> Description of problem:
>>
>> The RHEL automounter does try to do what’s inferred in the above notation, ie, make a bind mount.  What happens next, when the local filesystem doesn’t exist, is what’s causing us a problem.  It next spawns a mount request to an NFS service on localhost for the mount, either via IPV6, or via IPV4;  end-user hosts don’t serve NFS, so the problem arises due to how this is handled.
>>
>> On RHEL5, the mount request contacts the portmapper, which reports there is no program registered for NFS service, and the mount fails straight away.
>>
>> On RHEL6, if /etc/hosts has an entry for the IPV6 localhost address, an NFS mount from IPV6 localhost gets tried first, which fails immediately as expected, as IPV6 is not enabled, and no NFS service is registered on the host.  Automount reports the mount failure straight away.
>> Hi All,
>>
>> Recently we had a customer with some legacy maps using localhost run into a problem with NFSv4.
>>
>> They are not quite ready to get rid of those localhost maps.
>>
>> There is a fairly straight forward workaround/solution to get around this issue by modifying /etc/nfsmount.conf with
>>
>> [ Server "localhost" ]
>>     Nfsvers=3
>>
>> What we are asking for is better error handling so users can figure out what the problem is without having to debug as much.
>>
>> i.e. Have the code not retry and have the code send a notification to syslog saying something like "nfsv4 does not support localhost maps, man nfsmount.conf, use [ Server "localhost" ] with Nfsvers=3 mount option to restrict localhost maps to NFSv3"
>>
>> Here's an example of effort required to debug
>>
>> Description of problem:
>>
>> The RHEL automounter does try to do what’s inferred in the above notation, ie, make a bind mount.  What happens next, when the local filesystem doesn’t exist, is what’s causing us a problem.  It next spawns a mount request to an NFS service on localhost for the mount, either via IPV6, or via IPV4;  end-user hosts don’t serve NFS, so the problem arises due to how this is handled.
>>
>> On RHEL5, the mount request contacts the portmapper, which reports there is no program registered for NFS service, and the mount fails straight away.
>>
>> On RHEL6, if /etc/hosts has an entry for the IPV6 localhost address, an NFS mount from IPV6 localhost gets tried first, which fails immediately as expected, as IPV6 is not enabled, and no NFS service is registered on the host.  Automount reports the mount failure straight away.
>>
>> On RHEL6 however, if /etc/hosts does not have an entry for the IPV6 localhost address, automount tries the IPV4 localhost address; when the mount is requested using NFSv4, the RPC request seem to get localhost port 0 returned for NFS service, instead of being told there is no program registered for NFS service (if I’m reading this strace correctly):
>>
>> 1657  1416223283.572722 socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 3
>> 1657  1416223283.572764 bind(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
>> 1657  1416223283.572811 connect(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
>> 1657  1416223283.572849 getsockname(3, {sa_family=AF_INET, sin_port=htons(40477), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
>> 1657  1416223283.572896 mount("localhost:/local/0/blah", "/mnt/test", "nfs", 0, "vers=4,addr=127.0.0.1,clientaddr"...) = -1 ECONNREFUSED (Connection refused)
>>
>> The mount continues to pause and retry, until it reaches the retry limit, and then eventually fails.  Here’s what the bare mount command shows:
>>
>> [root@lonlx8001b13 bevaja]# mount -vvv -t nfs -o nfsvers=4 localhost:/local/0/blah /mnt/test
>> mount: fstab path: "/etc/fstab"
>> mount: mtab path:  "/etc/mtab"
>> mount: lock path:  "/etc/mtab~"
>> mount: temp path:  "/etc/mtab.tmp"
>> mount: UID:        0
>> mount: eUID:       0
>> mount: spec:  "localhost:/local/0/blah"
>> mount: node:  "/mnt/test"
>> mount: types: "nfs"
>> mount: opts:  "nfsvers=4"
>> final mount options: 'nfsvers=4'
>> mount: external mount: argv[0] = "/sbin/mount.nfs"
>> mount: external mount: argv[1] = "localhost:/local/0/blah"
>> mount: external mount: argv[2] = "/mnt/test"
>> mount: external mount: argv[3] = "-v"
>> mount: external mount: argv[4] = "-o"
>> mount: external mount: argv[5] = "rw,nfsvers=4"
>> mount.nfs: timeout set for Mon Nov 17 14:42:17 2014
>> mount.nfs: trying text-based options 'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
>> mount.nfs: mount(2): Connection refused
>> mount.nfs: trying text-based options 'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
>> mount.nfs: mount(2): Connection refused
>> mount.nfs: trying text-based options 'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
>> mount.nfs: mount(2): Connection refused
>> mount.nfs: trying text-based options 'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
>> mount.nfs: mount(2): Connection refused
>> mount.nfs: trying text-based options 'nfsvers=4,addr=127.0.0.1,clientaddr=127.0.0.1'
>> mount.nfs: mount(2): Connection refused
>>
>>
>> Thoughts?
> Have you tried using the -o timeo=1,retry=0 mount options? They might help...
>
But do you really want to have all your other mounts doing this too?

I don't think so, its less resilient right.

I think the /etc/nfsmount.conf fix

[ Server "localhost" ]
     Nfsvers=3


Is 100% sufficient, but it still would be nice to have this brought up 
to your attention easier through error handling/logging.

Unless in automount configuration you could force nfsv4 to do -o 
timeo=1,retry=0 just for localhost then fallback to nfsv3 with different 
set of options.

But I believe you would then globally being setting nfsv4 to use the 
same -o timeo=1,retry=0 options and I don't think there is fallback 
mechanism either.

So basically I still think to have these mounts land you still want the 
previously document solution.

/etc/nfsmount.conf fix

[ Server "localhost" ]
     Nfsvers=3

Thoughts?

> steved.
>


-- 

===================================================================
Dave Sullivan RHCE 			Email: dsulliva@redhat.com
Sr. Technical Account Manager 	

+1 312 660 3525 (Office)
+1 989 750 8385 (Cell)
===================================================================
Red Hat, Inc. | 100 East Davie St | Raleigh, NC | 27601

---
Learn. Network. Experience open source.
http://www.redhat.com
http://access.redhat.com
http://www.opensource.com

Re: possible better error handling for nfsv4 and localhost maps

[Steve Dickson]

On 12/10/2014 10:33 AM, Dave Sullivan wrote:
>> Have you tried using the -o timeo=1,retry=0 mount options? They might help...
>>
> But do you really want to have all your other mounts doing this too?
> 
> I don't think so, its less resilient right.
> 
> I think the /etc/nfsmount.conf fix
> 
> [ Server "localhost" ]
>     Nfsvers=3
> 
> 
> Is 100% sufficient, but it still would be nice to have this brought 
> up to your attention easier through error handling/logging.
Understood... 

> 
> Unless in automount configuration you could force nfsv4 to 
> do -o timeo=1,retry=0 just for localhost then fallback to 
> nfsv3 with different set of options.
> 
> But I believe you would then globally being setting nfsv4 to 
> use the same -o timeo=1,retry=0 options and I don't think 
> there is fallback mechanism either.
> 
> So basically I still think to have these mounts land you still 
> want the previously document solution.
Documentation has never been a strong suite with Linux NFS...
esp with debugging... Its probably a side effect of being Opensource,
the code is the documentation! ;-) 

That said, we have to be careful not to log common errors either... 
I too have been frustrated with mounts hang with "Connection refused"
because it was using the wrong address... But its a common error so we
have to be careful about flooding syslog with common occurrences... 

I'm not sure adding more debugging code to NFS kernel or user daemons 
is the answer... What I think the answer is making the current debugging
more user friendly... 

Being able to dynamically turn on and off debugging in daemons. Added
a user friendly address (possibly a GUI) to the current tracepoints.
Tailor the existing systemtab script to make debugging situations 
like this more straightforward... 

I think efforts in these areas would pay of more than just adding 
yet more debugging code... 

steved.