Linux NFS-HOWTO - improvement to Troubleshooting permission denied

Linux NFS-HOWTO - improvement to Troubleshooting permission denied

[Jari Aalto]

Thank you for the NFS howto (last modified 2006). Please add further
diagnostics to section:

	=> "I do not have permission to access files on the mounted volume"
	http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues

The error condition "Permission denied" also happens in certain NFS
(client/server) combinations and the workaround is to add additional
mount parameter at client:

        sec=sys

Please add this to the "#pemission_issues" (tag the word seems to have a
small typo).

See reported incidents:

        http://thread.gmane.org/gmane.linux.debian.user/333162
        http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=492970

Jari

----
CC: team http://sourceforge.net/projects/nfs

You're all in CC, as I did not know who is responsible for updating the
NFS-HOWTO at sourceforge.

Re: Linux NFS-HOWTO - improvement to Troubleshooting permission denied

[Chuck Lever]

Hi Jari-

On Mon, Aug 25, 2008 at 3:36 AM, Jari Aalto <jari.aalto-w6EBCRcXQ4deoWH0uzbU5w@public.gmane.org> wrote:
>
> Thank you for the NFS howto (last modified 2006). Please add further
> diagnostics to section:
>
>        => "I do not have permission to access files on the mounted volume"
>        http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues
>
> The error condition "Permission denied" also happens in certain NFS
> (client/server) combinations and the workaround is to add additional
> mount parameter at client:
>
>        sec=sys
>
> Please add this to the "#pemission_issues" (tag the word seems to have a
> small typo).

This is a known bug in nfs-utils 1.1.3 when used with older client
kernels that do not support text-based mounts.

Seems like an esoteric (ie not widespread) problem, but I could easily
be wrong about that.

-- 
Chuck Lever

Re: Linux NFS-HOWTO - improvement to Troubleshooting permission denied

[J. Bruce Fields]

On Mon, Aug 25, 2008 at 11:41:02AM -0400, Chuck Lever wrote:
> Hi Jari-
> 
> On Mon, Aug 25, 2008 at 3:36 AM, Jari Aalto <jari.aalto-w6EBCRcXQ4deoWH0uzbU5w@public.gmane.org> wrote:
> >
> > Thank you for the NFS howto (last modified 2006). Please add further
> > diagnostics to section:
> >
> >        => "I do not have permission to access files on the mounted volume"
> >        http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues
> >
> > The error condition "Permission denied" also happens in certain NFS
> > (client/server) combinations and the workaround is to add additional
> > mount parameter at client:
> >
> >        sec=sys
> >
> > Please add this to the "#pemission_issues" (tag the word seems to have a
> > small typo).
> 
> This is a known bug in nfs-utils 1.1.3 when used with older client
> kernels that do not support text-based mounts.
> 
> Seems like an esoteric (ie not widespread) problem, but I could easily
> be wrong about that.

People should be able to upgrade nfs-utils without upgrading their
kernel!

The referenced debian bug report:

	http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=492970

claims the first bad commit is a server-side commit that itself seems to
claim to work around a problem?  ("For mountd, it does not advertise
AUTH_NULL anymore. This is necessary to avoid backward compatibility
issue. If AUTH_NULL appears in the list, either the new or the old
client will choose that over AUTH_SYS.")

Did we ever find the actual root cause?

--b.

Re: Linux NFS-HOWTO - improvement to Troubleshooting permission denied

[Chuck Lever]

Some CC's corrected.

On Aug 25, 2008, at Aug 25, 2008, 11:55 AM, J. Bruce Fields wrote:
> On Mon, Aug 25, 2008 at 11:41:02AM -0400, Chuck Lever wrote:
>> Hi Jari-
>>
>> On Mon, Aug 25, 2008 at 3:36 AM, Jari Aalto <jari.aalto-w6EBCRcXQ4deoWH0uzbU5w@public.gmane.org>  
>> wrote:
>>>
>>> Thank you for the NFS howto (last modified 2006). Please add further
>>> diagnostics to section:
>>>
>>>       => "I do not have permission to access files on the mounted  
>>> volume"
>>>       http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues
>>>
>>> The error condition "Permission denied" also happens in certain NFS
>>> (client/server) combinations and the workaround is to add additional
>>> mount parameter at client:
>>>
>>>       sec=sys
>>>
>>> Please add this to the "#pemission_issues" (tag the word seems to  
>>> have a
>>> small typo).
>>
>> This is a known bug in nfs-utils 1.1.3 when used with older client
>> kernels that do not support text-based mounts.
>>
>> Seems like an esoteric (ie not widespread) problem, but I could  
>> easily
>> be wrong about that.
>
> People should be able to upgrade nfs-utils without upgrading their
> kernel!

Of course.  But Bugs Happen.  I'm not implying that nfs-utils 1.1.3  
requires a kernel upgrade, but merely describing the conditions where  
the issue appears.

In fact, you may recall a recent and rather protracted e-mail  
conversation where we determined that the text-based NFS mount  
implementation is not working correctly in this case either.

> The referenced debian bug report:
>
> 	http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=492970
>
> claims the first bad commit is a server-side commit that itself  
> seems to
> claim to work around a problem?  ("For mountd, it does not advertise
> AUTH_NULL anymore. This is necessary to avoid backward compatibility
> issue. If AUTH_NULL appears in the list, either the new or the old
> client will choose that over AUTH_SYS.")
>
> Did we ever find the actual root cause?

The bad commit does not live up to the comments it introduced, which  
claim that the legacy mount command will skip AUTH_NULL as a default  
if it receives it in the auth flavor list from mountd.  I haven't  
verified this, but otherwise, yes, we have an RCA.

I copied the patch author of record, bcwong-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org, on the recent  
discussion, but we haven't heard any response.  I think we will need  
to fix this ourselves.  IMO the proper fix is to eliminate the  
possibility that the client will ever use AUTH_NULL unless "sec=null"  
is explicitly specified on the mount command line, then to get nfs- 
utils 1.1.4 posted with this fix.

If I may also observe: the bad commit attempts to change both server- 
side and client-side behavior in one go.  Hindsight suggests the  
changes to nfsmount.c and mountd.c should have occurred in two  
separate commits so that automated bisection and surgical reversion  
are possible.

This reminds us again that we have an exponentially expanding test  
matrix for nfs-utils, especially considering the backwards- 
compatibility requirements.  I would love to see significant test  
resources committed in this area.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Linux NFS-HOWTO - improvement to Troubleshooting permission denied

[Jari Aalto]

"Chuck Lever" <chuck.lever@oracle.com> writes:
> On Mon, Aug 25, 2008 at 3:36 AM, Jari Aalto <jari.aalto-w6EBCRcXQ4deoWH0uzbU5w@public.gmane.org> wrote:
>>
>> Thank you for the NFS howto (last modified 2006). Please add further
>> diagnostics to section:
>>
>>        => "I do not have permission to access files on the mounted volume"
>>        http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues
>>
>> The error condition "Permission denied" also happens in certain NFS
>> (client/server) combinations and the workaround is to add additional
>> mount parameter at client:
>>
>>        sec=sys
>>
>> Please add this to the "#pemission_issues" (tag the word seems to have a
>> small typo).
>
> This is a known bug in nfs-utils 1.1.3 when used with older client
> kernels that do not support text-based mounts.
>
> Seems like an esoteric (ie not widespread) problem, but I could easily
> be wrong about that.

Please add the tip, although the bug may be fixed in later NFS releases.
There are great deal of hosts running old versions whose admins scratch
they head and seek for guidance from the HOWTO.

Mentioning "sec=sys" would have saved me countless of hours tracing what
was wrong in my setup[1]. Without to Bob <grantura <at> spamcop.net> and
his knowledge about Debian bug, I would never had resolved the issue in
decent time.

Jari

[1]
"NFS and no_root_squash - cannot write as root - permission denied"
http://thread.gmane.org/gmane.linux.debian.user/333162

http://bugs.debian.org/492970