Permission denied when mounting NFS (was okay before)

Permission denied when mounting NFS (was okay before)

[howard chen]

Hi,

I have a sudden problem when my server reboot and mount to a NFS
failed, so I neeed to mount
by hand. (But it was okay before...)

Error:

mount: trying xxxx prog 100003 vers 3 prot udp port 2049
mount: trying xxxx prog 100005 vers 3 prot udp port 891
mount: xxxx:/data0/tmp failed, reason given by server: Permission
denied

Many pages in the web said it is related to the hostname and ip
mapping, which made permission issues,
but in my NFS server,  I see the permission is ok:

(/var/log/message)

authenticated mount request from xxxx for yyyy...

Any idea for this funny problem?

Thanks.

Re: Permission denied when mounting NFS (was okay before)

[Talpey, Thomas]

Just because it's "authenticated" doesn't mean you have "permission".
What are the contents of your server's export file, and the umode of
the target /data0/tmp directory? And btw, what types/versions are
your client and server?

Tom.

At 07:32 AM 9/26/2008, howard chen wrote:
>Hi,
>
>I have a sudden problem when my server reboot and mount to a NFS
>failed, so I neeed to mount
>by hand. (But it was okay before...)
>
>Error:
>
>mount: trying xxxx prog 100003 vers 3 prot udp port 2049
>mount: trying xxxx prog 100005 vers 3 prot udp port 891
>mount: xxxx:/data0/tmp failed, reason given by server: Permission
>denied
>
>Many pages in the web said it is related to the hostname and ip
>mapping, which made permission issues,
>but in my NFS server,  I see the permission is ok:
>
>(/var/log/message)
>
>authenticated mount request from xxxx for yyyy...
>
>Any idea for this funny problem?
>

Re: Permission denied when mounting NFS (was okay before)

[howard chen]

Thanks.


On Fri, Sep 26, 2008 at 7:50 PM, Talpey, Thomas
<Thomas.Talpey@netapp.com> wrote:
> Just because it's "authenticated" doesn't mean you have "permission".
> What are the contents of your server's export file, and the umode of
> the target /data0/tmp directory? And btw, what types/versions are
> your client and server?
>

Thanks. Consider below as the full picture:

NFS Server (10.10.10.1)
=========================
/data0/tmp	10.10.10.2(rw,sync)

NFS Client (10.10.10.2)
=======================
# Add in /etc/rc.local
mount -t nfs -o
async,noatime,noexec,nosuid,hard,intr,udp,retry=3,rsize=32768,wsize=32768
10.10.10.1:/data0/tmp     /home/www/tmp

ls /data0/

drwxrwxr-x  261 root web  12288 Sep 26 22:58 tmp



Thanks.

Re: Permission denied when mounting NFS (was okay before)

[Talpey, Thomas]

At 11:05 AM 9/26/2008, howard chen wrote:
>Thanks.
>
>
>On Fri, Sep 26, 2008 at 7:50 PM, Talpey, Thomas
><Thomas.Talpey@netapp.com> wrote:
>> Just because it's "authenticated" doesn't mean you have "permission".
>> What are the contents of your server's export file, and the umode of
>> the target /data0/tmp directory? And btw, what types/versions are
>> your client and server?
>>
>
>Thanks. Consider below as the full picture:

You didn't indicate what the client and server were, btw.

Do you have multiple interfaces on the client? If the client routing
originates from a different address than 10.10.10.2, then the server
will deny it because you have specified a numerical address. You might
try exporting to "*" to see if this changes anything. Apart from that
possibility, the configuration looks plausible.

This isn't the source of the permissions error, but why are you doing
a UDP mount, and with only three retries? Generally, TCP will perform
better, and more robustly. Also, the "noatime" option is a no-op for
the NFS client (servers are in charge of maintaining atime).

Tom.

>
>NFS Server (10.10.10.1)
>=========================
>/data0/tmp	10.10.10.2(rw,sync)
>
>NFS Client (10.10.10.2)
>=======================
># Add in /etc/rc.local
>mount -t nfs -o
>async,noatime,noexec,nosuid,hard,intr,udp,retry=3,rsize=32768,wsize=32768
>10.10.10.1:/data0/tmp     /home/www/tmp
>
>ls /data0/
>
>drwxrwxr-x  261 root web  12288 Sep 26 22:58 tmp
>
>
>
>Thanks.

Re: Permission denied when mounting NFS (was okay before)

[howard chen]

Hi,

On Fri, Sep 26, 2008 at 11:25 PM, Talpey, Thomas
<Thomas.Talpey@netapp.com> wrote:
> At 11:05 AM 9/26/2008, howard chen wrote:
> You didn't indicate what the client and server were, btw.

client is 3, as I see when mount with verbose option: ... mount:
trying xxxx prog 100003 vers 3 prot udp port 2049

server should be also 3, default by CentOS 4.4, 64bit

>
> Do you have multiple interfaces on the client? If the client routing
> originates from a different address than 10.10.10.2, then the server
> will deny it because you have specified a numerical address.

Yes, both servers have interface for public IP, but as I can see the
10.10.10.2 in /var/log/message of NFS server, so I think routing is
ok.



> This isn't the source of the permissions error, but why are you doing
> a UDP mount, and with only three retries? Generally, TCP will perform
> better, and more robustly. Also, the "noatime" option is a no-op for
> the NFS client (servers are in charge of maintaining atime).

I agree TCP is more robust, but isn't UDP will have a better performance?

Thanks.

Re: Permission denied when mounting NFS (was okay before)

[Trond Myklebust]

On Fri, 2008-09-26 at 23:57 +0800, howard chen wrote:
> Hi,
> 
> On Fri, Sep 26, 2008 at 11:25 PM, Talpey, Thomas
> <Thomas.Talpey@netapp.com> wrote:
> > At 11:05 AM 9/26/2008, howard chen wrote:
> > You didn't indicate what the client and server were, btw.
> 
> client is 3, as I see when mount with verbose option: ... mount:
> trying xxxx prog 100003 vers 3 prot udp port 2049
> 
> server should be also 3, default by CentOS 4.4, 64bit

What does

  'rpcinfo -p 10.10.10.1'

give you? Also,

  'showmount -e 10.10.10.1'

Finally, what kind of filesystem are you exporting on /data0/tmp?

> >
> > Do you have multiple interfaces on the client? If the client routing
> > originates from a different address than 10.10.10.2, then the server
> > will deny it because you have specified a numerical address.
> 
> Yes, both servers have interface for public IP, but as I can see the
> 10.10.10.2 in /var/log/message of NFS server, so I think routing is
> ok.
> 
> 
> 
> > This isn't the source of the permissions error, but why are you doing
> > a UDP mount, and with only three retries? Generally, TCP will perform
> > better, and more robustly. Also, the "noatime" option is a no-op for
> > the NFS client (servers are in charge of maintaining atime).
> 
> I agree TCP is more robust, but isn't UDP will have a better performance?

That depends. In my experience, the difference in performance on an
unloaded network, then UDP will outperform TCP by ~10%. However, if you
have a heavily loaded network with lots of dropped packets, then TCP
will usually give much better performance than UDP.

Trond

Re: Permission denied when mounting NFS (was okay before)

[howard chen]

Hi,

Thanks all for your input first.

On Sat, Sep 27, 2008 at 12:06 AM, Trond Myklebust
<trond.myklebust@fys.uio.no> wrote:

> What does
>
>  'rpcinfo -p 10.10.10.1'

 program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100021    1   udp  32773  nlockmgr
    100021    3   udp  32773  nlockmgr
    100021    4   udp  32773  nlockmgr
    100021    1   tcp  32779  nlockmgr
    100021    3   tcp  32779  nlockmgr
    100021    4   tcp  32779  nlockmgr
    100024    1   udp    883  status
    100024    1   tcp    886  status
    100011    1   udp    821  rquotad
    100011    2   udp    821  rquotad
    100011    1   tcp    824  rquotad
    100011    2   tcp    824  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    1   udp    891  mountd
    100005    1   tcp    894  mountd
    100005    2   udp    891  mountd
    100005    2   tcp    894  mountd
    100005    3   udp    891  mountd
    100005    3   tcp    894  mountd


> give you? Also,
>
>  'showmount -e 10.10.10.1'
>


Export list for 10.10.10.1
/data0/tmp     10.10.10.2




> That depends. In my experience, the difference in performance on an
> unloaded network, then UDP will outperform TCP by ~10%. However, if you
> have a heavily loaded network with lots of dropped packets, then TCP
> will usually give much better performance than UDP.

Good to know! I will definitely have a test, any parameters are also
recommend together with TCP so I can do a fair benchmark?


Thank you again.

Re: Permission denied when mounting NFS (was okay before)

[Talpey, Thomas]

Have you tried my suggestion of replacing "10.10.10.2" with "*" in
your server's exports, as a test? It will tell us whether you have
an IP aliasing issue.

As for TCP, no special mount parameters are required. The timers
and algorithms are all adaptive and preconfiguring them is often
counterproductive. You still haven't shown "uname -a" for your
client, but if it's anything recent, you don't even need to specify
tcp, it's the default.

Tom.

At 12:13 PM 9/26/2008, howard chen wrote:
>Hi,
>
>Thanks all for your input first.
>
>On Sat, Sep 27, 2008 at 12:06 AM, Trond Myklebust
><trond.myklebust@fys.uio.no> wrote:
>
>> What does
>>
>>  'rpcinfo -p 10.10.10.1'
>
> program vers proto   port
>    100000    2   tcp    111  portmapper
>    100000    2   udp    111  portmapper
>    100021    1   udp  32773  nlockmgr
>    100021    3   udp  32773  nlockmgr
>    100021    4   udp  32773  nlockmgr
>    100021    1   tcp  32779  nlockmgr
>    100021    3   tcp  32779  nlockmgr
>    100021    4   tcp  32779  nlockmgr
>    100024    1   udp    883  status
>    100024    1   tcp    886  status
>    100011    1   udp    821  rquotad
>    100011    2   udp    821  rquotad
>    100011    1   tcp    824  rquotad
>    100011    2   tcp    824  rquotad
>    100003    2   udp   2049  nfs
>    100003    3   udp   2049  nfs
>    100003    4   udp   2049  nfs
>    100003    2   tcp   2049  nfs
>    100003    3   tcp   2049  nfs
>    100003    4   tcp   2049  nfs
>    100005    1   udp    891  mountd
>    100005    1   tcp    894  mountd
>    100005    2   udp    891  mountd
>    100005    2   tcp    894  mountd
>    100005    3   udp    891  mountd
>    100005    3   tcp    894  mountd
>
>
>> give you? Also,
>>
>>  'showmount -e 10.10.10.1'
>>
>
>
>Export list for 10.10.10.1
>/data0/tmp     10.10.10.2
>
>
>
>
>> That depends. In my experience, the difference in performance on an
>> unloaded network, then UDP will outperform TCP by ~10%. However, if you
>> have a heavily loaded network with lots of dropped packets, then TCP
>> will usually give much better performance than UDP.
>
>Good to know! I will definitely have a test, any parameters are also
>recommend together with TCP so I can do a fair benchmark?
>
>
>Thank you again.

Re: Permission denied when mounting NFS (was okay before)

[howard chen]

Hello,

On Sat, Sep 27, 2008 at 12:23 AM, Talpey, Thomas
<Thomas.Talpey@netapp.com> wrote:
> Have you tried my suggestion of replacing "10.10.10.2" with "*" in
> your server's exports, as a test? It will tell us whether you have
> an IP aliasing issue.
>

Yes, tested, and same error... Also tried "no_root_squash"...

Really have no idea what is going on...


Thanks.

Re: Permission denied when mounting NFS (was okay before)

[howard chen]

Hello all,

On Mon, Sep 29, 2008 at 12:21 PM, howard chen <howachen@gmail.com> wrote:
>
> Yes, tested, and same error... Also tried "no_root_squash"...
>
> Really have no idea what is going on...
>

Problem solved by adding the following line in fstab, according to
this site: http://www.linuxforums.org/forum/linux-networking/44779-nfs-permission-denied-error.html

nfsd /proc/fs/nfsd nfsd auto,defaults 0 0


This is really funny as I never have this option in my other NFS servers....

Anyone know the reason? Redhat's bug?

Thanks

Re: Permission denied when mounting NFS (was okay before)

[J. Bruce Fields]

On Mon, Sep 29, 2008 at 12:53:16PM +0800, howard chen wrote:
> Hello all,
> 
> On Mon, Sep 29, 2008 at 12:21 PM, howard chen <howachen@gmail.com> wrote:
> >
> > Yes, tested, and same error... Also tried "no_root_squash"...
> >
> > Really have no idea what is going on...
> >
> 
> Problem solved by adding the following line in fstab, according to
> this site: http://www.linuxforums.org/forum/linux-networking/44779-nfs-permission-denied-error.html

Huh.  Surely mountd or knfsd could have given a more helpful error
message, at least....

> nfsd /proc/fs/nfsd nfsd auto,defaults 0 0
> 
> 
> This is really funny as I never have this option in my other NFS servers....
> 
> Anyone know the reason? Redhat's bug?

It looks like nfsd is supposed to be mounted on load of the nfsd module,
by a line in /etc/modprobe.d/modprobe.conf.

(Maybe you built a new kernel with nfsd built-in instead of built as a
module?)

But I thought nfs-utils was supposed to fall back on old behavior when
the nfsd filesystem wasn't found.

--b.

Re: Permission denied when mounting NFS (was okay before)

[howard chen]

Hello,

On Tue, Sep 30, 2008 at 1:26 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> It looks like nfsd is supposed to be mounted on load of the nfsd module,
> by a line in /etc/modprobe.d/modprobe.conf.
>
> (Maybe you built a new kernel with nfsd built-in instead of built as a
> module?)
>
> But I thought nfs-utils was supposed to fall back on old behavior when
> the nfsd filesystem wasn't found.


Are there any related docs I should read?

As mentioned before, the server was working before the reboot, really
don't understand why it behave in this way...Need to find out the
reason.

THanks!

Re: Permission denied when mounting NFS (was okay before)

[J. Bruce Fields]

On Tue, Sep 30, 2008 at 09:51:08PM +0800, howard chen wrote:
> Hello,
> 
> On Tue, Sep 30, 2008 at 1:26 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> > It looks like nfsd is supposed to be mounted on load of the nfsd module,
> > by a line in /etc/modprobe.d/modprobe.conf.
> >
> > (Maybe you built a new kernel with nfsd built-in instead of built as a
> > module?)
> >
> > But I thought nfs-utils was supposed to fall back on old behavior when
> > the nfsd filesystem wasn't found.
> 
> 
> Are there any related docs I should read?

The "exportfs" man page explains a little about the "legacy" vs "new"
modes of mountd/exportfs operation.

> As mentioned before, the server was working before the reboot, really
> don't understand why it behave in this way...Need to find out the
> reason.

As I said, two things happened:
	1. The "nfsd" filesystem didn't get mounted under /proc/ as it's
	   suppose to be.
	2. The server therefore should have been using the "legacy" mode
	   of operation.  But it didn't work for some reason.

I'm not sure in either case.

If you do put an explicit nfsd mount in the fstab, then you won't see
this problem again.

--b.

Re: Permission denied when mounting NFS (was okay before)

[Benny Halevy]

On Sep. 29, 2008, 20:26 +0300, "J. Bruce Fields" <bfields@fieldses.org> wrote:
> On Mon, Sep 29, 2008 at 12:53:16PM +0800, howard chen wrote:
>> Hello all,
>>
>> On Mon, Sep 29, 2008 at 12:21 PM, howard chen <howachen@gmail.com> wrote:
>>> Yes, tested, and same error... Also tried "no_root_squash"...
>>>
>>> Really have no idea what is going on...
>>>
>> Problem solved by adding the following line in fstab, according to
>> this site: http://www.linuxforums.org/forum/linux-networking/44779-nfs-permission-denied-error.html
> 
> Huh.  Surely mountd or knfsd could have given a more helpful error
> message, at least....

I agree.
I've just hit this problem again today (running Fedora 9).
(and I think http://bugzilla.redhat.com/show_bug.cgi?id=218994 might
be the same thing BTW)

rpc.mountd just prints
authenticated mount request from 192.168.0.140:669 for /usr0/export
even when invoked with --debug all.

However it denies access.  Here's tshark -V output:

Remote Procedure Call, Type:Call XID:0xee9360da
    Fragment header: Last fragment, 124 bytes
        1... .... .... .... .... .... .... .... = Last Fragment: Yes
        .000 0000 0000 0000 0000 0000 0111 1100 = Fragment Length: 124
    XID: 0xee9360da (4002635994)
    Message Type: Call (0)
    RPC Version: 2
    Program: MOUNT (100005)
    Program Version: 3
    Procedure: MNT (1)
    Credentials
        Flavor: AUTH_UNIX (1)
        Length: 68
        Stamp: 0x004235f6
        Machine Name: pangw.bhalevy.com
            length: 17
            contents: pangw.bhalevy.com
            fill bytes: opaque data
        UID: 0
        GID: 0
        Auxiliary GIDs
            GID: 0
            GID: 1
            GID: 2
            GID: 3
            GID: 4
            GID: 6
            GID: 10
    Verifier
        Flavor: AUTH_NULL (0)
        Length: 0
Mount Service
    [Program Version: 3]
    [V3 Procedure: MNT (1)]
    Path: /usr0/export
        length: 12
        contents: /usr0/export

...
Remote Procedure Call, Type:Reply XID:0xee9360da
    Fragment header: Last fragment, 28 bytes
        1... .... .... .... .... .... .... .... = Last Fragment: Yes
        .000 0000 0000 0000 0000 0000 0001 1100 = Fragment Length: 28
    XID: 0xee9360da (4002635994)
    Message Type: Reply (1)
    [Program: MOUNT (100005)]
    [Program Version: 3]
    [Procedure: MNT (1)]
    Reply State: accepted (0)
    [This is a reply to a request in frame 3474]
    [Time from request: 0.000868000 seconds]
    Verifier
        Flavor: AUTH_NULL (0)
        Length: 0
    Accept State: RPC executed successfully (0)
Mount Service
    [Program Version: 3]
    [V3 Procedure: MNT (1)]
    Status: ERR_ACCESS (13)

Running strace rpc.mountd -F --debug all shows:
...
open("/proc/fs/nfsd/filehandle", O_RDWR) = -1 ENOENT (No such file or directory)
open("/proc/fs/nfs/filehandle", O_RDWR) = -1 ENOENT (No such file or directory)


> 
>> nfsd /proc/fs/nfsd nfsd auto,defaults 0 0

doping this manually using
mount -t nfsd nfsd /proc/fs/nfsd
does the same trick

>>
>>
>> This is really funny as I never have this option in my other NFS servers....
>>
>> Anyone know the reason? Redhat's bug?
> 
> It looks like nfsd is supposed to be mounted on load of the nfsd module,
> by a line in /etc/modprobe.d/modprobe.conf.

Right.

# grep nfsd /etc/modprobe.d/modprobe.conf.dist
install nfsd /sbin/modprobe --first-time --ignore-install nfsd && { /bin/mount -t nfsd nfsd /proc/fs/nfsd > /dev/null 2>&1 || :; }
remove nfsd { /bin/umount /proc/fs/nfsd > /dev/null 2>&1 || :; } ; /sbin/modprobe -r --first-time --ignore-remove nfsd

> 
> (Maybe you built a new kernel with nfsd built-in instead of built as a
> module?)
> 
> But I thought nfs-utils was supposed to fall back on old behavior when
> the nfsd filesystem wasn't found.

Doesn't seem so.

Steve: any chance this bug is fixed anywhere?

Benny

> 
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Permission denied when mounting NFS (was okay before)

[J. Bruce Fields]

On Tue, Mar 03, 2009 at 12:04:17PM +0200, Benny Halevy wrote:
> On Sep. 29, 2008, 20:26 +0300, "J. Bruce Fields" <bfields@fieldses.org> wrote:
> > 
> > (Maybe you built a new kernel with nfsd built-in instead of built as a
> > module?)
> > 
> > But I thought nfs-utils was supposed to fall back on old behavior when
> > the nfsd filesystem wasn't found.
> 
> Doesn't seem so.

Right.  The old behavior depends on mountd telling the server about each
client as it mounts, and that isn't adequate in some cases: for example,
since nfsv4 clients never talk directly to mountd at all (they first
call nfsd directly), they need to be able to call up to mountd to ask
about each client as it arrives, and the old interface doesn't provide
a mechanism to do that.

But in this case I don't see any reason the old interface shouldn't be
good enough.

> Steve: any chance this bug is fixed anywhere?

It shouldn't be hard to fix.

If someone has the time to work on it but doesn't know the code, just
testing with varied versions of nfs-utils and the kernel would help
determine whether the problem is in mountd or in the kernel, and when it
was introduced.

--b.