unable to mount nfs4 mount

unable to mount nfs4 mount

[daggs]

Greetings,

I'm trying to mount an nfs mount it seems ti fails, I'd appreciate some help on the matter.
the client is a gentoo setup with kernel 4.9.0, nfs-utils-1.3.4.
running showmount -e 10.0.0.10 (the server) returns this:
Export list for 10.0.0.10:
/mnt/nfs_exports/media 10.0.0.0/24
/mnt/nfs_exports       10.0.0.0/24

when I try to mount I get this:
mount -v -t nfs 10.0.0.10://mnt/nfs_exports/media /tmp/media -o vers=4,rw,async,auto
mount.nfs: timeout set for Fri Dec 23 14:30:56 2016
mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
mount.nfs: mount(2): Connection timed out
mount.nfs: Connection timed

the server is a odroidc2 board, the kernel is based 3.14.79 (I know it is old but there is still no full mainline kernel support for this board), nfs-utils-1.3.3 built with latest buildroot.
cat /etc/exportfs returns this:
/mnt/nfs_exports        10.0.0.0/24(rw,fsid=0,no_subtree_check)
/mnt/nfs_exports/media  10.0.0.0/24(rw,nohide,insecure,no_subtree_check)

ps aux | egrep "nfs|rpc" on the server returns this:
   37 root     [rpciod]
   41 root     [nfsiod]
  132 root     /usr/bin/rpcbind
  194 root     [nfsd4]
  195 root     [nfsd4_callbacks]
  199 root     [nfsd]
  200 root     [nfsd]
  233 root     rpc.statd
  237 root     /usr/sbin/rpc.idmapd
  241 root     rpc.mountd -V 3 -V 4

connection logs shows this:
[    4.278267] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[    4.278556] NFSD: starting 90-second grace period (net ffffffc001d42600)
[   20.777604] svc: socket ffffffc05bca4700 TCP (listen) state change 10
[   20.777805] svc: transport ffffffc05bc5d000 served by daemon ffffffc05bfec000
[   20.778100] svc: socket ffffffc05bf8ce00 TCP (listen) state change 1
[   20.778378] svc: tcp_accept ffffffc05bc5d000 sock ffffffc05ca10000
[   20.778618] nfsd: connect from 10.0.0.1, port=914
[   20.778811] svc: svc_setup_socket ffffffc05ca0ef00
[   20.779009] setting up TCP socket for reading
[   20.779190] svc: svc_setup_socket created ffffffc05c393000 (inet ffffffc05bf8ce00)
[   20.779501] svc: transport ffffffc05c393000 served by daemon ffffffc05bf8e000
[   20.779797] svc: server ffffffc05bf8e000, pool 0, transport ffffffc05c393000, inuse=3
[   20.780119] svc: tcp_recv ffffffc05c393000 data 1 conn 0 close 0
[   20.780368] svc: socket ffffffc05c393000 recvfrom(ffffffc05c3932bc, 0) = 4
[   20.780651] svc: TCP record, 40 bytes
[   20.780841] svc: socket ffffffc05c393000 recvfrom(ffffffc05be54028, 4056) = 40
[   20.781101] svc: TCP final record (40 bytes)
[   20.781292] svc: got len=40
[   20.781481] svc: svc_authenticate (0)
[   20.781670] svc: calling dispatcher
[   20.781860] svc: socket ffffffc05c393000 sendto([ffffffc05c270000 28... ], 28) = 28 (addr 10.0.0.1, port=914)
[   20.782232] svc: socket ffffffc05bf8ce00 TCP data ready (svsk ffffffc05c393000)
[   20.782532] svc: transport ffffffc05c393000 put into queue
[   20.782759] svc: transport ffffffc05c393000 busy, not enqueued
[   20.782999] svc: server ffffffc05bf8e000 waiting for data (to = 360000)
[   20.783273] svc: transport ffffffc05c393000 dequeued, inuse=2
[   20.783510] svc: server ffffffc05bf8e000, pool 0, transport ffffffc05c393000, inuse=3
[   20.783833] svc: tcp_recv ffffffc05c393000 data 1 conn 0 close 0
[   20.784081] svc: socket ffffffc05c393000 recvfrom(ffffffc05c3932bc, 0) = 4
[   20.784365] svc: TCP record, 172 bytes
[   20.784557] svc: socket ffffffc05c393000 recvfrom(ffffffc05be540ac, 3924) = 172
[   20.784821] svc: TCP final record (172 bytes)
[   20.785012] svc: got len=172
[   20.785200] svc: svc_authenticate (1)
[   20.785392] svc: transport ffffffc05bc5d000 put into queue
[   20.785579] svc: server ffffffc05bfec000 waiting for data (to = 360000)
[   20.785845] svc: transport ffffffc05bc5d000 dequeued, inuse=1
[   20.786082] svc: tcp_accept ffffffc05bc5d000 sock ffffffc05ca10000
[   20.786338] svc: server ffffffc05bfec000 waiting for data (to = 360000)
[   21.778372] svc: svc_process dropit
[   21.778565] svc: xprt ffffffc05c393000 dropped request
[   21.778757] svc: server ffffffc05bf8e000 waiting for data (to = 360000)
[   28.464973] svc: socket ffffffc05bf8ce00 TCP (connected) state change 8 (svsk ffffffc05c393000)
[   28.465297] svc: transport ffffffc05c393000 served by daemon ffffffc05bf8e000
[   28.465591] svc: socket ffffffc05bf8ce00 TCP data ready (svsk ffffffc05c393000)
[   28.465893] svc: transport ffffffc05c393000 busy, not enqueued
[   28.466147] svc_recv: found XPT_CLOSE
[   28.466340] svc: svc_delete_xprt(ffffffc05c393000)
[   28.466535] svc: svc_tcp_sock_detach(ffffffc05c393000)
[   28.466728] svc: svc_sock_detach(ffffffc05c393000)
[   28.466922] svc: server ffffffc05bf8e000 waiting for data (to = 360000)

cat /etc/idmapd.conf on the server returns:
[General]

Verbosity = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
Domain = localdomain

[Mapping]

Nobody-User = nobody
Nobody-Group = nogroup

any ideas what can be the issue?
is this a kernel bug that was fixed in later stages?

Thanks,

Dagg.

Re: unable to mount nfs4 mount

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1213 bytes --]

On Sat, Dec 24 2016, daggs wrote:

> Greetings,
>
> I'm trying to mount an nfs mount it seems ti fails, I'd appreciate some help on the matter.
> the client is a gentoo setup with kernel 4.9.0, nfs-utils-1.3.4.
> running showmount -e 10.0.0.10 (the server) returns this:
> Export list for 10.0.0.10:
> /mnt/nfs_exports/media 10.0.0.0/24
> /mnt/nfs_exports       10.0.0.0/24
>
> when I try to mount I get this:
> mount -v -t nfs 10.0.0.10://mnt/nfs_exports/media /tmp/media -o vers=4,rw,async,auto
> mount.nfs: timeout set for Fri Dec 23 14:30:56 2016
> mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
> mount.nfs: mount(2): Connection timed out
> mount.nfs: Connection timed
>
> the server is a odroidc2 board, the kernel is based 3.14.79 (I know it is old but there is still no full mainline kernel support for this board), nfs-utils-1.3.3 built with latest buildroot.
> cat /etc/exportfs returns this:
> /mnt/nfs_exports        10.0.0.0/24(rw,fsid=0,no_subtree_check)
                                         ^^^^^^^

Get rid of this, or get rid of "/mnt/nfs_exports" in your mount command.
i.e.
  mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: unable to mount nfs4 mount

[daggs]

Greetings,
> 
> > Greetings,
> >
> > I'm trying to mount an nfs mount it seems ti fails, I'd appreciate some help on the matter.
> > the client is a gentoo setup with kernel 4.9.0, nfs-utils-1.3.4.
> > running showmount -e 10.0.0.10 (the server) returns this:
> > Export list for 10.0.0.10:
> > /mnt/nfs_exports/media 10.0.0.0/24
> > /mnt/nfs_exports       10.0.0.0/24
> >
> > when I try to mount I get this:
> > mount -v -t nfs 10.0.0.10://mnt/nfs_exports/media /tmp/media -o vers=4,rw,async,auto
> > mount.nfs: timeout set for Fri Dec 23 14:30:56 2016
> > mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
> > mount.nfs: mount(2): Connection timed out
> > mount.nfs: Connection timed
> >
> > the server is a odroidc2 board, the kernel is based 3.14.79 (I know it is old but there is still no full mainline kernel support for this board), nfs-utils-1.3.3 built with latest buildroot.
> > cat /etc/exportfs returns this:
> > /mnt/nfs_exports        10.0.0.0/24(rw,fsid=0,no_subtree_check)
>                                          ^^^^^^^
> 
> Get rid of this, or get rid of "/mnt/nfs_exports" in your mount command.
> i.e.
>   mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto
> 

tried the mount line, same behavior.
client:
mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto
mount.nfs: timeout set for Sat Dec 24 09:07:17 2016
mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
mount.nfs: mount(2): Connection timed out
mount.nfs: Connection timed out

server:
[    7.585428] svc: socket ffffffc05bfaa000 TCP (listen) state change 10
[    7.585628] svc: transport ffffffc05c05c000 served by daemon ffffffc05bfd0000
[    7.585924] svc: socket ffffffc05c3bce00 TCP (listen) state change 1
[    7.586198] svc: tcp_accept ffffffc05c05c000 sock ffffffc05ca05180
[    7.586441] nfsd: connect from 10.0.0.1, port=737
[    7.586634] svc: svc_setup_socket ffffffc05c8ef400
[    7.586832] setting up TCP socket for reading
[    7.587011] svc: svc_setup_socket created ffffffc05bc70000 (inet ffffffc05c3bce00)
[    7.587324] svc: transport ffffffc05bc70000 put into queue
[    7.587551] svc: transport ffffffc05c05c000 put into queue
[    7.587777] svc: server ffffffc05bfd0000 waiting for data (to = 360000)
[    7.588050] svc: transport ffffffc05bc70000 dequeued, inuse=1
[    7.588287] svc: server ffffffc05bfd0000, pool 0, transport ffffffc05bc70000, inuse=2
[    7.588612] svc: tcp_recv ffffffc05bc70000 data 1 conn 0 close 0
[    7.588859] svc: socket ffffffc05bc70000 recvfrom(ffffffc05bc702bc, 0) = 4
[    7.589143] svc: TCP record, 40 bytes
[    7.589330] svc: socket ffffffc05bc70000 recvfrom(ffffffc05c272028, 4056) = 40
[    7.589592] svc: TCP final record (40 bytes)
[    7.589781] svc: got len=40
[    7.589970] svc: svc_authenticate (0)
[    7.590156] svc: calling dispatcher
[    7.590344] svc: socket ffffffc05bc70000 sendto([ffffffc05c3e8000 28... ], 28) = 28 (addr 10.0.0.1, port=737)
[    7.590715] svc: socket ffffffc05c3bce00 TCP data ready (svsk ffffffc05bc70000)
[    7.591015] svc: transport ffffffc05bc70000 put into queue
[    7.591242] svc: transport ffffffc05bc70000 busy, not enqueued
[    7.591482] svc: server ffffffc05bfd0000 waiting for data (to = 360000)
[    7.591756] svc: transport ffffffc05c05c000 dequeued, inuse=1
[    7.591993] svc: tcp_accept ffffffc05c05c000 sock ffffffc05ca05180
[    7.592249] svc: server ffffffc05bfd0000 waiting for data (to = 360000)
[    7.592521] svc: transport ffffffc05bc70000 dequeued, inuse=1
[    7.592758] svc: server ffffffc05bfd0000, pool 0, transport ffffffc05bc70000, inuse=2
[    7.593081] svc: tcp_recv ffffffc05bc70000 data 1 conn 0 close 0
[    7.593329] svc: socket ffffffc05bc70000 recvfrom(ffffffc05bc702bc, 0) = 4
[    7.593613] svc: TCP record, 172 bytes
[    7.593801] svc: socket ffffffc05bc70000 recvfrom(ffffffc05c2720ac, 3924) = 172
[    7.594070] svc: TCP final record (172 bytes)
[    7.594260] svc: got len=172
[    7.594448] svc: svc_authenticate (1)
[    7.594636] RPC:       Want update, refage=120, age=0
[    8.588359] svc: svc_process dropit
[    8.588550] svc: xprt ffffffc05bc70000 dropped request
[    8.588739] svc: server ffffffc05bfd0000 waiting for data (to = 360000)
[   10.518359] svc: svc_process dropit
[   10.518550] svc: xprt ffffffc05c396000 dropped request
[   10.518738] svc: transport ffffffc05c396000 busy, not enqueued
[   10.518956] svc: server ffffffc05b812000 waiting for data (to = 360000)
[   15.684420] svc: socket ffffffc05c3bce00 TCP (connected) state change 8 (svsk ffffffc05bc70000)
[   15.684740] svc: transport ffffffc05bc70000 served by daemon ffffffc05b812000
[   15.685035] svc: socket ffffffc05c3bce00 TCP data ready (svsk ffffffc05bc70000)
[   15.685337] svc: transport ffffffc05bc70000 busy, not enqueued
[   15.685592] svc_recv: found XPT_CLOSE
[   15.685785] svc: svc_delete_xprt(ffffffc05bc70000)
[   15.685979] svc: svc_tcp_sock_detach(ffffffc05bc70000)
[   15.686174] svc: svc_sock_detach(ffffffc05bc70000)
[   15.686370] svc: server ffffffc05b812000 waiting for data (to = 360000)

my /etc/exports if copied from the gentoo setup which has a working nfsv4 server.

Dagg.

Re: unable to mount nfs4 mount

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 6100 bytes --]

On Sat, Dec 24 2016, daggs wrote:

> Greetings,
>> 
>> > Greetings,
>> >
>> > I'm trying to mount an nfs mount it seems ti fails, I'd appreciate some help on the matter.
>> > the client is a gentoo setup with kernel 4.9.0, nfs-utils-1.3.4.
>> > running showmount -e 10.0.0.10 (the server) returns this:
>> > Export list for 10.0.0.10:
>> > /mnt/nfs_exports/media 10.0.0.0/24
>> > /mnt/nfs_exports       10.0.0.0/24
>> >
>> > when I try to mount I get this:
>> > mount -v -t nfs 10.0.0.10://mnt/nfs_exports/media /tmp/media -o vers=4,rw,async,auto
>> > mount.nfs: timeout set for Fri Dec 23 14:30:56 2016
>> > mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
>> > mount.nfs: mount(2): Connection timed out
>> > mount.nfs: Connection timed
>> >
>> > the server is a odroidc2 board, the kernel is based 3.14.79 (I know it is old but there is still no full mainline kernel support for this board), nfs-utils-1.3.3 built with latest buildroot.
>> > cat /etc/exportfs returns this:
>> > /mnt/nfs_exports        10.0.0.0/24(rw,fsid=0,no_subtree_check)
>>                                          ^^^^^^^
>> 
>> Get rid of this, or get rid of "/mnt/nfs_exports" in your mount command.
>> i.e.
>>   mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto
>> 
>
> tried the mount line, same behavior.
> client:
> mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto
> mount.nfs: timeout set for Sat Dec 24 09:07:17 2016
> mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
> mount.nfs: mount(2): Connection timed out
> mount.nfs: Connection timed out

Sorry - I should have known that.  I just responded to the first obvious
error I saw, without confirming that it would have the results
reported.  The error I saw wouldn't cause "Connection timed out".

This message
> [    8.588359] svc: svc_process dropit

might suggest that rpc.mountd isn't running. Is it?
If it is, is it producing any error messages in the logs?

NeilBrown


>
> server:
> [    7.585428] svc: socket ffffffc05bfaa000 TCP (listen) state change 10
> [    7.585628] svc: transport ffffffc05c05c000 served by daemon ffffffc05bfd0000
> [    7.585924] svc: socket ffffffc05c3bce00 TCP (listen) state change 1
> [    7.586198] svc: tcp_accept ffffffc05c05c000 sock ffffffc05ca05180
> [    7.586441] nfsd: connect from 10.0.0.1, port=737
> [    7.586634] svc: svc_setup_socket ffffffc05c8ef400
> [    7.586832] setting up TCP socket for reading
> [    7.587011] svc: svc_setup_socket created ffffffc05bc70000 (inet ffffffc05c3bce00)
> [    7.587324] svc: transport ffffffc05bc70000 put into queue
> [    7.587551] svc: transport ffffffc05c05c000 put into queue
> [    7.587777] svc: server ffffffc05bfd0000 waiting for data (to = 360000)
> [    7.588050] svc: transport ffffffc05bc70000 dequeued, inuse=1
> [    7.588287] svc: server ffffffc05bfd0000, pool 0, transport ffffffc05bc70000, inuse=2
> [    7.588612] svc: tcp_recv ffffffc05bc70000 data 1 conn 0 close 0
> [    7.588859] svc: socket ffffffc05bc70000 recvfrom(ffffffc05bc702bc, 0) = 4
> [    7.589143] svc: TCP record, 40 bytes
> [    7.589330] svc: socket ffffffc05bc70000 recvfrom(ffffffc05c272028, 4056) = 40
> [    7.589592] svc: TCP final record (40 bytes)
> [    7.589781] svc: got len=40
> [    7.589970] svc: svc_authenticate (0)
> [    7.590156] svc: calling dispatcher
> [    7.590344] svc: socket ffffffc05bc70000 sendto([ffffffc05c3e8000 28... ], 28) = 28 (addr 10.0.0.1, port=737)
> [    7.590715] svc: socket ffffffc05c3bce00 TCP data ready (svsk ffffffc05bc70000)
> [    7.591015] svc: transport ffffffc05bc70000 put into queue
> [    7.591242] svc: transport ffffffc05bc70000 busy, not enqueued
> [    7.591482] svc: server ffffffc05bfd0000 waiting for data (to = 360000)
> [    7.591756] svc: transport ffffffc05c05c000 dequeued, inuse=1
> [    7.591993] svc: tcp_accept ffffffc05c05c000 sock ffffffc05ca05180
> [    7.592249] svc: server ffffffc05bfd0000 waiting for data (to = 360000)
> [    7.592521] svc: transport ffffffc05bc70000 dequeued, inuse=1
> [    7.592758] svc: server ffffffc05bfd0000, pool 0, transport ffffffc05bc70000, inuse=2
> [    7.593081] svc: tcp_recv ffffffc05bc70000 data 1 conn 0 close 0
> [    7.593329] svc: socket ffffffc05bc70000 recvfrom(ffffffc05bc702bc, 0) = 4
> [    7.593613] svc: TCP record, 172 bytes
> [    7.593801] svc: socket ffffffc05bc70000 recvfrom(ffffffc05c2720ac, 3924) = 172
> [    7.594070] svc: TCP final record (172 bytes)
> [    7.594260] svc: got len=172
> [    7.594448] svc: svc_authenticate (1)
> [    7.594636] RPC:       Want update, refage=120, age=0
> [    8.588359] svc: svc_process dropit
> [    8.588550] svc: xprt ffffffc05bc70000 dropped request
> [    8.588739] svc: server ffffffc05bfd0000 waiting for data (to = 360000)
> [   10.518359] svc: svc_process dropit
> [   10.518550] svc: xprt ffffffc05c396000 dropped request
> [   10.518738] svc: transport ffffffc05c396000 busy, not enqueued
> [   10.518956] svc: server ffffffc05b812000 waiting for data (to = 360000)
> [   15.684420] svc: socket ffffffc05c3bce00 TCP (connected) state change 8 (svsk ffffffc05bc70000)
> [   15.684740] svc: transport ffffffc05bc70000 served by daemon ffffffc05b812000
> [   15.685035] svc: socket ffffffc05c3bce00 TCP data ready (svsk ffffffc05bc70000)
> [   15.685337] svc: transport ffffffc05bc70000 busy, not enqueued
> [   15.685592] svc_recv: found XPT_CLOSE
> [   15.685785] svc: svc_delete_xprt(ffffffc05bc70000)
> [   15.685979] svc: svc_tcp_sock_detach(ffffffc05bc70000)
> [   15.686174] svc: svc_sock_detach(ffffffc05bc70000)
> [   15.686370] svc: server ffffffc05b812000 waiting for data (to = 360000)
>
> my /etc/exports if copied from the gentoo setup which has a working nfsv4 server.
>
> Dagg.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: unable to mount nfs4 mount

[daggs]

Greeting Neil,

> > Greetings,
> >> 
> >> > Greetings,
> >> >
> >> > I'm trying to mount an nfs mount it seems ti fails, I'd appreciate some help on the matter.
> >> > the client is a gentoo setup with kernel 4.9.0, nfs-utils-1.3.4.
> >> > running showmount -e 10.0.0.10 (the server) returns this:
> >> > Export list for 10.0.0.10:
> >> > /mnt/nfs_exports/media 10.0.0.0/24
> >> > /mnt/nfs_exports       10.0.0.0/24
> >> >
> >> > when I try to mount I get this:
> >> > mount -v -t nfs 10.0.0.10://mnt/nfs_exports/media /tmp/media -o vers=4,rw,async,auto
> >> > mount.nfs: timeout set for Fri Dec 23 14:30:56 2016
> >> > mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
> >> > mount.nfs: mount(2): Connection timed out
> >> > mount.nfs: Connection timed
> >> >
> >> > the server is a odroidc2 board, the kernel is based 3.14.79 (I know it is old but there is still no full mainline kernel support for this board), nfs-utils-1.3.3 built with latest buildroot.
> >> > cat /etc/exportfs returns this:
> >> > /mnt/nfs_exports        10.0.0.0/24(rw,fsid=0,no_subtree_check)
> >>                                          ^^^^^^^
> >> 
> >> Get rid of this, or get rid of "/mnt/nfs_exports" in your mount command.
> >> i.e.
> >>   mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto
> >> 
> >
> > tried the mount line, same behavior.
> > client:
> > mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto
> > mount.nfs: timeout set for Sat Dec 24 09:07:17 2016
> > mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
> > mount.nfs: mount(2): Connection timed out
> > mount.nfs: Connection timed out
> 
> Sorry - I should have known that.  I just responded to the first obvious
> error I saw, without confirming that it would have the results
> reported.  The error I saw wouldn't cause "Connection timed out".
> 
> This message
> > [    8.588359] svc: svc_process dropit
> 
> might suggest that rpc.mountd isn't running. Is it?
> If it is, is it producing any error messages in the logs?
> 

from the original post:
ps aux | egrep "nfs|rpc" on the server returns this:
37 root [rpciod]
41 root [nfsiod]
132 root /usr/bin/rpcbind
194 root [nfsd4]
195 root [nfsd4_callbacks]
199 root [nfsd]
200 root [nfsd]
233 root rpc.statd
237 root /usr/sbin/rpc.idmapd
241 root rpc.mountd -V 3 -V 4

all the prints from the logs that I've found (/var/log/messages) exist in the previous mail I've sent.
is there another log I'm missing? if so, where can I find it?

Thanks.

Dagg.

Re: unable to mount nfs4 mount

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3134 bytes --]

On Sun, Dec 25 2016, daggs wrote:

> Greeting Neil,
>
>> > Greetings,
>> >> 
>> >> > Greetings,
>> >> >
>> >> > I'm trying to mount an nfs mount it seems ti fails, I'd appreciate some help on the matter.
>> >> > the client is a gentoo setup with kernel 4.9.0, nfs-utils-1.3.4.
>> >> > running showmount -e 10.0.0.10 (the server) returns this:
>> >> > Export list for 10.0.0.10:
>> >> > /mnt/nfs_exports/media 10.0.0.0/24
>> >> > /mnt/nfs_exports       10.0.0.0/24
>> >> >
>> >> > when I try to mount I get this:
>> >> > mount -v -t nfs 10.0.0.10://mnt/nfs_exports/media /tmp/media -o vers=4,rw,async,auto
>> >> > mount.nfs: timeout set for Fri Dec 23 14:30:56 2016
>> >> > mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
>> >> > mount.nfs: mount(2): Connection timed out
>> >> > mount.nfs: Connection timed
>> >> >
>> >> > the server is a odroidc2 board, the kernel is based 3.14.79 (I know it is old but there is still no full mainline kernel support for this board), nfs-utils-1.3.3 built with latest buildroot.
>> >> > cat /etc/exportfs returns this:
>> >> > /mnt/nfs_exports        10.0.0.0/24(rw,fsid=0,no_subtree_check)
>> >>                                          ^^^^^^^
>> >> 
>> >> Get rid of this, or get rid of "/mnt/nfs_exports" in your mount command.
>> >> i.e.
>> >>   mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto
>> >> 
>> >
>> > tried the mount line, same behavior.
>> > client:
>> > mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto
>> > mount.nfs: timeout set for Sat Dec 24 09:07:17 2016
>> > mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
>> > mount.nfs: mount(2): Connection timed out
>> > mount.nfs: Connection timed out
>> 
>> Sorry - I should have known that.  I just responded to the first obvious
>> error I saw, without confirming that it would have the results
>> reported.  The error I saw wouldn't cause "Connection timed out".
>> 
>> This message
>> > [    8.588359] svc: svc_process dropit
>> 
>> might suggest that rpc.mountd isn't running. Is it?
>> If it is, is it producing any error messages in the logs?
>> 
>
> from the original post:
> ps aux | egrep "nfs|rpc" on the server returns this:
> 37 root [rpciod]
> 41 root [nfsiod]
> 132 root /usr/bin/rpcbind
> 194 root [nfsd4]
> 195 root [nfsd4_callbacks]
> 199 root [nfsd]
> 200 root [nfsd]
> 233 root rpc.statd
> 237 root /usr/sbin/rpc.idmapd
> 241 root rpc.mountd -V 3 -V 4

Sorry, I missed that.
Still, it looks like mountd might be misbehaving.  That is a good place
to start anyway.
Can you strace mountd while you attempt a mount?
e.g.
  strace -o /tmp/trace -s 1000 -p 241

and send the /tmp/trace.
Also, after the attempt fails, run
 rpcdebug -m rpc -s cache
 grep . /proc/net/rpc/*/c*
 cat /proc/fs/nfsd/exports

and report the output.

NeilBrown

> all the prints from the logs that I've found (/var/log/messages) exist in the previous mail I've sent.
> is there another log I'm missing? if so, where can I find it?
>
> Thanks.
>
> Dagg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: unable to mount nfs4 mount

[daggs]

Greetings Neil,

> > Greeting Neil,
> >
> >> > Greetings,
> >> >> 
> >> >> > Greetings,
> >> >> >
> >> >> > I'm trying to mount an nfs mount it seems ti fails, I'd appreciate some help on the matter.
> >> >> > the client is a gentoo setup with kernel 4.9.0, nfs-utils-1.3.4.
> >> >> > running showmount -e 10.0.0.10 (the server) returns this:
> >> >> > Export list for 10.0.0.10:
> >> >> > /mnt/nfs_exports/media 10.0.0.0/24
> >> >> > /mnt/nfs_exports       10.0.0.0/24
> >> >> >
> >> >> > when I try to mount I get this:
> >> >> > mount -v -t nfs 10.0.0.10://mnt/nfs_exports/media /tmp/media -o vers=4,rw,async,auto
> >> >> > mount.nfs: timeout set for Fri Dec 23 14:30:56 2016
> >> >> > mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
> >> >> > mount.nfs: mount(2): Connection timed out
> >> >> > mount.nfs: Connection timed
> >> >> >
> >> >> > the server is a odroidc2 board, the kernel is based 3.14.79 (I know it is old but there is still no full mainline kernel support for this board), nfs-utils-1.3.3 built with latest buildroot.
> >> >> > cat /etc/exportfs returns this:
> >> >> > /mnt/nfs_exports        10.0.0.0/24(rw,fsid=0,no_subtree_check)
> >> >>                                          ^^^^^^^
> >> >> 
> >> >> Get rid of this, or get rid of "/mnt/nfs_exports" in your mount command.
> >> >> i.e.
> >> >>   mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto
> >> >> 
> >> >
> >> > tried the mount line, same behavior.
> >> > client:
> >> > mount -v -t nfs 10.0.0.10:/media /tmp/media -o vers=4,rw,async,auto
> >> > mount.nfs: timeout set for Sat Dec 24 09:07:17 2016
> >> > mount.nfs: trying text-based options 'vers=4,addr=10.0.0.10,clientaddr=10.0.0.1'
> >> > mount.nfs: mount(2): Connection timed out
> >> > mount.nfs: Connection timed out
> >> 
> >> Sorry - I should have known that.  I just responded to the first obvious
> >> error I saw, without confirming that it would have the results
> >> reported.  The error I saw wouldn't cause "Connection timed out".
> >> 
> >> This message
> >> > [    8.588359] svc: svc_process dropit
> >> 
> >> might suggest that rpc.mountd isn't running. Is it?
> >> If it is, is it producing any error messages in the logs?
> >> 
> >
> > from the original post:
> > ps aux | egrep "nfs|rpc" on the server returns this:
> > 37 root [rpciod]
> > 41 root [nfsiod]
> > 132 root /usr/bin/rpcbind
> > 194 root [nfsd4]
> > 195 root [nfsd4_callbacks]
> > 199 root [nfsd]
> > 200 root [nfsd]
> > 233 root rpc.statd
> > 237 root /usr/sbin/rpc.idmapd
> > 241 root rpc.mountd -V 3 -V 4
> 
> Sorry, I missed that.
> Still, it looks like mountd might be misbehaving.  That is a good place
> to start anyway.
> Can you strace mountd while you attempt a mount?
> e.g.
>   strace -o /tmp/trace -s 1000 -p 241
> 
> and send the /tmp/trace.
> Also, after the attempt fails, run
>  rpcdebug -m rpc -s cache
>  grep . /proc/net/rpc/*/c*
>  cat /proc/fs/nfsd/exports
> 
> and report the output.
> 
here:

# cat /tmp/trace
pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL) = 1 (in [3])
read(3, "nfsd 10.0.0.1\n", 32768)       = 14
openat(AT_FDCWD, "/run/nfs/etab", O_RDONLY) = 14
fstat(14, {st_mode=S_IFREG|0644, st_size=435, ...}) = 0
close(14)                               = 0
write(3, "nfsd 10.0.0.1 2079 10.0.0.0/24 \n", 32) = 32
pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL <detached ...>
# rpcdebug -m rpc -s cache
rpc        cache
# grep . /proc/net/rpc/*/c*
/proc/net/rpc/auth.unix.gid/content:#uid cnt: gids...
/proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
/proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
/proc/net/rpc/auth.unix.ip/content:#class IP domain
/proc/net/rpc/auth.unix.ip/content:# expiry=2079 refcnt=1 flags=1
/proc/net/rpc/auth.unix.ip/content:# nfsd 10.0.0.1 10.0.0.0/24
/proc/net/rpc/nfs4.idtoname/content:#domain type id [name]
/proc/net/rpc/nfs4.nametoid/content:#domain type name [id]
/proc/net/rpc/nfsd.export/content:#path domain(flags)
/proc/net/rpc/nfsd.fh/content:#domain fsidtype fsid [path]
# cat /proc/fs/nfsd/exports
# Version 1.1
# Path Client(Flags) # IPs

Dagg.

Re: unable to mount nfs4 mount

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2460 bytes --]

On Mon, Dec 26 2016, daggs wrote:

>> Can you strace mountd while you attempt a mount?
>> e.g.
>>   strace -o /tmp/trace -s 1000 -p 241
>> 
>> and send the /tmp/trace.
>> Also, after the attempt fails, run
>>  rpcdebug -m rpc -s cache
>>  grep . /proc/net/rpc/*/c*
>>  cat /proc/fs/nfsd/exports
>> 
>> and report the output.
>> 
> here:
>
> # cat /tmp/trace
> pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL) = 1 (in [3])
> read(3, "nfsd 10.0.0.1\n", 32768)       = 14
> openat(AT_FDCWD, "/run/nfs/etab", O_RDONLY) = 14
> fstat(14, {st_mode=S_IFREG|0644, st_size=435, ...}) = 0
> close(14)                               = 0
> write(3, "nfsd 10.0.0.1 2079 10.0.0.0/24 \n", 32) = 32

This is weird.
Here mountd is telling nfsd that when a request comes from IP address
10.0.0.1, it should look for export entries associated with the client
name "10.0.0.0/24", which is good.
However the expiry time for that information is "2079", which is back in
January 1970.
When mountd writes that number, it computes it as
   time(0) + DEFAULT_TTL
where DEFAULT_TTL is (30 * 60)
Which suggests time(0) is "279".

What is the current time on this system?

If it really was very early on Jan 1st 1970, it should work, however...


> pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL <detached ...>
> # rpcdebug -m rpc -s cache
> rpc        cache
> # grep . /proc/net/rpc/*/c*
> /proc/net/rpc/auth.unix.gid/content:#uid cnt: gids...
> /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
> /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
> /proc/net/rpc/auth.unix.ip/content:#class IP domain
> /proc/net/rpc/auth.unix.ip/content:# expiry=2079 refcnt=1 flags=1
> /proc/net/rpc/auth.unix.ip/content:# nfsd 10.0.0.1 10.0.0.0/24

...the fact that this line is commented out indicates that the entry in
the cache is already expired.  So the time must be after 2079.

Maybe the time is getting set from the network at an awkward time that
races with NFS service some how.
Can you find a way to run "exportfs -f" after the time has been set
correctly?

NeilBrown


> /proc/net/rpc/nfs4.idtoname/content:#domain type id [name]
> /proc/net/rpc/nfs4.nametoid/content:#domain type name [id]
> /proc/net/rpc/nfsd.export/content:#path domain(flags)
> /proc/net/rpc/nfsd.fh/content:#domain fsidtype fsid [path]
> # cat /proc/fs/nfsd/exports
> # Version 1.1
> # Path Client(Flags) # IPs
>
> Dagg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: unable to mount nfs4 mount

[daggs]

Greetings,

> On Mon, Dec 26 2016, daggs wrote:
> 
> >> Can you strace mountd while you attempt a mount?
> >> e.g.
> >>   strace -o /tmp/trace -s 1000 -p 241
> >> 
> >> and send the /tmp/trace.
> >> Also, after the attempt fails, run
> >>  rpcdebug -m rpc -s cache
> >>  grep . /proc/net/rpc/*/c*
> >>  cat /proc/fs/nfsd/exports
> >> 
> >> and report the output.
> >> 
> > here:
> >
> > # cat /tmp/trace
> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL) = 1 (in [3])
> > read(3, "nfsd 10.0.0.1\n", 32768)       = 14
> > openat(AT_FDCWD, "/run/nfs/etab", O_RDONLY) = 14
> > fstat(14, {st_mode=S_IFREG|0644, st_size=435, ...}) = 0
> > close(14)                               = 0
> > write(3, "nfsd 10.0.0.1 2079 10.0.0.0/24 \n", 32) = 32
> 
> This is weird.
> Here mountd is telling nfsd that when a request comes from IP address
> 10.0.0.1, it should look for export entries associated with the client
> name "10.0.0.0/24", which is good.
> However the expiry time for that information is "2079", which is back in
> January 1970.
> When mountd writes that number, it computes it as
>    time(0) + DEFAULT_TTL
> where DEFAULT_TTL is (30 * 60)
> Which suggests time(0) is "279".
> 
> What is the current time on this system?
> 
> If it really was very early on Jan 1st 1970, it should work, however...
> 
> 
> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL <detached ...>
> > # rpcdebug -m rpc -s cache
> > rpc        cache
> > # grep . /proc/net/rpc/*/c*
> > /proc/net/rpc/auth.unix.gid/content:#uid cnt: gids...
> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
> > /proc/net/rpc/auth.unix.ip/content:#class IP domain
> > /proc/net/rpc/auth.unix.ip/content:# expiry=2079 refcnt=1 flags=1
> > /proc/net/rpc/auth.unix.ip/content:# nfsd 10.0.0.1 10.0.0.0/24
> 
> ...the fact that this line is commented out indicates that the entry in
> the cache is already expired.  So the time must be after 2079.
> 
> Maybe the time is getting set from the network at an awkward time that
> races with NFS service some how.
> Can you find a way to run "exportfs -f" after the time has been set
> correctly?
> 
> NeilBrown
> 
> 

wait, I think I've seen this somewhere, does this feature needs rtc? this board doesn't have rtc component.
for example, I cannot use openssh as ssh server because it needs rtc. I have to use dropbear.
if so, this looks like it will affect nfsv3 mounts, am I right?

Dagg.

Re: unable to mount nfs4 mount

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3005 bytes --]

On Mon, Dec 26 2016, daggs wrote:

> Greetings,
>
>> On Mon, Dec 26 2016, daggs wrote:
>> 
>> >> Can you strace mountd while you attempt a mount?
>> >> e.g.
>> >>   strace -o /tmp/trace -s 1000 -p 241
>> >> 
>> >> and send the /tmp/trace.
>> >> Also, after the attempt fails, run
>> >>  rpcdebug -m rpc -s cache
>> >>  grep . /proc/net/rpc/*/c*
>> >>  cat /proc/fs/nfsd/exports
>> >> 
>> >> and report the output.
>> >> 
>> > here:
>> >
>> > # cat /tmp/trace
>> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL) = 1 (in [3])
>> > read(3, "nfsd 10.0.0.1\n", 32768)       = 14
>> > openat(AT_FDCWD, "/run/nfs/etab", O_RDONLY) = 14
>> > fstat(14, {st_mode=S_IFREG|0644, st_size=435, ...}) = 0
>> > close(14)                               = 0
>> > write(3, "nfsd 10.0.0.1 2079 10.0.0.0/24 \n", 32) = 32
>> 
>> This is weird.
>> Here mountd is telling nfsd that when a request comes from IP address
>> 10.0.0.1, it should look for export entries associated with the client
>> name "10.0.0.0/24", which is good.
>> However the expiry time for that information is "2079", which is back in
>> January 1970.
>> When mountd writes that number, it computes it as
>>    time(0) + DEFAULT_TTL
>> where DEFAULT_TTL is (30 * 60)
>> Which suggests time(0) is "279".
>> 
>> What is the current time on this system?
>> 
>> If it really was very early on Jan 1st 1970, it should work, however...
>> 
>> 
>> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL <detached ...>
>> > # rpcdebug -m rpc -s cache
>> > rpc        cache
>> > # grep . /proc/net/rpc/*/c*
>> > /proc/net/rpc/auth.unix.gid/content:#uid cnt: gids...
>> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
>> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
>> > /proc/net/rpc/auth.unix.ip/content:#class IP domain
>> > /proc/net/rpc/auth.unix.ip/content:# expiry=2079 refcnt=1 flags=1
>> > /proc/net/rpc/auth.unix.ip/content:# nfsd 10.0.0.1 10.0.0.0/24
>> 
>> ...the fact that this line is commented out indicates that the entry in
>> the cache is already expired.  So the time must be after 2079.
>> 
>> Maybe the time is getting set from the network at an awkward time that
>> races with NFS service some how.
>> Can you find a way to run "exportfs -f" after the time has been set
>> correctly?
>> 
>> NeilBrown
>> 
>> 
>
> wait, I think I've seen this somewhere, does this feature needs rtc? this board doesn't have rtc component.
> for example, I cannot use openssh as ssh server because it needs rtc. I have to use dropbear.
> if so, this looks like it will affect nfsv3 mounts, am I right?

No, you shouldn't need an RTC.
You need the synchronize the clock with ntp or similar, else time stamps
on files will look wrong.
Though I think we fixed issues with wall-clock-time jumping in 2.6.37...

If you could try using "exportfs -f", and explain what does happen with
time - do you use ntp ?? - we might be able to make progress.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: unable to mount nfs4 mount

[daggs]

Greetings Neil,

> On Mon, Dec 26 2016, daggs wrote:
> 
> > Greetings,
> >
> >> On Mon, Dec 26 2016, daggs wrote:
> >> 
> >> >> Can you strace mountd while you attempt a mount?
> >> >> e.g.
> >> >>   strace -o /tmp/trace -s 1000 -p 241
> >> >> 
> >> >> and send the /tmp/trace.
> >> >> Also, after the attempt fails, run
> >> >>  rpcdebug -m rpc -s cache
> >> >>  grep . /proc/net/rpc/*/c*
> >> >>  cat /proc/fs/nfsd/exports
> >> >> 
> >> >> and report the output.
> >> >> 
> >> > here:
> >> >
> >> > # cat /tmp/trace
> >> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL) = 1 (in [3])
> >> > read(3, "nfsd 10.0.0.1\n", 32768)       = 14
> >> > openat(AT_FDCWD, "/run/nfs/etab", O_RDONLY) = 14
> >> > fstat(14, {st_mode=S_IFREG|0644, st_size=435, ...}) = 0
> >> > close(14)                               = 0
> >> > write(3, "nfsd 10.0.0.1 2079 10.0.0.0/24 \n", 32) = 32
> >> 
> >> This is weird.
> >> Here mountd is telling nfsd that when a request comes from IP address
> >> 10.0.0.1, it should look for export entries associated with the client
> >> name "10.0.0.0/24", which is good.
> >> However the expiry time for that information is "2079", which is back in
> >> January 1970.
> >> When mountd writes that number, it computes it as
> >>    time(0) + DEFAULT_TTL
> >> where DEFAULT_TTL is (30 * 60)
> >> Which suggests time(0) is "279".
> >> 
> >> What is the current time on this system?
> >> 
> >> If it really was very early on Jan 1st 1970, it should work, however...
> >> 
> >> 
> >> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL <detached ...>
> >> > # rpcdebug -m rpc -s cache
> >> > rpc        cache
> >> > # grep . /proc/net/rpc/*/c*
> >> > /proc/net/rpc/auth.unix.gid/content:#uid cnt: gids...
> >> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
> >> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
> >> > /proc/net/rpc/auth.unix.ip/content:#class IP domain
> >> > /proc/net/rpc/auth.unix.ip/content:# expiry=2079 refcnt=1 flags=1
> >> > /proc/net/rpc/auth.unix.ip/content:# nfsd 10.0.0.1 10.0.0.0/24
> >> 
> >> ...the fact that this line is commented out indicates that the entry in
> >> the cache is already expired.  So the time must be after 2079.
> >> 
> >> Maybe the time is getting set from the network at an awkward time that
> >> races with NFS service some how.
> >> Can you find a way to run "exportfs -f" after the time has been set
> >> correctly?
> >> 
> >> NeilBrown
> >> 
> >> 
> >
> > wait, I think I've seen this somewhere, does this feature needs rtc? this board doesn't have rtc component.
> > for example, I cannot use openssh as ssh server because it needs rtc. I have to use dropbear.
> > if so, this looks like it will affect nfsv3 mounts, am I right?
> 
> No, you shouldn't need an RTC.
> You need the synchronize the clock with ntp or similar, else time stamps
> on files will look wrong.
> Though I think we fixed issues with wall-clock-time jumping in 2.6.37...
> 
> If you could try using "exportfs -f", and explain what does happen with
> time - do you use ntp ?? - we might be able to make progress.

I'll build ntp into the image and try. does this affects nfsv3 too?

what should I do with the "exportfs -f"? jsut run it and retry?

Thanks,

Dagg.

Re: unable to mount nfs4 mount

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3842 bytes --]

On Tue, Dec 27 2016, daggs wrote:

> Greetings Neil,
>
>> On Mon, Dec 26 2016, daggs wrote:
>> 
>> > Greetings,
>> >
>> >> On Mon, Dec 26 2016, daggs wrote:
>> >> 
>> >> >> Can you strace mountd while you attempt a mount?
>> >> >> e.g.
>> >> >>   strace -o /tmp/trace -s 1000 -p 241
>> >> >> 
>> >> >> and send the /tmp/trace.
>> >> >> Also, after the attempt fails, run
>> >> >>  rpcdebug -m rpc -s cache
>> >> >>  grep . /proc/net/rpc/*/c*
>> >> >>  cat /proc/fs/nfsd/exports
>> >> >> 
>> >> >> and report the output.
>> >> >> 
>> >> > here:
>> >> >
>> >> > # cat /tmp/trace
>> >> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL) = 1 (in [3])
>> >> > read(3, "nfsd 10.0.0.1\n", 32768)       = 14
>> >> > openat(AT_FDCWD, "/run/nfs/etab", O_RDONLY) = 14
>> >> > fstat(14, {st_mode=S_IFREG|0644, st_size=435, ...}) = 0
>> >> > close(14)                               = 0
>> >> > write(3, "nfsd 10.0.0.1 2079 10.0.0.0/24 \n", 32) = 32
>> >> 
>> >> This is weird.
>> >> Here mountd is telling nfsd that when a request comes from IP address
>> >> 10.0.0.1, it should look for export entries associated with the client
>> >> name "10.0.0.0/24", which is good.
>> >> However the expiry time for that information is "2079", which is back in
>> >> January 1970.
>> >> When mountd writes that number, it computes it as
>> >>    time(0) + DEFAULT_TTL
>> >> where DEFAULT_TTL is (30 * 60)
>> >> Which suggests time(0) is "279".
>> >> 
>> >> What is the current time on this system?
>> >> 
>> >> If it really was very early on Jan 1st 1970, it should work, however...
>> >> 
>> >> 
>> >> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL <detached ...>
>> >> > # rpcdebug -m rpc -s cache
>> >> > rpc        cache
>> >> > # grep . /proc/net/rpc/*/c*
>> >> > /proc/net/rpc/auth.unix.gid/content:#uid cnt: gids...
>> >> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
>> >> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
>> >> > /proc/net/rpc/auth.unix.ip/content:#class IP domain
>> >> > /proc/net/rpc/auth.unix.ip/content:# expiry=2079 refcnt=1 flags=1
>> >> > /proc/net/rpc/auth.unix.ip/content:# nfsd 10.0.0.1 10.0.0.0/24
>> >> 
>> >> ...the fact that this line is commented out indicates that the entry in
>> >> the cache is already expired.  So the time must be after 2079.
>> >> 
>> >> Maybe the time is getting set from the network at an awkward time that
>> >> races with NFS service some how.
>> >> Can you find a way to run "exportfs -f" after the time has been set
>> >> correctly?
>> >> 
>> >> NeilBrown
>> >> 
>> >> 
>> >
>> > wait, I think I've seen this somewhere, does this feature needs rtc? this board doesn't have rtc component.
>> > for example, I cannot use openssh as ssh server because it needs rtc. I have to use dropbear.
>> > if so, this looks like it will affect nfsv3 mounts, am I right?
>> 
>> No, you shouldn't need an RTC.
>> You need the synchronize the clock with ntp or similar, else time stamps
>> on files will look wrong.
>> Though I think we fixed issues with wall-clock-time jumping in 2.6.37...
>> 
>> If you could try using "exportfs -f", and explain what does happen with
>> time - do you use ntp ?? - we might be able to make progress.
>
> I'll build ntp into the image and try. does this affects nfsv3 too?

Having correct time is quite important for any version of NFS.  With out
it, time stamps on files get confused.  "make" doesn't cope at all,
"tar" often complains, other tools might experience other problems.

I still cannot quite see why having an incorrect clock would cause the
particular symptoms you are experiencing, but it is worth fixing anyway.

>
> what should I do with the "exportfs -f"? jsut run it and retry?

Yes.

NeilBrown

>
> Thanks,
>
> Dagg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: unable to mount nfs4 mount

[daggs]

Greetings,

> Subject: Re: unable to mount nfs4 mount
>
> On Tue, Dec 27 2016, daggs wrote:
> 
> > Greetings Neil,
> >
> >> On Mon, Dec 26 2016, daggs wrote:
> >> 
> >> > Greetings,
> >> >
> >> >> On Mon, Dec 26 2016, daggs wrote:
> >> >> 
> >> >> >> Can you strace mountd while you attempt a mount?
> >> >> >> e.g.
> >> >> >>   strace -o /tmp/trace -s 1000 -p 241
> >> >> >> 
> >> >> >> and send the /tmp/trace.
> >> >> >> Also, after the attempt fails, run
> >> >> >>  rpcdebug -m rpc -s cache
> >> >> >>  grep . /proc/net/rpc/*/c*
> >> >> >>  cat /proc/fs/nfsd/exports
> >> >> >> 
> >> >> >> and report the output.
> >> >> >> 
> >> >> > here:
> >> >> >
> >> >> > # cat /tmp/trace
> >> >> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL) = 1 (in [3])
> >> >> > read(3, "nfsd 10.0.0.1\n", 32768)       = 14
> >> >> > openat(AT_FDCWD, "/run/nfs/etab", O_RDONLY) = 14
> >> >> > fstat(14, {st_mode=S_IFREG|0644, st_size=435, ...}) = 0
> >> >> > close(14)                               = 0
> >> >> > write(3, "nfsd 10.0.0.1 2079 10.0.0.0/24 \n", 32) = 32
> >> >> 
> >> >> This is weird.
> >> >> Here mountd is telling nfsd that when a request comes from IP address
> >> >> 10.0.0.1, it should look for export entries associated with the client
> >> >> name "10.0.0.0/24", which is good.
> >> >> However the expiry time for that information is "2079", which is back in
> >> >> January 1970.
> >> >> When mountd writes that number, it computes it as
> >> >>    time(0) + DEFAULT_TTL
> >> >> where DEFAULT_TTL is (30 * 60)
> >> >> Which suggests time(0) is "279".
> >> >> 
> >> >> What is the current time on this system?
> >> >> 
> >> >> If it really was very early on Jan 1st 1970, it should work, however...
> >> >> 
> >> >> 
> >> >> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL <detached ...>
> >> >> > # rpcdebug -m rpc -s cache
> >> >> > rpc        cache
> >> >> > # grep . /proc/net/rpc/*/c*
> >> >> > /proc/net/rpc/auth.unix.gid/content:#uid cnt: gids...
> >> >> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
> >> >> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
> >> >> > /proc/net/rpc/auth.unix.ip/content:#class IP domain
> >> >> > /proc/net/rpc/auth.unix.ip/content:# expiry=2079 refcnt=1 flags=1
> >> >> > /proc/net/rpc/auth.unix.ip/content:# nfsd 10.0.0.1 10.0.0.0/24
> >> >> 
> >> >> ...the fact that this line is commented out indicates that the entry in
> >> >> the cache is already expired.  So the time must be after 2079.
> >> >> 
> >> >> Maybe the time is getting set from the network at an awkward time that
> >> >> races with NFS service some how.
> >> >> Can you find a way to run "exportfs -f" after the time has been set
> >> >> correctly?
> >> >> 
> >> >> NeilBrown
> >> >> 
> >> >> 
> >> >
> >> > wait, I think I've seen this somewhere, does this feature needs rtc? this board doesn't have rtc component.
> >> > for example, I cannot use openssh as ssh server because it needs rtc. I have to use dropbear.
> >> > if so, this looks like it will affect nfsv3 mounts, am I right?
> >> 
> >> No, you shouldn't need an RTC.
> >> You need the synchronize the clock with ntp or similar, else time stamps
> >> on files will look wrong.
> >> Though I think we fixed issues with wall-clock-time jumping in 2.6.37...
> >> 
> >> If you could try using "exportfs -f", and explain what does happen with
> >> time - do you use ntp ?? - we might be able to make progress.
> >
> > I'll build ntp into the image and try. does this affects nfsv3 too?
> 
> Having correct time is quite important for any version of NFS.  With out
> it, time stamps on files get confused.  "make" doesn't cope at all,
> "tar" often complains, other tools might experience other problems.
> 
> I still cannot quite see why having an incorrect clock would cause the
> particular symptoms you are experiencing, but it is worth fixing anyway.
I worked around the issue by setting the date manually. the issue still presists.

> 
> >
> > what should I do with the "exportfs -f"? jsut run it and retry?
> 
> Yes.
exportfs -f didn't made any difference.

Thanks,

Dagg.

Re: unable to mount nfs4 mount

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 4765 bytes --]

On Fri, Jan 06 2017, daggs wrote:

> Greetings,
>
>> Subject: Re: unable to mount nfs4 mount
>>
>> On Tue, Dec 27 2016, daggs wrote:
>> 
>> > Greetings Neil,
>> >
>> >> On Mon, Dec 26 2016, daggs wrote:
>> >> 
>> >> > Greetings,
>> >> >
>> >> >> On Mon, Dec 26 2016, daggs wrote:
>> >> >> 
>> >> >> >> Can you strace mountd while you attempt a mount?
>> >> >> >> e.g.
>> >> >> >>   strace -o /tmp/trace -s 1000 -p 241
>> >> >> >> 
>> >> >> >> and send the /tmp/trace.
>> >> >> >> Also, after the attempt fails, run
>> >> >> >>  rpcdebug -m rpc -s cache
>> >> >> >>  grep . /proc/net/rpc/*/c*
>> >> >> >>  cat /proc/fs/nfsd/exports
>> >> >> >> 
>> >> >> >> and report the output.
>> >> >> >> 
>> >> >> > here:
>> >> >> >
>> >> >> > # cat /tmp/trace
>> >> >> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL) = 1 (in [3])
>> >> >> > read(3, "nfsd 10.0.0.1\n", 32768)       = 14
>> >> >> > openat(AT_FDCWD, "/run/nfs/etab", O_RDONLY) = 14
>> >> >> > fstat(14, {st_mode=S_IFREG|0644, st_size=435, ...}) = 0
>> >> >> > close(14)                               = 0
>> >> >> > write(3, "nfsd 10.0.0.1 2079 10.0.0.0/24 \n", 32) = 32
>> >> >> 
>> >> >> This is weird.
>> >> >> Here mountd is telling nfsd that when a request comes from IP address
>> >> >> 10.0.0.1, it should look for export entries associated with the client
>> >> >> name "10.0.0.0/24", which is good.
>> >> >> However the expiry time for that information is "2079", which is back in
>> >> >> January 1970.
>> >> >> When mountd writes that number, it computes it as
>> >> >>    time(0) + DEFAULT_TTL
>> >> >> where DEFAULT_TTL is (30 * 60)
>> >> >> Which suggests time(0) is "279".
>> >> >> 
>> >> >> What is the current time on this system?
>> >> >> 
>> >> >> If it really was very early on Jan 1st 1970, it should work, however...
>> >> >> 
>> >> >> 
>> >> >> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL <detached ...>
>> >> >> > # rpcdebug -m rpc -s cache
>> >> >> > rpc        cache
>> >> >> > # grep . /proc/net/rpc/*/c*
>> >> >> > /proc/net/rpc/auth.unix.gid/content:#uid cnt: gids...
>> >> >> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
>> >> >> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
>> >> >> > /proc/net/rpc/auth.unix.ip/content:#class IP domain
>> >> >> > /proc/net/rpc/auth.unix.ip/content:# expiry=2079 refcnt=1 flags=1
>> >> >> > /proc/net/rpc/auth.unix.ip/content:# nfsd 10.0.0.1 10.0.0.0/24
>> >> >> 
>> >> >> ...the fact that this line is commented out indicates that the entry in
>> >> >> the cache is already expired.  So the time must be after 2079.
>> >> >> 
>> >> >> Maybe the time is getting set from the network at an awkward time that
>> >> >> races with NFS service some how.
>> >> >> Can you find a way to run "exportfs -f" after the time has been set
>> >> >> correctly?
>> >> >> 
>> >> >> NeilBrown
>> >> >> 
>> >> >> 
>> >> >
>> >> > wait, I think I've seen this somewhere, does this feature needs rtc? this board doesn't have rtc component.
>> >> > for example, I cannot use openssh as ssh server because it needs rtc. I have to use dropbear.
>> >> > if so, this looks like it will affect nfsv3 mounts, am I right?
>> >> 
>> >> No, you shouldn't need an RTC.
>> >> You need the synchronize the clock with ntp or similar, else time stamps
>> >> on files will look wrong.
>> >> Though I think we fixed issues with wall-clock-time jumping in 2.6.37...
>> >> 
>> >> If you could try using "exportfs -f", and explain what does happen with
>> >> time - do you use ntp ?? - we might be able to make progress.
>> >
>> > I'll build ntp into the image and try. does this affects nfsv3 too?
>> 
>> Having correct time is quite important for any version of NFS.  With out
>> it, time stamps on files get confused.  "make" doesn't cope at all,
>> "tar" often complains, other tools might experience other problems.
>> 
>> I still cannot quite see why having an incorrect clock would cause the
>> particular symptoms you are experiencing, but it is worth fixing anyway.
> I worked around the issue by setting the date manually. the issue still presists.
>
>> 
>> >
>> > what should I do with the "exportfs -f"? jsut run it and retry?
>> 
>> Yes.
> exportfs -f didn't made any difference.

OK. Please:
 - make sure the time is (nearly) correct - e.g. running date manually.
 - run "exportfs -f"
 - run "strace" on mountd
 - try to mount the filesystem
 - stop the strace
 - run
      rpcdebug -m rpc -s cache
      grep . /proc/net/rpc/*/*
      cat /proc/fs/nfsd/exports

  (note that it isn't quite the same 'grep' as before).

then post the output of the 'strace', the 'grep', and the 'cat'.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: unable to mount nfs4 mount

[daggs]

Greetings,

> On Fri, Jan 06 2017, daggs wrote:
> 
> > Greetings,
> >
> >> Subject: Re: unable to mount nfs4 mount
> >>
> >> On Tue, Dec 27 2016, daggs wrote:
> >> 
> >> > Greetings Neil,
> >> >
> >> >> On Mon, Dec 26 2016, daggs wrote:
> >> >> 
> >> >> > Greetings,
> >> >> >
> >> >> >> On Mon, Dec 26 2016, daggs wrote:
> >> >> >> 
> >> >> >> >> Can you strace mountd while you attempt a mount?
> >> >> >> >> e.g.
> >> >> >> >>   strace -o /tmp/trace -s 1000 -p 241
> >> >> >> >> 
> >> >> >> >> and send the /tmp/trace.
> >> >> >> >> Also, after the attempt fails, run
> >> >> >> >>  rpcdebug -m rpc -s cache
> >> >> >> >>  grep . /proc/net/rpc/*/c*
> >> >> >> >>  cat /proc/fs/nfsd/exports
> >> >> >> >> 
> >> >> >> >> and report the output.
> >> >> >> >> 
> >> >> >> > here:
> >> >> >> >
> >> >> >> > # cat /tmp/trace
> >> >> >> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL) = 1 (in [3])
> >> >> >> > read(3, "nfsd 10.0.0.1\n", 32768)       = 14
> >> >> >> > openat(AT_FDCWD, "/run/nfs/etab", O_RDONLY) = 14
> >> >> >> > fstat(14, {st_mode=S_IFREG|0644, st_size=435, ...}) = 0
> >> >> >> > close(14)                               = 0
> >> >> >> > write(3, "nfsd 10.0.0.1 2079 10.0.0.0/24 \n", 32) = 32
> >> >> >> 
> >> >> >> This is weird.
> >> >> >> Here mountd is telling nfsd that when a request comes from IP address
> >> >> >> 10.0.0.1, it should look for export entries associated with the client
> >> >> >> name "10.0.0.0/24", which is good.
> >> >> >> However the expiry time for that information is "2079", which is back in
> >> >> >> January 1970.
> >> >> >> When mountd writes that number, it computes it as
> >> >> >>    time(0) + DEFAULT_TTL
> >> >> >> where DEFAULT_TTL is (30 * 60)
> >> >> >> Which suggests time(0) is "279".
> >> >> >> 
> >> >> >> What is the current time on this system?
> >> >> >> 
> >> >> >> If it really was very early on Jan 1st 1970, it should work, however...
> >> >> >> 
> >> >> >> 
> >> >> >> > pselect6(1024, [3 4 5 7 8 9 10 11 12], NULL, NULL, NULL, NULL <detached ...>
> >> >> >> > # rpcdebug -m rpc -s cache
> >> >> >> > rpc        cache
> >> >> >> > # grep . /proc/net/rpc/*/c*
> >> >> >> > /proc/net/rpc/auth.unix.gid/content:#uid cnt: gids...
> >> >> >> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
> >> >> >> > /proc/net/rpc/auth.unix.ip/channel:nfsd 10.0.0.1
> >> >> >> > /proc/net/rpc/auth.unix.ip/content:#class IP domain
> >> >> >> > /proc/net/rpc/auth.unix.ip/content:# expiry=2079 refcnt=1 flags=1
> >> >> >> > /proc/net/rpc/auth.unix.ip/content:# nfsd 10.0.0.1 10.0.0.0/24
> >> >> >> 
> >> >> >> ...the fact that this line is commented out indicates that the entry in
> >> >> >> the cache is already expired.  So the time must be after 2079.
> >> >> >> 
> >> >> >> Maybe the time is getting set from the network at an awkward time that
> >> >> >> races with NFS service some how.
> >> >> >> Can you find a way to run "exportfs -f" after the time has been set
> >> >> >> correctly?
> >> >> >> 
> >> >> >> NeilBrown
> >> >> >> 
> >> >> >> 
> >> >> >
> >> >> > wait, I think I've seen this somewhere, does this feature needs rtc? this board doesn't have rtc component.
> >> >> > for example, I cannot use openssh as ssh server because it needs rtc. I have to use dropbear.
> >> >> > if so, this looks like it will affect nfsv3 mounts, am I right?
> >> >> 
> >> >> No, you shouldn't need an RTC.
> >> >> You need the synchronize the clock with ntp or similar, else time stamps
> >> >> on files will look wrong.
> >> >> Though I think we fixed issues with wall-clock-time jumping in 2.6.37...
> >> >> 
> >> >> If you could try using "exportfs -f", and explain what does happen with
> >> >> time - do you use ntp ?? - we might be able to make progress.
> >> >
> >> > I'll build ntp into the image and try. does this affects nfsv3 too?
> >> 
> >> Having correct time is quite important for any version of NFS.  With out
> >> it, time stamps on files get confused.  "make" doesn't cope at all,
> >> "tar" often complains, other tools might experience other problems.
> >> 
> >> I still cannot quite see why having an incorrect clock would cause the
> >> particular symptoms you are experiencing, but it is worth fixing anyway.
> > I worked around the issue by setting the date manually. the issue still presists.
> >
> >> 
> >> >
> >> > what should I do with the "exportfs -f"? jsut run it and retry?
> >> 
> >> Yes.
> > exportfs -f didn't made any difference.
> 
> OK. Please:
>  - make sure the time is (nearly) correct - e.g. running date manually.
>  - run "exportfs -f"
>  - run "strace" on mountd
>  - try to mount the filesystem
>  - stop the strace
>  - run
>       rpcdebug -m rpc -s cache
>       grep . /proc/net/rpc/*/*
>       cat /proc/fs/nfsd/exports
> 
>   (note that it isn't quite the same 'grep' as before).
> 
> then post the output of the 'strace', the 'grep', and the 'cat'.
> 

so it took so long, I was able to fix the timer issue and now indeed I can mount as nfsv4.
but I've encountered another issue, copying a large file over nfs takes 3 minutes longer with nfsv4 than nfsv3.
I want to be sure this isn't a config issue, so here is the config:
# cat /etc/exports 
/mnt/media      10.0.0.0/24(rw,no_subtree_check,nohide,insecure)

the kernel in question is based on 3.14 (yeah, I know it is old but that is currently the only kernel fully supports the board in question).

can it be the config or the kernel?

Thanks,

Dagg.