`ls` shows no files in mounted nfs folder, but ls 'file' works

`ls` shows no files in mounted nfs folder, but ls 'file' works

[Andreas Fenkart]

Found regression introduced by this patch

311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"

# cd /bin
-sh: cd: /bin: Not a directory
root@192:/# pwd
/
root@192:/# ls /bin
ls: /bin
root@192:/# ls -l /bin/sh
lrwxrwxrwx    1 root     root             9 Jun 16 13:59 /bin/sh -> /bin/bash

the mount options of the client, an embedded system, mounting nfsroot:
nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
vers: 3.16.0-rc1

the server exports:
/exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux

reverting above patch, dirlisting with 'ls /' works again

/Andreas

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[J. Bruce Fields]

On Mon, Jun 23, 2014 at 12:07:22PM +0200, Andreas Fenkart wrote:
> Found regression introduced by this patch
> 
> 311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
> 
> # cd /bin
> -sh: cd: /bin: Not a directory
> root@192:/# pwd
> /
> root@192:/# ls /bin
> ls: /bin
> root@192:/# ls -l /bin/sh
> lrwxrwxrwx    1 root     root             9 Jun 16 13:59 /bin/sh -> /bin/bash
> 
> the mount options of the client, an embedded system, mounting nfsroot:
> nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
> vers: 3.16.0-rc1
> 
> the server exports:
> /exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
> vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
> 
> reverting above patch, dirlisting with 'ls /' works again

Is this still happening?

--b.

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[Andreas Fenkart]

2014-07-10 19:51 GMT+02:00 J. Bruce Fields <bfields@fieldses.org>:
> On Mon, Jun 23, 2014 at 12:07:22PM +0200, Andreas Fenkart wrote:
>> Found regression introduced by this patch
>>
>> 311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
>>
>> # cd /bin
>> -sh: cd: /bin: Not a directory
>> root@192:/# pwd
>> /
>> root@192:/# ls /bin
>> ls: /bin
>> root@192:/# ls -l /bin/sh
>> lrwxrwxrwx    1 root     root             9 Jun 16 13:59 /bin/sh -> /bin/bash
>>
>> the mount options of the client, an embedded system, mounting nfsroot:
>> nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
>> vers: 3.16.0-rc1
>>
>> the server exports:
>> /exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
>> vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
>>
>> reverting above patch, dir listing with 'ls /' works again
>
> Is this still happening?

yes it is, I just reverted the aforementioned commit.

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[Trond Myklebust]

On Sun, Jul 13, 2014 at 11:44 PM, Andreas Fenkart <afenkart@gmail.com> wrote:
> 2014-07-10 19:51 GMT+02:00 J. Bruce Fields <bfields@fieldses.org>:
>> On Mon, Jun 23, 2014 at 12:07:22PM +0200, Andreas Fenkart wrote:
>>> Found regression introduced by this patch
>>>
>>> 311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
>>>
>>> # cd /bin
>>> -sh: cd: /bin: Not a directory
>>> root@192:/# pwd
>>> /
>>> root@192:/# ls /bin
>>> ls: /bin
>>> root@192:/# ls -l /bin/sh
>>> lrwxrwxrwx    1 root     root             9 Jun 16 13:59 /bin/sh -> /bin/bash
>>>
>>> the mount options of the client, an embedded system, mounting nfsroot:
>>> nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
>>> vers: 3.16.0-rc1
>>>
>>> the server exports:
>>> /exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
>>> vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
>>>
>>> reverting above patch, dir listing with 'ls /' works again
>>
>> Is this still happening?
>
> yes it is, I just reverted the aforementioned commit.

Can you reapply, and then do an "ls /" while recording the traffic to
the NFS server using wireshark, and then send us the resulting dump?

If the client is on an embedded system which can't run wireshark, then
it is fine to record it on the server, but please use something like

'tshark -w /var/tmp/dump.out host <client-ip>'

Cheers
  Trond


-- 
Trond Myklebust

Linux NFS client maintainer, PrimaryData

trond.myklebust@primarydata.com

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[Andreas Fenkart]

2014-07-16 1:54 GMT+02:00 Trond Myklebust <trond.myklebust@primarydata.com>:
> On Sun, Jul 13, 2014 at 11:44 PM, Andreas Fenkart <afenkart@gmail.com> wrote:
>> 2014-07-10 19:51 GMT+02:00 J. Bruce Fields <bfields@fieldses.org>:
>>> On Mon, Jun 23, 2014 at 12:07:22PM +0200, Andreas Fenkart wrote:
>>>> Found regression introduced by this patch
>>>>
>>>> 311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
>>>>
>>>> # cd /bin
>>>> -sh: cd: /bin: Not a directory
>>>> root@192:/# pwd
>>>> /
>>>> root@192:/# ls /bin
>>>> ls: /bin
>>>> root@192:/# ls -l /bin/sh
>>>> lrwxrwxrwx    1 root     root             9 Jun 16 13:59 /bin/sh -> /bin/bash
>>>>
>>>> the mount options of the client, an embedded system, mounting nfsroot:
>>>> nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
>>>> vers: 3.16.0-rc1
>>>>
>>>> the server exports:
>>>> /exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
>>>> vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
>>>>
>>>> reverting above patch, dir listing with 'ls /' works again
>>>
>>> Is this still happening?
>>
>> yes it is, I just reverted the aforementioned commit.
>
> Can you reapply, and then do an "ls /" while recording the traffic to
> the NFS server using wireshark, and then send us the resulting dump?

http://www.pcapr.net/view/afenkart/2014/6/3/1/nfs-client.dump.html
you have to create a login... simple. Or did you mean attach to email?

> If the client is on an embedded system which can't run wireshark, then
> it is fine to record it on the server, but please use something like
>
> 'tshark -w /var/tmp/dump.out host <client-ip>'

captured on embedded system:
tcpdump -w /tmp/nfs-client.dump

>
> Cheers
>   Trond

thanks, sincerely
Andreas

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[Trond Myklebust]

On Wed, Jul 16, 2014 at 1:21 AM, Andreas Fenkart <afenkart@gmail.com> wrote:
> 2014-07-16 1:54 GMT+02:00 Trond Myklebust <trond.myklebust@primarydata.com>:
>> On Sun, Jul 13, 2014 at 11:44 PM, Andreas Fenkart <afenkart@gmail.com> wrote:
>>> 2014-07-10 19:51 GMT+02:00 J. Bruce Fields <bfields@fieldses.org>:
>>>> On Mon, Jun 23, 2014 at 12:07:22PM +0200, Andreas Fenkart wrote:
>>>>> Found regression introduced by this patch
>>>>>
>>>>> 311324ad NFS: Be more aggressive in using readdirplus for 'ls -l' situations"
>>>>>
>>>>> # cd /bin
>>>>> -sh: cd: /bin: Not a directory
>>>>> root@192:/# pwd
>>>>> /
>>>>> root@192:/# ls /bin
>>>>> ls: /bin
>>>>> root@192:/# ls -l /bin/sh
>>>>> lrwxrwxrwx    1 root     root             9 Jun 16 13:59 /bin/sh -> /bin/bash
>>>>>
>>>>> the mount options of the client, an embedded system, mounting nfsroot:
>>>>> nfs(rw,relatime,vers=2,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.xx.xx,mountvers=1,mountproto=udp,local_lock=all,addr=192.168.xx.xx)
>>>>> vers: 3.16.0-rc1
>>>>>
>>>>> the server exports:
>>>>> /exports/rootfs 192.168.xx.xx(rw,no_root_squash,subtree_check)
>>>>> vers: 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux
>>>>>
>>>>> reverting above patch, dir listing with 'ls /' works again
>>>>
>>>> Is this still happening?
>>>
>>> yes it is, I just reverted the aforementioned commit.
>>
>> Can you reapply, and then do an "ls /" while recording the traffic to
>> the NFS server using wireshark, and then send us the resulting dump?
>
> http://www.pcapr.net/view/afenkart/2014/6/3/1/nfs-client.dump.html
> you have to create a login... simple. Or did you mean attach to email?
>
>> If the client is on an embedded system which can't run wireshark, then
>> it is fine to record it on the server, but please use something like
>>
>> 'tshark -w /var/tmp/dump.out host <client-ip>'
>
> captured on embedded system:
> tcpdump -w /tmp/nfs-client.dump
>
>>
>> Cheers
>>   Trond
>
> thanks, sincerely
> Andreas

That unfortunately only shows a few READ requests, probably because
the readdirs etc are cached. Could you try again with a directory that
is not in cache and/or try "echo 3 >/proc/sys/vm/drop_caches" before
retrying the 'ls /'?

Note: I am a little surprised that there are no GETATTRs in that dump.
Even if the readdir is cached, the client is supposed to revalidate
the directory.

-- 
Trond Myklebust

Linux NFS client maintainer, PrimaryData

trond.myklebust@primarydata.com

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[Andreas Fenkart]

2014-07-16 16:47 GMT+02:00 Trond Myklebust <trond.myklebust@primarydata.com>:
> On Wed, Jul 16, 2014 at 1:21 AM, Andreas Fenkart <afenkart@gmail.com> wrote:

> That unfortunately only shows a few READ requests, probably because
> the readdirs etc are cached. Could you try again with a directory that
> is not in cache and/or try "echo 3 >/proc/sys/vm/drop_caches" before
> retrying the 'ls /'?

# tcpdump -s 0 -w /tmp/nfs-client_v4.dump
# echo 3 >/proc/sys/vm/drop_caches
# ls -l /

file is logged to tmpfs, then copied
http://www.pcapr.net/view/afenkart/2014/6/3/8/nfs-client_v4.dump.html

> Note: I am a little surprised that there are no GETATTRs in that dump.
> Even if the readdir is cached, the client is supposed to revalidate
> the directory.

should be there now

Andreas

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[Trond Myklebust]

On Wed, Jul 16, 2014 at 11:45 AM, Andreas Fenkart <afenkart@gmail.com> wrote:
> 2014-07-16 16:47 GMT+02:00 Trond Myklebust <trond.myklebust@primarydata.com>:
>> On Wed, Jul 16, 2014 at 1:21 AM, Andreas Fenkart <afenkart@gmail.com> wrote:
>
>> That unfortunately only shows a few READ requests, probably because
>> the readdirs etc are cached. Could you try again with a directory that
>> is not in cache and/or try "echo 3 >/proc/sys/vm/drop_caches" before
>> retrying the 'ls /'?
>
> # tcpdump -s 0 -w /tmp/nfs-client_v4.dump
> # echo 3 >/proc/sys/vm/drop_caches
> # ls -l /
>
> file is logged to tmpfs, then copied
> http://www.pcapr.net/view/afenkart/2014/6/3/8/nfs-client_v4.dump.html
>
>> Note: I am a little surprised that there are no GETATTRs in that dump.
>> Even if the readdir is cached, the client is supposed to revalidate
>> the directory.
>
> should be there now

Still not seeing the readdir, but the getattrs are there and all looks OK.
Question: the server is knfsd from a Linux 3.14.4 kernel, is that
correct? So it does have commit 83b19121522 (nfsd: revert v2 half of
"nfsd: don't return high mode bits")?

-- 
Trond Myklebust

Linux NFS client maintainer, PrimaryData

trond.myklebust@primarydata.com

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[J. Bruce Fields]

On Fri, Jul 25, 2014 at 06:12:40PM -0400, Trond Myklebust wrote:
> On Wed, Jul 16, 2014 at 11:45 AM, Andreas Fenkart <afenkart@gmail.com> wrote:
> > 2014-07-16 16:47 GMT+02:00 Trond Myklebust <trond.myklebust@primarydata.com>:
> >> On Wed, Jul 16, 2014 at 1:21 AM, Andreas Fenkart <afenkart@gmail.com> wrote:
> >
> >> That unfortunately only shows a few READ requests, probably because
> >> the readdirs etc are cached. Could you try again with a directory that
> >> is not in cache and/or try "echo 3 >/proc/sys/vm/drop_caches" before
> >> retrying the 'ls /'?
> >
> > # tcpdump -s 0 -w /tmp/nfs-client_v4.dump
> > # echo 3 >/proc/sys/vm/drop_caches
> > # ls -l /
> >
> > file is logged to tmpfs, then copied
> > http://www.pcapr.net/view/afenkart/2014/6/3/8/nfs-client_v4.dump.html
> >
> >> Note: I am a little surprised that there are no GETATTRs in that dump.
> >> Even if the readdir is cached, the client is supposed to revalidate
> >> the directory.
> >
> > should be there now
> 
> Still not seeing the readdir, but the getattrs are there and all looks OK.
> Question: the server is knfsd from a Linux 3.14.4 kernel, is that
> correct? So it does have commit 83b19121522 (nfsd: revert v2 half of
> "nfsd: don't return high mode bits")?

Yeah, you can even see the high bits set (correctly) in the stat
returns, so he has that patch.

Andreas, are you still seeing this problem?

--b.

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[Andreas Fenkart]

Hi

2014-08-26 21:50 GMT+02:00 J. Bruce Fields <bfields@fieldses.org>:
>>
>> Still not seeing the readdir, but the getattrs are there and all looks OK.
>> Question: the server is knfsd from a Linux 3.14.4 kernel, is that
>> correct? So it does have commit 83b19121522 (nfsd: revert v2 half of
>> "nfsd: don't return high mode bits")?
>
> Yeah, you can even see the high bits set (correctly) in the stat
> returns, so he has that patch.
>
> Andreas, are you still seeing this problem?

No it's gone, but I upgraded my (server-)system meanwhile (debian unstable)
I also tried downgrading my kernel to the version I had back then, but
the problem doesn't show up anymore.

sorry the noise
/Andreas


$ cat /proc/version
Linux version 3.14-1-amd64 (debian-kernel@lists.debian.org) (gcc
version 4.8.2 (Debian 4.8.2-21) ) #1 SMP Debian 3.14.4-1 (2014-05-13)

http://snapshot.debian.org/archive/debian/20140514T103724Z/pool/main/l/linux/linux-image-3.14-1-amd64_3.14.4-1_i386.deb

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[J. Bruce Fields]

On Wed, Aug 27, 2014 at 10:30:34AM +0200, Andreas Fenkart wrote:
> Hi
> 
> 2014-08-26 21:50 GMT+02:00 J. Bruce Fields <bfields@fieldses.org>:
> >>
> >> Still not seeing the readdir, but the getattrs are there and all looks OK.
> >> Question: the server is knfsd from a Linux 3.14.4 kernel, is that
> >> correct? So it does have commit 83b19121522 (nfsd: revert v2 half of
> >> "nfsd: don't return high mode bits")?
> >
> > Yeah, you can even see the high bits set (correctly) in the stat
> > returns, so he has that patch.
> >
> > Andreas, are you still seeing this problem?
> 
> No it's gone, but I upgraded my (server-)system meanwhile (debian unstable)
> I also tried downgrading my kernel to the version I had back then, but
> the problem doesn't show up anymore.

Well, maybe there was a server bug in the old server that this change
just happened to expose.

What was the server upgrade, out of curiosity?  (From what kernel
version, to what kernel version?)

--b.

Re: `ls` shows no files in mounted nfs folder, but ls 'file' works

[Andreas Fenkart]

Hi Bruce,

http://snapshot.debian.org/package/linux/3.14.4-1/#linux-image-3.14-1-amd64_3.14.4-1
  <- version when reporting bug
http://snapshot.debian.org/package/linux/3.14.15-2/#linux-image-3.14-2-amd64_3.14.15-2
<- current version

even when downgrading to the previous version, I can't reproduce the
bug anymore...
just realized that I didn't mention the tip of my embedded system
kernel, bad me, but it was some 3.16 rc,

/Andreas


2014-08-27 15:26 GMT+02:00 J. Bruce Fields <bfields@fieldses.org>:
> On Wed, Aug 27, 2014 at 10:30:34AM +0200, Andreas Fenkart wrote:
>> Hi
>>
>> 2014-08-26 21:50 GMT+02:00 J. Bruce Fields <bfields@fieldses.org>:
>> >>
>> >> Still not seeing the readdir, but the getattrs are there and all looks OK.
>> >> Question: the server is knfsd from a Linux 3.14.4 kernel, is that
>> >> correct? So it does have commit 83b19121522 (nfsd: revert v2 half of
>> >> "nfsd: don't return high mode bits")?
>> >
>> > Yeah, you can even see the high bits set (correctly) in the stat
>> > returns, so he has that patch.
>> >
>> > Andreas, are you still seeing this problem?
>>
>> No it's gone, but I upgraded my (server-)system meanwhile (debian unstable)
>> I also tried downgrading my kernel to the version I had back then, but
>> the problem doesn't show up anymore.
>
> Well, maybe there was a server bug in the old server that this change
> just happened to expose.
>
> What was the server upgrade, out of curiosity?  (From what kernel
> version, to what kernel version?)
>
> --b.