New file doesn't show up if cached as missing

New file doesn't show up if cached as missing

[Simon Kirby]

Hello!

On Linux 2.6.28.10 server and client, with NFSv3 or NFSv4, I can
fairly easily reproduce a case where creating a file (open,write,close)
and then attempting to open it on another client mounting the same server
returns ENOENT.  If I run "ls" before I try to cat it, it works.

To reproduce:

ssh root@lps01 'rm -f /shared/ssl/test; stat /shared/ssl/test'; \
ssh root@lps02 'echo hi > /shared/ssl/test'; \
ssh root@lps01 'cat /shared/ssl/test'

The original stat (or cat or open) is important as it seems to cache that
the file does not exist.  I can then even log in and poke around:

lps01:~# cd /shared
lps01:/shared# cat test
cat: test: No such file or directory
lps01:/shared# stat test
stat: cannot stat `test': No such file or directory
lps01:/shared# ls test
ls: test: No such file or directory
lps01:/shared# ls
<files including "test">
lps01lsh:/shared# cat test
hi

Is this expected behaviour?  The file system is mounted on both clients
with: rw,hard,intr,...,acdirmin=0,acdirmax=0,acregmin=0,acregmax=0,noac

(eg: we can't find a way to make it not happen.)

Cheers,

Simon-

Re: New file doesn't show up if cached as missing

[Trond Myklebust]

On Fri, 2009-05-29 at 10:52 -0700, Simon Kirby wrote:
> Hello!
> 
> On Linux 2.6.28.10 server and client, with NFSv3 or NFSv4, I can
> fairly easily reproduce a case where creating a file (open,write,close)
> and then attempting to open it on another client mounting the same server
> returns ENOENT.  If I run "ls" before I try to cat it, it works.
> 
> To reproduce:
> 
> ssh root@lps01 'rm -f /shared/ssl/test; stat /shared/ssl/test'; \
> ssh root@lps02 'echo hi > /shared/ssl/test'; \
> ssh root@lps01 'cat /shared/ssl/test'
> 
> The original stat (or cat or open) is important as it seems to cache that
> the file does not exist.  I can then even log in and poke around:
> 
> lps01:~# cd /shared
> lps01:/shared# cat test
> cat: test: No such file or directory
> lps01:/shared# stat test
> stat: cannot stat `test': No such file or directory
> lps01:/shared# ls test
> ls: test: No such file or directory
> lps01:/shared# ls
> <files including "test">
> lps01lsh:/shared# cat test
> hi
> 
> Is this expected behaviour?  The file system is mounted on both clients
> with: rw,hard,intr,...,acdirmin=0,acdirmax=0,acregmin=0,acregmax=0,noac
> 
> (eg: we can't find a way to make it not happen.)

That is most likely to be a consequence of poor mtime resolution on the
server (i.e. the directory mtime failing to change because the file
creation occurred within < 1 second of the 'rm'), combined with negative
lookup caching.

Try using the '-olookupcache=positive' or '-olookupcache=none' mount
options (requires a relatively recent version of nfs-utils).

Trond

Re: New file doesn't show up if cached as missing

[Simon Kirby]

On Fri, May 29, 2009 at 02:36:46PM -0400, Trond Myklebust wrote:

> That is most likely to be a consequence of poor mtime resolution on the
> server (i.e. the directory mtime failing to change because the file
> creation occurred within < 1 second of the 'rm'), combined with negative
> lookup caching.

As in "ls --full-time" shows 0 for the fractional second part?

Hrm, I thought newer ext3 had nanosecond mtime, but it seems not...

> Try using the '-olookupcache=positive' or '-olookupcache=none' mount
> options (requires a relatively recent version of nfs-utils).

After backporting nfs-utils, either of those options seem to this
particular case work as desired.  Excellent!

I guess all of the attribute-caching-related options are unrelated to
this case.  Is this consistent with older operation and that of other
OSes?  It might be a little surprising for some (me) when "noac" still
ends up doing some caching.

Thanks,

Simon-

Re: New file doesn't show up if cached as missing

[Trond Myklebust]

On Fri, 2009-05-29 at 15:04 -0700, Simon Kirby wrote:
> On Fri, May 29, 2009 at 02:36:46PM -0400, Trond Myklebust wrote:
> 
> > That is most likely to be a consequence of poor mtime resolution on the
> > server (i.e. the directory mtime failing to change because the file
> > creation occurred within < 1 second of the 'rm'), combined with negative
> > lookup caching.
> 
> As in "ls --full-time" shows 0 for the fractional second part?
> 
> Hrm, I thought newer ext3 had nanosecond mtime, but it seems not...

No. You'll have to migrate to ext4 (or xfs) if you want finer grained
timestamps.

> > Try using the '-olookupcache=positive' or '-olookupcache=none' mount
> > options (requires a relatively recent version of nfs-utils).
> 
> After backporting nfs-utils, either of those options seem to this
> particular case work as desired.  Excellent!
> 
> I guess all of the attribute-caching-related options are unrelated to
> this case.  Is this consistent with older operation and that of other
> OSes?  It might be a little surprising for some (me) when "noac" still
> ends up doing some caching.

noac is all about attribute caching, whereas this is lookup/dentry
caching. You can compare it to data caching: 'noac' doesn't force the
cache to be flushed on every read() request, but it does force the mtime
to be checked, and so cache consistency is relatively stronger (provided
the mtime is accurate enough).

Cheers
  Trond