Force client page cache invalidation?

Force client page cache invalidation?

[ralph gonzalez]

We tend to run multiple applications on a single client, which all
mmap (usually readonly) the same file on an nfs-mounted volume.
If the file changes on the server, then we would like clients to
see these changes upon closing/reopening this file.

On Solaris (which we are moving from), this would work fine. However
Linux 2.4.22-10mdk (nfs utils 1.0.5) seems to be unwilling to reload
the file cache *when another application already has the file mmapped*.

Therefore each time the file is editted on the server, we have to
force all applications mapping this file (on the client in question)
to close the file *simultaneously*, and then allow them to reopen the
mapping.

*Question*: Instead, is there a way to force the client to reload
the file cache (exclusively for this file)? (So when we subsequently
close/open the file in each application asynchronously, they see the
correct pages.) Maybe there's a simple option or utility I'm missing?

A separate problem is that the file attributes are cached, so the
file cache is only reloaded if we force the client to reread the
file attributes whenever the file is editted on the server. This we
accomplish by the trick of locking/unlocking the file immediately
after opening it, to avoid disabling mount-level attribute caching
via the noac option.

Thanks for any suggestions!
-- 
Ralph Gonzalez
rgonzale@ibl.bm



-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Force client page cache invalidation?

[trond.myklebust]

På ty , 24/02/2004 klokka 10:30, skreiv ralph gonzalez:
> We tend to run multiple applications on a single client, which all mmap
(usually readonly) the same file on an nfs-mounted volume.
> If the file changes on the server, then we would like clients to see
these changes upon closing/reopening this file.
>
> On Solaris (which we are moving from), this would work fine. However
Linux 2.4.22-10mdk (nfs utils 1.0.5) seems to be unwilling to reload the
file cache *when another application already has the file mmapped*.

Which is quite sane semantics for mmap().

> Therefore each time the file is editted on the server, we have to force
all applications mapping this file (on the client in question) to close
the file *simultaneously*, and then allow them to reopen the mapping.
>
> *Question*: Instead, is there a way to force the client to reload the
file cache (exclusively for this file)? (So when we subsequently
close/open the file in each application asynchronously, they see the
correct pages.) Maybe there's a simple option or utility I'm missing?

Linux does not support this sort of kludge and is unlikely to do so for
the near future. The VM simply cannot support it.

In any case, the fact is that strict POSIX mmap() semantics assume
exclusive access to the file on the server while it is mapped on a given
client. If you're not providing that, then you are going to be prone to
some nasty races whether or not the cache invalidation works like Solaris
does.

Cheers,
  Trond





-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Force client page cache invalidation?

[Olaf Kirch]

> Linux does not support this sort of kludge and is unlikely to do so for
> the near future. The VM simply cannot support it.

What about using fadvise(POSIX_FADV_DONTNEED) - shouldn't this
do the trick? Or does it refuse to invalidate mapped pages?

Olaf
-- 
Olaf Kirch     |  Stop wasting entropy - start using predictable
okir@suse.de   |  tempfile names today!
---------------+ 


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Force client page cache invalidation?

[trond.myklebust]

På to , 26/02/2004 klokka 00:57, skreiv Olaf Kirch:
> > Linux does not support this sort of kludge and is unlikely to do so
for
> > the near future. The VM simply cannot support it.
>
> What about using fadvise(POSIX_FADV_DONTNEED) - shouldn't this
> do the trick? Or does it refuse to invalidate mapped pages?

fadvise(POSIX_FADV_DONTNEED) uses the same interface that NFS does to
invalidate pages (invalidate_mapping_pages()), so it has the same
weaknesses: if you have unmapped those pages since whenever it was NFS
tried to invalidate them, then it will work, but there is no way for the
user to guarantee that it will.
Moreover, fadvise() is only available under Linux 2.6.x...

I did, however, see an encouraging posting by Daniel Phillips on the
linux-kernel list yesterday, which indicates he is working on this
problem. See the thread about "Distributed mmap API".

Cheers,
  Trond





-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Force client page cache invalidation?

[ralph gonzalez]

[snip]
 >> Linux 2.4.22-10mdk (nfs utils 1.0.5) seems to be unwilling to reload the
 >> file cache *when another application already has the file mmapped*.
 >
 > Which is quite sane semantics for mmap().

Agreed...

[snip]
 >>*Question*: Instead, is there a way to force the client to reload the
 >
 > file cache (exclusively for this file)? (So when we subsequently
 > close/open the file in each application asynchronously, they see the
 > correct pages.) Maybe there's a simple option or utility I'm missing?
 >
 > Linux does not support this sort of kludge and is unlikely to do so for
 > the near future. The VM simply cannot support it.
 >
 > In any case, the fact is that strict POSIX mmap() semantics assume
 > exclusive access to the file on the server while it is mapped on a given
 > client. If you're not providing that, then you are going to be prone to
 > some nasty races whether or not the cache invalidation works like Solaris
 > does.

I wasn't aware of the POSIX semantics. It's a shame, since we've been
using this approach -- multiple readers and single writer -- on Solaris
for 8 years with excellent stability and performance. When we switched to
Linux we found this unexpected and unfriendly behavior:

   If NFS client has the file mmapped while it is edited on the server,
   and the client then opens a second application which mmaps the file
   (without closing the first application), then no matter what you do
   -- close both clients, etc -- the file cache never releases the old
   pages!

-- 
Ralph Gonzalez
rgonzale@ibl.bm



-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs