Cache flushing

Cache flushing

[Timo Sirainen]

[-- Attachment #1.1: Type: text/plain, Size: 1679 bytes --]

Solaris and BSDs support flushing attribute cache safely using
fchown(fd, (uid_t)-1, (gid_t)-1). Could Linux be changed to support this
as well? If I'm looking at the sources right, this might work
(completely untested):

--- inode.c.old 2007-11-16 22:18:46.000000000 +0200
+++ inode.c     2007-11-16 22:19:44.000000000 +0200
@@ -322,6 +322,7 @@
 nfs_setattr(struct dentry *dentry, struct iattr *attr)
 {
        struct inode *inode = dentry->d_inode;
+       struct nfs_inode *nfsi = NFS_I(inode);
        struct nfs_fattr fattr;
        int error;
 
@@ -334,8 +335,10 @@
 
        /* Optimization: if the end result is no change, don't RPC */
        attr->ia_valid &= NFS_VALID_ATTRS;
-       if (attr->ia_valid == 0)
+       if (attr->ia_valid == 0) {
+               nfsi->cache_validity |= NFS_INO_INVALID_ATTR;
                return 0;
+       }
 
        lock_kernel();
        nfs_begin_data_update(inode);

Another problem I have is that it's difficult to get a file's data cache
flushed. The only way I found was to successfully fcntl() lock the file.
This is pretty bad from performance point of view since often I don't
want/need to lock the file.

Solaris and BSDs invalidate also a file's data cache when its attribute
cache is invalidated. It would be nicer if there was a separate way, but
I'd settle for fchown(fd, (uid_t)-1, (gid_t)-1) invalidating data cache
as well.

Actually I did also look at posix_fadvise(fd, 0, 0,
POSIX_FADVN_DONTNEED). It appears to work, but I'm a bit worried about
race conditions that causes pages to randomly not get dropped because
invalidate_mapping_pages() doesn't drop locked pages.

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Trond Myklebust]

On Sat, 2007-11-17 at 02:11 +0200, Timo Sirainen wrote:
> Solaris and BSDs support flushing attribute cache safely using
> fchown(fd, (uid_t)-1, (gid_t)-1). Could Linux be changed to support this
> as well? If I'm looking at the sources right, this might work
> (completely untested):
> 
> --- inode.c.old 2007-11-16 22:18:46.000000000 +0200
> +++ inode.c     2007-11-16 22:19:44.000000000 +0200
> @@ -322,6 +322,7 @@
>  nfs_setattr(struct dentry *dentry, struct iattr *attr)
>  {
>         struct inode *inode = dentry->d_inode;
> +       struct nfs_inode *nfsi = NFS_I(inode);
>         struct nfs_fattr fattr;
>         int error;
>  
> @@ -334,8 +335,10 @@
>  
>         /* Optimization: if the end result is no change, don't RPC */
>         attr->ia_valid &= NFS_VALID_ATTRS;
> -       if (attr->ia_valid == 0)
> +       if (attr->ia_valid == 0) {
> +               nfsi->cache_validity |= NFS_INO_INVALID_ATTR;
>                 return 0;
> +       }
>  
>         lock_kernel();
>         nfs_begin_data_update(inode);

Why is this needed?

> Another problem I have is that it's difficult to get a file's data cache
> flushed. The only way I found was to successfully fcntl() lock the file.
> This is pretty bad from performance point of view since often I don't
> want/need to lock the file.
> 
> Solaris and BSDs invalidate also a file's data cache when its attribute
> cache is invalidated. It would be nicer if there was a separate way, but
> I'd settle for fchown(fd, (uid_t)-1, (gid_t)-1) invalidating data cache
> as well.
> 
> Actually I did also look at posix_fadvise(fd, 0, 0,
> POSIX_FADVN_DONTNEED). It appears to work, but I'm a bit worried about
> race conditions that causes pages to randomly not get dropped because
> invalidate_mapping_pages() doesn't drop locked pages.

Again, why do you need this level of data cache invalidation? If you
don't want cached i/o, then use O_DIRECT.

Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Timo Sirainen]

[-- Attachment #1.1: Type: text/plain, Size: 3327 bytes --]

On 17.11.2007, at 21.46, Trond Myklebust wrote:

> On Sat, 2007-11-17 at 02:11 +0200, Timo Sirainen wrote:
>> Solaris and BSDs support flushing attribute cache safely using
>> fchown(fd, (uid_t)-1, (gid_t)-1). Could Linux be changed to  
>> support this
>> as well? If I'm looking at the sources right, this might work
>> (completely untested):
>>
>> --- inode.c.old 2007-11-16 22:18:46.000000000 +0200
>> +++ inode.c     2007-11-16 22:19:44.000000000 +0200
>> @@ -322,6 +322,7 @@
>>  nfs_setattr(struct dentry *dentry, struct iattr *attr)
>>  {
>>         struct inode *inode = dentry->d_inode;
>> +       struct nfs_inode *nfsi = NFS_I(inode);
>>         struct nfs_fattr fattr;
>>         int error;
>>
>> @@ -334,8 +335,10 @@
>>
>>         /* Optimization: if the end result is no change, don't RPC */
>>         attr->ia_valid &= NFS_VALID_ATTRS;
>> -       if (attr->ia_valid == 0)
>> +       if (attr->ia_valid == 0) {
>> +               nfsi->cache_validity |= NFS_INO_INVALID_ATTR;
>>                 return 0;
>> +       }
>>
>>         lock_kernel();
>>         nfs_begin_data_update(inode);
>
> Why is this needed?

Do you mean why is flushing attribute cache needed, or why is this  
particular way to flush it needed?

I need to be able to find out if a file has changed, so I need to get  
its attribute cache flushed. fchown()ing to -1, -1 would work safely  
in all situations because it's guaranteed not to change the file in  
any way.

>> Another problem I have is that it's difficult to get a file's data  
>> cache
>> flushed. The only way I found was to successfully fcntl() lock the  
>> file.
>> This is pretty bad from performance point of view since often I don't
>> want/need to lock the file.
>>
>> Solaris and BSDs invalidate also a file's data cache when its  
>> attribute
>> cache is invalidated. It would be nicer if there was a separate  
>> way, but
>> I'd settle for fchown(fd, (uid_t)-1, (gid_t)-1) invalidating data  
>> cache
>> as well.
>>
>> Actually I did also look at posix_fadvise(fd, 0, 0,
>> POSIX_FADVN_DONTNEED). It appears to work, but I'm a bit worried  
>> about
>> race conditions that causes pages to randomly not get dropped because
>> invalidate_mapping_pages() doesn't drop locked pages.
>
> Again, why do you need this level of data cache invalidation? If you
> don't want cached i/o, then use O_DIRECT.

Because O_DIRECT is optional and I can't trust that support for it  
has been compiled into kernel. It defaults to off, and even its  
description makes it sound like it's a bad idea to enable normally.

Also O_DIRECT is a bit too much for my use case. I do want the file  
to be cached for the most part, but there are times occationally when  
parts of it can be overwritten, and I need to make sure that in those  
situations the newest data is read.

If you want a wider description of what I'm trying to do: I'm  
developing Dovecot IMAP server. A lot of people store mails on NFS  
and want to have multiple IMAP servers be able to access the mails.  
Dovecot uses somewhat complex index files to speed up accessing the  
mailboxes, and it's mainly for these index files that I need this  
explicit control over caching. If two servers are accessing the same  
mailbox at the same time, the index files get easily corrupted if I  
can't control the caching.

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Trond Myklebust]

On Sat, 2007-11-17 at 22:12 +0200, Timo Sirainen wrote:
> On 17.11.2007, at 21.46, Trond Myklebust wrote:
> > Why is this needed?
> 
> Do you mean why is flushing attribute cache needed, or why is this  
> particular way to flush it needed?
> 
> I need to be able to find out if a file has changed, so I need to get  
> its attribute cache flushed. fchown()ing to -1, -1 would work safely  
> in all situations because it's guaranteed not to change the file in  
> any way.

Why can't you simply close(), and then re-open() the file? That is _the_
standard way to force an attribute cache revalidation on all NFS
versions. The close-to-open caching model, which is implemented on most
NFS clients guarantees this.

> Also O_DIRECT is a bit too much for my use case. I do want the file  
> to be cached for the most part, but there are times occationally when  
> parts of it can be overwritten, and I need to make sure that in those  
> situations the newest data is read.
> 
> If you want a wider description of what I'm trying to do: I'm  
> developing Dovecot IMAP server. A lot of people store mails on NFS  
> and want to have multiple IMAP servers be able to access the mails.  
> Dovecot uses somewhat complex index files to speed up accessing the  
> mailboxes, and it's mainly for these index files that I need this  
> explicit control over caching. If two servers are accessing the same  
> mailbox at the same time, the index files get easily corrupted if I  
> can't control the caching.

So how are you ensuring that both servers don't try writing to the same
locations? You must have some form of synchronisation scheme for this to
work.

Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Timo Sirainen]

[-- Attachment #1.1: Type: text/plain, Size: 3036 bytes --]

On Sat, 2007-11-17 at 15:41 -0500, Trond Myklebust wrote:
> On Sat, 2007-11-17 at 22:12 +0200, Timo Sirainen wrote:
> > On 17.11.2007, at 21.46, Trond Myklebust wrote:
> > > Why is this needed?
> > 
> > Do you mean why is flushing attribute cache needed, or why is this  
> > particular way to flush it needed?
> > 
> > I need to be able to find out if a file has changed, so I need to get  
> > its attribute cache flushed. fchown()ing to -1, -1 would work safely  
> > in all situations because it's guaranteed not to change the file in  
> > any way.
> 
> Why can't you simply close(), and then re-open() the file? That is _the_
> standard way to force an attribute cache revalidation on all NFS
> versions. The close-to-open caching model, which is implemented on most
> NFS clients guarantees this.

Interesting. Too bad this is the first time I have heard of it (where as
I've seen fchown()/chown() suggested in several mailing lists before). I
understood NFS FAQ's close-to-open caching description to mean only how
data caching is handled.

close()+open() would have been difficult to handle because open() can
fail, but looks like opening another file descriptor and closing it
works just as well. Also looks like it works for flushing directories'
attribute cache (which doesn't seem to work with FreeBSD though).

There's one potential problem with closing a file descriptor though. It
loses all fcntl locks for that file from all fds. I'm not sure if this a
problem for me though.

> > Also O_DIRECT is a bit too much for my use case. I do want the file  
> > to be cached for the most part, but there are times occationally when  
> > parts of it can be overwritten, and I need to make sure that in those  
> > situations the newest data is read.
> > 
> > If you want a wider description of what I'm trying to do: I'm  
> > developing Dovecot IMAP server. A lot of people store mails on NFS  
> > and want to have multiple IMAP servers be able to access the mails.  
> > Dovecot uses somewhat complex index files to speed up accessing the  
> > mailboxes, and it's mainly for these index files that I need this  
> > explicit control over caching. If two servers are accessing the same  
> > mailbox at the same time, the index files get easily corrupted if I  
> > can't control the caching.
> 
> So how are you ensuring that both servers don't try writing to the same
> locations? You must have some form of synchronisation scheme for this to
> work.

Well, there's no simple answer for that. :) There are 3 different kinds
of index files with completely different locking behavior, because I try
to avoid long lasting locks. I do use write locks, but reads are mainly
lockless. There's this transaction log file which tells me when
something has changed, so I know when data cache need to be flushed.

Anyway, all of this is working already, but I'd just like to get the
performance a bit better with Linux by avoiding those unnecessary lock
+unlock sequences.

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Trond Myklebust]

On Sun, 2007-11-18 at 00:11 +0200, Timo Sirainen wrote:
> On Sat, 2007-11-17 at 15:41 -0500, Trond Myklebust wrote:
> > Why can't you simply close(), and then re-open() the file? That is _the_
> > standard way to force an attribute cache revalidation on all NFS
> > versions. The close-to-open caching model, which is implemented on most
> > NFS clients guarantees this.
> 
> Interesting. Too bad this is the first time I have heard of it (where as
> I've seen fchown()/chown() suggested in several mailing lists before). I
> understood NFS FAQ's close-to-open caching description to mean only how
> data caching is handled.

No. It handles attribute caching also. In NFSv2/v3, the mtime is used to
decide whether or not the data has changed. In NFSv4 it is the change
attribute. In either case, you need to force an attribute cache
update...

> close()+open() would have been difficult to handle because open() can
> fail, but looks like opening another file descriptor and closing it
> works just as well. Also looks like it works for flushing directories'
> attribute cache (which doesn't seem to work with FreeBSD though).
> 
> There's one potential problem with closing a file descriptor though. It
> loses all fcntl locks for that file from all fds. I'm not sure if this a
> problem for me though.

Right. I understood that you were not using fcntl() locks. Those will in
any case ensure cache revalidation, so you wouldn't have to use anything
else.

> > > Also O_DIRECT is a bit too much for my use case. I do want the file  
> > > to be cached for the most part, but there are times occationally when  
> > > parts of it can be overwritten, and I need to make sure that in those  
> > > situations the newest data is read.
> > > 
> > > If you want a wider description of what I'm trying to do: I'm  
> > > developing Dovecot IMAP server. A lot of people store mails on NFS  
> > > and want to have multiple IMAP servers be able to access the mails.  
> > > Dovecot uses somewhat complex index files to speed up accessing the  
> > > mailboxes, and it's mainly for these index files that I need this  
> > > explicit control over caching. If two servers are accessing the same  
> > > mailbox at the same time, the index files get easily corrupted if I  
> > > can't control the caching.
> > 
> > So how are you ensuring that both servers don't try writing to the same
> > locations? You must have some form of synchronisation scheme for this to
> > work.
> 
> Well, there's no simple answer for that. :) There are 3 different kinds
> of index files with completely different locking behavior, because I try
> to avoid long lasting locks. I do use write locks, but reads are mainly
> lockless. There's this transaction log file which tells me when
> something has changed, so I know when data cache need to be flushed.
> 
> Anyway, all of this is working already, but I'd just like to get the
> performance a bit better with Linux by avoiding those unnecessary lock
> +unlock sequences.

Again, you can use the close-to-open trick of simply reopening the file
whenever you need to revalidate the data cache.

The problem, however, is that on the most common Linux filesystems
(ext2/ext3, reiserfs,...) the time resolution on the mtime is 1 second.
If your NFS server is running one of those filesystems, then the data
cache revalidation may fail to detect a write that happens within 1
second of the previous write.

Cheers
  Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Timo Sirainen]

[-- Attachment #1.1: Type: text/plain, Size: 2126 bytes --]

On 18.11.2007, at 1.52, Trond Myklebust wrote:

>> close()+open() would have been difficult to handle because open() can
>> fail, but looks like opening another file descriptor and closing it
>> works just as well. Also looks like it works for flushing  
>> directories'
>> attribute cache (which doesn't seem to work with FreeBSD though).
>>
>> There's one potential problem with closing a file descriptor  
>> though. It
>> loses all fcntl locks for that file from all fds. I'm not sure if  
>> this a
>> problem for me though.
>
> Right. I understood that you were not using fcntl() locks. Those  
> will in
> any case ensure cache revalidation, so you wouldn't have to use  
> anything
> else.

That assumes that locks are used in a typical "read lock" and "write  
lock" way. Dovecot currently does, but not necessarily in future. For  
example UW-IMAP's mbx format requires grabbing a shared lock when the  
mailbox is opened and keeping it until the mailbox is closed. The  
shared lock prevents messages from being deleted from the file, but  
it doesn't prevent message flag updates or appending new messages to  
the file. To see if there are new messages would require flushing  
attribute cache to get the file size updated, but close+open wouldn't  
work because it would drop the shared lock and drop the guarantee  
that no messages get deleted while the mailbox is open.

> Again, you can use the close-to-open trick of simply reopening the  
> file
> whenever you need to revalidate the data cache.
>
> The problem, however, is that on the most common Linux filesystems
> (ext2/ext3, reiserfs,...) the time resolution on the mtime is 1  
> second.
> If your NFS server is running one of those filesystems, then the data
> cache revalidation may fail to detect a write that happens within 1
> second of the previous write.

Most people are using NetApps, and they also seem to have 1 second  
resolution (at least the one I just tested had). So this isn't a  
solution. "Let's hope that there are no writes less than 1 second  
apart" isn't really a solution either. People like their mailboxes  
uncorrupted.

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Trond Myklebust]

On Sun, 2007-11-18 at 02:26 +0200, Timo Sirainen wrote:
> Most people are using NetApps, and they also seem to have 1 second  
> resolution (at least the one I just tested had). So this isn't a  
> solution. "Let's hope that there are no writes less than 1 second  
> apart" isn't really a solution either. People like their mailboxes  
> uncorrupted.

No. NetApp filers and WAFL have nanosecond resolution. They should be
fine.

Cheers
  Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Timo Sirainen]

[-- Attachment #1.1: Type: text/plain, Size: 1356 bytes --]

On Sat, 2007-11-17 at 19:46 -0500, Trond Myklebust wrote:

> Right. I understood that you were not using fcntl() locks. Those will
> in
> any case ensure cache revalidation, so you wouldn't have to use
> anything
> else.

I just noticed that FreeBSD's fcntl() locking doesn't flush either
attribute or data cache. So things seem to be a lot more difficult to
handle with it.

> On Sun, 2007-11-18 at 02:26 +0200, Timo Sirainen wrote:
> > Most people are using NetApps, and they also seem to have 1 second  
> > resolution (at least the one I just tested had). So this isn't a  
> > solution. "Let's hope that there are no writes less than 1 second  
> > apart" isn't really a solution either. People like their mailboxes  
> > uncorrupted.
> 
> No. NetApp filers and WAFL have nanosecond resolution. They should be
> fine.

I guess I ran my test accidentally on local filesystem. I see only
microsecond resolution though, the last 3 digits in st_mtim.tv_nsec are
always zeroes. But maybe that's enough to be trusted.

Looks like this mtime checking works only with close+open, but not with
fchown()ing the file. That explains a few other things I was mistaken
about as well.

BTW. I've been writing these things down to
http://iki.fi/tss/nfs-coding-howto.html. Would be nice to know if there
are still some mistakes in it.

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Timo Sirainen]

[-- Attachment #1.1: Type: text/plain, Size: 1153 bytes --]

On Sun, 2007-11-18 at 00:11 +0200, Timo Sirainen wrote:
> > Why can't you simply close(), and then re-open() the file? That is _the_
> > standard way to force an attribute cache revalidation on all NFS
> > versions. The close-to-open caching model, which is implemented on most
> > NFS clients guarantees this.
..
> close()+open() would have been difficult to handle because open() can
> fail, but looks like opening another file descriptor and closing it
> works just as well. Also looks like it works for flushing directories'
> attribute cache (which doesn't seem to work with FreeBSD though).

Actually it works for flushing a directory's attribute cache in
v2.6.17-rc2, but not in v2.6.22. chown() works in v2.6.22 also. Is there
a reason for this change? I guess it anyway means that I'm back to using
chown() for flushing a directory's attribute cache.

The reason for flushing a directory's attribute cache is so that I can
see with stat() if an open file under it has been replaced (if its inode
has changed).

http://dovecot.org/tools/nfstest.c can be used to easily test what works
and what doesn't for cache flushes.


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Trond Myklebust]

On Tue, 2007-11-20 at 04:14 +0200, Timo Sirainen wrote:
> On Sun, 2007-11-18 at 00:11 +0200, Timo Sirainen wrote:
> > > Why can't you simply close(), and then re-open() the file? That is _the_
> > > standard way to force an attribute cache revalidation on all NFS
> > > versions. The close-to-open caching model, which is implemented on most
> > > NFS clients guarantees this.
> ..
> > close()+open() would have been difficult to handle because open() can
> > fail, but looks like opening another file descriptor and closing it
> > works just as well. Also looks like it works for flushing directories'
> > attribute cache (which doesn't seem to work with FreeBSD though).
> 
> Actually it works for flushing a directory's attribute cache in
> v2.6.17-rc2, but not in v2.6.22. chown() works in v2.6.22 also. Is there
> a reason for this change? I guess it anyway means that I'm back to using
> chown() for flushing a directory's attribute cache.

close-to-open caching works fine for me in v2.6.22. It does indeed send
a GETATTR and revalidate the inode.

Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Timo Sirainen]

[-- Attachment #1.1: Type: text/plain, Size: 2140 bytes --]

On Tue, 2007-11-20 at 18:47 -0500, Trond Myklebust wrote:
> On Tue, 2007-11-20 at 04:14 +0200, Timo Sirainen wrote:
> > On Sun, 2007-11-18 at 00:11 +0200, Timo Sirainen wrote:
> > > > Why can't you simply close(), and then re-open() the file? That is _the_
> > > > standard way to force an attribute cache revalidation on all NFS
> > > > versions. The close-to-open caching model, which is implemented on most
> > > > NFS clients guarantees this.
> > ..
> > > close()+open() would have been difficult to handle because open() can
> > > fail, but looks like opening another file descriptor and closing it
> > > works just as well. Also looks like it works for flushing directories'
> > > attribute cache (which doesn't seem to work with FreeBSD though).
> > 
> > Actually it works for flushing a directory's attribute cache in
> > v2.6.17-rc2, but not in v2.6.22. chown() works in v2.6.22 also. Is there
> > a reason for this change? I guess it anyway means that I'm back to using
> > chown() for flushing a directory's attribute cache.
> 
> close-to-open caching works fine for me in v2.6.22. It does indeed send
> a GETATTR and revalidate the inode.

I don't have my own NFS test setup, so I can't check what actually
happens in the network, but I can easily reproduce this as a user:

1. touch foo.1 foo.2

2. Run on NFS client 1:

rm -f foo;ln foo.1 foo;sleep 0.5;rm -f foo;ln foo.2 foo

3. Run on NFS client 2 within that 0.5 secs:

echo *>/dev/null;stat foo;sleep 0.5;echo * >/dev/null;stat foo

After repeating this a few (2-5) times stat will return foo.1's inode
for both stats and running the command over and over again will return
only foo.1's inode. The only way to get NFS client 2 to notice the
change and return foo.2's inode is to either wait for attribute cache to
timeout or to run "chown 0 ."

Now that I think of it, I guess the reason is that my 2.6.22 setup has
Linux+ext3 as NFS server and 2.6.17-rc2 setup has NetApp as server. So
even though open()+close() flushes the directory's attribute cache, it
doesn't flush the file name -> NFS handle cache unless mtime changes?

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Trond Myklebust]

On Wed, 2007-11-21 at 15:14 +0200, Timo Sirainen wrote:

> Now that I think of it, I guess the reason is that my 2.6.22 setup has
> Linux+ext3 as NFS server and 2.6.17-rc2 setup has NetApp as server. So
> even though open()+close() flushes the directory's attribute cache, it
> doesn't flush the file name -> NFS handle cache unless mtime changes?

Right. You're hitting the principal limitation of ext3 as an NFS backend
filesystem.
As I said in an earlier mail, the resolution on the time is 1 second, so
client 2 can basically not see any changes that happen within < 1 second
on client 1. The reason is that the mtime stays the same.

Cheers
  Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Timo Sirainen]

[-- Attachment #1.1: Type: text/plain, Size: 1076 bytes --]

On 21.11.2007, at 15.56, Trond Myklebust wrote:

> On Wed, 2007-11-21 at 15:14 +0200, Timo Sirainen wrote:
>
>> Now that I think of it, I guess the reason is that my 2.6.22 setup  
>> has
>> Linux+ext3 as NFS server and 2.6.17-rc2 setup has NetApp as  
>> server. So
>> even though open()+close() flushes the directory's attribute  
>> cache, it
>> doesn't flush the file name -> NFS handle cache unless mtime changes?
>
> Right. You're hitting the principal limitation of ext3 as an NFS  
> backend
> filesystem.
> As I said in an earlier mail, the resolution on the time is 1  
> second, so
> client 2 can basically not see any changes that happen within < 1  
> second
> on client 1. The reason is that the mtime stays the same.

With files it's possible to work around this limitation with fcntl  
locking, but directories can't be locked. So back to my original  
question: Would it be possible to have chmod(dir, (uid_t)-1,  
(gid_t)-1) flush its cache? I don't see what harm there could be in  
this, since normally no-one would do it anyway and the change is simple.


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Trond Myklebust]

On Wed, 2007-11-21 at 22:36 +0200, Timo Sirainen wrote:
> On 21.11.2007, at 15.56, Trond Myklebust wrote:
> 
> > On Wed, 2007-11-21 at 15:14 +0200, Timo Sirainen wrote:
> >
> >> Now that I think of it, I guess the reason is that my 2.6.22 setup  
> >> has
> >> Linux+ext3 as NFS server and 2.6.17-rc2 setup has NetApp as  
> >> server. So
> >> even though open()+close() flushes the directory's attribute  
> >> cache, it
> >> doesn't flush the file name -> NFS handle cache unless mtime changes?
> >
> > Right. You're hitting the principal limitation of ext3 as an NFS  
> > backend
> > filesystem.
> > As I said in an earlier mail, the resolution on the time is 1  
> > second, so
> > client 2 can basically not see any changes that happen within < 1  
> > second
> > on client 1. The reason is that the mtime stays the same.
> 
> With files it's possible to work around this limitation with fcntl  
> locking, but directories can't be locked. So back to my original  
> question: Would it be possible to have chmod(dir, (uid_t)-1,  
> (gid_t)-1) flush its cache? I don't see what harm there could be in  
> this, since normally no-one would do it anyway and the change is simple.

I thought you said this was for flushing the attribute cache?

Flushing the attribute cache won't help in this case, since the crux of
the problem is that the attributes haven't actually changed at all.

Cheers
  Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Timo Sirainen]

[-- Attachment #1.1: Type: text/plain, Size: 1518 bytes --]

On Wed, 2007-11-21 at 15:39 -0500, Trond Myklebust wrote:
> On Wed, 2007-11-21 at 22:36 +0200, Timo Sirainen wrote:
> > On 21.11.2007, at 15.56, Trond Myklebust wrote:
> > 
> > > On Wed, 2007-11-21 at 15:14 +0200, Timo Sirainen wrote:
> > >
> > >> Now that I think of it, I guess the reason is that my 2.6.22 setup  
> > >> has
> > >> Linux+ext3 as NFS server and 2.6.17-rc2 setup has NetApp as  
> > >> server. So
> > >> even though open()+close() flushes the directory's attribute  
> > >> cache, it
> > >> doesn't flush the file name -> NFS handle cache unless mtime changes?
> > >
> > > Right. You're hitting the principal limitation of ext3 as an NFS  
> > > backend
> > > filesystem.
> > > As I said in an earlier mail, the resolution on the time is 1  
> > > second, so
> > > client 2 can basically not see any changes that happen within < 1  
> > > second
> > > on client 1. The reason is that the mtime stays the same.
> > 
> > With files it's possible to work around this limitation with fcntl  
> > locking, but directories can't be locked. So back to my original  
> > question: Would it be possible to have chmod(dir, (uid_t)-1,  
> > (gid_t)-1) flush its cache? I don't see what harm there could be in  
> > this, since normally no-one would do it anyway and the change is simple-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [NFS] Cache flushing

[Trond Myklebust]

On Wed, 2007-11-21 at 22:56 +0200, Timo Sirainen wrote:
> On Wed, 2007-11-21 at 15:39 -0500, Trond Myklebust wrote:
> > I thought you said this was for flushing the attribute cache?
> 
> That was it originally, but this one's more important. :) Also I
> originally thought that directory's attribute cache flushing also
> flushed its file handle cache. I'm not actually sure what file handle
> cache should even really be called. It looks like Linux handles it with
> page cache the same way as for files?
> 
> So I guess I'm asking for chown(-1,-1) call to invalidate the file's
> attribute and page cache. Although it's interesting why chown(0,-1)
> flushes a directory's file handle cache (==page cache?), but the same
> call for a file doesn't flush its page cache. Maybe I'm still
> misunderstanding something how all of this works.

No. I'm not at all comfortable with the idea of overloading chown() to
invalidate data caches.

It is one thing to have it revalidate the attribute cache (you might be
able to argue that this is consistent with the mission of chown(),
although I'm not yet convinced of that) but the only effect that chown()
has on data caching is that we force a sync to disk in order to avoid
trouble with cached writes that are no longer accepted by the server. It
has never forced a cache invalidation.

IMO, the right interface for manipulating the page cache is rather
posix_fadvise(). I can certainly see an argument for adding a
"POSIX_FADV_UNCACHE" option in order to force a cache invalidation for
those applications that need stronger cache consistency.

Note, however, that even if such a thing is accepted into the mainstream
kernel, you still have to convince all the distros to backport this
interface to their older kernels. If not, you will still have a problem
with legacy clients...

Cheers
  Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Cache flushing

[Timo Sirainen]

[-- Attachment #1.1: Type: text/plain, Size: 3831 bytes --]

On Wed, 2007-11-21 at 16:15 -0500, Trond Myklebust wrote:
> On Wed, 2007-11-21 at 22:56 +0200, Timo Sirainen wrote:
> > On Wed, 2007-11-21 at 15:39 -0500, Trond Myklebust wrote:
> > > I thought you said this was for flushing the attribute cache?
> > 
> > That was it originally, but this one's more important. :) Also I
> > originally thought that directory's attribute cache flushing also
> > flushed its file handle cache. I'm not actually sure what file handle
> > cache should even really be called. It looks like Linux handles it with
> > page cache the same way as for files?
> > 
> > So I guess I'm asking for chown(-1,-1) call to invalidate the file's
> > attribute and page cache. Although it's interesting why chown(0,-1)
> > flushes a directory's file handle cache (==page cache?), but the same
> > call for a file doesn't flush its page cache. Maybe I'm still
> > misunderstanding something how all of this works.
> 
> No. I'm not at all comfortable with the idea of overloading chown() to
> invalidate data caches.
> 
> It is one thing to have it revalidate the attribute cache (you might be
> able to argue that this is consistent with the mission of chown(),
> although I'm not yet convinced of that) but the only effect that chown()
> has on data caching is that we force a sync to disk in order to avoid
> trouble with cached writes that are no longer accepted by the server. It
> has never forced a cache invalidation.

I finally set up my own NFS test setup and figured out how this really
works. As long as nfs_setattr() has something to do, it calls
nfs_end_data_update() at the end, which in turn unconditionally sets
"nfsi->cache_change_attribute = jiffies", which causes the file handle
cache to be flushed on next access.

So the only special case here is if nfs_setattr() has nothing to do.
Just moving that check a bit later would be enough:

--- inode.c.old	2007-11-16 22:18:46.000000000 +0200
+++ inode.c	2007-11-22 02:50:04.000000000 +0200
@@ -323,7 +323,7 @@
 {
 	struct inode *inode = dentry->d_inode;
 	struct nfs_fattr fattr;
-	int error;
+	int error = 0;
 
 	nfs_inc_stats(inode, NFSIOS_VFSSETATTR);
 
@@ -332,11 +332,6 @@
 			attr->ia_valid &= ~ATTR_SIZE;
 	}
 
-	/* Optimization: if the end result is no change, don't RPC */
-	attr->ia_valid &= NFS_VALID_ATTRS;
-	if (attr->ia_valid == 0)
-		return 0;
-
 	lock_kernel();
 	nfs_begin_data_update(inode);
 	/* Write all dirty data */
@@ -349,9 +344,14 @@
 	 */
 	if ((attr->ia_valid & (ATTR_MODE|ATTR_UID|ATTR_GID)) != 0)
 		nfs_inode_return_delegation(inode);
-	error = NFS_PROTO(inode)->setattr(dentry, &fattr, attr);
-	if (error == 0)
-		nfs_refresh_inode(inode, &fattr);
+
+	/* Optimization: if the end result is no change, don't RPC */
+	attr->ia_valid &= NFS_VALID_ATTRS;
+	if (attr->ia_valid != 0) {
+		error = NFS_PROTO(inode)->setattr(dentry, &fattr, attr);
+		if (error == 0)
+			nfs_refresh_inode(inode, &fattr);
+	}
 	nfs_end_data_update(inode);
 	unlock_kernel();
 	return error;


> IMO, the right interface for manipulating the page cache is rather
> posix_fadvise(). I can certainly see an argument for adding a
> "POSIX_FADV_UNCACHE" option in order to force a cache invalidation for
> those applications that need stronger cache consistency.

Yes, that would be nice. Perhaps I'll try to implement it some day.

> Note, however, that even if such a thing is accepted into the mainstream
> kernel, you still have to convince all the distros to backport this
> interface to their older kernels. If not, you will still have a problem
> with legacy clients...

I don't mind as long as I can say a newer kernel will solve the problem.
It's still a lot easier to upgrade the kernel than requiring to change
the whole NFS server/filesystem.

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs