Can we flush directory data when the ctime changes?

Can we flush directory data when the ctime changes?

[Neil Brown]

Hi,
 We have a scenario where readdir in the linux client gets confused by 
responses from a Netapp fileserver.

What happens is that a readdir collects and caches the directory.
Then a subsequent readdir finds the first page is still in the cache
but the second page is missing.
The nfs client uses a cookie from the first page to request subsequent
entries and received a list of entries starting from the beginning of
the directory rather than from the point that it was up to.   It seems
that the cookies have changed.

The GETATTR call that the client makes to validate the cache reports
that the mtime hasn't change, *but the ctime has*.

I patched the (SLES9) kernel to invalidate the cache for directories
when the ctime changes and this fixed the problem (which was quite
repeatable). 

While it seems that it should be necessary to flush the dir cache on a
ctime change, it doesn't seem harmful either.

So: would it be acceptable to do this.  The following patch should
achieve this in current -mm (which is quite different from the 2.6.5
kernel where I tested this).

In case it is interesting, I have this commentary from one of our
support personnel.  I'm not sure if the content comes from someone at
Netapp or elsewhere, but it sounds plausible.

Thanks,
NeilBrown

  ......

  For example, NetApp filer support NFS but their file system is not
  simply a true unix/linux style file system.  They emulate that for
  NFS purposes but they also handle the file system in special ways
  for account for CIFS and other access.

  Things go on like files name conversions to unicode, for other name
  spaces.  These events can take place at much later times than the
  original file name creation through NFS.  This results in changes in
  the ctime stamp, without changes to mtime, and it also results in
  new cookie indexes being generated for the NFS file system.  If an
  NFS client does not invalidate it's cache upon such a condition, if
  can result in inconsistent cache information, as it has in the
  example for which this bug report was filed.


Signed-off-by: Neil Brown <neilb@suse.de>

### Diffstat output
 ./fs/nfs/inode.c |    5 +++++
 1 file changed, 5 insertions(+)

diff .prev/fs/nfs/inode.c ./fs/nfs/inode.c
--- .prev/fs/nfs/inode.c	2006-08-08 17:03:15.000000000 +1000
+++ ./fs/nfs/inode.c	2006-08-08 17:04:48.000000000 +1000
@@ -943,6 +943,11 @@ static int nfs_update_inode(struct inode
 	/* If ctime has changed we should definitely clear access+acl caches */
 	if (!timespec_equal(&inode->i_ctime, &fattr->ctime)) {
 		invalid |= NFS_INO_INVALID_ACCESS|NFS_INO_INVALID_ACL;
+		if (S_ISDIR(inode->i_mode))
+			/* Some servers (netapp) can change cookies when the
+			 * ctime changes, without changing mtime...
+			 */
+			invalid |= NFS_INO_INVALID_DATA;
 		memcpy(&inode->i_ctime, &fattr->ctime, sizeof(inode->i_ctime));
 		nfsi->cache_change_attribute = jiffies;
 	}

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Can we flush directory data when the ctime changes?

[Chuck Lever]

On 8/8/06, Neil Brown <neilb@suse.de> wrote:
>  We have a scenario where readdir in the linux client gets confused by
> responses from a Netapp fileserver.
>
> What happens is that a readdir collects and caches the directory.
> Then a subsequent readdir finds the first page is still in the cache
> but the second page is missing.
> The nfs client uses a cookie from the first page to request subsequent
> entries and received a list of entries starting from the beginning of
> the directory rather than from the point that it was up to.   It seems
> that the cookies have changed.
>
> The GETATTR call that the client makes to validate the cache reports
> that the mtime hasn't change, *but the ctime has*.
>
> I patched the (SLES9) kernel to invalidate the cache for directories
> when the ctime changes and this fixed the problem (which was quite
> repeatable).
>
> While it seems that it should be necessary to flush the dir cache on a
> ctime change, it doesn't seem harmful either.
>
> So: would it be acceptable to do this.  The following patch should
> achieve this in current -mm (which is quite different from the 2.6.5
> kernel where I tested this).
>
> In case it is interesting, I have this commentary from one of our
> support personnel.  I'm not sure if the content comes from someone at
> Netapp or elsewhere, but it sounds plausible.
>
> Thanks,
> NeilBrown
>
>   ......
>
>   For example, NetApp filer support NFS but their file system is not
>   simply a true unix/linux style file system.  They emulate that for
>   NFS purposes but they also handle the file system in special ways
>   for account for CIFS and other access.
>
>   Things go on like files name conversions to unicode, for other name
>   spaces.  These events can take place at much later times than the
>   original file name creation through NFS.  This results in changes in
>   the ctime stamp, without changes to mtime, and it also results in
>   new cookie indexes being generated for the NFS file system.  If an
>   NFS client does not invalidate it's cache upon such a condition, if
>   can result in inconsistent cache information, as it has in the
>   example for which this bug report was filed.

This is reasonable.  Another scenario where ctime could change but
mtime might not is during a restore -- either snaprestore, NDMP copy,
or even an rsync.

However, I thought the latest kernels actually did account for ctime
changes like this....

-- 
"We who cut mere stones must always be envisioning cathedrals"
   -- Quarry worker's creed

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Can we flush directory data when the ctime changes?

[Trond Myklebust]

On Tue, 2006-08-08 at 17:26 +1000, Neil Brown wrote:
> Hi,
>  We have a scenario where readdir in the linux client gets confused by 
> responses from a Netapp fileserver.
> 
> What happens is that a readdir collects and caches the directory.
> Then a subsequent readdir finds the first page is still in the cache
> but the second page is missing.
> The nfs client uses a cookie from the first page to request subsequent
> entries and received a list of entries starting from the beginning of
> the directory rather than from the point that it was up to.   It seems
> that the cookies have changed.
> 
> The GETATTR call that the client makes to validate the cache reports
> that the mtime hasn't change, *but the ctime has*.

Growl! If the directory contents change, then the mtime should too. Page
22 of RFC1813:

        Mtime is the time when the file data was last modified.  Ctime
        is the time when the attributes of the file were last changed.

Directory cookies are not file or directory attributes!

In addition, NFSv3 has the cookie verifier mechanism for the server to
specifically inform the client that the cookies are invalid.

IOW: this really needs to be fixed on the _server_. I'm not happy taking
new patches that may cause the client GETATTR calls to skyrocket again.

Cheers,
  Trond

> I patched the (SLES9) kernel to invalidate the cache for directories
> when the ctime changes and this fixed the problem (which was quite
> repeatable). 
> 
> While it seems that it should be necessary to flush the dir cache on a
> ctime change, it doesn't seem harmful either.
> 
> So: would it be acceptable to do this.  The following patch should
> achieve this in current -mm (which is quite different from the 2.6.5
> kernel where I tested this).
> 
> In case it is interesting, I have this commentary from one of our
> support personnel.  I'm not sure if the content comes from someone at
> Netapp or elsewhere, but it sounds plausible.
> 
> Thanks,
> NeilBrown
> 
>   ......
> 
>   For example, NetApp filer support NFS but their file system is not
>   simply a true unix/linux style file system.  They emulate that for
>   NFS purposes but they also handle the file system in special ways
>   for account for CIFS and other access.
> 
>   Things go on like files name conversions to unicode, for other name
>   spaces.  These events can take place at much later times than the
>   original file name creation through NFS.  This results in changes in
>   the ctime stamp, without changes to mtime, and it also results in
>   new cookie indexes being generated for the NFS file system.  If an
>   NFS client does not invalidate it's cache upon such a condition, if
>   can result in inconsistent cache information, as it has in the
>   example for which this bug report was filed.
> 
> 
> Signed-off-by: Neil Brown <neilb@suse.de>
> 
> ### Diffstat output
>  ./fs/nfs/inode.c |    5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff .prev/fs/nfs/inode.c ./fs/nfs/inode.c
> --- .prev/fs/nfs/inode.c	2006-08-08 17:03:15.000000000 +1000
> +++ ./fs/nfs/inode.c	2006-08-08 17:04:48.000000000 +1000
> @@ -943,6 +943,11 @@ static int nfs_update_inode(struct inode
>  	/* If ctime has changed we should definitely clear access+acl caches */
>  	if (!timespec_equal(&inode->i_ctime, &fattr->ctime)) {
>  		invalid |= NFS_INO_INVALID_ACCESS|NFS_INO_INVALID_ACL;
> +		if (S_ISDIR(inode->i_mode))
> +			/* Some servers (netapp) can change cookies when the
> +			 * ctime changes, without changing mtime...
> +			 */
> +			invalid |= NFS_INO_INVALID_DATA;
>  		memcpy(&inode->i_ctime, &fattr->ctime, sizeof(inode->i_ctime));
>  		nfsi->cache_change_attribute = jiffies;
>  	}


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Can we flush directory data when the ctime changes?

[Peter Staubach]

Trond Myklebust wrote:
> On Tue, 2006-08-08 at 17:26 +1000, Neil Brown wrote:
>   
>> Hi,
>>  We have a scenario where readdir in the linux client gets confused by 
>> responses from a Netapp fileserver.
>>
>> What happens is that a readdir collects and caches the directory.
>> Then a subsequent readdir finds the first page is still in the cache
>> but the second page is missing.
>> The nfs client uses a cookie from the first page to request subsequent
>> entries and received a list of entries starting from the beginning of
>> the directory rather than from the point that it was up to.   It seems
>> that the cookies have changed.
>>
>> The GETATTR call that the client makes to validate the cache reports
>> that the mtime hasn't change, *but the ctime has*.
>>     
>
> Growl! If the directory contents change, then the mtime should too. Page
> 22 of RFC1813:
>
>         Mtime is the time when the file data was last modified.  Ctime
>         is the time when the attributes of the file were last changed.
>
> Directory cookies are not file or directory attributes!
>
> In addition, NFSv3 has the cookie verifier mechanism for the server to
> specifically inform the client that the cookies are invalid.
>
> IOW: this really needs to be fixed on the _server_. I'm not happy taking
> new patches that may cause the client GETATTR calls to skyrocket again.

This really does seem like a bug in the NetApp server.  With a change
like this, a simple chmod(2) could cause the client to invalidate its
caches...

       ps

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Can we flush directory data when the ctime changes?

[Trond Myklebust]

On Tue, 2006-08-08 at 10:27 -0400, Peter Staubach wrote:
> This really does seem like a bug in the NetApp server.  With a change
> like this, a simple chmod(2) could cause the client to invalidate its
> caches...

Agreed.

Now what we could do to improve the client is to try to detect the loops
this sort of bug causes. I'll try to think a bit about that...

Cheers,
  Trond


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Can we flush directory data when the ctime changes?

[Neil Brown]

On Tuesday August 8, trond.myklebust@fys.uio.no wrote:
> On Tue, 2006-08-08 at 17:26 +1000, Neil Brown wrote:
> > 
> > The GETATTR call that the client makes to validate the cache reports
> > that the mtime hasn't change, *but the ctime has*.
> 
> Growl! If the directory contents change, then the mtime should too. Page
> 22 of RFC1813:
> 
>         Mtime is the time when the file data was last modified.  Ctime
>         is the time when the attributes of the file were last changed.
> 
> Directory cookies are not file or directory attributes!
> 
> In addition, NFSv3 has the cookie verifier mechanism for the server to
> specifically inform the client that the cookies are invalid.

What would you expect the user experience to be if the server returned 
NFS3ERR_BAD_COOKIE ?
I cannot really tell from the nfs client code, but I would guess that
the only option would be to start again from the beginning of the
directory, and that would lead to duplicate entries being returned,
which is exactly the problem are seeing (though the server doesn't
return BAD_COOKIE, it just starts again from the beginning).

> 
> IOW: this really needs to be fixed on the _server_. I'm not happy taking
> new patches that may cause the client GETATTR calls to skyrocket again.
> 

I certainly have a lot of sympathy for this position.

Thanks,
NeilBrown

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Can we flush directory data when the ctime changes?

[Peter Staubach]

Neil Brown wrote:
> On Tuesday August 8, trond.myklebust@fys.uio.no wrote:
>   
>> On Tue, 2006-08-08 at 17:26 +1000, Neil Brown wrote:
>>     
>>> The GETATTR call that the client makes to validate the cache reports
>>> that the mtime hasn't change, *but the ctime has*.
>>>       
>> Growl! If the directory contents change, then the mtime should too. Page
>> 22 of RFC1813:
>>
>>         Mtime is the time when the file data was last modified.  Ctime
>>         is the time when the attributes of the file were last changed.
>>
>> Directory cookies are not file or directory attributes!
>>
>> In addition, NFSv3 has the cookie verifier mechanism for the server to
>> specifically inform the client that the cookies are invalid.
>>     
>
> What would you expect the user experience to be if the server returned 
> NFS3ERR_BAD_COOKIE ?
> I cannot really tell from the nfs client code, but I would guess that
> the only option would be to start again from the beginning of the
> directory, and that would lead to duplicate entries being returned,
> which is exactly the problem are seeing (though the server doesn't
> return BAD_COOKIE, it just starts again from the beginning).
>
>   

The only thing that the client can do is return an error, perhaps something
like EIO.  The real recovery needs to happen in the application, but there
isn't a way to convey the correct information to the application so that it
could do the recovery.

As was noted, transparently starting again at the beginning of the 
directory,
whether in the kernel or not, can lead to bad things happening.

In retrospect, NFS3ERR_BAD_COOKIE wasn't such a good idea.  There just
isn't enough infrastructure in the rest of the system to correctly
support the semantic that it represents.

>> IOW: this really needs to be fixed on the _server_. I'm not happy taking
>> new patches that may cause the client GETATTR calls to skyrocket again.
>>
>>     
>
> I certainly have a lot of sympathy for this position.

I would agree as well.  I think that represents a constraint on the file
systems which can be used on the server.  Although it is not a huge issue,
it sure would be nice to have it fixed.  Perhaps we need to construct the
infrastructure required in order to be able to detect and recover from
directory cookies which change mid-stream.

    Thanx...

       ps

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs