Re: nfs client readdir caching issue?

[Trond Myklebust <trond.myklebust@fys.uio.no>]

On Wed, 2008-07-02 at 12:03 +0100, Andy Chittenden wrote:
> Very rarely, we're seeing various problems on a linux kernel client
> (seen on various versions) with ls on directories from an NFS server
> that haven't changed:
> 
> * looping ls (strace -v shows getdents returning the same names over
> again).
> * duplicate directory entries.
> * missing directory entries.
> 
> I've hunted google but can only see problems where NFS servers have
> returned duplicate cookies. I've packet captured the readdirplus on one
> of the directories and see no duplicate cookies. The problems remain
> until the directory is touched, the NFS server is unmounted or some
> other event happens (the data is flushed from the cache?).
> 
> I think we then got lucky and got two packet captures from different
> clients running the same linux kernel. On these clients, the ls output
> was ok - no loops, no duplicates, no missing entries. Both captures
> showed two readdirplus requests returning the same entries in the same
> order but the amount of data in the responses was different. One capture
> showed the server returned 1724 bytes, 10 entries, last cookie of 12,
> followed by the next readdirplus returning a length of 948 bytes, 5
> entries, a first cookie value of 13. In the other capture, the responses
> returned 2204 bytes, 13 entries, a last cookie of 17 and 468 bytes, 2
> entries, a first cookie of 19.
> 
> In the past we've found that ls has returned duplicate entries on this
> directory (but didn't have a capture at the time) and those duplicate
> entries are the ones that are returned as the last 3 entries in the
> first response of the second capture and the first 3 entries in the
> second response of the first capture.
> 
> So what I think has happened in this particular case, is that at some
> point in the past, the directory was read OK with packets similar to the
> first capture. Next, the client decided to get rid of the first page of
> cached readdir responses from memory for some reason (running low on
> memory?) but kept the second page. Subsequently, the readdir cache needs
> repopulating so the client sends a readdirplus specifying cookie of 0
> and this time it gets a response which is similar to the first packet of
> the second capture and thus we now have in cache duplicate names and
> cookie values.
> 
> So is this possible? Is there some easy way to provoke it? Does this
> mean the client's readdir cache is broken?

If so, then invalidate_inode_pages2_range() would have to be broken: we
always clear the readdir cache immediately after reading in the page
with index 0 (i.e. the first readdir page).

It shouldn't be possible for another thread to race with that cache
invalidation either since the entire readdir() call is protected by the
parent directory's inode->i_mutex.

Cheers
  Trond