FLushing cached writes in nfs_getattr() and stat() delay

[Alex Sidorenko <asid@hp.com>]

Hello,

I am an HP engineer participating in L3 Linux support. Recently we have found 
that current design of nfs_getattr() might create huge delays in stat() on 
the file we are writing to (this is important for big files only, >2Gb).

The problem
-----------

Assuming that /nfs is an NFS-mounted FS:

1. In one shell, start
$ dd if=/dev/zero of=/nfs/dir/big bs=1G count=20

2. In another shell, start

$ ls -l /nfs/dir

or

$ ls -l /nfs/dir/big

'ls' does not return until the whole /nfs/dir/big is written.

Analysis
--------

Kernel 2.6.16 has introduced the following change in nfs_getattr():


    NFS: Make stat() return updated mtimes after a write()

    The SuS states that a call to write() will cause mtime to be updated on
    the file. In order to satisfy that requirement, we need to flush out
    any cached writes in nfs_getattr().
    Speed things up slightly by not committing the writes.

    Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>

+	/* Flush out writes to the server in order to update c/mtime */
+	nfs_sync_inode(inode, 0, 0, FLUSH_WAIT|FLUSH_NOCOMMIT);

Then later:

2.6.16-rc6:

http://www.linux-nfs.org/Linux-2.6.x/2.6.16-rc6/linux-2.6.16-99-fix_nfs_sync_inode_race.dif

 	/* Flush out writes to the server in order to update c/mtime */
-	nfs_sync_inode(inode, 0, 0, FLUSH_WAIT|FLUSH_NOCOMMIT);
+	nfs_sync_inode_wait(inode, 0, 0, FLUSH_NOCOMMIT);


I understand the reasoning behind that. From application point of view, NFS 
file/directory should behave the same as on local FS. If we have queued many 
writes, without this patch stat() will return incorrect results, both for 
mtime and file length. Some applications may depend on stat() results being 
correct.

At the same time, the fact that we have to wait forever while copying big 
files and doing 'ls -l' on that directory (or on the file being written) is 
not very good either (two HP customers have complained about this after 
migrating from RHEL4 to RHEL5).

The problem is still there in 2.6.27. I am not sure what can be done to both 
reduce the stat() delay and guarantee reasonable stat() results. 

It is interesting that with 'noac' stat() returns much faster (just 1-3s 
delay).

Best regards,
Alex

-- 
------------------------------------------------------------------
Alexandre Sidorenko             email: asid@hp.com
Global Solutions Engineering:   Unix Networking
Hewlett-Packard (Canada)
------------------------------------------------------------------