Re: extremely slow nfs when sync enabled

[Daniel Pocock <daniel@pocock.com.au>]

On 08/05/12 12:45, J. Bruce Fields wrote:
> On Tue, May 08, 2012 at 12:06:59PM +0000, Daniel Pocock wrote:
>>
>>
>> On 07/05/12 17:18, J. Bruce Fields wrote:
>>> How many file creates per second?
>>>
>>
>> I ran:
>> nfsstat -s -o all -l -Z5
>> and during the test (unpacking the tarball), I see numbers like these
>> every 5 seconds for about 2 minutes:
>>
>> nfs v3 server        total:      319
>> ------------- ------------- --------
>> nfs v3 server      getattr:        1
>> nfs v3 server      setattr:      126
>> nfs v3 server       access:        6
>> nfs v3 server        write:       61
>> nfs v3 server       create:       61
>> nfs v3 server        mkdir:        3
>> nfs v3 server       commit:       61
> 
> OK, so it's probably creating about 60 new files, each requiring a
> create, write, commit, and two setattrs?
> 
> Each of those operations is synchronous, so probably has to wait for at
> least one disk seek.  About 300 such operations every 5 seconds is about
> 60 per second, or about 16ms each.  That doesn't sound so far off.
> 
> (I wonder why it needs two setattrs?)

I checked that with wireshark:
- first SETATTR sets mode=0644 and atime=mtime=`set to server time'
- second SETATTR sets atime and mtime using client values (atime=now,
mtime= some time in the past)

Is this likely to be an application issue (e.g. in tar), or should the
NFS client be able to merge those two requests somehow?

If I add `m' to my tar command (tar xzmf) the nfsstat results change:

nfs v3 server        total:      300
------------- ------------- --------
nfs v3 server      setattr:       82
nfs v3 server       lookup:       17
nfs v3 server       access:        5
nfs v3 server        write:       90
nfs v3 server       create:       83
nfs v3 server        mkdir:        3
nfs v3 server       remove:       15
nfs v3 server       commit:        5

and iostat reports that w/s is about the same (290/sec) but throughput
is now in the region 1.1 - 1.5MBytes/sec


Without using `m', here are the full stats from the Z800 acting as NFSv3
server:

nfs v3 server        total:      299
------------- ------------- --------
nfs v3 server      setattr:      132
nfs v3 server       access:        6
nfs v3 server        write:       88
nfs v3 server       create:       62
nfs v3 server        mkdir:        6
nfs v3 server       commit:        5

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
avgrq-sz avgqu-sz   await  svctm  %util

dm-10             0.00     0.00    0.00  294.00     0.00     1.15
8.00     1.04    3.55   3.26  95.92

The other thing that stands out is this:
avgrq-sz = 8  (units = 512 byte sectors)

If the server filesystem is btrfs, the wMB/s figure is the same, but I
notice avgrq-sz = 16, so it seems to be combining more requests into
bigger writes


Here is the entry from /proc/mounts:
/dev/mapper/vg01-test1 /mnt/test1 ext4
rw,relatime,barrier=1,data=ordered 0 0