Re: extremely slow nfs when sync enabled

["J. Bruce Fields" <bfields@fieldses.org>]

On Tue, May 08, 2012 at 12:06:59PM +0000, Daniel Pocock wrote:
> 
> 
> On 07/05/12 17:18, J. Bruce Fields wrote:
> > How many file creates per second?
> > 
> 
> I ran:
> nfsstat -s -o all -l -Z5
> and during the test (unpacking the tarball), I see numbers like these
> every 5 seconds for about 2 minutes:
> 
> nfs v3 server        total:      319
> ------------- ------------- --------
> nfs v3 server      getattr:        1
> nfs v3 server      setattr:      126
> nfs v3 server       access:        6
> nfs v3 server        write:       61
> nfs v3 server       create:       61
> nfs v3 server        mkdir:        3
> nfs v3 server       commit:       61

OK, so it's probably creating about 60 new files, each requiring a
create, write, commit, and two setattrs?

Each of those operations is synchronous, so probably has to wait for at
least one disk seek.  About 300 such operations every 5 seconds is about
60 per second, or about 16ms each.  That doesn't sound so far off.

(I wonder why it needs two setattrs?)

> I decided to expand the scope of my testing too, I want to rule out the
> possibility that my HP Microserver with onboard SATA is the culprit.  I
> set up two other NFS servers (all Debian 6, kernel 2.6.38):
> 
> HP Z800 Xeon workstation
> Intel Corporation 82801 SATA RAID Controller (operating as AHCI)
> VB0250EAVER (250GB 7200rpm)
> 
> Lenovo Thinkpad X220
> Intel Corporation Cougar Point 6 port SATA AHCI Controller (rev 04)
> SSDSA2BW160G3L  (160GB SSD)
> 
> Both the Z800 and X220 run as NFSv3 servers
> Each one has a fresh 10GB logical volume formatted ext4,
> mount options: barrier=1,data=ordered
> write cache (hdparm -W 1): enabled
> 
> Results:
> NFS client: X220
> NFS server: Z800 (regular disk)
> iostat reports about 1,000kbytes/sec when unpacking the tarball
> This is just as slow as the original NFS server

Again, reporting kbytes/second alone isn't useful--data throughput isn't
interesting for a workload like unpacking a tarball with a lot of small
files.  The limiting factor is the synchronous operations.

> NFS client: Z800
> NFS server: X220 (SSD disk)
> iostat reports about 22,000kbytes/sec when unpacking the tarball
> 
> It seems that buying a pair of SSDs for my HP MicroServer will let me
> use NFS `sync' and enjoy healthy performance - 20x faster.

And an SSD has much lower write latency, so this isn't surprising.

> However, is there really no other way to get more speed out of NFS when
> using the `sync' option?

I've heard reports of people being able to get better performance on
this kind of workload by using an external journal on an SSD.

(Last I tried this--with a machine at home, using (if I remember
correctly) ext4 on software raid with the journal on an intel x25-m, I
wasn't able to get any improvement.  I didn't try to figure out why.)

--b.