(More) NFS Performance woes

[Christopher Smith <csmith@nighthawkrad.net>]

While I am waiting for Red Hat to sort out the weird Dell SAS 5/i and 
NFS bug I've found, we have temporarily thrown some WD Raptor drives 
into the machines to tide us over.  Performance is as good as the 
existing boxes, but I'm having trouble making it better.

The server machine is a Dell PE860, 4G RAM, Dual-core 2.4Ghz Xeon, 2* 
150G, 10k RPM SATA hard disk.  OS is CentOS 4, kernel 
2.6.9-42.0.3.ELsmp, x86_64.
The client is a Dell PE1425, 2G RAM, 2*3.2Ghz Xeon, 160G 7200rpm SATA 
drive.  OS is CentOS 4, kernel 2.6.9-42.0.2.ELsmp, x86.

The network between them is clean gigabit.  It does go through a Cisco 
3750 switch, but there's only about 5 other machines connected and the 
"storage network" has a dedicated vlan with only the clients and server 
on it.

The first test is copying a single large file of about 700M from the 
(client) local disk to the NFS mount.
The second test is untarring a 700M tarfile from the (client) local disk 
to the NFS mount (~5000 files).

Test #1 consistently finishes in about 17 seconds, delivering about 40 
MiB/s.
Test #2 consistently finishes in about 85 seconds, delivering about 8 MiB/s

For reference, untarring the same tarfile locally on the server machine 
takes 7-8 seconds (about 80 MiB/s).

Please note that test #2 is most representative of this machine's 
real-life usage (while there will be reads as well, we expect with 4G of 
RAM the vast majority will come from the cache and not physical disk). 
Test #1 is primarily to demonstrate that there aren't any obvious or 
significant problems with the underlying network and disk IO.

I have tried various values for rsize and wsize from 2048 up to 32768, 
with both TCP and UDP mounts (TCP seems _marginally_ slower - 0.5s test 
#1, 3s test #2).  With exception of r/wsize=2048 for test #1 (which was 
about 20% slower), all combinations give basically the same performance 
+/- a couple of percent.

I'm not expecting to get the same performance for test #2 as test #1, 
but surely it can do better than ~8MiB/sec (or even 40MiB/sec for test 
#1) ?  How close can NFS get to saturating gigabit ?  Do I need to try 
FreeBSD or Solaris ;) ?

Cheers,
CS

-- 
Christopher Smith

Systems Administrator
Nighthawk Radiology Services
Level 11, Suite 1101
Grosvenor Place
225 George Street
Sydney NSW 2000
Australia

T:  612 - 8211 2300 (IP: x8163)
F:  612 - 8211 2333
M:  61  - 407 397 563
USA Toll free:  866 241 6635
E: csmith@nighthawkrad.net
I: www.nighthawkrad.net

CONFIDENTIALITY NOTICE:   This email, including any attachments, 
contains information from NightHawk Radiology Services, which may be 
confidential or privileged. The information is intended to be for the 
use of the individual or entity named above. If you are not the intended 
recipient, be aware that any disclosure, copying, distribution or use of 
the contents of this information is prohibited. If you have received 
this email in error, please notify NightHawk Radiology Services 
immediately by forwarding message to postmaster@nighthawkrad.net and 
destroy all electronic and hard copies of the communication, including 
attachments.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs