(More) NFS Performance woes

(More) NFS Performance woes

[Christopher Smith]

While I am waiting for Red Hat to sort out the weird Dell SAS 5/i and 
NFS bug I've found, we have temporarily thrown some WD Raptor drives 
into the machines to tide us over.  Performance is as good as the 
existing boxes, but I'm having trouble making it better.

The server machine is a Dell PE860, 4G RAM, Dual-core 2.4Ghz Xeon, 2* 
150G, 10k RPM SATA hard disk.  OS is CentOS 4, kernel 
2.6.9-42.0.3.ELsmp, x86_64.
The client is a Dell PE1425, 2G RAM, 2*3.2Ghz Xeon, 160G 7200rpm SATA 
drive.  OS is CentOS 4, kernel 2.6.9-42.0.2.ELsmp, x86.

The network between them is clean gigabit.  It does go through a Cisco 
3750 switch, but there's only about 5 other machines connected and the 
"storage network" has a dedicated vlan with only the clients and server 
on it.

The first test is copying a single large file of about 700M from the 
(client) local disk to the NFS mount.
The second test is untarring a 700M tarfile from the (client) local disk 
to the NFS mount (~5000 files).

Test #1 consistently finishes in about 17 seconds, delivering about 40 
MiB/s.
Test #2 consistently finishes in about 85 seconds, delivering about 8 MiB/s

For reference, untarring the same tarfile locally on the server machine 
takes 7-8 seconds (about 80 MiB/s).

Please note that test #2 is most representative of this machine's 
real-life usage (while there will be reads as well, we expect with 4G of 
RAM the vast majority will come from the cache and not physical disk). 
Test #1 is primarily to demonstrate that there aren't any obvious or 
significant problems with the underlying network and disk IO.

I have tried various values for rsize and wsize from 2048 up to 32768, 
with both TCP and UDP mounts (TCP seems _marginally_ slower - 0.5s test 
#1, 3s test #2).  With exception of r/wsize=2048 for test #1 (which was 
about 20% slower), all combinations give basically the same performance 
+/- a couple of percent.

I'm not expecting to get the same performance for test #2 as test #1, 
but surely it can do better than ~8MiB/sec (or even 40MiB/sec for test 
#1) ?  How close can NFS get to saturating gigabit ?  Do I need to try 
FreeBSD or Solaris ;) ?

Cheers,
CS

-- 
Christopher Smith

Systems Administrator
Nighthawk Radiology Services
Level 11, Suite 1101
Grosvenor Place
225 George Street
Sydney NSW 2000
Australia

T:  612 - 8211 2300 (IP: x8163)
F:  612 - 8211 2333
M:  61  - 407 397 563
USA Toll free:  866 241 6635
E: csmith@nighthawkrad.net
I: www.nighthawkrad.net

CONFIDENTIALITY NOTICE:   This email, including any attachments, 
contains information from NightHawk Radiology Services, which may be 
confidential or privileged. The information is intended to be for the 
use of the individual or entity named above. If you are not the intended 
recipient, be aware that any disclosure, copying, distribution or use of 
the contents of this information is prohibited. If you have received 
this email in error, please notify NightHawk Radiology Services 
immediately by forwarding message to postmaster@nighthawkrad.net and 
destroy all electronic and hard copies of the communication, including 
attachments.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: (More) NFS Performance woes

[J. Bruce Fields]

On Fri, Dec 15, 2006 at 12:39:51AM +1100, Christopher Smith wrote:
> The first test is copying a single large file of about 700M from the 
> (client) local disk to the NFS mount.
> The second test is untarring a 700M tarfile from the (client) local disk 
> to the NFS mount (~5000 files).
> 
> Test #1 consistently finishes in about 17 seconds, delivering about 40 
> MiB/s.
> Test #2 consistently finishes in about 85 seconds, delivering about 8 MiB/s

That's about 17ms per file?  Each of those 5000 file creations is a
synchronous operation--the server doesn't respond until it's actually
committed the operation to disk.  I don't really know what expected
numbers are for your hardware, but given that for each file we have to
wait for both the initial creation and the final flush of the data to
disk on close, 17ms doesn't sound too far off.

> For reference, untarring the same tarfile locally on the server machine 
> takes 7-8 seconds (about 80 MiB/s).

Locally there's not the same requirement that stuff be committed to disk
before certain operations can complete.  For all I know it could be
performing that entire operation without touching the disk.

--b.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: (More) NFS Performance woes

[Trond Myklebust]

On Fri, 2006-12-15 at 00:39 +1100, Christopher Smith wrote:
> While I am waiting for Red Hat to sort out the weird Dell SAS 5/i and 
> NFS bug I've found, we have temporarily thrown some WD Raptor drives 
> into the machines to tide us over.  Performance is as good as the 
> existing boxes, but I'm having trouble making it better.
> 
> The server machine is a Dell PE860, 4G RAM, Dual-core 2.4Ghz Xeon, 2* 
> 150G, 10k RPM SATA hard disk.  OS is CentOS 4, kernel 
> 2.6.9-42.0.3.ELsmp, x86_64.
> The client is a Dell PE1425, 2G RAM, 2*3.2Ghz Xeon, 160G 7200rpm SATA 
> drive.  OS is CentOS 4, kernel 2.6.9-42.0.2.ELsmp, x86.
> 
> The network between them is clean gigabit.  It does go through a Cisco 
> 3750 switch, but there's only about 5 other machines connected and the 
> "storage network" has a dedicated vlan with only the clients and server 
> on it.
> 
> The first test is copying a single large file of about 700M from the 
> (client) local disk to the NFS mount.
> The second test is untarring a 700M tarfile from the (client) local disk 
> to the NFS mount (~5000 files).
> 
> Test #1 consistently finishes in about 17 seconds, delivering about 40 
> MiB/s.
> Test #2 consistently finishes in about 85 seconds, delivering about 8 MiB/s
> 
> For reference, untarring the same tarfile locally on the server machine 
> takes 7-8 seconds (about 80 MiB/s).
> 
> Please note that test #2 is most representative of this machine's 
> real-life usage (while there will be reads as well, we expect with 4G of 
> RAM the vast majority will come from the cache and not physical disk). 
> Test #1 is primarily to demonstrate that there aren't any obvious or 
> significant problems with the underlying network and disk IO.
> 
> I have tried various values for rsize and wsize from 2048 up to 32768, 
> with both TCP and UDP mounts (TCP seems _marginally_ slower - 0.5s test 
> #1, 3s test #2).  With exception of r/wsize=2048 for test #1 (which was 
> about 20% slower), all combinations give basically the same performance 
> +/- a couple of percent.
> 
> I'm not expecting to get the same performance for test #2 as test #1, 
> but surely it can do better than ~8MiB/sec (or even 40MiB/sec for test 
> #1) ?  How close can NFS get to saturating gigabit ?  Do I need to try 
> FreeBSD or Solaris ;) ?

File creation is a synchronous operation. Creating 5000 files and then
filling them with 700M worth of data will therefore always be orders of
magnitude slower than creating a single file with 700M worth of data.

Cheers,
  Trond


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: (More) NFS Performance woes

[Christian Robottom Reis]

On Thu, Dec 14, 2006 at 11:03:59AM -0500, J. Bruce Fields wrote:
> On Fri, Dec 15, 2006 at 12:39:51AM +1100, Christopher Smith wrote:
> > The first test is copying a single large file of about 700M from the 
> > (client) local disk to the NFS mount.
> > The second test is untarring a 700M tarfile from the (client) local disk 
> > to the NFS mount (~5000 files).
> > 
> > Test #1 consistently finishes in about 17 seconds, delivering about 40 
> > MiB/s.
> > Test #2 consistently finishes in about 85 seconds, delivering about 8 MiB/s
> 
> That's about 17ms per file?  Each of those 5000 file creations is a
> synchronous operation--the server doesn't respond until it's actually
> committed the operation to disk.  I don't really know what expected
> numbers are for your hardware, but given that for each file we have to
> wait for both the initial creation and the final flush of the data to
> disk on close, 17ms doesn't sound too far off.

I actually found the numbers that Christopher is seeing pretty good. If
I could get a 700MB write of 5000 files to an NFS mount in 85s I'd be
as happy as could be. I'm imagining that if this wasn't Gigabit this
would be a lot slower -- am I wrong?
-- 
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3376 0125

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs