NFS over TCP performance

NFS over TCP performance

[Felix Radensky]

Hi,

I'm testing NFS server performance in the following environment:

NFS server :
 
Compaq Proliant ML 350,
dual 1.33 GHz CPU,
1G of RAM,
132G RAID5 based on Compaq Smart Array 532 (cciss driver),
linux-2.4.19-pre10 SMP kernel with latest NFS_ALL and NFS TCP patches,
ext3 filesystem with internal 400M journal,
NFS version 3,
nfs-utils 0.3.3,
e100 NIC driver version 2.0.30,
16 nfs server threads,
rmem_default and rmem_max are set to 1048576
 
NFS client:

Dual 1G CPU board based on ServerWorks chipset,
1G of RAM,
linux-2.4.18,
NFS version 3,
mount-2.11b,
e100 NIC driver version 2.0.30

Client and server are connected to 10/100 Cisco switch,
which is not congested, and both are synchronized to
100 Base Tx Full Duplex.

Client mounts /users directory like this:

mount -o rsize=8192,wsize=8192,hard,intr,tcp server_ip:/users /users

My test, as explained in Linux NFS howto, consists of copying a 2G
file, from client to server, like this

time dd if=/dev/zero of=testfile bs=8k count=262144

It takes about 4.5 min to complete the write. The same test, but over UDP,
takes around 3 min. My question is: is NFS over TCP supposed to be much
slower than over UDP, and if no, how can I improve the performance.
I've tried other values for rsize/wsize, but 8k seems to give the best 
performance.
I've also tried mounting ext3 filesystem with data=journal, but then 
test runs
about 4 times longer.

TIA,

Felix.


_______________________________________________________________

Sponsored by:
ThinkGeek at http://www.ThinkGeek.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS over TCP performance

[Tom McNeal]

Hi -

There is a cost for using TCP, but it might not be as much 
system-wide as this test indicates.  To get a really broad
picture, I'd suggest using IOzone, which isn't hard to set
up, and gives you a lot of data over a broad spectrum of 
file access types.  The parameters you might want to use
are disucssed in the performance section of the howto, at
http://nfs.sourceforge.net/nfs-howto/performance.html

I think you are probably right about 8K being the best 
transfer size for your particular tests, but I'd have to 
go back and look at my IOzone runs to see what might be
best in genernal.  You can still get a writeup plus the
benchmark data that I generated late last year, at 
http://www.missioncriticallinux.com/orph/

It also looks like you are not using jumbo frames, which 
Trond has mentioned as a big performance boost for tcp in 
100 baseT full duplex environments.

Also, since you're using 0.3.3 utils, I assume you are 
using sync exports; otherwise there are costs at the server 
which are hidden from the benchmark data.

Tom

Felix Radensky wrote:
> 
> Hi,
> 
> I'm testing NFS server performance in the following environment:
> 
> NFS server :
> 
> Compaq Proliant ML 350,
> dual 1.33 GHz CPU,
> 1G of RAM,
> 132G RAID5 based on Compaq Smart Array 532 (cciss driver),
> linux-2.4.19-pre10 SMP kernel with latest NFS_ALL and NFS TCP patches,
> ext3 filesystem with internal 400M journal,
> NFS version 3,
> nfs-utils 0.3.3,
> e100 NIC driver version 2.0.30,
> 16 nfs server threads,
> rmem_default and rmem_max are set to 1048576
> 
> NFS client:
> 
> Dual 1G CPU board based on ServerWorks chipset,
> 1G of RAM,
> linux-2.4.18,
> NFS version 3,
> mount-2.11b,
> e100 NIC driver version 2.0.30
> 
> Client and server are connected to 10/100 Cisco switch,
> which is not congested, and both are synchronized to
> 100 Base Tx Full Duplex.
> 
> Client mounts /users directory like this:
> 
> mount -o rsize=8192,wsize=8192,hard,intr,tcp server_ip:/users /users
> 
> My test, as explained in Linux NFS howto, consists of copying a 2G
> file, from client to server, like this
> 
> time dd if=/dev/zero of=testfile bs=8k count=262144
> 
> It takes about 4.5 min to complete the write. The same test, but over UDP,
> takes around 3 min. My question is: is NFS over TCP supposed to be much
> slower than over UDP, and if no, how can I improve the performance.
> I've tried other values for rsize/wsize, but 8k seems to give the best
> performance.
> I've also tried mounting ext3 filesystem with data=journal, but then
> test runs
> about 4 times longer.
> 
> TIA,
> 
> Felix.
> 
> _______________________________________________________________
> 
> Sponsored by:
> ThinkGeek at http://www.ThinkGeek.com/
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs

--
------------------------------------------------------------
Tom McNeal       trmcneal@attbi.com     (650)906-0761 (cell) 
------------------------------------------------------------

_______________________________________________________________

Sponsored by:
ThinkGeek at http://www.ThinkGeek.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs