Performance Difference Between Linux NFS Server and Netapp

Performance Difference Between Linux NFS Server and Netapp

[Hugh Caley]

I've been doing a little limited benchmarking of NFS performance, and
doing a comparison between Linux NFS servers and a Netapp.  All tests
done over gigabit, clients are Linux (i386 and Opteron), OS is Fedora
Core 2,
kernel is 2.6.10-1.771_FC2smp, filesystem is Reiser.  I basically copy a
205
megabyte file to the server three times and average the results.  Sure,
limited,
but it seems to have real world relevance here.

The difference in performance is kind of startling.  I had originally
put the Linux-based servers (with a Nexsan Atabeast backend) in as a
replacement for an older EMC Celerra and a few older Suns.  On both the
Suns and Celerra I would see single-client performance topping out at
between 120 and 150 megabits per second.

The Linux setup (Supermicro Dual 2.6 Ghz Xeons with 4 gigabytes of RAM,
recently boosted to 8 gigabytes of RAM) regularly returns about 300
megabits per second.  Given the difference in price between the Linux
boxes and the Celerra, everyone's pretty happy ;)

However, recently I've been trying the same testing between Linux
clients and a Netapp, and can achieve between 500 and 600 megabits per
second!  Darned impressive.

I realize that the disk subsystem of the Netapp is going to stomp on my
Atabeast, but neither 300 nor 600 megabits is anywhere near the speed I
can get if I run the same test locally on the Linux server boxes, which
can regularly top 2000 megabits per second, so I really don't think that
the differences in disk architecture should make this much difference.

I guess my question is, are my Linux NFS servers seriously
underperforming?  Could I expect to achieve better?  I realize that my
"benchmark" is pretty limited, but the difference is pretty big.  Since
the same client
on all the Linux boxen gets these results, I'm guessing that the client end
has little to do with it.

How does Netapp do it?  Don't suppose they'd tell me ...

Hugh

--
Hugh Caley | Unix Systems Administrator | CIS
AFFYMETRIX INC. | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
Tel: 510-428-8537 | Hugh_Caley@affymetrix.com

-- 
"Brain-eating mutants are bad for business" - Battle Angel
Batmensch <hcaley@plasmabat.com>



-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP, 
AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Performance Difference Between Linux NFS Server and Netapp

[Roger Heflin]

Locally disk cache will cause the results to be junk,
locally the cache will be used, with NFS the cache
will not be used for writes.

With Suse 9 Enterprise and a disk subsystem that will
sustain rates of 125MiB/second I can get write rates of
95MiB/second and read rates of 115MiB/second, this is
with sustained over 12 minutes of writes.

2000mbit/second is a rate that no ide disk subsystem
will obtain, so double check your local performance, 
if it is fast single disk it will get 50mb/second, if
you are using a built-in ata controller, with all channels
having fast disks you won't be able exceed 90mb/second
no matter what disk you use or how many or how you stripe
them, as the build-in mb controllers are not the best, 
and this is on the  more expensive dual-cpu mbs, it may
be worse on the single cpu machines.

You will need to use a test larger than the ram size of 
the machine to get a realistic rate, at least 2x, and
probably 4x, and maybe both just to see how things fall
over.

                         Roger

> -----Original Message-----
> From: nfs-admin@lists.sourceforge.net 
> [mailto:nfs-admin@lists.sourceforge.net] On Behalf Of Hugh Caley
> Sent: Monday, July 11, 2005 5:56 PM
> To: nfs@lists.sourceforge.net
> Subject: [NFS] Performance Difference Between Linux NFS 
> Server and Netapp
> 
> I've been doing a little limited benchmarking of NFS 
> performance, and doing a comparison between Linux NFS servers 
> and a Netapp.  All tests done over gigabit, clients are Linux 
> (i386 and Opteron), OS is Fedora Core 2, kernel is 
> 2.6.10-1.771_FC2smp, filesystem is Reiser.  I basically copy a
> 205
> megabyte file to the server three times and average the 
> results.  Sure, limited, but it seems to have real world 
> relevance here.
> 
> The difference in performance is kind of startling.  I had 
> originally put the Linux-based servers (with a Nexsan 
> Atabeast backend) in as a replacement for an older EMC 
> Celerra and a few older Suns.  On both the Suns and Celerra I 
> would see single-client performance topping out at between 
> 120 and 150 megabits per second.
> 
> The Linux setup (Supermicro Dual 2.6 Ghz Xeons with 4 
> gigabytes of RAM, recently boosted to 8 gigabytes of RAM) 
> regularly returns about 300 megabits per second.  Given the 
> difference in price between the Linux boxes and the Celerra, 
> everyone's pretty happy ;)
> 
> However, recently I've been trying the same testing between 
> Linux clients and a Netapp, and can achieve between 500 and 
> 600 megabits per second!  Darned impressive.
> 
> I realize that the disk subsystem of the Netapp is going to 
> stomp on my Atabeast, but neither 300 nor 600 megabits is 
> anywhere near the speed I can get if I run the same test 
> locally on the Linux server boxes, which can regularly top 
> 2000 megabits per second, so I really don't think that the 
> differences in disk architecture should make this much difference.
> 
> I guess my question is, are my Linux NFS servers seriously 
> underperforming?  Could I expect to achieve better?  I 
> realize that my "benchmark" is pretty limited, but the 
> difference is pretty big.  Since the same client on all the 
> Linux boxen gets these results, I'm guessing that the client 
> end has little to do with it.
> 
> How does Netapp do it?  Don't suppose they'd tell me ...
> 
> Hugh
> 
> --
> Hugh Caley | Unix Systems Administrator | CIS AFFYMETRIX INC. 
> | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
> Tel: 510-428-8537 | Hugh_Caley@affymetrix.com
> 
> --
> "Brain-eating mutants are bad for business" - Battle Angel 
> Batmensch <hcaley@plasmabat.com>
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by the 'Do More With Dual!' 
> webinar happening
> July 14 at 8am PDT/11am EDT. We invite you to explore the 
> latest in dual
> core and dual graphics technology at this free one hour event 
> hosted by HP, 
> AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
> 



-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP, 
AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Performance Difference Between Linux NFS Server and Netapp

[Joshua Baker-LePain]

On Tue, 12 Jul 2005 at 11:24am, Roger Heflin wrote

> 2000mbit/second is a rate that no ide disk subsystem
> will obtain, so double check your local performance, 

I have a 3ware based server that begs to differ with you -- over 150MB/s 
writing, over 300MB/s reading.  That's on a software RAID0 stripe across 2 
12 port controllers, with each controller doing hardware RAID5.

> You will need to use a test larger than the ram size of 
> the machine to get a realistic rate, at least 2x, and
> probably 4x, and maybe both just to see how things fall
> over.

Absolutely.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University


-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP, 
AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance Difference Between Linux NFS Server and Netapp

[Hugh Caley]

The Atabeast is an ATA-raid array, but as far as the server knows it is 
a 2 Gb fibre-attached SCSI disk.  I get 800-900 Mb/sec sequential writes 
using ext3; with Reiser or JFS it is much faster.

Regardless, I don't think this is very relevant.   My only point was 
that the disk subsystem is capable of much better performance than I'm 
currently seeing over NFS.  300 Mb/sec seems slow to me when I have a 
Dual-2.6 Ghz Linux box running kernel 2.6.10 (Fedora core 2) attached to 
a 10 disk RAID5, and the box is doing nothing else but serving this 
single NFS server.

Hugh

Joshua Baker-LePain wrote:

>On Tue, 12 Jul 2005 at 11:24am, Roger Heflin wrote
>
>  
>
>>2000mbit/second is a rate that no ide disk subsystem
>>will obtain, so double check your local performance, 
>>    
>>
>
>I have a 3ware based server that begs to differ with you -- over 150MB/s 
>writing, over 300MB/s reading.  That's on a software RAID0 stripe across 2 
>12 port controllers, with each controller doing hardware RAID5.
>
>  
>
>>You will need to use a test larger than the ram size of 
>>the machine to get a realistic rate, at least 2x, and
>>probably 4x, and maybe both just to see how things fall
>>over.
>>    
>>
>
>Absolutely.
>
>  
>


-- 
"Take off your pants and get in the reptile tank." - Joe Rogan, Fear Factor 
Hugh Caley hcaley@plasmabat.com



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance Difference Between Linux NFS Server and Netapp

[Sten Spans]

On Mon, 11 Jul 2005, Hugh Caley wrote:

> How does Netapp do it?  Don't suppose they'd tell me ...

Quite simple really, you pay them loads of money, they
give part of it to engineers doing work on nfs tuning.

They generally have 3 performance advantages:

- battery backed cache with full os support,
   which should make it perform better than battery backed
   cache on normal raid cards. They use this cache for
   journaling. I suppose one could try something similar
   with pci-ram cards with battery and store the journal
   on these.

- special filesystem which allows writing data on any
   disk, this allows distributing the workload across the
   available disks, more disks more performance
   ( if the fibrechannel can handle it ).

- Highly tuned nfs inplementation with dedicated operating system.

The bruteforce linux approach is quite good for the money spent,
but don't expect the features/performance of a way more expensive
enterprise solution.

-- 
Sten Spans

"There is a crack in everything, that's how the light gets in."
Leonard Cohen - Anthem


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance Difference Between Linux NFS Server and Netapp

[Hugh Caley]

Sten Spans wrote:

> On Mon, 11 Jul 2005, Hugh Caley wrote:
>
>> How does Netapp do it?  Don't suppose they'd tell me ...
>
>
> Quite simple really, you pay them loads of money, they
> give part of it to engineers doing work on nfs tuning.
>
> They generally have 3 performance advantages:
>
> - battery backed cache with full os support,
>   ...
>
> - special filesystem which allows writing data on any
>   ...

I actually don't think filesystem type is relevant to my question.  I 
tested this out pretty thoroughly with all the journaling filesystems 
available from the Fedora Core 2 kernels (Ext3, Reiser, XFS, JFS), and 
although Ext3 had by far the worst local performance, it's performance 
over NFS with my simple sequential write test was exactly the same as 
the others.  I mean, I'm talking about a single sequential write to a 
10-disk RAID5 attached with 2 Gb fibre channel; whichever file system I 
use on the RAID should be a whole lot faster than over NFS, and indeed 
they all are, Ext3 about 3 times, the others much more.

So, unless there are hooks between the filesystem and the NFSd that are 
not obvious to a user (and the fact that changing filesystems doesn't 
seem to make much difference seems to rule that out) and the fact that 
the NFS client seems capable of much better performance when used 
against a Netapp, it makes me wonder what's going on with the server 
portion.

> - Highly tuned nfs inplementation with dedicated operating system.
>
> The bruteforce linux approach is quite good for the money spent,
> but don't expect the features/performance of a way more expensive
> enterprise solution.
>
A valid point, of course, but I don't think I'm actually expecting a 
single NFSd to act like an expensive Netapp.  I do think that wondering 
why the Netapp is twice as fast for a sequential write is a valid 
question, even if the OS and NFS server subsystem are free.  I was kind 
of hoping someone would just say "you're getting what you should expect 
to get" or "wow, that's slow, try this and this and this".

And I did mention that moving to the Linux/NFS/Atabeast solution was 
quite a bit better than what we had before ;)  I'm not really 
complaining, just wondering here.

Hugh

-- 
Whatsoever hath no fins nor scales in the waters, that shall be an abomination unto you. - Leviticus 11:9-12 
http://godhatesshrimp.com
Hugh Caley hcaley@plasmabat.com



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance Difference Between Linux NFS Server and Netapp

[Chris Penney]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

On 7/14/05, Hugh Caley <hcaley@plasmabat.com> wrote: 
> 
> A valid point, of course, but I don't think I'm actually expecting a
> single NFSd to act like an expensive Netapp. I do think that wondering
> why the Netapp is twice as fast for a sequential write is a valid
> question, even if the OS and NFS server subsystem are free. I was kind
> of hoping someone would just say "you're getting what you should expect
> to get" or "wow, that's slow, try this and this and this".

 You referenced that you were getting 300 megabits (or 37MB/s). I have 
several SLES 9 nfs servers (using self compiled 2.6.11.5
<http://2.6.11.5>kernel) running on IBM x345 hardware (dual cpu pentum
4, 2gb ram, dual
qlogic hbas) connected to a single LSI storage array that presents four luns 
(two from controller A and two from B). Each lun is 1TB and made from 
hardware raid 8+1. Luns are merged together using device mapper.
 It's not uncommon with my setup to get a sustained write speed of 75MB/s on 
one of our SLES 9 compute systems (AMD Opterons) when doing a sequential 
write of an 8GB file. With two systems writing at the same time I get 
aggregate bandwidth better than 75MB/s (can't recall what it is).
 I use tcp/nfs3 and for write testing I use 'iozone -c -e -s 8192m -i 0'. I 
use 128 nfsds, export with 'rw,sync,no_subtree_check,no_root_squash' and add 
the following to sysctl.conf:
 net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
 On nfs clients (Sun, Linux, IRIX) I use the mount options: 
nosuid,rw,bg,hard,intr,vers=3,proto=tcp,rsize=32768,wsize=32768. On AIX I 
use the same options, but also add the critical 'combehind' (without it 
writes of large files [ie. close to the size of physical mem] is just 
horrid). 
  Chris

[-- Attachment #2: Type: text/html, Size: 2494 bytes --]

Re: Performance Difference Between Linux NFS Server and Netapp

[Hugh Caley]

Thank you!  Just the sort of thing I'm looking for ...

Hugh

>From: Chris Penney <cpenney@gmail.com>
>Reply-To: penney@msu.edu
>To: nfs@lists.sourceforge.net
>Subject: Re: [NFS] Performance Difference Between Linux NFS Server and Netapp
>
>
>On 7/14/05, Hugh Caley <hcaley@plasmabat.com> wrote:=20
>  
>
>>>=20
>>> A valid point, of course, but I don't think I'm actually expecting a
>>> single NFSd to act like an expensive Netapp. I do think that wondering
>>> why the Netapp is twice as fast for a sequential write is a valid
>>> question, even if the OS and NFS server subsystem are free. I was kind
>>> of hoping someone would just say "you're getting what you should expect
>>> to get" or "wow, that's slow, try this and this and this".
>>    
>>
>
> You referenced that you were getting 300 megabits (or 37MB/s). I have=20
>several SLES 9 nfs servers (using self compiled 2.6.11.5
><http://2.6.11.5>kernel) running on IBM x345 hardware (dual cpu pentum
>4, 2gb ram, dual
>qlogic hbas) connected to a single LSI storage array that presents four lun=
>s=20 ...
>

-- 
Whatsoever hath no fins nor scales in the waters, that shall be an abomination unto you. - Leviticus 11:9-12 
http://godhatesshrimp.com
Hugh Caley hcaley@plasmabat.com



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs