ReasmFails increases / NFS performance on Linux Cluster

ReasmFails increases / NFS performance on Linux Cluster

[Patrice Seyed]

We have 1 ibm x345 as a management node, and 1 ibm x35 as a storage =
node,
(both dual cpu 2.8 xeon and 2.5gb memory) which is fiber channeled to a =
Raid
5 array with ~900 GB usable. Then we have 10 IBM BladeCenters which =
contain
14 blades per chassis except the last one (has 8). So 134 nodes with =
dual
xeon at 2.8Ghz, mostly with 1GB ram and 20 with 2GB.

All nodes including x345s are set to 1000/full and cabled into a 3750
Cataylst. Bear in mind each BladeCenter chassis include a switch module,
which is also set to 1000/full.

In testing dd writes from /dev/zero to an nfs mount and back in a large
number of batch jobs, I'm seeing high load on the storage node and heavy
slowdowns for example for ssh login, df, or ls on the head node.=20
In my attempt to tune the storage node, I have tried 32k and now 8k
(rsize,wsize in autofs) with no improvement in the slowness. Other
parameters I am using are hard,intr,noatime,retrans=3D20,timeo=3D25. I =
am
currently running 64 nfsd daemons. Also I now have ipfrag_low_thresh and
ipfrag_high_thread both set to 1045876. When I had doubled the default
values for these settings a few weeks ago, it appeared to solve I/O =
errors
that were appearing in the logs for many of the nodes. Still now the
ReasmFails value in /proc/net/snmp steadily increases (for 8k or 32k) =
when I
submit a moderate number of jobs (20-40) that are heavy on I/O.

More info (on the storage node):
$ netstat -s | less
Ip:
    226750504 total packets received
    0 forwarded
    501200 incoming packets discarded
    164658619 incoming packets delivered
    161895650 requests sent out
    5006 fragments dropped after timeout
    80794698 reassemblies required
    13723023 packets reassembled ok
    126665 packet reassembles failed
    2858536 fragments received ok
    14006786 fragments created

Udp:
    159845714 packets received
    142 packets to unknown port received.
    501200 packet receive errors
    152909713 packets sent


I welcome any suggestions or recommendations.

Regards,
=20
Patrice Seyed
Linux System Administrator - SIG
RHCE, SCSA
Boston University School of Medicine




-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: ReasmFails increases / NFS performance on Linux Cluster

[Olaf Kirch]

On Sat, Aug 14, 2004 at 03:46:44AM -0400, Patrice Seyed wrote:
> I welcome any suggestions or recommendations.

Try using NFS over TCP, or, if all your NICs support it, use jumbograms
to avoid fragmentation.

Olaf
-- 
Olaf Kirch     |  The Hardware Gods hate me.
okir@suse.de   |
---------------+ 


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: ReasmFails increases / NFS performance on Linux Cluster

[Patrice Seyed]

Thanks Olaf, when I tried nfs over tcp on the clients the ReasmFails =
value
increased much significantly slower. At this time of testing the value =
is
not increasing at all.

Still, slowness on logging in (or running ls on that mount) continues, =
which
is not good since many users will be logging into the headnode during
production.

In addition, I have enabled jumbo packets on the cisco, and set all =
nodes on
the cluster network to mtu 9000. Also I have set read and write size to =
32k
as a document suggested with tcp and mtu at 9000 for better thoroughput.
Still I see the slowness when logging in.

 I am not sure if there is now any performance improvement as I am not
currently using any intelligent benchmark or mbit/sec analysis. Any
recommendation on doing these measurements and benchmarking is =
appreciated
greatly. Also I think I should consider the setup and the feasibility of
putting this type of load on this cluster (I/O intensive usually calls =
for
pvfs or multiple nfs servers, or both). E.g. one of my tests uses dd to
write 256mb files to nfs from 28 nodes at once (all going to the storage
node), and also running 2 jobs from each node (134 total node) that use =
dd
to write 4MB files and also run nbench. I.e. Should an nfs server be =
able to
handle this amount of load anyway?

Like I said I'm now running 64 nfs daemons, and the th line looks like =
this
now:

th 64 2380739 168.120 128.860 68.350 261.730 184.990 121.680 145.930 =
122.090
178.390 1749.090


Thanks Olaf for your suggestions on tcp and jumbo packets. Now I'm =
wondering
what I can use to determine if actually speed performance was improved, =
and
if the "bottleneck" I'm seeing now should be expected for my setup. Any
comments, further suggestions, appreciated.

Cheers,
-Patrice

-----Original Message-----
From: Olaf Kirch [mailto:okir@suse.de]=20
Sent: Saturday, August 14, 2004 5:24 AM
To: Patrice Seyed
Cc: nfs@lists.sourceforge.net
Subject: Re: [NFS] ReasmFails increases / NFS performance on Linux =
Cluster

On Sat, Aug 14, 2004 at 03:46:44AM -0400, Patrice Seyed wrote:
> I welcome any suggestions or recommendations.

Try using NFS over TCP, or, if all your NICs support it, use jumbograms
to avoid fragmentation.

Olaf
--=20
Olaf Kirch     |  The Hardware Gods hate me.
okir@suse.de   |
---------------+=20



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: ReasmFails increases / NFS performance on Linux Cluster

[Joshua Baker-LePain]

> 
> Message: 1
> From: "Patrice Seyed" <apseyed@bu.edu>
> To: <nfs@lists.sourceforge.net>
> Cc: "'Patrice Seyed'" <apseyed@bu.edu>
> Subject: RE: [NFS] ReasmFails increases / NFS performance on Linux Cluster
> Date: Sun, 15 Aug 2004 17:52:13 -0400
> 
> Like I said I'm now running 64 nfs daemons, and the th line looks like =
> this
> now:
> 
> th 64 2380739 168.120 128.860 68.350 261.730 184.990 121.680 145.930 =
> 122.090
> 178.390 1749.090

You don't have enough threads.  That 2nd number (after the number of 
threads) should be as close to zero as possible -- that's the number of 
times all threads have been busy when a request came in.  IIRC, you have 
plenty of memory on the server -- crank up the number of threads and you 
should see some difference.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: ReasmFails increases / NFS performance on Linux Cluster

[Patrice Seyed]

Joshua,

Hmmm...

I had read somewhere that when you have nfs daemons over 100 it can =
cause
throttled and hence slowdowns on the server. I did test with 128 daemons
today and at one point it appears things grinded to a complete halt ( I
couldn't get a login session through ssh at all). Still I had increased =
the
size of the dd writes from 4MB to 8MB, coming from each 134 node to the =
nfs
server simultaneously. With this change my nfsd looks like this:

th 128 1018239 4.510 8.650 6.110 7.290 2.280 4.040 6.360 6.460 20.590
1467.200=20

In the tldp howto it indicates that the last 10 numbers of the th line =
is
the number of seconds the thread usage was at that percentage of maximum
allowable, and if you have large numbers in the top three deciles, =
increase
# of nfsd.=20

I was unaware of the interpretation of the second number, thanks.

Patrice

-----Original Message-----
From: Joshua Baker-LePain [mailto:jlb17@duke.edu]=20
Sent: Monday, August 16, 2004 8:54 AM
To: nfs@lists.sourceforge.net; Patrice Seyed
Subject: Re: [NFS] ReasmFails increases / NFS performance on Linux =
Cluster

>=20
> Message: 1
> From: "Patrice Seyed" <apseyed@bu.edu>
> To: <nfs@lists.sourceforge.net>
> Cc: "'Patrice Seyed'" <apseyed@bu.edu>
> Subject: RE: [NFS] ReasmFails increases / NFS performance on Linux =
Cluster
> Date: Sun, 15 Aug 2004 17:52:13 -0400
>=20
> Like I said I'm now running 64 nfs daemons, and the th line looks like =
=3D
> this
> now:
>=20
> th 64 2380739 168.120 128.860 68.350 261.730 184.990 121.680 145.930 =
=3D
> 122.090
> 178.390 1749.090

You don't have enough threads.  That 2nd number (after the number of=20
threads) should be as close to zero as possible -- that's the number of=20
times all threads have been busy when a request came in.  IIRC, you have =

plenty of memory on the server -- crank up the number of threads and you =

should see some difference.

--=20
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: ReasmFails increases / NFS performance on Linux Cluster

[Olaf Kirch]

On Mon, Aug 16, 2004 at 05:05:09PM -0400, Patrice Seyed wrote:
> You don't have enough threads.  That 2nd number (after the number of 
> threads) should be as close to zero as possible -- that's the number of 
> times all threads have been busy when a request came in.  IIRC, you have 
> plenty of memory on the server -- crank up the number of threads and you 
> should see some difference.

Not necessarily. We're talking about nfsd threads hung in a COMMIT call -
these can take over a second, and will block all WRITE calls to the same
file for the entire duration. So a single client doing write-behind can
easily tie up 8 or more server threads.

Olaf
-- 
Olaf Kirch     |  The Hardware Gods hate me.
okir@suse.de   |
---------------+ 


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs