How many nfsd's can I run?

How many nfsd's can I run?

[Hugh Caley]

Often when a cluster run (100+ nodes) is started my NFS server will get 
very slow; users with their home directories on the server will lock up 
for short periods, and the load average crawls higher.

The output of cat /proc/net/rpc/nfsd:

th 125 264502 142948.997 38666.037 22620.179 16494.577 10072.689 
5700.113 2623.322 1094.785 474.752 1900.536

This would seem to mean that my nfsd threads are runnning at 100% for 
much of the time, meaning I need more nfsd's, correct?  But is there a 
limit to the amount of nfsd's I should run?  I had had it as high as 150 
at one time, but it seemed to me this made the server less stable.  
There may have been other reasons for this, however.

Anyway I have since doubled the ram on the server (to 8 gig).  Machine 
is a dual Xeon 2.6 Ghz machine running Fedora Core 2, kernel version 
2.6.10-1.771_FC2smp, gigabit ethernet.  Can I run more nfsd's, or should 
I be looking for other problems?

Hugh

-- 
Hugh Caley | Unix Systems Administrator | CIS
AFFYMETRIX, INC. | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
Tel: 510-428-8537 | Hugh_Caley@affymetrix.com



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: How many nfsd's can I run?

[Neil Brown]

On Tuesday June 21, Hugh_Caley@affymetrix.com wrote:
> Often when a cluster run (100+ nodes) is started my NFS server will get 
> very slow; users with their home directories on the server will lock up 
> for short periods, and the load average crawls higher.
> 
> The output of cat /proc/net/rpc/nfsd:
> 
> th 125 264502 142948.997 38666.037 22620.179 16494.577 10072.689 
> 5700.113 2623.322 1094.785 474.752 1900.536
> 
> This would seem to mean that my nfsd threads are runnning at 100% for 
> much of the time, meaning I need more nfsd's, correct?  But is there a 
> limit to the amount of nfsd's I should run?  I had had it as high as 150 
> at one time, but it seemed to me this made the server less stable.  
> There may have been other reasons for this, however.
> 
> Anyway I have since doubled the ram on the server (to 8 gig).  Machine 
> is a dual Xeon 2.6 Ghz machine running Fedora Core 2, kernel version 
> 2.6.10-1.771_FC2smp, gigabit ethernet.  Can I run more nfsd's, or should 
> I be looking for other problems?

There should be very little cost in pushing the number of nfsd
substantially higher, but there might not be much gain.

So definitely run more nfsd - say 256 (you can just run "nfsd 256" and
it will increase the number of threads to that number).
But also look into how the filesystem and storage is performing.

What filesys and what storage are you using?

NeilBrown


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: How many nfsd's can I run?

[Roger Heflin]

My experience is that above 16 or so I have not seen a gain
in the actual performance delivered to the network.

There also appears (on RHEL3 2.4 kernel) to be some sort
of kernel resource that gets overused by having too
many nfsds and causes other issues, this would not a 
conclusive test, but appears to be a problem, we took
the nfsds back down from 256 to 64 because of this 
perceived problem, and because it did not change any
of the performance issues that we had.

                        Roger
				Atipa Technologies

> -----Original Message-----
> From: nfs-admin@lists.sourceforge.net 
> [mailto:nfs-admin@lists.sourceforge.net] On Behalf Of Hugh Caley
> Sent: Tuesday, June 21, 2005 9:02 PM
> To: nfs@lists.sourceforge.net
> Subject: [NFS] How many nfsd's can I run?
> 
> Often when a cluster run (100+ nodes) is started my NFS 
> server will get very slow; users with their home directories 
> on the server will lock up for short periods, and the load 
> average crawls higher.
> 
> The output of cat /proc/net/rpc/nfsd:
> 
> th 125 264502 142948.997 38666.037 22620.179 16494.577 10072.689
> 5700.113 2623.322 1094.785 474.752 1900.536
> 
> This would seem to mean that my nfsd threads are runnning at 
> 100% for much of the time, meaning I need more nfsd's, 
> correct?  But is there a limit to the amount of nfsd's I 
> should run?  I had had it as high as 150 at one time, but it 
> seemed to me this made the server less stable.  
> There may have been other reasons for this, however.
> 
> Anyway I have since doubled the ram on the server (to 8 gig). 
>  Machine is a dual Xeon 2.6 Ghz machine running Fedora Core 
> 2, kernel version 2.6.10-1.771_FC2smp, gigabit ethernet.  Can 
> I run more nfsd's, or should I be looking for other problems?
> 
> Hugh
> 
> --
> Hugh Caley | Unix Systems Administrator | CIS AFFYMETRIX, 
> INC. | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
> Tel: 510-428-8537 | Hugh_Caley@affymetrix.com
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> from IBM. Find simple to follow Roadmaps, straightforward articles,
> informative Webcasts and more! Get everything you need to get up to
> speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
> 



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: How many nfsd's can I run?

[Hugh Caley]

Thanks for the advice.  So, if I run "nfsd 256", it will increase the 
currently running 125 daemons to 256?

Filesystem is reiser, and the disk device is a Nexsan Atabeast. 

Hugh

Neil Brown wrote:

>On Tuesday June 21, Hugh_Caley@affymetrix.com wrote:
>  
>
>>Often when a cluster run (100+ nodes) is started my NFS server will get 
>>very slow; users with their home directories on the server will lock up 
>>for short periods, and the load average crawls higher.
>>
>>The output of cat /proc/net/rpc/nfsd:
>>
>>th 125 264502 142948.997 38666.037 22620.179 16494.577 10072.689 
>>5700.113 2623.322 1094.785 474.752 1900.536
>>
>>This would seem to mean that my nfsd threads are runnning at 100% for 
>>much of the time, meaning I need more nfsd's, correct?  But is there a 
>>limit to the amount of nfsd's I should run?  I had had it as high as 150 
>>at one time, but it seemed to me this made the server less stable.  
>>There may have been other reasons for this, however.
>>
>>Anyway I have since doubled the ram on the server (to 8 gig).  Machine 
>>is a dual Xeon 2.6 Ghz machine running Fedora Core 2, kernel version 
>>2.6.10-1.771_FC2smp, gigabit ethernet.  Can I run more nfsd's, or should 
>>I be looking for other problems?
>>    
>>
>
>There should be very little cost in pushing the number of nfsd
>substantially higher, but there might not be much gain.
>
>So definitely run more nfsd - say 256 (you can just run "nfsd 256" and
>it will increase the number of threads to that number).
>But also look into how the filesystem and storage is performing.
>
>What filesys and what storage are you using?
>
>NeilBrown
>  
>


-- 
Hugh Caley | Unix Systems Administrator | CIS
AFFYMETRIX, INC. | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
Tel: 510-428-8537 | Hugh_Caley@affymetrix.com



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: How many nfsd's can I run?

[Neil Brown]

On Wednesday June 22, Hugh_Caley@affymetrix.com wrote:
> Thanks for the advice.  So, if I run "nfsd 256", it will increase the 
> currently running 125 daemons to 256?

Yes.

> 
> Filesystem is reiser, and the disk device is a Nexsan Atabeast. 
> 

I don't know of any particular performance issues or tuning
possibilities with reiserfs, and I've never heard of an Atabeast..

So I don't think I can be any more helpful.

NeilBrown


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs