Re: [patch 3/3] knfsd: add file to export stats about nfsd pools

["J. Bruce Fields" <bfields@fieldses.org>]

On Thu, Feb 19, 2009 at 05:42:49PM +1100, Greg Banks wrote:
> J. Bruce Fields wrote:
> > On Tue, Jan 13, 2009 at 09:26:36PM +1100, Greg Banks wrote:
> >   
> >> Add /proc/fs/nfsd/pool_stats to export to userspace various
> >> statistics about the operation of rpc server thread pools.
> >>     
> >
> > Could you explainw hy these specific statistics (total packets,
> > sockets_queued, threads_woken, overloads_avoided, threads_timedout) are
> > the important ones to capture?  Could you give examples of what sort of
> > problems could be solved using them?
> >   
> Actually I originally added these stats to help debug the
> overload-avoiding patch.  Then I thought to use them to drive a
> userspace control loop for controlling the number of nfsds, which I
> never finished writing.
> > As you said, an important question for the sysadmin is "should I
> > configure more nfsds?"  How do they answer that?
> >   
> You can work that out, but it's not obvious, i.e. not human-friendly.  
> Firstly, you need to rate convert all the stats.
> 
> The total_packets stat tells you how many NFS packets are arriving on
> each thread pool.  This is your primary load metric, i.e. with more load
> you want more nfsd threads.
> 
> The sockets_queued stat tells you that calls are arriving which are not
> being immediately serviced by threads, i.e. you're either thread-limited
> or CPU-limited rather than network-limited and you might get better
> throughput if there were more nfsd threads.
> 
> Conversely the overloads_avoided stat tells you if there are more
> threads than can usefully be made runnable on the available CPUs, so
> that adding more nfsd threads is unlikely to be helpful.
> 
> The threads_timedout stat will give you a first-level approximation of
> whether there are threads that are completely idle, i.e. don't see any
> calls for the svc_recv() timeout (which I reduced to IIRC 10 sec as part
> of the original version of this patch).  This is a clue that you can now
> reduce the number of threads.

OK, thanks, that makes sense, but: it would be really fantastic if we
could update and/or replace some of the howto's and faq's that are out
there.  The above would be useful, as would be some of the material from
your nfs-tuning presentation.

Queued up this and the old-stats removal for 2.6.30.

--b.