Re: [PATCH 8/8] nfsd: default to a more sane number of threads

[Jeff Layton <jlayton@kernel.org>]

On Wed, 2024-03-20 at 10:10 -0400, Chuck Lever wrote:
> On Wed, Mar 20, 2024 at 09:11:31AM -0400, Jeff Layton wrote:
> > Add a small script that can estimate the number of threads for
> > nfsd. We default to 32 threads for every GB of guest memory.
> > 
> > Signed-off-by: Jeff Layton <jlayton@kernel.org>
> > ---
> >  kconfigs/Kconfig.nfsd      |  2 +-
> >  scripts/nfsd-thread-est.sh | 15 +++++++++++++++
> >  2 files changed, 16 insertions(+), 1 deletion(-)
> > 
> > diff --git a/kconfigs/Kconfig.nfsd b/kconfigs/Kconfig.nfsd
> > index dec98c1e964f..7c28ad98955a 100644
> > --- a/kconfigs/Kconfig.nfsd
> > +++ b/kconfigs/Kconfig.nfsd
> > @@ -69,7 +69,7 @@ config NFSD_EXPORT_OPTIONS
> >  
> >  config NFSD_THREADS
> >  	int "Number of nfsd threads to spawn"
> > -	default 8
> > +	default $(shell, scripts/nfsd-thread-est.sh)
> >  	help
> >  	  Number of nfsd threads to start up for testing.
> >  
> > diff --git a/scripts/nfsd-thread-est.sh b/scripts/nfsd-thread-est.sh
> > new file mode 100755
> > index 000000000000..dc5a1deb1215
> > --- /dev/null
> > +++ b/scripts/nfsd-thread-est.sh
> > @@ -0,0 +1,15 @@
> > +#!/bin/bash
> > +#
> > +# The default number of 8 nfsd threads is pitifully low!
> > +#
> > +#
> > +# Each nfsd thread consumes ~1MB of memory, long-term:
> > +#
> > +# stack, kthread overhead, svc_rqst, and the rq_pages array, plus
> > +# temporary working space.
> > +#
> > +# Allow 32 threads for each GB of guest memory:
> 
> I agree that 8 is a small number, but that multiplier seems
> excessive. On my 16GB test systems, that would create 512 threads.
> I set the nfsd thread count to 32 on my test NFS server, and I've
> never seen more than 10-15 threads for any test. (OK, maybe I should
> run tests that drive the test server harder).
> 

How are you measuring that? I'm open to a different formula. This one is
based on a total SWAG.

> Plus, no distribution uses that configuration. Maybe we should stick
> to what's a common deployment scenario?
> 

They don't use that configuration because rpc.nfsd still defaults to 8.
Admins have to _know_ to increase the thread count, which sort of sucks.
I'd like to make that more dynamic, but that's a separate project.
Still, maybe worthwhile to look at soon...

> Do you have data that shows creating more threads improves test run
> time?
> 

No, but I think the overhead from that number of threads is pretty
minimal and the whole purpose of the kdevops nfsd host is to serve NFS
requests. Also, this _is_ just a default.

BTW, do we collect stats on how often we can't find a thread to run? I
guess that's all the svc_defer stuff?

> 
> > +
> > +. "${TOPDIR}/.config"
> > +
> > +echo "$(( 32 * $CONFIG_LIBVIRT_MEM_MB / 1024 ))"
> 
> How would this work on cloud (non-libvirt) systems?
> 

Yeah, that is a problem. The way you specify guest sizes on cloud hosts
is totally different (and different for each cloud provider). For that,
we should probably just assume a 4GB guest. With this formula that's 128
threads which I guess is still excessive by your estimation.

-- 
Jeff Layton <jlayton@kernel.org>