Re: [PATCH] sunrpc: use better NUMA affinities

[NeilBrown <neilb-l3A5Bk7waGM@public.gmane.org>]

On Sat, 30 Jul 2011 06:34:44 +1000 Greg Banks <gnb-97jfqw80gc6171pxa8y+qA@public.gmane.org> wrote:

> 
> 
> Sent from my iPhone
> 
> On 30/07/2011, at 2:53, "J. Bruce Fields" <bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org> wrote:
> 
> > On Fri, Jul 29, 2011 at 12:48:36PM -0400, bfields wrote:
> >> On Fri, Jul 29, 2011 at 11:30:05PM +1000, Greg Banks wrote:
> >>>
> >>>
> >>> Sent from my iPhone
> >>>
> >>> On 29/07/2011, at 22:11, Eric Dumazet <eric.dumazet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>  
> >>> wrote:
> >>>
> >>>> Le vendredi 29 juillet 2011 à 21:58 +1000, Greg Banks a écrit :
> >>>>
> >>>>>
> >>>>> Sure, and a whole lot of the callsites are ("..._%d", cpu),  
> >>>>> hence the
> >>>>> unfortune :(
> >>>>
> >>>> BTW, we could name nfsd threads differently :
> >>>>
> >>>> Currently, they all are named : "nfsd"
> >>>>
> >>>> If SVC_POOL_PERCPU is selected, we could name them :
> >>>> nfsd_c0 -> nfsd_cN
> >>>>
> >>>> If SVC_POOL_PERNODE is selected, we could name them :
> >>>> nfsd_n0  -> nfsd_nN
> >>>>
> >>>> That would help to check with "ps aux" which cpu/nodes are under
> >>>> stress.
> >>>>
> >>>>
> >>>
> >>> I like it!
> >>
> >> Yup, patch welcomed.--b.
> >
> > (Annoying fact: some initscripts stop nfsd using a rough equivalent of
> > "killall nfsd".  So the name of the threads is arguably ABI.  I think
> > those initscripts are nuts and deserve what they get, but that may be
> > because I'm forgetting the reason they do that.)
> >
> 
> We had this discussion in May-June 2008; it's because the nfsds were  
> once  many many years ago userspace threads.

Even when they became kernel threads, 'kill' was the only way to kill them -
at first.

> 
> The "killall nfsd" semantics in those scripts are awful and lead to  
> problems shutting down when there are lots of threads. It would  
> probably be an improvement to provide a better shutdown mechanism and  
> force distros to use it.

rpc.nfsd 0

will stop all nfsd threads.  Follow with

  exportfs -f

and you should be done.  I'm not 100% sure about the nfsv4 thread though -
would need to check.

And yes - I would love it if distros could standardise on start/stop scripts
so upstreams could maintain them sensibly.  This is my personal number 1
reason for liking systemd - it pushes for this standardisation.

> 
> Or, you could preserve the effective semantics by having a single  
> "nfsd" thread whose purpose is to notice that it's being signalled and  
> perform a clean shutdown (perhaps blocking the thread doing the kill()  
> call until the shutdown has completed).

That's kinda neat.  It would be an ugly wart to have to keep around, but
sometimes that the price we pay for "no regressions".

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html