Re: dlm: Use cmwq for send and receive workqueues

[Steven Whitehouse <swhiteho@redhat.com>]

Hi,

On Fri, 2010-11-12 at 11:12 -0500, David Teigland wrote:
> On Fri, Nov 12, 2010 at 12:12:29PM +0000, Steven Whitehouse wrote:
> > 
> > So far as I can tell, there is no reason to use a single-threaded
> > send workqueue for dlm, since it may need to send to several sockets
> > concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid
> > any possible deadlocks, WQ_HIGHPRI since locking traffic is highly
> > latency sensitive (and to avoid a priority inversion wrt GFS2's
> > glock_workqueue) and WQ_FREEZABLE just in case someone needs to do
> > that (even though with current cluster infrastructure, it doesn't
> > make sense as the node will most likely land up ejected from the
> > cluster) in the future.
> 
> Thanks, I'll want to do some testing with this, but my test machines do
> not seem to create more than one dlm_recv workqueue thread (prior to this
> patch).  Have you tested in any cases where many threads end up being
> created?  I've noticed while debugging some many-cpu machines a huge
> number of dlm_recv threads, which is just excessive.  Does this patch
> address that?
> 
> 
Yes, one of the features of the cmwq is that you land up with only as
many threads as required. When threads block, new ones are created to
avoid stalling the workqueue. Workqueues marked with WQ_MEM_RECLAIM
create a single rescuer thread, otherwise the threads are shared with
all other users of cmwq,

Steve.

> > Signed-off-by: Steven Whitehouse <swhiteho@redhat.com>
> > Cc: Tejun Heo <tj@kernel.org>
> > 
> > diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
> > index 37a34c2..0893b30 100644
> > --- a/fs/dlm/lowcomms.c
> > +++ b/fs/dlm/lowcomms.c
> > @@ -1431,14 +1431,16 @@ static void work_stop(void)
> >  static int work_start(void)
> >  {
> >  	int error;
> > -	recv_workqueue = create_workqueue("dlm_recv");
> > +	recv_workqueue = alloc_workqueue("dlm_recv", WQ_MEM_RECLAIM |
> > +					 WQ_HIGHPRI | WQ_FREEZEABLE, 0);
> >  	error = IS_ERR(recv_workqueue);
> >  	if (error) {
> >  		log_print("can't start dlm_recv %d", error);
> >  		return error;
> >  	}
> >  
> > -	send_workqueue = create_singlethread_workqueue("dlm_send");
> > +	send_workqueue = alloc_workqueue("dlm_send", WQ_MEM_RECLAIM |
> > +					 WQ_HIGHPRI | WQ_FREEZEABLE, 0);
> >  	error = IS_ERR(send_workqueue);
> >  	if (error) {
> >  		log_print("can't start dlm_send %d", error);
> > 
> >