* [Cluster-devel] dlm: Use cmwq for send and receive workqueues @ 2010-11-12 12:12 ` Steven Whitehouse 0 siblings, 0 replies; 8+ messages in thread From: Steven Whitehouse @ 2010-11-12 12:12 UTC (permalink / raw) To: cluster-devel.redhat.com So far as I can tell, there is no reason to use a single-threaded send workqueue for dlm, since it may need to send to several sockets concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid any possible deadlocks, WQ_HIGHPRI since locking traffic is highly latency sensitive (and to avoid a priority inversion wrt GFS2's glock_workqueue) and WQ_FREEZABLE just in case someone needs to do that (even though with current cluster infrastructure, it doesn't make sense as the node will most likely land up ejected from the cluster) in the future. Signed-off-by: Steven Whitehouse <swhiteho@redhat.com> Cc: Tejun Heo <tj@kernel.org> diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c index 37a34c2..0893b30 100644 --- a/fs/dlm/lowcomms.c +++ b/fs/dlm/lowcomms.c @@ -1431,14 +1431,16 @@ static void work_stop(void) static int work_start(void) { int error; - recv_workqueue = create_workqueue("dlm_recv"); + recv_workqueue = alloc_workqueue("dlm_recv", WQ_MEM_RECLAIM | + WQ_HIGHPRI | WQ_FREEZEABLE, 0); error = IS_ERR(recv_workqueue); if (error) { log_print("can't start dlm_recv %d", error); return error; } - send_workqueue = create_singlethread_workqueue("dlm_send"); + send_workqueue = alloc_workqueue("dlm_send", WQ_MEM_RECLAIM | + WQ_HIGHPRI | WQ_FREEZEABLE, 0); error = IS_ERR(send_workqueue); if (error) { log_print("can't start dlm_send %d", error); ^ permalink raw reply related [flat|nested] 8+ messages in thread
* dlm: Use cmwq for send and receive workqueues @ 2010-11-12 12:12 ` Steven Whitehouse 0 siblings, 0 replies; 8+ messages in thread From: Steven Whitehouse @ 2010-11-12 12:12 UTC (permalink / raw) To: cluster-devel, linux-kernel, David Teigland; +Cc: Tejun Heo So far as I can tell, there is no reason to use a single-threaded send workqueue for dlm, since it may need to send to several sockets concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid any possible deadlocks, WQ_HIGHPRI since locking traffic is highly latency sensitive (and to avoid a priority inversion wrt GFS2's glock_workqueue) and WQ_FREEZABLE just in case someone needs to do that (even though with current cluster infrastructure, it doesn't make sense as the node will most likely land up ejected from the cluster) in the future. Signed-off-by: Steven Whitehouse <swhiteho@redhat.com> Cc: Tejun Heo <tj@kernel.org> diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c index 37a34c2..0893b30 100644 --- a/fs/dlm/lowcomms.c +++ b/fs/dlm/lowcomms.c @@ -1431,14 +1431,16 @@ static void work_stop(void) static int work_start(void) { int error; - recv_workqueue = create_workqueue("dlm_recv"); + recv_workqueue = alloc_workqueue("dlm_recv", WQ_MEM_RECLAIM | + WQ_HIGHPRI | WQ_FREEZEABLE, 0); error = IS_ERR(recv_workqueue); if (error) { log_print("can't start dlm_recv %d", error); return error; } - send_workqueue = create_singlethread_workqueue("dlm_send"); + send_workqueue = alloc_workqueue("dlm_send", WQ_MEM_RECLAIM | + WQ_HIGHPRI | WQ_FREEZEABLE, 0); error = IS_ERR(send_workqueue); if (error) { log_print("can't start dlm_send %d", error); ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [Cluster-devel] dlm: Use cmwq for send and receive workqueues 2010-11-12 12:12 ` Steven Whitehouse @ 2010-11-12 16:12 ` David Teigland -1 siblings, 0 replies; 8+ messages in thread From: David Teigland @ 2010-11-12 16:12 UTC (permalink / raw) To: cluster-devel.redhat.com On Fri, Nov 12, 2010 at 12:12:29PM +0000, Steven Whitehouse wrote: > > So far as I can tell, there is no reason to use a single-threaded > send workqueue for dlm, since it may need to send to several sockets > concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid > any possible deadlocks, WQ_HIGHPRI since locking traffic is highly > latency sensitive (and to avoid a priority inversion wrt GFS2's > glock_workqueue) and WQ_FREEZABLE just in case someone needs to do > that (even though with current cluster infrastructure, it doesn't > make sense as the node will most likely land up ejected from the > cluster) in the future. Thanks, I'll want to do some testing with this, but my test machines do not seem to create more than one dlm_recv workqueue thread (prior to this patch). Have you tested in any cases where many threads end up being created? I've noticed while debugging some many-cpu machines a huge number of dlm_recv threads, which is just excessive. Does this patch address that? > Signed-off-by: Steven Whitehouse <swhiteho@redhat.com> > Cc: Tejun Heo <tj@kernel.org> > > diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c > index 37a34c2..0893b30 100644 > --- a/fs/dlm/lowcomms.c > +++ b/fs/dlm/lowcomms.c > @@ -1431,14 +1431,16 @@ static void work_stop(void) > static int work_start(void) > { > int error; > - recv_workqueue = create_workqueue("dlm_recv"); > + recv_workqueue = alloc_workqueue("dlm_recv", WQ_MEM_RECLAIM | > + WQ_HIGHPRI | WQ_FREEZEABLE, 0); > error = IS_ERR(recv_workqueue); > if (error) { > log_print("can't start dlm_recv %d", error); > return error; > } > > - send_workqueue = create_singlethread_workqueue("dlm_send"); > + send_workqueue = alloc_workqueue("dlm_send", WQ_MEM_RECLAIM | > + WQ_HIGHPRI | WQ_FREEZEABLE, 0); > error = IS_ERR(send_workqueue); > if (error) { > log_print("can't start dlm_send %d", error); > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: dlm: Use cmwq for send and receive workqueues @ 2010-11-12 16:12 ` David Teigland 0 siblings, 0 replies; 8+ messages in thread From: David Teigland @ 2010-11-12 16:12 UTC (permalink / raw) To: Steven Whitehouse; +Cc: cluster-devel, linux-kernel, Tejun Heo On Fri, Nov 12, 2010 at 12:12:29PM +0000, Steven Whitehouse wrote: > > So far as I can tell, there is no reason to use a single-threaded > send workqueue for dlm, since it may need to send to several sockets > concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid > any possible deadlocks, WQ_HIGHPRI since locking traffic is highly > latency sensitive (and to avoid a priority inversion wrt GFS2's > glock_workqueue) and WQ_FREEZABLE just in case someone needs to do > that (even though with current cluster infrastructure, it doesn't > make sense as the node will most likely land up ejected from the > cluster) in the future. Thanks, I'll want to do some testing with this, but my test machines do not seem to create more than one dlm_recv workqueue thread (prior to this patch). Have you tested in any cases where many threads end up being created? I've noticed while debugging some many-cpu machines a huge number of dlm_recv threads, which is just excessive. Does this patch address that? > Signed-off-by: Steven Whitehouse <swhiteho@redhat.com> > Cc: Tejun Heo <tj@kernel.org> > > diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c > index 37a34c2..0893b30 100644 > --- a/fs/dlm/lowcomms.c > +++ b/fs/dlm/lowcomms.c > @@ -1431,14 +1431,16 @@ static void work_stop(void) > static int work_start(void) > { > int error; > - recv_workqueue = create_workqueue("dlm_recv"); > + recv_workqueue = alloc_workqueue("dlm_recv", WQ_MEM_RECLAIM | > + WQ_HIGHPRI | WQ_FREEZEABLE, 0); > error = IS_ERR(recv_workqueue); > if (error) { > log_print("can't start dlm_recv %d", error); > return error; > } > > - send_workqueue = create_singlethread_workqueue("dlm_send"); > + send_workqueue = alloc_workqueue("dlm_send", WQ_MEM_RECLAIM | > + WQ_HIGHPRI | WQ_FREEZEABLE, 0); > error = IS_ERR(send_workqueue); > if (error) { > log_print("can't start dlm_send %d", error); > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* [Cluster-devel] dlm: Use cmwq for send and receive workqueues 2010-11-12 16:12 ` David Teigland @ 2010-11-12 16:20 ` Steven Whitehouse -1 siblings, 0 replies; 8+ messages in thread From: Steven Whitehouse @ 2010-11-12 16:20 UTC (permalink / raw) To: cluster-devel.redhat.com Hi, On Fri, 2010-11-12 at 11:12 -0500, David Teigland wrote: > On Fri, Nov 12, 2010 at 12:12:29PM +0000, Steven Whitehouse wrote: > > > > So far as I can tell, there is no reason to use a single-threaded > > send workqueue for dlm, since it may need to send to several sockets > > concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid > > any possible deadlocks, WQ_HIGHPRI since locking traffic is highly > > latency sensitive (and to avoid a priority inversion wrt GFS2's > > glock_workqueue) and WQ_FREEZABLE just in case someone needs to do > > that (even though with current cluster infrastructure, it doesn't > > make sense as the node will most likely land up ejected from the > > cluster) in the future. > > Thanks, I'll want to do some testing with this, but my test machines do > not seem to create more than one dlm_recv workqueue thread (prior to this > patch). Have you tested in any cases where many threads end up being > created? I've noticed while debugging some many-cpu machines a huge > number of dlm_recv threads, which is just excessive. Does this patch > address that? > > Yes, one of the features of the cmwq is that you land up with only as many threads as required. When threads block, new ones are created to avoid stalling the workqueue. Workqueues marked with WQ_MEM_RECLAIM create a single rescuer thread, otherwise the threads are shared with all other users of cmwq, Steve. > > Signed-off-by: Steven Whitehouse <swhiteho@redhat.com> > > Cc: Tejun Heo <tj@kernel.org> > > > > diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c > > index 37a34c2..0893b30 100644 > > --- a/fs/dlm/lowcomms.c > > +++ b/fs/dlm/lowcomms.c > > @@ -1431,14 +1431,16 @@ static void work_stop(void) > > static int work_start(void) > > { > > int error; > > - recv_workqueue = create_workqueue("dlm_recv"); > > + recv_workqueue = alloc_workqueue("dlm_recv", WQ_MEM_RECLAIM | > > + WQ_HIGHPRI | WQ_FREEZEABLE, 0); > > error = IS_ERR(recv_workqueue); > > if (error) { > > log_print("can't start dlm_recv %d", error); > > return error; > > } > > > > - send_workqueue = create_singlethread_workqueue("dlm_send"); > > + send_workqueue = alloc_workqueue("dlm_send", WQ_MEM_RECLAIM | > > + WQ_HIGHPRI | WQ_FREEZEABLE, 0); > > error = IS_ERR(send_workqueue); > > if (error) { > > log_print("can't start dlm_send %d", error); > > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: dlm: Use cmwq for send and receive workqueues @ 2010-11-12 16:20 ` Steven Whitehouse 0 siblings, 0 replies; 8+ messages in thread From: Steven Whitehouse @ 2010-11-12 16:20 UTC (permalink / raw) To: David Teigland; +Cc: cluster-devel, linux-kernel, Tejun Heo Hi, On Fri, 2010-11-12 at 11:12 -0500, David Teigland wrote: > On Fri, Nov 12, 2010 at 12:12:29PM +0000, Steven Whitehouse wrote: > > > > So far as I can tell, there is no reason to use a single-threaded > > send workqueue for dlm, since it may need to send to several sockets > > concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid > > any possible deadlocks, WQ_HIGHPRI since locking traffic is highly > > latency sensitive (and to avoid a priority inversion wrt GFS2's > > glock_workqueue) and WQ_FREEZABLE just in case someone needs to do > > that (even though with current cluster infrastructure, it doesn't > > make sense as the node will most likely land up ejected from the > > cluster) in the future. > > Thanks, I'll want to do some testing with this, but my test machines do > not seem to create more than one dlm_recv workqueue thread (prior to this > patch). Have you tested in any cases where many threads end up being > created? I've noticed while debugging some many-cpu machines a huge > number of dlm_recv threads, which is just excessive. Does this patch > address that? > > Yes, one of the features of the cmwq is that you land up with only as many threads as required. When threads block, new ones are created to avoid stalling the workqueue. Workqueues marked with WQ_MEM_RECLAIM create a single rescuer thread, otherwise the threads are shared with all other users of cmwq, Steve. > > Signed-off-by: Steven Whitehouse <swhiteho@redhat.com> > > Cc: Tejun Heo <tj@kernel.org> > > > > diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c > > index 37a34c2..0893b30 100644 > > --- a/fs/dlm/lowcomms.c > > +++ b/fs/dlm/lowcomms.c > > @@ -1431,14 +1431,16 @@ static void work_stop(void) > > static int work_start(void) > > { > > int error; > > - recv_workqueue = create_workqueue("dlm_recv"); > > + recv_workqueue = alloc_workqueue("dlm_recv", WQ_MEM_RECLAIM | > > + WQ_HIGHPRI | WQ_FREEZEABLE, 0); > > error = IS_ERR(recv_workqueue); > > if (error) { > > log_print("can't start dlm_recv %d", error); > > return error; > > } > > > > - send_workqueue = create_singlethread_workqueue("dlm_send"); > > + send_workqueue = alloc_workqueue("dlm_send", WQ_MEM_RECLAIM | > > + WQ_HIGHPRI | WQ_FREEZEABLE, 0); > > error = IS_ERR(send_workqueue); > > if (error) { > > log_print("can't start dlm_send %d", error); > > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* [Cluster-devel] dlm: Use cmwq for send and receive workqueues 2010-11-12 16:20 ` Steven Whitehouse @ 2010-11-12 20:03 ` David Teigland -1 siblings, 0 replies; 8+ messages in thread From: David Teigland @ 2010-11-12 20:03 UTC (permalink / raw) To: cluster-devel.redhat.com On Fri, Nov 12, 2010 at 04:20:35PM +0000, Steven Whitehouse wrote: > Hi, > > On Fri, 2010-11-12 at 11:12 -0500, David Teigland wrote: > > On Fri, Nov 12, 2010 at 12:12:29PM +0000, Steven Whitehouse wrote: > > > > > > So far as I can tell, there is no reason to use a single-threaded > > > send workqueue for dlm, since it may need to send to several sockets > > > concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid > > > any possible deadlocks, WQ_HIGHPRI since locking traffic is highly > > > latency sensitive (and to avoid a priority inversion wrt GFS2's > > > glock_workqueue) and WQ_FREEZABLE just in case someone needs to do > > > that (even though with current cluster infrastructure, it doesn't > > > make sense as the node will most likely land up ejected from the > > > cluster) in the future. > > > > Thanks, I'll want to do some testing with this, but my test machines do > > not seem to create more than one dlm_recv workqueue thread (prior to this > > patch). Have you tested in any cases where many threads end up being > > created? I've noticed while debugging some many-cpu machines a huge > > number of dlm_recv threads, which is just excessive. Does this patch > > address that? > > > > > Yes, one of the features of the cmwq is that you land up with only as > many threads as required. When threads block, new ones are created to > avoid stalling the workqueue. Workqueues marked with WQ_MEM_RECLAIM > create a single rescuer thread, otherwise the threads are shared with > all other users of cmwq, Thanks, pushed to git://git.kernel.org/pub/scm/linux/kernel/git/teigland/dlm.git#next ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: dlm: Use cmwq for send and receive workqueues @ 2010-11-12 20:03 ` David Teigland 0 siblings, 0 replies; 8+ messages in thread From: David Teigland @ 2010-11-12 20:03 UTC (permalink / raw) To: Steven Whitehouse; +Cc: cluster-devel, linux-kernel, Tejun Heo On Fri, Nov 12, 2010 at 04:20:35PM +0000, Steven Whitehouse wrote: > Hi, > > On Fri, 2010-11-12 at 11:12 -0500, David Teigland wrote: > > On Fri, Nov 12, 2010 at 12:12:29PM +0000, Steven Whitehouse wrote: > > > > > > So far as I can tell, there is no reason to use a single-threaded > > > send workqueue for dlm, since it may need to send to several sockets > > > concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid > > > any possible deadlocks, WQ_HIGHPRI since locking traffic is highly > > > latency sensitive (and to avoid a priority inversion wrt GFS2's > > > glock_workqueue) and WQ_FREEZABLE just in case someone needs to do > > > that (even though with current cluster infrastructure, it doesn't > > > make sense as the node will most likely land up ejected from the > > > cluster) in the future. > > > > Thanks, I'll want to do some testing with this, but my test machines do > > not seem to create more than one dlm_recv workqueue thread (prior to this > > patch). Have you tested in any cases where many threads end up being > > created? I've noticed while debugging some many-cpu machines a huge > > number of dlm_recv threads, which is just excessive. Does this patch > > address that? > > > > > Yes, one of the features of the cmwq is that you land up with only as > many threads as required. When threads block, new ones are created to > avoid stalling the workqueue. Workqueues marked with WQ_MEM_RECLAIM > create a single rescuer thread, otherwise the threads are shared with > all other users of cmwq, Thanks, pushed to git://git.kernel.org/pub/scm/linux/kernel/git/teigland/dlm.git#next ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2010-11-12 20:04 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-11-12 12:12 [Cluster-devel] dlm: Use cmwq for send and receive workqueues Steven Whitehouse 2010-11-12 12:12 ` Steven Whitehouse 2010-11-12 16:12 ` [Cluster-devel] " David Teigland 2010-11-12 16:12 ` David Teigland 2010-11-12 16:20 ` [Cluster-devel] " Steven Whitehouse 2010-11-12 16:20 ` Steven Whitehouse 2010-11-12 20:03 ` [Cluster-devel] " David Teigland 2010-11-12 20:03 ` David Teigland
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.