Re: [PATCH RFC] nohz,blk-mq: do not create blk-mq workqueues on nohz dedicated CPUs

[Luiz Capitulino <lcapitulino@redhat.com>]

On Tue, 31 Mar 2015 20:18:36 -0300
Marcelo Tosatti <mtosatti@redhat.com> wrote:

> On Tue, Mar 31, 2015 at 05:02:38PM +0200, Frederic Weisbecker wrote:
> > On Tue, Mar 31, 2015 at 10:27:26AM -0400, Rik van Riel wrote:
> > > CPUs with nohz_full do not want disruption from timer interrupts,
> > > or other random system things.  This includes block mq work.
> > > 
> > > There is another issue with block mq vs. realtime tasks that run
> > > 100% of the time, which is not uncommon on systems that have CPUs
> > > dedicated to real time use with isolcpus= and nohz_full=
> > > 
> > > Specifically, on systems like that, a block work item may never
> > > get to run, which could lead to filesystems getting stuck forever.
> > > 
> > > We can avoid both issues by not scheduling blk-mq workqueues on
> > > cpus in nohz_full mode.
> > > 
> > > Question for Jens: should we try to spread out the load for
> > > currently offline and nohz CPUs across the remaining CPUs in
> > > the system, to get the full benefit of blk-mq in these situations?
> > > 
> > > If so, do you have any preference on how I should implement that?
> > > 
> > > Cc: Frederic Weisbecker <fweisbec@redhat.com>
> > > Cc: Ingo Molnar <mingo@kernel.org>
> > > Cc: Jens Axboe <axboe@kernel.org>
> > > Signed-off-by: Rik van Riel <riel@redhat.com>
> > > ---
> > >  block/blk-mq.c | 5 +++++
> > >  1 file changed, 5 insertions(+)
> > > 
> > > diff --git a/block/blk-mq.c b/block/blk-mq.c
> > > index 4f4bea21052e..1004d6817fa4 100644
> > > --- a/block/blk-mq.c
> > > +++ b/block/blk-mq.c
> > > @@ -21,6 +21,7 @@
> > >  #include <linux/sched/sysctl.h>
> > >  #include <linux/delay.h>
> > >  #include <linux/crash_dump.h>
> > > +#include <linux/tick.h>
> > >  
> > >  #include <trace/events/block.h>
> > >  
> > > @@ -1760,6 +1761,10 @@ static void blk_mq_init_cpu_queues(struct request_queue *q,
> > >  		if (!cpu_online(i))
> > >  			continue;
> > >  
> > > +		/* Do not schedule work on nohz full dedicated CPUs. */
> > > +		if (tick_nohz_full_cpu(i))
> > > +			continue;
> > 
> > I guess in this case, the queue for this CPU will be handled by another CPU?
> > Is this an unbound workqueue? I guess it's not but if it is, we should wait for
> > the workqueue affinity patchset.
> 
> Where is the latest version of that patchset again ?

I think you're referring to this series:

https://marc.info/?l=linux-kernel&m=142613630606404&w=2

> 
> > 
> > Also, since we are doing a lot of kernel pre-setting behind nohz full, it would
> > be nice to warn the user about each of them in dmesg.
> > 
> > > +
> > >  		hctx = q->mq_ops->map_queue(q, i);
> > >  		cpumask_set_cpu(i, hctx->cpumask);
> > >  		hctx->nr_ctx++;
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > Please read the FAQ at  http://www.tux.org/lkml/
>