Re: [PATCH RFC] nohz,blk-mq: do not create blk-mq workqueues on nohz dedicated CPUs

[Jens Axboe <axboe@kernel.dk>]

On 03/31/2015 05:17 PM, Marcelo Tosatti wrote:
> On Tue, Mar 31, 2015 at 10:27:26AM -0400, Rik van Riel wrote:
>> CPUs with nohz_full do not want disruption from timer interrupts,
>> or other random system things.  This includes block mq work.
>>
>> There is another issue with block mq vs. realtime tasks that run
>> 100% of the time, which is not uncommon on systems that have CPUs
>> dedicated to real time use with isolcpus= and nohz_full=
>>
>> Specifically, on systems like that, a block work item may never
>> get to run, which could lead to filesystems getting stuck forever.
>>
>> We can avoid both issues by not scheduling blk-mq workqueues on
>> cpus in nohz_full mode.
>>
>> Question for Jens: should we try to spread out the load for
>> currently offline and nohz CPUs across the remaining CPUs in
>> the system, to get the full benefit of blk-mq in these situations?
>>
>> If so, do you have any preference on how I should implement that?
>>
>> Cc: Frederic Weisbecker <fweisbec@redhat.com>
>> Cc: Ingo Molnar <mingo@kernel.org>
>> Cc: Jens Axboe <axboe@kernel.org>
>> Signed-off-by: Rik van Riel <riel@redhat.com>
>> ---
>>   block/blk-mq.c | 5 +++++
>>   1 file changed, 5 insertions(+)
>>
>> diff --git a/block/blk-mq.c b/block/blk-mq.c
>> index 4f4bea21052e..1004d6817fa4 100644
>> --- a/block/blk-mq.c
>> +++ b/block/blk-mq.c
>> @@ -21,6 +21,7 @@
>>   #include <linux/sched/sysctl.h>
>>   #include <linux/delay.h>
>>   #include <linux/crash_dump.h>
>> +#include <linux/tick.h>
>>
>>   #include <trace/events/block.h>
>>
>> @@ -1760,6 +1761,10 @@ static void blk_mq_init_cpu_queues(struct request_queue *q,
>>   		if (!cpu_online(i))
>>   			continue;
>>
>> +		/* Do not schedule work on nohz full dedicated CPUs. */
>> +		if (tick_nohz_full_cpu(i))
>> +			continue;
>> +
>>   		hctx = q->mq_ops->map_queue(q, i);
>>   		cpumask_set_cpu(i, hctx->cpumask);
>>   		hctx->nr_ctx++;
>
> Rik,
>
> I suppose any bound workqueue queued on isolated CPUs should be moved at
> queue time to other CPUs (sacrifficing performance).
>
> So that by doing "queue_work" on an isolated CPU would move that
> work somewhere else.

That wont work for blk-mq, we rely on the characteristics of bound 
workqueues. So it would have to be handled up front, like in the patch I 
sent out.

-- 
Jens Axboe