From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754557AbbDGNPG (ORCPT ); Tue, 7 Apr 2015 09:15:06 -0400 Received: from mx1.redhat.com ([209.132.183.28]:37357 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753727AbbDGNPA (ORCPT ); Tue, 7 Apr 2015 09:15:00 -0400 Date: Tue, 7 Apr 2015 09:14:54 -0400 From: Luiz Capitulino To: Marcelo Tosatti Cc: Frederic Weisbecker , Rik van Riel , axboe@kernel.org, fweisbec@redhat.com, mingo@kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH RFC] nohz,blk-mq: do not create blk-mq workqueues on nohz dedicated CPUs Message-ID: <20150407091454.5ac42e86@redhat.com> In-Reply-To: <20150331231836.GC4161@amt.cnet> References: <20150331102726.076a6860@annuminas.surriel.com> <20150331150236.GA28093@lerouge> <20150331231836.GC4161@amt.cnet> Organization: Red Hat MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 31 Mar 2015 20:18:36 -0300 Marcelo Tosatti wrote: > On Tue, Mar 31, 2015 at 05:02:38PM +0200, Frederic Weisbecker wrote: > > On Tue, Mar 31, 2015 at 10:27:26AM -0400, Rik van Riel wrote: > > > CPUs with nohz_full do not want disruption from timer interrupts, > > > or other random system things. This includes block mq work. > > > > > > There is another issue with block mq vs. realtime tasks that run > > > 100% of the time, which is not uncommon on systems that have CPUs > > > dedicated to real time use with isolcpus= and nohz_full= > > > > > > Specifically, on systems like that, a block work item may never > > > get to run, which could lead to filesystems getting stuck forever. > > > > > > We can avoid both issues by not scheduling blk-mq workqueues on > > > cpus in nohz_full mode. > > > > > > Question for Jens: should we try to spread out the load for > > > currently offline and nohz CPUs across the remaining CPUs in > > > the system, to get the full benefit of blk-mq in these situations? > > > > > > If so, do you have any preference on how I should implement that? > > > > > > Cc: Frederic Weisbecker > > > Cc: Ingo Molnar > > > Cc: Jens Axboe > > > Signed-off-by: Rik van Riel > > > --- > > > block/blk-mq.c | 5 +++++ > > > 1 file changed, 5 insertions(+) > > > > > > diff --git a/block/blk-mq.c b/block/blk-mq.c > > > index 4f4bea21052e..1004d6817fa4 100644 > > > --- a/block/blk-mq.c > > > +++ b/block/blk-mq.c > > > @@ -21,6 +21,7 @@ > > > #include > > > #include > > > #include > > > +#include > > > > > > #include > > > > > > @@ -1760,6 +1761,10 @@ static void blk_mq_init_cpu_queues(struct request_queue *q, > > > if (!cpu_online(i)) > > > continue; > > > > > > + /* Do not schedule work on nohz full dedicated CPUs. */ > > > + if (tick_nohz_full_cpu(i)) > > > + continue; > > > > I guess in this case, the queue for this CPU will be handled by another CPU? > > Is this an unbound workqueue? I guess it's not but if it is, we should wait for > > the workqueue affinity patchset. > > Where is the latest version of that patchset again ? I think you're referring to this series: https://marc.info/?l=linux-kernel&m=142613630606404&w=2 > > > > > Also, since we are doing a lot of kernel pre-setting behind nohz full, it would > > be nice to warn the user about each of them in dmesg. > > > > > + > > > hctx = q->mq_ops->map_queue(q, i); > > > cpumask_set_cpu(i, hctx->cpumask); > > > hctx->nr_ctx++; > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > Please read the FAQ at http://www.tux.org/lkml/ >