From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756539AbaFUAO1 (ORCPT ); Fri, 20 Jun 2014 20:14:27 -0400 Received: from e38.co.us.ibm.com ([32.97.110.159]:56438 "EHLO e38.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752253AbaFUAOZ (ORCPT ); Fri, 20 Jun 2014 20:14:25 -0400 Date: Fri, 20 Jun 2014 17:14:18 -0700 From: "Paul E. McKenney" To: josh@joshtriplett.org Cc: linux-kernel@vger.kernel.org, mingo@kernel.org, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, dvhart@linux.intel.com, fweisbec@gmail.com, oleg@redhat.com, sbw@mit.edu Subject: Re: [PATCH tip/core/rcu 0/5] Fix for cond_resched performance regression Message-ID: <20140621001418.GF4615@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <20140620183249.GA6325@linux.vnet.ibm.com> <20140620191236.GA10340@linux.vnet.ibm.com> <20140620212423.GA22886@cloud> <20140620221120.GD4615@linux.vnet.ibm.com> <20140620223951.GA23677@cloud> <20140620233033.GE4615@linux.vnet.ibm.com> <20140620235215.GA24026@cloud> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140620235215.GA24026@cloud> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 14062100-1344-0000-0000-00000255417A Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jun 20, 2014 at 04:52:15PM -0700, josh@joshtriplett.org wrote: > On Fri, Jun 20, 2014 at 04:30:33PM -0700, Paul E. McKenney wrote: > > On Fri, Jun 20, 2014 at 03:39:51PM -0700, josh@joshtriplett.org wrote: > > > On Fri, Jun 20, 2014 at 03:11:20PM -0700, Paul E. McKenney wrote: > > > > On Fri, Jun 20, 2014 at 02:24:23PM -0700, josh@joshtriplett.org wrote: > > > > > On Fri, Jun 20, 2014 at 12:12:36PM -0700, Paul E. McKenney wrote: > > > > > > o Make cond_resched() a no-op for PREEMPT=y. This might well turn > > > > > > out to be a good thing, but it doesn't help give RCU the quiescent > > > > > > states that it needs. > > > > > > > > > > What about doing this, together with letting the fqs logic poke > > > > > un-quiesced kernel code as needed? That way, rather than having > > > > > cond_resched do any work, you have the fqs logic recognize that a > > > > > particular CPU has gone too long without quiescing, without disturbing > > > > > that CPU at all if it hasn't gone too long. > > > > > > > > My next stop is to post the previous series, but with a couple of > > > > exports and one bug fix uncovered by testing thus far, but after > > > > another round of testing. Then I am going to take a close look at > > > > this one: > > > > > > > > o Push the checks further into cond_resched(), so that the > > > > fastpath does the same sequence of instructions that the original > > > > did. This might work well, but requires IPIs, which are not so > > > > good for latencies on the remote CPU. It nevertheless might be a > > > > decent long-term solution given that if your CPU is spending many > > > > jiffies looping in the kernel, you aren't getting good latencies > > > > anyway. It also has the benefit of allowing RCU to take advantage > > > > of the implicit quiescent states of all cond_resched() calls, > > > > and of eliminating the need for a separate cond_resched_rcu_qs() > > > > and for RCU_COND_RESCHED_QS. > > > > > > > > The one you call out is of course interesting as well. But there are > > > > a couple of questions: > > > > > > > > 1. Why wasn't cond_resched() a no-op in CONFIG_PREEMPT to start > > > > with? It just seems to obvious a thing to do for it to possibly > > > > be an oversight. (What, me paranoid?) > > > > > > > > 2. When RCU recognizes that a particular CPU has gone too long, > > > > exactly what are you suggesting that RCU do about it? When > > > > formulating your answer, please give due consideration to the > > > > implications of that CPU being a NO_HZ_FULL CPU. ;-) > > > > > > Send it an IPI that either causes it to flag a quiescent state > > > immediately if currently quiesced or causes it to quiesce at the next > > > opportunity if not. > > > > OK. But if we are in a !PREEMPT kernel, > > That's not the case I was suggesting. Fair enough, but we still need to support !PREEMPT kernels. > *If* the kernel is fully > preemptible, then it makes little sense to put any code in cond_resched, > when instead another thread can simply cause a preemption if it needs a > quiescent state. That has the advantage of not imposing any unnecessary > polling on code running in the kernel. OK. Exactly which thread are you suggesting should cause the preemption? > In a !PREEMPT kernel, it makes a bit more sense to have cond_resched as > a voluntary preemption point. But voluntary preemption points don't > make as much sense in a kernel prepared to preempt a thread anywhere. That does sound intuitive, but I am not yet prepared to believe that the scheduler guys missed this trick. There might well be some good reason for cond_resched() doing something, though I cannot think what it might be (something to do with preempt_enable_no_resched(), perhaps?). We should at least ask them, although if you want to do some testing before asking them, I of course have no objection to your doing so. Thanx, Paul