From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Steven Rostedt <rostedt@goodmis.org>,
LKML <linux-kernel@vger.kernel.org>,
Alessio Igor Bogani <abogani@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Avi Kivity <avi@redhat.com>, Chris Metcalf <cmetcalf@tilera.com>,
Christoph Lameter <cl@linux.com>,
Geoff Levand <geoff@infradead.org>,
Gilad Ben Yossef <gilad@benyossef.com>,
Hakan Akkan <hakanakkan@gmail.com>,
Ingo Molnar <mingo@kernel.org>,
Paul Gortmaker <paul.gortmaker@windriver.com>,
Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>,
Li Zhong <zhong@linux.vnet.ibm.com>
Subject: Re: [ANNOUNCE] 3.7-nohz1
Date: Sat, 29 Dec 2012 19:56:38 -0800 [thread overview]
Message-ID: <20121230035638.GM2542@linux.vnet.ibm.com> (raw)
In-Reply-To: <CAFTL4hy_k8rWg5EAFnh-owey7NJCQbK+ULrkNrSjgdynzEseTw@mail.gmail.com>
On Mon, Dec 24, 2012 at 12:43:25AM +0100, Frederic Weisbecker wrote:
> 2012/12/21 Steven Rostedt <rostedt@goodmis.org>:
> > On Thu, 2012-12-20 at 19:32 +0100, Frederic Weisbecker wrote:
> >> Let's imagine you have 4 CPUs. We keep the CPU 0 to offline RCU callbacks there and to
> >> handle the timekeeping. We set the rest as full dynticks. So you need the following kernel
> >> parameters:
> >>
> >> rcu_nocbs=1-3 full_nohz=1-3
> >>
> >> (Note rcu_nocbs value must always be the same as full_nohz).
> >
> > Why? You can't have: rcu_nocbs=1-4 full_nohz=1-3
>
> That should be allowed.
>
> > or: rcu_nocbs=1-3 full_nohz=1-4 ?
>
> But that not.
>
> You need to have: rcu_nocbs & full_nohz == full_nohz. This is because
> the tick is not there to maintain the local RCU callbacks anymore. So
> this must be offloaded to the rcu_nocb threads.
>
> I just have a doubt with rcu_nocb. Do we still need the tick to
> complete the grace period for local rcu callbacks? I need to discuss
> that with Paul.
The tick is only needed if rcu_needs_cpu() returns false. Of course,
this means that if you don't invoke rcu_needs_cpu() before returning to
adaptive-idle usermode execution, you are correct that a full_nohz CPU
would also have to be a rcu_nocbs CPU.
That said, I am getting close to having an rcu_needs_cpu() that only
returns false if there are callbacks immediately ready to invoke, at
least if RCU_FAST_NO_HZ=y.
Thanx, Paul
> > That needs to be fixed. Either with a warning, and/or to force the two
> > to be the same. That is, if they specify:
> >
> > rcu_nocbs=1-3 full_nohz=1-4
> >
> > Then set rcu_nocbs=1-4 with a warning about it. Or simply set
> > full_nohz=1-3.
>
> Yep, will do.
>
> Thanks!
>
> >
> > -- Steve
> >
> >>
> >> Now if you want proper isolation you need to:
> >>
> >> * Migrate your processes adequately
> >> * Migrate your irqs to CPU 0
> >> * Migrate the RCU nocb threads to CPU 0. Example with the above configuration:
> >>
> >> for p in $(ps -o pid= -C rcuo1,rcuo2,rcuo3)
> >> do
> >> taskset -cp 0 $p
> >> done
> >>
> >> Then run what you want on the full dynticks CPUs. For best results, run 1 task
> >> per CPU, mostly in userspace and mostly CPU bound (otherwise more IO = more kernel
> >> mode execution = more chances to get IPIs, tick restarted, workqueues, kthreads, etc...)
> >>
> >> This page contains a good reminder for those interested in CPU isolation: https://github.com/gby/linux/wiki
> >>
> >> But keep in mind that my tree is not yet ready for serious production.
> >>
> >
> >
>
next prev parent reply other threads:[~2012-12-30 3:57 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-12-20 18:32 [ANNOUNCE] 3.7-nohz1 Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 01/24] context_tracking: Add comments on interface and internals Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 02/24] cputime: Generic on-demand virtual cputime accounting Frederic Weisbecker
2012-12-21 5:11 ` Steven Rostedt
2012-12-26 8:19 ` Li Zhong
2012-12-29 13:15 ` Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 03/24] cputime: Allow dynamic switch between tick/virtual based " Frederic Weisbecker
2012-12-21 15:05 ` Steven Rostedt
2012-12-22 17:43 ` Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 04/24] cputime: Use accessors to read task cputime stats Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 05/24] cputime: Safely read cputime of full dynticks CPUs Frederic Weisbecker
2012-12-21 15:09 ` Steven Rostedt
2012-12-22 17:51 ` Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 06/24] nohz: Basic full dynticks interface Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 07/24] nohz: Assign timekeeping duty to a non-full-nohz CPU Frederic Weisbecker
2012-12-21 16:13 ` Steven Rostedt
2012-12-22 16:39 ` Frederic Weisbecker
2012-12-22 17:05 ` Steven Rostedt
2012-12-20 18:32 ` [PATCH 08/24] nohz: Trace timekeeping update Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 09/24] nohz: Wake up full dynticks CPUs when a timer gets enqueued Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 10/24] rcu: Restart the tick on non-responding full dynticks CPUs Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 11/24] sched: Comment on rq->clock correctness in ttwu_do_wakeup() in nohz Frederic Weisbecker
2012-12-20 18:32 ` [PATCH 12/24] sched: Update rq clock on nohz CPU before migrating tasks Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 13/24] sched: Update rq clock on nohz CPU before setting fair group shares Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 14/24] sched: Update rq clock on tickless CPUs before calling check_preempt_curr() Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 15/24] sched: Update rq clock earlier in unthrottle_cfs_rq Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 16/24] sched: Update clock of nohz busiest rq before balancing Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 17/24] sched: Update rq clock before idle balancing Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 18/24] sched: Update nohz rq clock before searching busiest group on load balancing Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 19/24] nohz: Move nohz load balancer selection into idle logic Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 20/24] nohz: Full dynticks mode Frederic Weisbecker
2012-12-26 6:12 ` Namhyung Kim
2012-12-26 7:02 ` Namhyung Kim
2012-12-29 13:21 ` Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 21/24] nohz: Only stop the tick on RCU nocb CPUs Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 22/24] nohz: Don't turn off the tick if rcu needs it Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 23/24] nohz: Don't stop the tick if posix cpu timers are running Frederic Weisbecker
2012-12-20 18:33 ` [PATCH 24/24] nohz: Add some tracing Frederic Weisbecker
2012-12-21 2:35 ` [ANNOUNCE] 3.7-nohz1 Steven Rostedt
2012-12-23 23:43 ` Frederic Weisbecker
2012-12-30 3:56 ` Paul E. McKenney [this message]
2013-01-04 23:42 ` Frederic Weisbecker
2013-01-07 13:06 ` Paul E. McKenney
2012-12-21 5:20 ` Hakan Akkan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121230035638.GM2542@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=abogani@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=avi@redhat.com \
--cc=cl@linux.com \
--cc=cmetcalf@tilera.com \
--cc=fweisbec@gmail.com \
--cc=geoff@infradead.org \
--cc=gilad@benyossef.com \
--cc=hakanakkan@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=paul.gortmaker@windriver.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=zhong@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).