Re: [PATCH 1/2] nohz: Affine unpinned timers to housekeepers

[Frederic Weisbecker <fweisbec@gmail.com>]

On Fri, Sep 04, 2015 at 11:56:33AM -0400, Luiz Capitulino wrote:
> On Tue, 1 Sep 2015 22:47:24 +0200
> Frederic Weisbecker <fweisbec@gmail.com> wrote:
> 
> > On Tue, Sep 01, 2015 at 07:14:13PM +0000, Jiang, Yunhong wrote:
> > > > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > > > index 8b864ec..0902e4d 100644
> > > > --- a/kernel/sched/core.c
> > > > +++ b/kernel/sched/core.c
> > > > @@ -623,18 +623,21 @@ int get_nohz_timer_target(void)
> > > >  	int i, cpu = smp_processor_id();
> > > >  	struct sched_domain *sd;
> > > > 
> > > > -	if (!idle_cpu(cpu))
> > > > +	if (!idle_cpu(cpu) && is_housekeeping_cpu(cpu))
> > > >  		return cpu;
> > > > 
> > > >  	rcu_read_lock();
> > > >  	for_each_domain(cpu, sd) {
> > > >  		for_each_cpu(i, sched_domain_span(sd)) {
> > > > -			if (!idle_cpu(i)) {
> > > > +			if (!idle_cpu(i) && is_housekeeping_cpu(cpu)) {
> > > 
> > > Hi, Frederic, sorry for a naive question. Per my understanding, the tick_nohz_full_mask is added to cpu_isolated_map in
> > > sched_init_smp(), and the cpu_isolated_map is excluded from sched_domain in init_sched_domains(), so why check here?
> > 
> > Very good observation! But it's better to keep this check in the domain loop in
> > case things change in the future such as removing that cpu_isolated_map inclusion
> > or other suprises.
> 
> I have another observation.
> 
> As nohz_full cores are already excluded from the sched domains, this patch
> boils down to migrating timers which are queued by the nohz_full core
> itself. However, it only fully works for lowres timers. hrtimers may
> still fire on nohz_full cores, because hrtimers are only migrated if
> the core returned by get_nohz_timer_target() has a queued timer expiring
> before than the timer being migrated.

That's right. Probably some of them may never move to another CPU if they have
very small delays (smaller than those of hrtimer enqueued on other housekeepers).
And if they reach a housekeeper they can still move back to nohz full CPUs later
if they are modified from them.

The only safe solution is to always drive unbound hrtimers from housekeepers.