Re: [PATCH RESEND] sched/nohz: Affine unpinned timers to housekeepers

[Frederic Weisbecker <fweisbec@gmail.com>]

On Tue, Aug 25, 2015 at 10:29:04AM +0200, Ingo Molnar wrote:
> 
> * Frederic Weisbecker <fweisbec@gmail.com> wrote:
> 
> > On Mon, Aug 24, 2015 at 08:44:12AM +0200, Ingo Molnar wrote:
> > > 2)
> > > 
> > > What happens if the boot CPU is offlined? (under CONFIG_BOOTPARAM_HOTPLUG_CPU0=y)
> > > 
> > > I don't see CPU hotplug callbacks fixing up the housekeeping_mask if the boot CPU 
> > > is offlined.
> > 
> > We have tick_nohz_cpu_down_callback() which makes sure that the timekeeper, which
> > is the boot CPU in nohz full, never gets offlined.
> 
> That solution really sucks - it essentially regresses a feature the user 
> explicitly asked for! I also see no way for the user to migrate the timekeeping 
> functionality over to another CPU without rebooting.
> 
> If this is the last timekeeping CPU then it should migrate the timekeeping 
> functionality to another CPU, and perhaps printk a warning if all other CPUs are 
> nohz-full and we have to mark one of them as the timekeeper.
> 
> Also, the nohz-full and timekeeper functionality should not be a boot parameter 
> only thing, but should be runtime configurable.

When I tried to allow moving the timekeeping duty over all housekeeping CPUs, Thomas got
angry because it broke the KISS current nohz full code. Indeed, there must be at least
one running all the time on behalf of nohz full CPUs that can run anytime. Thus balancing
the timekeeping duty over housekeepers is a bit more complicated than in normal
configurations.

Now surely we can do that using an IPI from CPU_DOWN_PREPARE to a housekeeper if
any remains or to a nohz full one. Then we must make sure the new timekeeper never
goes to idle.

But nohz_full is a corner usecase and I'm not sure it's worth the complexity. If a
nohz full user came and complained about CPU0 hotplog not working, I would definetly
retry it but I haven't heard about that yet. Besides, hotplug is very isolation-unfriendly
in general due to stop machine.