Will CPU 0 be forever prohibited from NO_HZ_FULL status?

Will CPU 0 be forever prohibited from NO_HZ_FULL status?

[Paul E. McKenney]

Hello, Frederic,

Just realized that I could further decrease RT latency of one of my "shut
up RCU on NO_HZ_FULL CPUs" patches if I relied on CPU 0 always having
a scheduling-clock tick unless the entire system is idle.  The trick
is that I could then rely on CPU 0 to detect RCU CPU stall warnings,
and remove the checking from the other CPUs.

Thoughts?

							Thanx, Paul

Re: Will CPU 0 be forever prohibited from NO_HZ_FULL status?

[Frederic Weisbecker]

On Wed, Dec 04, 2013 at 11:39:57AM -0800, Paul E. McKenney wrote:
> Hello, Frederic,
> 
> Just realized that I could further decrease RT latency of one of my "shut
> up RCU on NO_HZ_FULL CPUs" patches if I relied on CPU 0 always having
> a scheduling-clock tick unless the entire system is idle.  The trick
> is that I could then rely on CPU 0 to detect RCU CPU stall warnings,
> and remove the checking from the other CPUs.
> 
> Thoughts?

You're right on time as I'm currently working on that :)
So the plan is to allow timekeeping to be handled by a set of CPUs (cpu_housekeeping_mask
which I guess should be ~nohz_full_mask & cpu_online_mask). I think it will be better
for powersaving. I guess you could balance the RCU stall checks in this
set of housekeeping CPUs?

It should be very easy to make the rcu sysidle stuff to support that housekeeping set,
I just looked into it and all we need to do is to turn the several "cpu == tick_do_timer"
checks into something like is_housekeeping_cpu(cpu). And may be a few easy details, like which
CPU from the housekeeping set should get the kick IPI, well the first one available should be a good start,
of course I expect some issues with cpu hotplug.
But other than that, RCU sysidle detection is mostly ready to support tracking only a given subset
of CPUs instead of all of them. That's in fact what it already does currently by excluding the
fixed boot timekeeping CPU.

So I'm working on that and should have some patches ready soon.

In fact I just realized that all the sysidle detection infrastructure is there and working
but we forgot to plug it in the tick engine, and thus we are still running
with periodic CPU 0 even with CONFIG_NO_HZ_FULL_SYSIDLE=y. Anyway I have a few changes
ready to enable that, lets hope testing will be ok :)

> 
> 							Thanx, Paul
> 

Re: Will CPU 0 be forever prohibited from NO_HZ_FULL status?

[Paul E. McKenney]

On Thu, Dec 05, 2013 at 02:20:55AM +0100, Frederic Weisbecker wrote:
> On Wed, Dec 04, 2013 at 11:39:57AM -0800, Paul E. McKenney wrote:
> > Hello, Frederic,
> > 
> > Just realized that I could further decrease RT latency of one of my "shut
> > up RCU on NO_HZ_FULL CPUs" patches if I relied on CPU 0 always having
> > a scheduling-clock tick unless the entire system is idle.  The trick
> > is that I could then rely on CPU 0 to detect RCU CPU stall warnings,
> > and remove the checking from the other CPUs.
> > 
> > Thoughts?
> 
> You're right on time as I'm currently working on that :)
> So the plan is to allow timekeeping to be handled by a set of CPUs (cpu_housekeeping_mask
> which I guess should be ~nohz_full_mask & cpu_online_mask). I think it will be better
> for powersaving. I guess you could balance the RCU stall checks in this
> set of housekeeping CPUs?
> 
> It should be very easy to make the rcu sysidle stuff to support that housekeeping set,
> I just looked into it and all we need to do is to turn the several "cpu == tick_do_timer"
> checks into something like is_housekeeping_cpu(cpu). And may be a few easy details, like which
> CPU from the housekeeping set should get the kick IPI, well the first one available should be a good start,
> of course I expect some issues with cpu hotplug.
> But other than that, RCU sysidle detection is mostly ready to support tracking only a given subset
> of CPUs instead of all of them. That's in fact what it already does currently by excluding the
> fixed boot timekeeping CPU.
> 
> So I'm working on that and should have some patches ready soon.

Thank you for the info!  Nice to know that RCU will continue to be able
to rely on there being at least one housekeeping CPU.  ;-)

At that point, tick_nohz_full_cpu() would still be a good way for RCU
to distinguish housekeeping CPUs from working CPUs, correct?

> In fact I just realized that all the sysidle detection infrastructure is there and working
> but we forgot to plug it in the tick engine, and thus we are still running
> with periodic CPU 0 even with CONFIG_NO_HZ_FULL_SYSIDLE=y. Anyway I have a few changes
> ready to enable that, lets hope testing will be ok :)

Indeed!  ;-)

The CONFIG_NO_HZ_FULL_SYSIDLE=y might complicate things a bit.  But I
guess the problem would be a corner case -- the system entered sysidle
mode with a grace period pending, which should eventually wake up the
corresponding grace-period kthread, which might be prevented from ever
running due to high load or something.  If that problem arises, I will
fix it.

So there!  ;-)

							Thanx, Paul

Re: Will CPU 0 be forever prohibited from NO_HZ_FULL status?

[Frederic Weisbecker]

On Wed, Dec 04, 2013 at 06:50:37PM -0800, Paul E. McKenney wrote:
> On Thu, Dec 05, 2013 at 02:20:55AM +0100, Frederic Weisbecker wrote:
> > On Wed, Dec 04, 2013 at 11:39:57AM -0800, Paul E. McKenney wrote:
> > > Hello, Frederic,
> > > 
> > > Just realized that I could further decrease RT latency of one of my "shut
> > > up RCU on NO_HZ_FULL CPUs" patches if I relied on CPU 0 always having
> > > a scheduling-clock tick unless the entire system is idle.  The trick
> > > is that I could then rely on CPU 0 to detect RCU CPU stall warnings,
> > > and remove the checking from the other CPUs.
> > > 
> > > Thoughts?
> > 
> > You're right on time as I'm currently working on that :)
> > So the plan is to allow timekeeping to be handled by a set of CPUs (cpu_housekeeping_mask
> > which I guess should be ~nohz_full_mask & cpu_online_mask). I think it will be better
> > for powersaving. I guess you could balance the RCU stall checks in this
> > set of housekeeping CPUs?
> > 
> > It should be very easy to make the rcu sysidle stuff to support that housekeeping set,
> > I just looked into it and all we need to do is to turn the several "cpu == tick_do_timer"
> > checks into something like is_housekeeping_cpu(cpu). And may be a few easy details, like which
> > CPU from the housekeeping set should get the kick IPI, well the first one available should be a good start,
> > of course I expect some issues with cpu hotplug.
> > But other than that, RCU sysidle detection is mostly ready to support tracking only a given subset
> > of CPUs instead of all of them. That's in fact what it already does currently by excluding the
> > fixed boot timekeeping CPU.
> > 
> > So I'm working on that and should have some patches ready soon.
> 
> Thank you for the info!  Nice to know that RCU will continue to be able
> to rely on there being at least one housekeeping CPU.  ;-)
> 
> At that point, tick_nohz_full_cpu() would still be a good way for RCU
> to distinguish housekeeping CPUs from working CPUs, correct?

Correct!

> 
> > In fact I just realized that all the sysidle detection infrastructure is there and working
> > but we forgot to plug it in the tick engine, and thus we are still running
> > with periodic CPU 0 even with CONFIG_NO_HZ_FULL_SYSIDLE=y. Anyway I have a few changes
> > ready to enable that, lets hope testing will be ok :)
> 
> Indeed!  ;-)
> 
> The CONFIG_NO_HZ_FULL_SYSIDLE=y might complicate things a bit.  But I
> guess the problem would be a corner case -- the system entered sysidle
> mode with a grace period pending, which should eventually wake up the
> corresponding grace-period kthread, which might be prevented from ever
> running due to high load or something.  If that problem arises, I will
> fix it.

I see. Well we'll find out.
In the meantime I successfully plugged sysidle detection with full dynticks and it
surprisingly works like a charm. Which makes me think there must be some bug in my patches that make things
working by accident :)

I'll post soon.

Thanks.

Re: Will CPU 0 be forever prohibited from NO_HZ_FULL status?

[Paul E. McKenney]

On Tue, Dec 10, 2013 at 03:37:03PM +0100, Frederic Weisbecker wrote:
> On Wed, Dec 04, 2013 at 06:50:37PM -0800, Paul E. McKenney wrote:
> > On Thu, Dec 05, 2013 at 02:20:55AM +0100, Frederic Weisbecker wrote:
> > > On Wed, Dec 04, 2013 at 11:39:57AM -0800, Paul E. McKenney wrote:
> > > > Hello, Frederic,
> > > > 
> > > > Just realized that I could further decrease RT latency of one of my "shut
> > > > up RCU on NO_HZ_FULL CPUs" patches if I relied on CPU 0 always having
> > > > a scheduling-clock tick unless the entire system is idle.  The trick
> > > > is that I could then rely on CPU 0 to detect RCU CPU stall warnings,
> > > > and remove the checking from the other CPUs.
> > > > 
> > > > Thoughts?
> > > 
> > > You're right on time as I'm currently working on that :)
> > > So the plan is to allow timekeeping to be handled by a set of CPUs (cpu_housekeeping_mask
> > > which I guess should be ~nohz_full_mask & cpu_online_mask). I think it will be better
> > > for powersaving. I guess you could balance the RCU stall checks in this
> > > set of housekeeping CPUs?
> > > 
> > > It should be very easy to make the rcu sysidle stuff to support that housekeeping set,
> > > I just looked into it and all we need to do is to turn the several "cpu == tick_do_timer"
> > > checks into something like is_housekeeping_cpu(cpu). And may be a few easy details, like which
> > > CPU from the housekeeping set should get the kick IPI, well the first one available should be a good start,
> > > of course I expect some issues with cpu hotplug.
> > > But other than that, RCU sysidle detection is mostly ready to support tracking only a given subset
> > > of CPUs instead of all of them. That's in fact what it already does currently by excluding the
> > > fixed boot timekeeping CPU.
> > > 
> > > So I'm working on that and should have some patches ready soon.
> > 
> > Thank you for the info!  Nice to know that RCU will continue to be able
> > to rely on there being at least one housekeeping CPU.  ;-)
> > 
> > At that point, tick_nohz_full_cpu() would still be a good way for RCU
> > to distinguish housekeeping CPUs from working CPUs, correct?
> 
> Correct!

Cool!  Maybe I should start future-proofing RCU in that manner.

> > > In fact I just realized that all the sysidle detection infrastructure is there and working
> > > but we forgot to plug it in the tick engine, and thus we are still running
> > > with periodic CPU 0 even with CONFIG_NO_HZ_FULL_SYSIDLE=y. Anyway I have a few changes
> > > ready to enable that, lets hope testing will be ok :)
> > 
> > Indeed!  ;-)
> > 
> > The CONFIG_NO_HZ_FULL_SYSIDLE=y might complicate things a bit.  But I
> > guess the problem would be a corner case -- the system entered sysidle
> > mode with a grace period pending, which should eventually wake up the
> > corresponding grace-period kthread, which might be prevented from ever
> > running due to high load or something.  If that problem arises, I will
> > fix it.
> 
> I see. Well we'll find out.
> In the meantime I successfully plugged sysidle detection with full dynticks and it
> surprisingly works like a charm. Which makes me think there must be some bug in my patches that make things
> working by accident :)

Must be some mistake!  ;-)

							Thanx, Paul

> I'll post soon.
> 
> Thanks.
>