Re: [PATCH] nohz1: Documentation

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Thu, Mar 21, 2013 at 08:04:08PM +0000, Christoph Lameter wrote:
> On Thu, 21 Mar 2013, Paul E. McKenney wrote:
> 
> > > Yeah doing that right now but I'd like to see it handled without manual
> > > intervention.
> >
> > Given that RCU has no idea where you want them to run, some manual
> > intervention would most likely be required even if RCU spawned them
> > dynamically, right?
> 
> If rcuoXX is a SCHED_OTHER process/thread then the kernel will move it to
> another processor from the one running the SCHED_FIFO task. There would be
> no manual intervention required.

Assuming that the SCHED_FIFO task was running at the time that RCU
decided to spawn the kthread, and assuming that there was at least
one CPU not running a SCHED_FIFO task, agreed.  But these assumptions
do not hold in general.

> > So, again, removing scheduling-clock interrupts in more situations is
> > a good future enhancement.
> 
> The point here is that the check for a single runnable process is wrong
> because it accounts for tasks in all scheduling classes.

Incomplete, yes.  Only a starting point, yes.  Wrong, no.

> It would be better to check if there is only one runnable task in the
> highest scheduling class. That would work and defer the SCHED_OTHER kernel
> threads for the SCHED_FIFO thread.

Agreed, that would be better.  Hopefully we will handle that and other
similar cases at some point.

> I am wondering how you actually can get NOHZ to work right? There is
> always a kernel thread that is scheduled in a couple of ticks.
> 
> I guess what will happens with this patchset is:
> 
> 1. SCHED_FIFO thread begins to run. There is only a single runnable task
> so adaptive tick mode is enabled.

Yep.

> 2. After 2 seconds or so some or other thing needs to run (keventd thread
> needs to run vm statistics f.e.). It becomes runnable. nr_running > 1.
> Adaptive tick mode is disabled? Occurs on my system. Or is there some
> other trick to avoid kernel threads becoming runnable?

Yes, adaptive tick mode would be disabled at that point.

> 3. Now there are 2 runnable processes. The SCHED_FIFO thread continues to
> run with the tick. The kernel thread is also runnable but will not be
> given cpu time since the SCHED_FIFO thread has priority?

Yep.

> So the SCHED_FIFO thread enjoys 2 seconds of no tick time and then ticks
> occur uselessly from there on?

If the SCHED_FIFO thread never sleeps at all, this would be the outcome.
On the other hand, if the SCHED_FIFO thread never sleeps at all, the
various per-CPU kthreads are deferred forever, which might not be so
good long term.

If the SCHED_FIFO thread does sleep at some point, the SCHED_OTHER threads
would run, the CPU would go idle, and then when the SCHED_OTHER thread
started up again, it would start up in adaptive-idle mode.

> I have not been able to consistently get the tick switched off with
> the nohz patchset. How do others use nohz? Is it only usable for short
> periods of less than 2 seconds?

I believe that many other SCHED_FIFO users run their SCHED_FIFO threads
in short bursts to respond to some real-time event.  They would not tend
to have a SCHED_FIFO thread with a busy period exceeding two seconds,
and therefore would be less likely to encounter this issue.

So, how long of busy periods are you contemplating for your SCHED_FIFO
threads?  Is it possible to tune/adjust the offending per-CPU ktheads
to wake up less frequently than that time?

							Thanx, Paul