Re: [PATCH 07/10] sched: Migrate sched to use new tick dependency mask model

[Frederic Weisbecker <fweisbec@gmail.com>]

On Mon, Aug 03, 2015 at 07:09:11PM +0200, Peter Zijlstra wrote:
> On Mon, Aug 03, 2015 at 04:50:33PM +0200, Frederic Weisbecker wrote:
> > I think I could remove the context switch part. But then I need to find a
> > way to perform these checks on enqueue and dequeue task time:
> 
> Uhm, but you already do!?

Sure but I would like to avoid adding a context switch step, although
dequeuing itself often happens on context switch but we don't have
the choice but to check at that step.

> > So we can divide the dependency into:
> > 
> >          struct rq {
> > 	     ...
> > 	     int nr_fifo;
> > 	     int nr_rr;
> 
> Those are currently summed together in: rq->rt.rt_nr_total, I suppose we
> can split RR out.

Right

> 
> > 	     int nr_normal;
> 
> That's called: rq->cfs.h_nr_running

Ok.

> 
> But you've forgotten about SCHED_DEADLINE, we count those in:
> rq->dl.dl_nr_running.

Indeed. Hmm, there is no preemption between SCHED_DEALINE tasks, right?
So I can treat it like SCHED_FIFO.

> > 	}
> > 
> > 
> > 	int rq_update_tick_dep(struct rq *rq)
> > 	{
> > 	     if (rq->nr_fifo && (rq->nr_rr > 1 || rq->nr_normal > 1))

Oops I meant:

    if (rq->nr_fifo || (rq->nr_rr < 2 && rq->nr_normal < 2))
       clear_dep()
    else
       set_dep()

> > 	         tick_nohz_set_dep(SCHED_TICK_DEP);
> >              else
> > 	         tick_nohz_set_dep(SCHED_TICK_DEP)
> >         }
> > 
> > Then we add or dec the relevant counter fields from the various
> > sched_class::enqueue/dequeue.  I think I saw some of these counters
> > already exist but perhaps not all of them. There are per class rqs but
> > rt_nr_running counts tasks without distinction of policies.
> 
> Right. At which point you'll end up with:
> 
> 	if (rq->dl.dl_nr_running > 1 || rq->rt.rr_nr_total > 1 || rq->cfs.h_nr_running > 1)
> 		tick_nohz_set_dep(SCHED_TICK_DEP)
> 	else
> 		tick_nohz_clear_dep(SCHED_TICK_DEP)

It there is no preemption between deadline tasks, and SCHED_DEALINE is of higher
priority than SCHED_RR that would rather be:

        if (rq->dl.dl_nr_running || rq->rt.ff_nr_running || (rq->rt.rr_nr_running < 2 && rq->cfs.h_nr_running < 2))
	    clear_dep()
	else
	    set_dep()

> 
> But I fear that'll still be rather expensive in some cases. Imagine a
> case where we frequently flip between 1-2 tasks on the queue for any one
> of those classes, then we'll do a whole bunch of dep flips, which is an
> atomic op.

Indeed. Now doing such a thing on a nohz full CPU sounds insane.