Re: [RFC PATCH 0/4] Gang scheduling in CFS

[Peter Zijlstra <peterz@infradead.org>]

On Mon, 2012-02-20 at 17:32 +0530, Srivatsa Vaddagiri wrote:
> * Nikunj A Dadhania <nikunj@linux.vnet.ibm.com> [2012-02-20 17:23:16]:
> 
> > On Mon, 20 Feb 2012 11:51:13 +0100, Peter Zijlstra <peterz@infradead.org> wrote:
> > > On Mon, 2012-02-20 at 13:38 +0530, Nikunj A Dadhania wrote:
> > > > +#ifdef CONFIG_PARAVIRT_FLUSH_TLB
> > > > +               while (!cpumask_empty(to_cpumask(f->flush_cpumask)) && --loop)
> > > > +                       cpu_relax();
> > > > +               if (!loop && !cpumask_empty(to_cpumask(f->flush_cpumask)))
> > > > +                       halt();
> > > 
> > > 
> > > That's just vile, you don't need to wait for it, all you need to make
> > > sure is that when that vcpu wakes up it does the flush.
> > >
> > Yes, but we are not sure the vcpu will be sleeping or running. In the
> > case where vcpus are running, it might be beneficial to wait a while.
> 
> I guess one possibility is for host scheduler to export run/preempt
> information to guest OS, as was discussed in the context of this thread
> as well:
> 
> http://lkml.org/lkml/2010/4/6/223

Doesn't need to be the host scheduler, KVM itself can do that just fine
on guest entry/exit.

> Essentially guest OS will know (using such exported information) which of its 
> vcpus are currently running and thus do a busy-wait for them.

Right, do something like:

again:
  for_each_cpu_in_mask(cpu, flush_cpumask) {
    if !vcpu-running {
      set_flush-on-enter(cpu)
      if !vcpu-running
        cpumask_clear(flush_cpumask, cpu); // vm-enter will do  
  }
  wait-a-while-for-mask-to-clear
  if (!cpumask_empty)
    goto again;    

with the appropriate memory barriers and atomic instructions, that way
you can skip waiting for vcpus that are not in guest mode and vm-enter
will fixup.