Re: About vcpu wakeup and runq tickling in credit

[George Dunlap <george.dunlap@eu.citrix.com>]

On 16/11/12 12:00, Dario Faggioli wrote:
> On Fri, 2012-11-16 at 11:53 +0100, Dario Faggioli wrote:
>> On Thu, 2012-11-15 at 12:18 +0000, George Dunlap wrote:
>>> Maybe what we should do is do the wake-up based on who is likely to run
>>> on the current cpu: i.e., if "current" is likely to be pre-empted, look
>>> at idlers based on "current"'s mask; if "new" is likely to be put on the
>>> queue, look at idlers based on "new"'s mask.
>>>
>> Ok, find attached the two (trivial) patches that I produced and am
>> testing in these days. Unfortunately, early results shows that I/we
>> might be missing something.
>>
> I'm just came to thinking that this approach, although more, say,
> correct, could have a bad impact on caches and locality in general.

One thing that xenalyze will already tell you is statistics on how a 
vcpu migrates over pcpus.  For example:

  cpu affinity:     242 7009916158 {621089444|5643356292|19752063006}
    [0]:      15 6940230676 {400952|5643531152|27013831272}
    [1]:      19 6366861827 {117462|5031404806|19751998114}
    [2]:      31 6888557514 {1410800684|5643015454|19752100009}
    [3]:      18 7790887470 {109764|5920027975|25395539566}
...

The general format is: "$number $average_cycles {5th percentile|50th 
percentile|95th percentile}".  The first line includes samples from 
*all* cpus (i.e,. so it migrated a total of 242 times, averaging 7 
billion cycles each time); the subsequent numbers show statistics on 
specific pcpus (i.e., it had 15 sessions on pcpu 0, averaging 6.94 
billion cycles, &c).

You should be able to use this to do a basic verification of your 
hypothesis that vcpus are migrating more often.

> In fact, suppose a new vcpu N wakes up on pcpu #x where another vcpu C
> is running, with prio(N)>prio(C).
>
> What upstream does is asking to #x and to all the idlers that can
> execute N to reschedule. Doing both is, I think, wrong, as there's the
> chance of ending up with N being scheduled on #x and C being runnable
> but not running (in #x's runqueue) even if there are idle cpus that
> could run it, as they're not poked (as already and repeatedly said).
>
> What the patches do, in this case (remember (prio(N)>prio(C)), is asking
> #x and all the idlers that can run C to reschedule, the effect being
> that N will likely run on #x, after a context switch, and C will run
> somewhere else, after a migration, potentially wasting its cache-hotness
> (it is running after all!).
>
> It looks like we can do better... Something like the below:
>   + if there are no idlers where N can run, ask #x and the idlers where
>     C can run to reschedule (exactly what the patches do, although, they
>     do that _unconditionally_), as there isn't anything else we can do
>     to try to make sure they both will run;
>   + if *there*are* idlers where N can run, _do_not_ ask #x to reschedule
>     and only poke them to come pick N up. In fact, in this case, it is
>     not necessary to send C away for having both the vcpus ruunning, and
>     it seems better to have N experience the migration as, since it's
>     waking-up, it's more likely for him than for C to be cache-cold.

I think that makes a lot of sense -- look forward to seeing the results. :-)

There may be some other tricks we could look at.  For example, if N and 
C are both going to do a significant chunk of computation, then this 
strategy will work best.  But suppose that C does a significant junk of 
computation, but N is only going to run for a few hundred microseconds 
and then go to sleep again?  In that case, it may be easier to just run 
N on the current processor and not bother with IPIs and such; C will run 
again in a few microseconds.   Conversely, if N will do a significant 
chunk of work but C is fairly short, we might as well let C continue 
running, as N will shortly get to run.

How to know if the next time this vcpu runs will be long or short? We 
could try tracking the runtimes of the last N (maybe 3 or 5) this was 
scheduled, and using that to predict the results.

Do you have traces for any of those runs you did?  I might just take a 
look at them and see if I can make an analysis of cache "temperature" 
wrt scheduling. :-)

  -George

  -George