Re: credit scheduler and HYPERVISOR_yield()

[Emmanuel Ackaouy <ackaouy@gmail.com>]

On Oct 14, 2007, at 20:45, John Levon wrote:
> Emmanuel Ackaouy wrote:
>> It may be worthwhile to consider if yield() can be replaced with
>> more intelligent mechanisms for VCPU synchronization of SMP
>> guests. In the case of ACKed IPIs for example, if all target VCPUs
>> are not running at the time of the IPI initiation, it might be a good
>> idea to put the source to sleep until all targets have ACKed.
>> If all target VCPUs are running though, I suspect things will work
>> best if the IPI initiator does not yield at all.
>
> This seems like a bad idea since we may be IPIing to several CPUs and 
> we
> don't want to sleep whilst we can usefully move on and IPI the other
> CPUs (even if they can't quite respond yet).

Why can't you initiate the IPI to all the destination CPUs first
and then wait for them to ACK, going to sleep if it looks like at
least one of them won't be able to ACK in a reasonable
timeframe (for example if it is asleep or on the run queue of
the running VCPU's physical CPU)?

I'm probably not understanding what you're trying to do?


> On Tue, Oct 09, 2007 at 02:22:13PM +0100, George Dunlap wrote:
>> A simple option would be, for the credit scheduler, to temporarily
>> reduce the priority from TS_UNDER to TS_OVER.  This will cause it to
>
> We prototyped this change and it made quite a difference (though didn't
> solve our problems entirely). Would it be possible to get a proper fix
> available?

Doing the change that George proposed may help in your case
but I suspect that, as I described in my previous post, it will cause
problems for other workloads.

I think it is reasonable for a yield() operation to yield to runnable
VCPUs of equal or higher priority than the running VCPU. That
is the behavior of the scheduler today. Maybe your problem can
be addressed without changing the behavior of yield?

With that said, it's unlikely that I'll be making a change to the
scheduler myself: I haven't worked at XenSource for some time
now and don't have the resources (not to mention time) to test
any such change. I'm happy to learn about your problem and
suggest potential fixes but I'm probably not the person you need
to convince if you want to make a significant scheduler change
these days. Arguably, a number of things need to be done in
the Xen scheduler and synchronization primitives to improve
the performance of SMP guests. It may be worthwhile to have
a generic discussion about that on top of the specific problem
you're encountering.

Cheers,
Emmanuel.