CPU scheduling of domains and vcpus

CPU scheduling of domains and vcpus

[Nauman Rafique]

[-- Attachment #1.1: Type: text/plain, Size: 821 bytes --]

My question is related to an earlier discussion on the list (
http://lists.xensource.com/archives/html/xen-devel/2007-03/msg01395.html).
Is there any plan for implementing gang scheduling (of all vcpus of a
domain) in xen? I see that gang scheduling is mentioned in almost all the
latest talks about xen (posted on xen.org).
It seems to me that gang scheduling would be desirable, specially if there
are synchronization points (lock and semaphores) in guest OS and
applications. I understand that gang scheduling can lead to wasted cycles,
but that might be acceptable in certain situations. In fact, wasted cycles
can probably be avoided by doing opportunisitic gang scheduling (i.e. gang
schedule, unless there would be wasted cycles) and by carefully choosing the
number of vcpus.

--
Nauman Rafique
Purdue University

[-- Attachment #1.2: Type: text/html, Size: 962 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: CPU scheduling of domains and vcpus

[Samuel Thibault]

Hello,

Nauman Rafique, le Mon 21 Apr 2008 13:52:21 -0400, a écrit :
> In fact, wasted cycles can probably be avoided by doing opportunisitic
> gang scheduling (i.e. gang schedule, unless there would be wasted
> cycles)

How do you detect that there would be wasted cycles?

Samuel

Re: CPU scheduling of domains and vcpus

[Mike D. Day]

On 21/04/08 18:56 +0100, Samuel Thibault wrote:
> Hello,
> 
> Nauman Rafique, le Mon 21 Apr 2008 13:52:21 -0400, a écrit :
> > In fact, wasted cycles can probably be avoided by doing opportunisitic
> > gang scheduling (i.e. gang schedule, unless there would be wasted
> > cycles)
> 
> How do you detect that there would be wasted cycles?

The only way is for a very self-aware guest to use a paravirtual
feature to give a hint to the scheduler. 

Which is also the way to solve the original problem: the
paravirtualized guest can provide a hint to the scheduler that it is
holding a contended lock.

Alternatively, the scheduler can notify the guest that it is about to
be preempted by the hypervisor and now would be a good time to sleep
before gaining a contended spinlock.

In either case lock contention within a multi-vcpu guest is probably
not as bad a problem as the various solutions cause.

If a workload is suffering so much, give it multiple vcpus that are
mapped 1:1 to physical cpus. 

Mike

-- 
Mike D. Day
IBM LTC
Cell: 919 412-3900
Sametime: ncmike@us.ibm.com AIM: ncmikeday  Yahoo: ultra.runner
PGP key: http://www.ncultra.org/ncmike/pubkey.asc

Re: CPU scheduling of domains and vcpus

[Samuel Thibault]

Mike D. Day, le Mon 21 Apr 2008 16:32:18 -0400, a écrit :
> On 21/04/08 18:56 +0100, Samuel Thibault wrote:
> > Nauman Rafique, le Mon 21 Apr 2008 13:52:21 -0400, a écrit :
> > > In fact, wasted cycles can probably be avoided by doing opportunisitic
> > > gang scheduling (i.e. gang schedule, unless there would be wasted
> > > cycles)
> > 
> > How do you detect that there would be wasted cycles?
> 
> The only way is for a very self-aware guest to use a paravirtual
> feature to give a hint to the scheduler. 

Agreed :)

> Alternatively, the scheduler can notify the guest that it is about to
> be preempted by the hypervisor and now would be a good time to sleep
> before gaining a contended spinlock.

Mmm, "about to be" means "on the next tick" then.  That may be a long
time.

Samuel

Re: CPU scheduling of domains and vcpus

[Nauman Rafique]

[-- Attachment #1.1: Type: text/plain, Size: 1531 bytes --]

If we know that a particular VM is running a compute bound workload with
synchronization, we can mark that VM to be gang scheduled. That would work
better than pinning to physical cpus, since scheduler can still load balance
across all physical cpus.

Nevertheless, as Mike mentioned, idle cycles can be accounted for by
providing hints from guest OS. Hints can be rolled into the multicall done
at context switch.

It seems to me that it would be desirable to have gang scheduling available
as an one of the scheduler options.

--
Nauman Rafique
Purdue University

On Mon, Apr 21, 2008 at 5:34 PM, Samuel Thibault <
samuel.thibault@eu.citrix.com> wrote:

> Mike D. Day, le Mon 21 Apr 2008 16:32:18 -0400, a écrit :
> > On 21/04/08 18:56 +0100, Samuel Thibault wrote:
> > > Nauman Rafique, le Mon 21 Apr 2008 13:52:21 -0400, a écrit :
> > > > In fact, wasted cycles can probably be avoided by doing
> opportunisitic
> > > > gang scheduling (i.e. gang schedule, unless there would be wasted
> > > > cycles)
> > >
> > > How do you detect that there would be wasted cycles?
> >
> > The only way is for a very self-aware guest to use a paravirtual
> > feature to give a hint to the scheduler.
>
> Agreed :)
>
> > Alternatively, the scheduler can notify the guest that it is about to
> > be preempted by the hypervisor and now would be a good time to sleep
> > before gaining a contended spinlock.
>
> Mmm, "about to be" means "on the next tick" then.  That may be a long
> time.
>
> Samuel
>

[-- Attachment #1.2: Type: text/html, Size: 2028 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel