unfair VCPU scheduling: slow HVM guest boot

unfair VCPU scheduling: slow HVM guest boot

[Christoph Egger]

Hi,

Launch a HVM guest with twice as many VCPUs as physical CPUs are in the
machine. You will notice the guest boots slow.

With xentop you see, the first VCPU is rarely scheduled once the other
VCPUs are up in the guest.
If the boot process is just waiting for the first VCPU to finish something
(e.g. handling an interrupt), then the whole boot process "freezes" until the 
first VCPU gets scheduled.

Here is an xentop line showing how unfair the VCPUs are scheduled:

VCPUs(sec):   0:         44s  1:         94s  2:         96s  3:        140s


Christoph


-- 
AMD Saxony, Dresden, Germany
Operating System Research Center

Legal Information:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift):
   Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär:
   AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC:
   Dr. Hans-R. Deppe, Thomas McCoy

Re: unfair VCPU scheduling: slow HVM guest boot

[George Dunlap]

The fact that there's different amounts of cpu time isn't evidence
that the scheduler is unfair.  The vcpus may be blocked, or may be
coming up at different times.

Is there a particular reason you want to run with more VCPUs than physical cpus?

Given that cpu synchronization primitives like spinlocks and IPIs were
generally designed with the assumption that they're running on bare
metal and are not pre-empted, it's not surprising that when vcpus are
trying to work together but not able to run at the same time, there
will be performance problems.

 -George

On Fri, Aug 15, 2008 at 12:29 PM, Christoph Egger
<Christoph.Egger@amd.com> wrote:
>
> Hi,
>
> Launch a HVM guest with twice as many VCPUs as physical CPUs are in the
> machine. You will notice the guest boots slow.
>
> With xentop you see, the first VCPU is rarely scheduled once the other
> VCPUs are up in the guest.
> If the boot process is just waiting for the first VCPU to finish something
> (e.g. handling an interrupt), then the whole boot process "freezes" until the
> first VCPU gets scheduled.
>
> Here is an xentop line showing how unfair the VCPUs are scheduled:
>
> VCPUs(sec):   0:         44s  1:         94s  2:         96s  3:        140s
>
>
> Christoph
>
>
> --
> AMD Saxony, Dresden, Germany
> Operating System Research Center
>
> Legal Information:
> AMD Saxony Limited Liability Company & Co. KG
> Sitz (Geschäftsanschrift):
>   Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
> Registergericht Dresden: HRA 4896
> vertretungsberechtigter Komplementär:
>   AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
> Geschäftsführer der AMD Saxony LLC:
>   Dr. Hans-R. Deppe, Thomas McCoy
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: unfair VCPU scheduling: slow HVM guest boot

[Christoph Egger]

On Friday 15 August 2008 13:39:45 George Dunlap wrote:
> The fact that there's different amounts of cpu time isn't evidence
> that the scheduler is unfair.  The vcpus may be blocked, or may be
> coming up at different times.
>
> Is there a particular reason you want to run with more VCPUs than physical
> cpus?

Yes. You can have a virtual test/crash box for some development work, for
example.

> Given that cpu synchronization primitives like spinlocks and IPIs were
> generally designed with the assumption that they're running on bare
> metal and are not pre-empted, it's not surprising that when vcpus are
> trying to work together but not able to run at the same time, there
> will be performance problems.

Yes, overcommitting always causes performance problems.
The thing I am observing with xentop is that the last activated VCPU
seems to be prefered over the others and the virtual BSP runs very rarely. 
That is what I mean by "unfair".

Christoph

>  -George
>
> On Fri, Aug 15, 2008 at 12:29 PM, Christoph Egger
>
> <Christoph.Egger@amd.com> wrote:
> > Hi,
> >
> > Launch a HVM guest with twice as many VCPUs as physical CPUs are in the
> > machine. You will notice the guest boots slow.
> >
> > With xentop you see, the first VCPU is rarely scheduled once the other
> > VCPUs are up in the guest.
> > If the boot process is just waiting for the first VCPU to finish
> > something (e.g. handling an interrupt), then the whole boot process
> > "freezes" until the first VCPU gets scheduled.
> >
> > Here is an xentop line showing how unfair the VCPUs are scheduled:
> >
> > VCPUs(sec):   0:         44s  1:         94s  2:         96s  3:       
> > 140s
> >
> >
> > Christoph


-- 
AMD Saxony, Dresden, Germany
Operating System Research Center

Legal Information:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift):
   Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär:
   AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC:
   Dr. Hans-R. Deppe, Thomas McCoy