Performance of Xen VCPU Scheduling

Performance of Xen VCPU Scheduling

[Marcus Granado]

Hello,

I observed that a configuration where dom0 vcpus were pinned to a set of 
pcpus in the host using dom0_vcpus_pin and the guests were prevented 
from running on the dom0 pcpus (here called "exclusively-pinned dom0 
vcpus", or xpin) caused the general performance of the guests in a host 
with around 24 pcpus or more to increase during bootstorms and high 
density of guests, even though there were less pcpus available to the 
guests.

While trying to understand why this increased performance is not present 
in the default non-pinned state and if it would be possible to obtain 
this extra performance in the default non-pinned dom0 configuration, 
Matthew Portas and I evaluated lots of Xen 4.2 parameters and patches:

http://wiki.xenproject.org/wiki/Performance_of_Xen_VCPU_Scheduling

We put some of the results that we thought would be of interest to this 
list in the link above, and we invite everybody here to have a look. We 
are especially interested in having some feedback on the prototype 
section, and to find out if anyone has further ideas for patches and if 
the listed patches are reasonable or could have side-effects.

Interesting results in the link above:
- xpin decreases startup time of vms in a bootstorm
- xpin has a pathological case when the vms are burning lots of cpu
- nopin produces an interesting cluster of high event channel latency 
between 100us-1000us when enough vms are using lots of cpu
- experimental tweaks on the xen credit1 scheduler code that cause the 
xpin pathological case to go away, and other increases (and sometimes 
decreases) in bootstorm and vm density performance.

cheers,
Marcus

Re: Performance of Xen VCPU Scheduling

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1490 bytes --]

On mer, 2013-07-10 at 14:00 +0100, Marcus Granado wrote:
> Hello,
> 
Hey!

> [...]
>
> While trying to understand why this increased performance is not present 
> in the default non-pinned state and if it would be possible to obtain 
> this extra performance in the default non-pinned dom0 configuration, 
> Matthew Portas and I evaluated lots of Xen 4.2 parameters and patches:
> 
> http://wiki.xenproject.org/wiki/Performance_of_Xen_VCPU_Scheduling
> 
I've just glanced at it, and there's really a lot of data there! :-) 

I'll have a seriously deep look in the next hours and will provide some
feedback about it... For now, let me thank you a lot for both doing and
sharing this (especially for putting it on the Wiki! :-D).

Just one point. This by any means make all this study/effort less
valuable (quite the contrary, actually), but, with George, we started to
discuss about bringing credit2 into a better shape and, at some point,
switch to it by default. It's unclear whether this could be done for
4.4, but still... So, do you have any numbers about it already? How hard
will it be to, at some point, perform the same measurements and analysis
with it?

Thanks again and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Performance of Xen VCPU Scheduling

[Marcus Granado]

On 10/07/13 14:23, Dario Faggioli wrote:
> On mer, 2013-07-10 at 14:00 +0100, Marcus Granado wrote:
>>
>> While trying to understand why this increased performance is not present
>> in the default non-pinned state and if it would be possible to obtain
>> this extra performance in the default non-pinned dom0 configuration,
>> Matthew Portas and I evaluated lots of Xen 4.2 parameters and patches:
>>
>
> Just one point. This by any means make all this study/effort less
> valuable (quite the contrary, actually), but, with George, we started to
> discuss about bringing credit2 into a better shape and, at some point,
> switch to it by default. It's unclear whether this could be done for
> 4.4, but still... So, do you have any numbers about it already?

Yes, I believe we have a couple of results for credit2, I'll put them on 
the page.

> How hard
> will it be to, at some point, perform the same measurements and analysis
> with it?
>

Easy enough, all the measurements used automated tests to improve 
reproducibility. We would need a url with binaries or rpms as input, and 
the output is a set of graphs for analysis.

cheers,
Marcus

Re: Performance of Xen VCPU Scheduling

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1401 bytes --]

On mer, 2013-07-10 at 15:46 +0100, Marcus Granado wrote:
> > Just one point. This by any means make all this study/effort less
> > valuable (quite the contrary, actually), but, with George, we started to
> > discuss about bringing credit2 into a better shape and, at some point,
> > switch to it by default. It's unclear whether this could be done for
> > 4.4, but still... So, do you have any numbers about it already?
> 
> Yes, I believe we have a couple of results for credit2, I'll put them on 
> the page.
> 
Cool. Thanks. :-)

> > How hard
> > will it be to, at some point, perform the same measurements and analysis
> > with it?
> >
> 
> Easy enough, all the measurements used automated tests to improve 
> reproducibility. We would need a url with binaries or rpms as input, and 
> the output is a set of graphs for analysis.
> 
Yeah, that was what I was hoping for! ;-P BTW, it's definitely lesser of
a priority right now, but that is (how to run the benchmarks, I mean)
another thing that it would be worthwhile doc-ing properly (at least
internally, provided it's not done already)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel