Re: Virt overehead with HT [was: Re: Xen 4.5 development update]

[Gordan Bobic <gordan@bobich.net>]

On 07/15/2014 03:30 AM, Dario Faggioli wrote:
> On mar, 2014-07-15 at 01:10 +0100, Gordan Bobic wrote:
>> On 07/14/2014 11:44 PM, Dario Faggioli wrote:
>
>>> If you pin VCPU#1 to PCPU#1 and VCPU#2 to PCPU#2, with PCPU#1 and PCPU#2
>>> being HT siblings, what prevents Linux (in the guest) to run two of the
>>> four build jobs on VCPU#1 and VCPU#2 (i.e., on siblings PCPUs!!) for all
>>> the length of the benchmark? Nothing, I think.
>>
>> That would imply that Xen can somehow make a better decision that the
>> domU's kernel scheduler, something that doesn't seem that likely.
>>
> Well, as far as SMT load balancing is concerned, that is _exactly_ the
> case. The reason is simple: Xen knows the hw topology, and hence knows
> whether the sibling of an idle core is idle or busy. The guest kernel
> sees nothing about this, it just treat all its (V)CPUs as full cores, so
> it most likely will do a bad job in this case.
>
>>> And in fact, pinning would also result in good (near to native,
>>> perhaps?) performance, if we were exposing the SMT topology details to
>>> guests as, in that case, Linux would do the balancing properly. However,
>>> that's not the case either. :-(
>>
>> I see, so you are referring specifically to the HT case.
>>
> Yeah, well, that's what this benchmarks where all about  :-)
>
>> I can see how
>> that could cause a problem. Does pinning improve the performance with HT
>> disabled?
>>
> HT disabled had pretty goo perf. already. Anyhow, I tried:
>
> Average Half load -j 2 Run (std deviation):
>   Elapsed Time 56.462 (0.109179)
> Average Optimal load -j 4 Run (std deviation):
>   Elapsed Time 31.526 (0.224789)
> Average Maximal load -j Run (std deviation):
>   Elapsed Time 33.04 (0.439147)
>
> So a lot similar to the no-HT unpinned case, which on it's turn was a
> lot similar to baremetal without HT.

Just out of interest - in cases where there is a non-negligible 
performance discrepancy with HT enabled (bare metal or Xen), does 
disabling the C6 CPU state support in the BIOS help? C6 state 
selectively disables CPU threads when the CPU is idle for power saving 
purposes, but the disabling threshold can be too sensitive. Does Xen 
handle this?

Gordan