Re: Poor SMP performance pv_ops domU

[Jeremy Fitzhardinge <jeremy@goop.org>]

(Re-added cc: xen-devel)

On 05/19/2010 12:41 PM, John Morrison wrote:
> xentop for the cpu usage.
>
> We see the performance of a single core in domU when running a pv_ops kernel.
> Reboot domU with 2.6.18.8-xenU and performance jumps nearly 8 fold.
>   

Could you reproduce my experiment?  If you look at the CPU time
accumulated by each vcpu, is it incrementing at less than 1 vcpu
second/second?

> Pinned all 8 cpu's -  still the same results.
>
> Tried bare metal much better results.
>   

What do you mean by "much better"?  How does it compare to domu 2.6.18?

> We have seen this over 18 months on all pv kernel's we try.
>
> It's not any specific kernel - all pv kernel's we try have the same performance impact.
>   

Do you mean pvops, or all PV Xen kernels?  How do the recent Novell
Xenlinux kernels perform?  Have you verified there are no expensive
debug options enabled?

BTW, is it a 32 or 64-bit guest?

    J

> John
>
> On 19 May 2010, at 18:44, Jeremy Fitzhardinge wrote:
>
>   
>> On 05/19/2010 09:24 AM, John Morrison wrote:
>>     
>>> I've tried with various kernel's today - pv_ops seems to only use 1 core out of 8.
>>>
>>> PV spinlocks makes no difference.
>>>
>>> The thing that sticks out most is I cannot get the dom0 (xen-3.4.2) to show more that about 99.7% cpu usage for any pv_ops kernel.
>>>
>>> #!/usr/bin/perl
>>>
>>> while () {}
>>>
>>> running 8 of these loads 2.6.18.8-xenU with nearly 800% cpu as shown in dom0
>>> running the same 8 in any pv_ops kernel's only gets as high as about 99.7%
>>>
>>>       
>> What tool are you using to show CPU use?
>>
>>     
>>> Inside the pv and xenU kernels top -s show all 8 cores being used.
>>>
>>>       
>> I tried to reproduce this:
>>
>>   1. I created a 4 vcpu pvops PV domain (4 pcpu host)
>>   2. Confirmed that all 4 vcpus are present with "cat /proc/cpuinfo" in
>>      the domain
>>   3. Ran 4 instances of ``perl -e "while(){}"&'' in the domain
>>   4. "top" within the domain shows 99% overall user time, no stolen
>>      time, with the perl processes each using 99% cpu time
>>   5. in dom0 "watch -n 1 xl vcpu-list <domain>" shows all 4 vcpus are
>>      consuming 1 vcpu second per second
>>   6. running a spin loop in dom0 makes top within the domain show
>>      16-25% stolen time
>>
>> Aside from top showing "99%" rather than ~400% as one might expect, it
>> all seems OK, and it looks like the vcpus are actually getting all the
>> CPU they're asking for.  I think the 99 vs 400 difference is just a
>> change in how the kernel shows its accounting (since there's been a lot
>> of change in that area between .18 and .32, including a whole new
>> scheduler).
>>
>> If you're seeing a real performance regression between .18 and .32,
>> that's interesting, but it would be useful to make sure you're comparing
>> apples to apples; in particular, isolating any performance effect
>> inherent in Linux's performance change from .18 -> .32, compared to
>> pvops vs xenU.
>>
>> So, things to try:
>>
>>    * make sure all the vcpus are actually enabled within your domain;
>>      if your adding them after the domain has booted, you need to make
>>      sure they get hot-plugged properly
>>    * make sure you don't have any expensive debug options enabled in
>>      your kernel config
>>    * run your benchmark on the 2.6.32 kernel booted native and compare
>>      it to pvops running under xen
>>    * compare it with the Novell 2.6.32 non-pvops kernel
>>    * try pinning the vcpus to physical cpus to eliminate any Xen
>>      scheduler effects
>>
>> Thanks,
>>    J
>>
>>     
>