Re: pinning, tsc and apic

[Anthony Liguori <anthony@codemonkey.ws>]

Ryan Harper wrote:
> * Anthony Liguori <anthony@codemonkey.ws> [2008-05-12 15:05]:
>   
>> Ryan Harper wrote:
>>     
>>> I've been digging into some of the instability we see when running
>>> larger numbers of guests at the same time.  The test I'm currently using
>>> involves launching 64 1vcpu guests on an 8-way AMD box.
>>>       
>> Note this is a Barcelona system and therefore should have a 
>> fixed-frequency TSC.
>>
>>     
>>>  With the latest
>>> kvm-userspace git and kvm.git + Gerd's kvmclock fixes, I can launch all
>>> 64 of these 1 second apart,
>>>       
>> BTW, what if you don't pace-out the startups?  Do we still have issues 
>> with that?
>>     
>
> Do you mean without the 1 second delay or with a longer delay?  My
> experience is that delay helps (fewer hangs), but doesn't solve things
> completely.
>   

So you see problems when using numactrl to pin and using a 0-second 
delay?  The short delay may help reduce the number of CPU migrations 
which would explain your observation.

If there are problems when doing a 0-second delay and numactl, then 
perhaps it's not just a cpu-migration issue.

> In svm.c, I do think we account for most of that time since the delta
> calculation will shift the guest time forward to the tsc value read in
> svm_vcpu_load().  We'll still miss the time between fixing the offset
> and when the guest can actually read its tsc.
>   

Yes, which is the duration that the guest isn't scheduled on any 
processor and the next time it runs happens to be on a different process.

>> A possible way to fix this (that's only valid on a processor with a 
>> fixed-frequency TSC), is to take a high-res timestamp on vcpu_put, and 
>> then on vcpu_load, take the delta timestamp since the old TSC was saved, 
>> and use the TSC frequency on the new pcpu to calculate the number of 
>> elapsed cycles.
>>
>> Assuming a fixed frequency TSC, and a calibrated TSC across all 
>> processors, you could get the same affects by using the VT tsc delta 
>> logic.  Basically, it always uses the new CPU's TSC unless that would 
>> cause the guest to move backwards in time.  As long as you have a 
>> stable, calibrated TSC, this would work out.
>>
>> Can you try your old patch that did this and see if it fixes the problem?
>>     
>
> Yeah, I'll give it a spin.
>   

Thanks,

Anthony Liguori



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft 
Defy all challenges. Microsoft(R) Visual Studio 2008. 
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/