Re: RHEL5.5, 32-bit VM repeatedly locks up due to kvmclock

["David S. Ahern" <daahern@cisco.com>]

On 04/23/2010 03:39 PM, Zachary Amsden wrote:
> On 04/23/2010 10:39 AM, Brian Jackson wrote:
>> On Friday 23 April 2010 12:08:22 David S. Ahern wrote:
>>   
>>> After a few days of debugging I think kvmclock is the source of lockups
>>> for a RHEL5.5-based VM. The VM works fine on one host, but repeatedly
>>> locks up on another.
>>>
>>> Server 1 - VM locks up repeatedly
>>> -- DL580 G5
>>> -- 4 quad-core X7350 processors at 2.93GHz
>>> -- 48GB RAM
>>>
>>> Server 2 - VM works just fine
>>> -- DL380 G6
>>> -- 2 quad-core E5540 processors at 2.53GHz
>>> -- 24GB RAM
>>>
>>> Both host servers are running Fedora Core 12, 2.6.32.11-99.fc12.x86_64
>>> kernel. I have tried various versions of qemu-kvm -- the version in
>>> FC-12 and the version for FC-12 in virt-preview. In both cases the
>>> qemu-kvm command line is identical.
>>>
>>> VM
>>> - RHEL5.5, PAE kernel (also tried standard 32-bit)
>>> - 2 vcpus
>>> - 3GB RAM
>>> - virtio network and disk
>>>
>>> When the VM locks up both vcpu threads are spinning at 100%. Changing
>>> the clocksource to jiffies appears to have addressed the problem.
>>>      
>>
>> Does changing the guest to -smp 1 help?
>>
>>    
> 
> Based on our current understanding of the problem, it should help, but
> it may not prevent the problem entirely.
> 
> There are three issues with kvmclock due to sampling:
> 
> 1) smp clock alignment may be slightly off due to timing conditions
> 2) kvmclock is resampled at each switch of vcpu to another pcpu
> 3) kvmclock granularity exceeds that of kernel timespec, which means
> sampling errors may show even on UP
> 
> Recommend using a different clocksource (tsc is great if you have stable
> TSC and don't migrate across different-speed machines) until we have all
> the fixes in place.

That's my plan for now. As I recall jiffies was the default in early
RHEL5 versions. Not sure what that means hardware wise.

The biggest problem for me is that RHEL5.5 defaults to kvmclock; I'll
find some workaround for it.

David

> 
> Zach