Re: [PATCH RFC: kvm tsc virtualization 15/20] Fix longstanding races

Marcelo Tosatti
2009-12-16T14:42:14Z

On Tue, Dec 15, 2009 at 11:26:59AM -1000, Zachary Amsden wrote:
> On 12/15/2009 08:21 AM, Marcelo Tosatti wrote:
>> On Mon, Dec 14, 2009 at 06:08:42PM -1000, Zachary Amsden wrote:
>>
>> <snip>
>>
>> + atomic_set(&per_cpu(cpu_tsc_synchronized, freq->cpu),
>> 0);
>> + spin_lock(&kvm_lock);
>> + list_for_each_entry(kvm,&vm_list, vm_list) {
>> + kvm_for_each_vcpu(i, vcpu, kvm) {
>> + if (vcpu->cpu != freq->cpu)
>> + continue;
>> + if (vcpu->cpu != smp_processor_id())
>> + send_ipi++;
>> + kvm_request_guest_time_update(vcpu);
>>
>> There is some overlap here between KVM_REQ_KVMCLOCK_UPDATE and
>> cpu_tsc_synchronized. Its the same information (frequency for a CPU has
>> changed) stored in two places.
>>
>> Later you do:
>>
>> spin_lock(&kvm_lock);
>> list_for_each_entry(kvm,&vm_list, vm_list) {
>> kvm_for_each_vcpu(i, vcpu, kvm) {
>> if (vcpu->cpu != freq->cpu)
>> continue;
>> if (vcpu->cpu != smp_processor_id())
>> send_ipi++;
>> kvm_request_guest_time_update(vcpu);
>> }
>> }
>> spin_unlock(&kvm_lock);
>> <--- a remote CPU could have updated kvmclock information
>> with stale cpu_tsc_khz, clearing the
>> KVM_REQ_KVMCLOCK_UPDATE bit.
>> smp_call_function(evict) (which sets cpu_tsc_synchronized
>> to zero)
>>
>> Maybe worthwhile to unify it. Perhaps use the per cpu tsc generation in
>> addition to vcpu_load to update kvmclock info (on arch vcpu_load update
>> kvmclock store generation, update again on generation change).
>>
>
> Yes, that is an excellent point. The generation counter, the
> tsc_synchronized variable and the per-vcpu clock counter all have some
> amount of redundancy of information.
>
> Perhaps instead of overlapping, they should be layered?
>
> A rule for kvmclock: can't update kvmclock info until cpu is synchronized?

How about update kvmclock on:

- cpu switch (kvm_arch_vcpu_load). Then store cpu tsc generation in
vcpu->arch.
- on vcpu_enter_guest, if tsc generation changes.

If kvm_arch_vcpu_load stored stale cpu_khz into kvmclock, the tsc
generation will have changed by the time guest entry succeeds.

Then you can kill KVM_REQ_KVMCLOCK_UPDATE and the kvm_for_each_vcpu()
loop in the cpufreq callback.