Re: [PATCH 2/2] KVM: Protect race-condition between VMCS and current_vmcs on VMX hardware

Re: [PATCH 2/2] KVM: Protect race-condition between VMCS and current_vmcs on VMX hardware

[Gregory Haskins]

On Thu, 2007-07-26 at 18:03 +0300, Avi Kivity wrote:
> Gregory Haskins wrote:
> > We need to provide locking around the current_vmcs/VMCS interactions to
> > protect against race conditions.
> >
> >   
> 
> Can you explain the race?

Sure.  It can happen with two VMs are running simultaneously.  Lets call
them VM-a and VM-b.  Assume the scenario: VM-a is on CPU-x, gets
migrated to CPU-y, and VM-b gets scheduled in on CPU-x.  There is a race
on CPU-x with the VMCS handling logic between the VM-b process context,
and the IPI to execute the __vcpu_clear for VM-a. 

Disabling interrupts was chosen as the sync-primitive, because the code
will always be on the CPU in question.



-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: [PATCH 2/2] KVM: Protect race-condition between VMCS and current_vmcs on VMX hardware

[Avi Kivity]

Gregory Haskins wrote:
> On Thu, 2007-07-26 at 18:03 +0300, Avi Kivity wrote:
>   
>> Gregory Haskins wrote:
>>     
>>> We need to provide locking around the current_vmcs/VMCS interactions to
>>> protect against race conditions.
>>>
>>>   
>>>       
>> Can you explain the race?
>>     
>
> Sure.  It can happen with two VMs are running simultaneously.  Lets call
> them VM-a and VM-b.  Assume the scenario: VM-a is on CPU-x, gets
> migrated to CPU-y, and VM-b gets scheduled in on CPU-x.  There is a race
> on CPU-x with the VMCS handling logic between the VM-b process context,
> and the IPI to execute the __vcpu_clear for VM-a. 
>
>   

A race indeed, good catch.

I think the race is only on the per_cpu(current_vmcs) variable, no?  The 
actual vmcs ptr (as loaded by vmptrld) is handled by the processor.

> Disabling interrupts was chosen as the sync-primitive, because the code
> will always be on the CPU in question.
>
>   

Looks a bit heavy handed.  How about replacing (in __vcpu_clear())

    if (per_cpu(current_vmcs, cpu) == vcpu->vmcs)
        per_cpu(current_vmcs, cpu) = NULL;

by

    cmpxchg_local(&per_cpu(current_vmcs, cpu), vcpu->vmcs, NULL);

?

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: [PATCH 2/2] KVM: Protect race-condition between VMCS and current_vmcs on VMX hardware

[Avi Kivity]

Avi Kivity wrote:
>>
>> Sure.  It can happen with two VMs are running simultaneously.  Lets call
>> them VM-a and VM-b.  Assume the scenario: VM-a is on CPU-x, gets
>> migrated to CPU-y, and VM-b gets scheduled in on CPU-x.  There is a race
>> on CPU-x with the VMCS handling logic between the VM-b process context,
>> and the IPI to execute the __vcpu_clear for VM-a.
>>   
>
> A race indeed, good catch.
>
> I think the race is only on the per_cpu(current_vmcs) variable, no?  
> The actual vmcs ptr (as loaded by vmptrld) is handled by the processor.

btw, I think the race is benign.  if __vcpu_clear() wins, vcpu_load() 
gets to set current_vmcs and all is well.  If vcpu_load() wins, 
__vcpu_clear() stomps on current_vmcs, but the only effect of that the 
next time vcpu_load() is called, it issues an unnecessary vmptrld.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: [PATCH 2/2] KVM: Protect race-condition betweenVMCS and current_vmcs on VMX hardware

[Dong, Eddie]

kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
> On Thu, 2007-07-26 at 18:03 +0300, Avi Kivity wrote:
>> Gregory Haskins wrote:
>>> We need to provide locking around the current_vmcs/VMCS
>>> interactions to protect against race conditions.
>>> 
>>> 
>> 
>> Can you explain the race?
> 
> Sure.  It can happen with two VMs are running simultaneously.
> Lets call
> them VM-a and VM-b.  Assume the scenario: VM-a is on CPU-x, gets
> migrated to CPU-y, and VM-b gets scheduled in on CPU-x.  There
> is a race
> on CPU-x with the VMCS handling logic between the VM-b process
> context, and the IPI to execute the __vcpu_clear for VM-a.

I may miss something, why does that matter? __vcpu_clear will eventually
get executed though it is a little bit delayed. vmclear will eventually
dump 
internal state of VM-a VMCS to memory and VM-b get its own VMCS 
loaded.  Here the point is vmclear has a parameter to identify which
VM's VMCS to dump, not only a memory address. Jun, please correct me if
I am wrong.

> 
> Disabling interrupts was chosen as the sync-primitive, because the
> code will always be on the CPU in question.
> 

thx, eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: [PATCH 2/2] KVM: Protect race-condition betweenVMCS and current_vmcs on VMX hardware

[Avi Kivity]

Dong, Eddie wrote:
> kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
>   
>> On Thu, 2007-07-26 at 18:03 +0300, Avi Kivity wrote:
>>     
>>> Gregory Haskins wrote:
>>>       
>>>> We need to provide locking around the current_vmcs/VMCS
>>>> interactions to protect against race conditions.
>>>>
>>>>
>>>>         
>>> Can you explain the race?
>>>       
>> Sure.  It can happen with two VMs are running simultaneously.
>> Lets call
>> them VM-a and VM-b.  Assume the scenario: VM-a is on CPU-x, gets
>> migrated to CPU-y, and VM-b gets scheduled in on CPU-x.  There
>> is a race
>> on CPU-x with the VMCS handling logic between the VM-b process
>> context, and the IPI to execute the __vcpu_clear for VM-a.
>>     
>
> I may miss something, why does that matter? __vcpu_clear will eventually
> get executed though it is a little bit delayed. vmclear will eventually
> dump 
> internal state of VM-a VMCS to memory and VM-b get its own VMCS 
> loaded.  Here the point is vmclear has a parameter to identify which
> VM's VMCS to dump, not only a memory address. Jun, please correct me if
> I am wrong.
>
>   

The vmclear instruction itself cannot race (because, as you say, the 
vmcs is a parameter).  However access to the current_vmcs variable is 
racy.  The race is benign and cannot lead to any problems, so we're not 
changing any code for that.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: [PATCH 2/2] KVM: Protect race-condition betweenVMCS and current_vmcs on VMX hardware

[Gregory Haskins]

On Tue, 2007-07-31 at 17:18 +0800, Dong, Eddie wrote:

> 
> I may miss something, why does that matter?

As it turns out, it doesn't ;)  So we have dropped the patch.  But not
for the reason you are suggesting.


>  __vcpu_clear will eventually
> get executed though it is a little bit delayed. vmclear will eventually
> dump 
> internal state of VM-a VMCS to memory and VM-b get its own VMCS 
> loaded.  Here the point is vmclear has a parameter to identify which
> VM's VMCS to dump, not only a memory address. Jun, please correct me if
> I am wrong.

The race is against per_cpu(current_vmcs), not the actual VMCS.
However, Avi pointed out that the race is benign so the race doesn't
matter.


-Greg


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/