Nested SVM and migration

Nested SVM and migration

[Zachary Amsden]

Perhaps I am misunderstanding, but I don't see how nested SVM instances 
can be properly migrated.  How does one extract and rebuild the nested 
hsave control block?

If it isn't done already, one possible way to add it as an extension 
might be to represent the data as additional MSRs which are saved and 
restored with migration.

Actually, looking deeper, there doesn't even appear to be any way to 
export the nested CPU data at all, meaning basic features like 
suspending and restoring the VM are not possible.  Is there any plan to 
make it work in the near future?  I'm not complaining; if my 
understanding is correct, this actually makes my current task easier.

Thanks,

Zach

Re: Nested SVM and migration

[Joerg Roedel]

On Sat, Feb 20, 2010 at 09:14:06AM -1000, Zachary Amsden wrote:
> Perhaps I am misunderstanding, but I don't see how nested SVM instances  
> can be properly migrated.  How does one extract and rebuild the nested  
> hsave control block?

Migrating guests which run in nested mode could not be migrated in a
save way currently but there are plans to fix that :-)
The first step is to save the l1 cpu state in the guest supplied hsave
area. But that is not sufficient because this does not work for all l1
state.

> If it isn't done already, one possible way to add it as an extension  
> might be to represent the data as additional MSRs which are saved and  
> restored with migration.

This sounds complicated.

> Actually, looking deeper, there doesn't even appear to be any way to  
> export the nested CPU data at all, meaning basic features like  
> suspending and restoring the VM are not possible.  Is there any plan to  
> make it work in the near future?  I'm not complaining; if my  
> understanding is correct, this actually makes my current task easier.

I think we should introduce a flag to indicate userspace if a vcpu is in
a state that could be migrated in a save way together with a way for
userspace to request that the vcpu enters a migratable state. In the
kernel we could do something like that:

nested_svm_vmrun(...)
{
	/* ... */
	kvm_migration_disable(vcpu);
	/* ... */
}

nested_svm_vmexit(...)
{
	/* ... */
	kvm_migration_enable(vcpu);
	/* ... */
}

and somewhere in the vcpu_run loop:

if (vcpu->arch.migration_win_req)
	nested_svm_vmexit(INTR);

This might be helpful in other situations too. Thoughts?

	Joerg

Re: Nested SVM and migration

[Zachary Amsden]

>
> On Sat, Feb 20, 2010 at 09:14:06AM -1000, Zachary Amsden wrote:
>    
>> Perhaps I am misunderstanding, but I don't see how nested SVM instances
>> can be properly migrated.  How does one extract and rebuild the nested
>> hsave control block?
>>      
> Migrating guests which run in nested mode could not be migrated in a
> save way currently but there are plans to fix that :-)
> The first step is to save the l1 cpu state in the guest supplied hsave
> area. But that is not sufficient because this does not work for all l1
> state.
>    

Thanks for the fast response!  I am glad to know both my reading of the 
code is correct and also that there are plans to fix it.  For now, it 
gives me freedom to fix a couple outstanding bugs and not worry about 
breaking a complex feature as nested migration through a bisectable 
patch-set.

>> If it isn't done already, one possible way to add it as an extension
>> might be to represent the data as additional MSRs which are saved and
>> restored with migration.
>>      
> This sounds complicated.
>
>    

I think it's actually pretty easy.

The infrastructure is already there to import / export and migrate MSR 
settings.  MSRs are also 64-bit, and hold "model-specific" settings, so 
if you don't mind thinking of the nested feature as a model-specific 
feature of the KVM-SVM CPU, it's even somewhat well defined in terms of 
the architecture.

In that case, the simplest approach, mapping a set of MSRs 1-1 onto the 
vmcb could be one possible implementation of a migration solution.  I 
don't think you would even need very much code; it could be simply 
blasted one qword at a time into the struct.  You would need only 
minimal error checking - most checks are done by hardware - and the only 
security concern, exposing host pages to the guest - is actually really 
a point of correctness - on migration, any physical page frames 
referenced in the hardware struct will obviously need to be reallocated 
anyway.

Mostly the problem is figuring out what chunk of MSR space to use.

Zach

Re: Nested SVM and migration

[Avi Kivity]

On 02/20/2010 10:18 PM, Joerg Roedel wrote:
>
>> Actually, looking deeper, there doesn't even appear to be any way to
>> export the nested CPU data at all, meaning basic features like
>> suspending and restoring the VM are not possible.  Is there any plan to
>> make it work in the near future?  I'm not complaining; if my
>> understanding is correct, this actually makes my current task easier.
>>      
> I think we should introduce a flag to indicate userspace if a vcpu is in
> a state that could be migrated in a save way together with a way for
> userspace to request that the vcpu enters a migratable state. In the
> kernel we could do something like that:
>
> nested_svm_vmrun(...)
> {
> 	/* ... */
> 	kvm_migration_disable(vcpu);
> 	/* ... */
> }
>
> nested_svm_vmexit(...)
> {
> 	/* ... */
> 	kvm_migration_enable(vcpu);
> 	/* ... */
> }
>
> and somewhere in the vcpu_run loop:
>
> if (vcpu->arch.migration_win_req)
> 	nested_svm_vmexit(INTR);
>
> This might be helpful in other situations too. Thoughts?
>    

This doesn't work if the guest disables INTR intercepts, or if the guest 
checks that an interrupt was actually received.  Of course no sane guest 
does this.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Nested SVM and migration

[Gleb Natapov]

On Sun, Feb 21, 2010 at 09:23:40AM +0200, Avi Kivity wrote:
> On 02/20/2010 10:18 PM, Joerg Roedel wrote:
> >
> >>Actually, looking deeper, there doesn't even appear to be any way to
> >>export the nested CPU data at all, meaning basic features like
> >>suspending and restoring the VM are not possible.  Is there any plan to
> >>make it work in the near future?  I'm not complaining; if my
> >>understanding is correct, this actually makes my current task easier.
> >I think we should introduce a flag to indicate userspace if a vcpu is in
> >a state that could be migrated in a save way together with a way for
> >userspace to request that the vcpu enters a migratable state. In the
> >kernel we could do something like that:
> >
> >nested_svm_vmrun(...)
> >{
> >	/* ... */
> >	kvm_migration_disable(vcpu);
> >	/* ... */
> >}
> >
> >nested_svm_vmexit(...)
> >{
> >	/* ... */
> >	kvm_migration_enable(vcpu);
> >	/* ... */
> >}
> >
> >and somewhere in the vcpu_run loop:
> >
> >if (vcpu->arch.migration_win_req)
> >	nested_svm_vmexit(INTR);
> >
> >This might be helpful in other situations too. Thoughts?
> 
> This doesn't work if the guest disables INTR intercepts, or if the
> guest checks that an interrupt was actually received.  Of course no
> sane guest does this.
> 
Malicious guest may do this on purpose just to prevent migration.
Relying on vcpu state controllable by a guest for migration is not a
good idea.

--
			Gleb.

Re: Nested SVM and migration

[Avi Kivity]

On 02/21/2010 09:46 AM, Gleb Natapov wrote:
> On Sun, Feb 21, 2010 at 09:23:40AM +0200, Avi Kivity wrote:
>    
>> On 02/20/2010 10:18 PM, Joerg Roedel wrote:
>>      
>>>        
>>>> Actually, looking deeper, there doesn't even appear to be any way to
>>>> export the nested CPU data at all, meaning basic features like
>>>> suspending and restoring the VM are not possible.  Is there any plan to
>>>> make it work in the near future?  I'm not complaining; if my
>>>> understanding is correct, this actually makes my current task easier.
>>>>          
>>> I think we should introduce a flag to indicate userspace if a vcpu is in
>>> a state that could be migrated in a save way together with a way for
>>> userspace to request that the vcpu enters a migratable state. In the
>>> kernel we could do something like that:
>>>
>>> nested_svm_vmrun(...)
>>> {
>>> 	/* ... */
>>> 	kvm_migration_disable(vcpu);
>>> 	/* ... */
>>> }
>>>
>>> nested_svm_vmexit(...)
>>> {
>>> 	/* ... */
>>> 	kvm_migration_enable(vcpu);
>>> 	/* ... */
>>> }
>>>
>>> and somewhere in the vcpu_run loop:
>>>
>>> if (vcpu->arch.migration_win_req)
>>> 	nested_svm_vmexit(INTR);
>>>
>>> This might be helpful in other situations too. Thoughts?
>>>        
>> This doesn't work if the guest disables INTR intercepts, or if the
>> guest checks that an interrupt was actually received.  Of course no
>> sane guest does this.
>>
>>      
> Malicious guest may do this on purpose just to prevent migration.
>    

We #vmexit unconditionally; the malicious guest will break if it doesn't 
handle INTR intercepts even when it did not enable them, but migration 
will succeed.

> Relying on vcpu state controllable by a guest for migration is not a
> good idea.
>    

I agree, but that is not the case here.  Regardless, I'd prefer to 
migrate the guest in its original state (and not force a #vmexit), so 
long as it isn't horribly complicated.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Nested SVM and migration

[Joerg Roedel]

On Sat, Feb 20, 2010 at 01:26:49PM -1000, Zachary Amsden wrote:
> The infrastructure is already there to import / export and migrate MSR  
> settings.  MSRs are also 64-bit, and hold "model-specific" settings, so  
> if you don't mind thinking of the nested feature as a model-specific  
> feature of the KVM-SVM CPU, it's even somewhat well defined in terms of  
> the architecture.

There is a lot of additional state to migrate if the vcpu is running
nested. To be architecturally correct you need to transfer 6kb of data
through MSRs only for the msr permission bitmap. The rest comes down to
the nested intercept masks and some small bits like the global interrupt
flag and the nested vmcb address. It is doable but I still think its
complicated to get this right. The simplest approach would be to
disallow migration when the vcpu is running in guest mode.

> Mostly the problem is figuring out what chunk of MSR space to use.

And hoping that this MSR space is not used by real hardware in the
future ;-)

	Joerg

Re: Nested SVM and migration

[Joerg Roedel]

On Sun, Feb 21, 2010 at 09:23:40AM +0200, Avi Kivity wrote:
> On 02/20/2010 10:18 PM, Joerg Roedel wrote:
>> I think we should introduce a flag to indicate userspace if a vcpu is in
>> a state that could be migrated in a save way together with a way for
>> userspace to request that the vcpu enters a migratable state. In the
>> kernel we could do something like that:
>>
>> nested_svm_vmrun(...)
>> {
>> 	/* ... */
>> 	kvm_migration_disable(vcpu);
>> 	/* ... */
>> }
>>
>> nested_svm_vmexit(...)
>> {
>> 	/* ... */
>> 	kvm_migration_enable(vcpu);
>> 	/* ... */
>> }
>>
>> and somewhere in the vcpu_run loop:
>>
>> if (vcpu->arch.migration_win_req)
>> 	nested_svm_vmexit(INTR);
>>
>> This might be helpful in other situations too. Thoughts?
>>    
>
> This doesn't work if the guest disables INTR intercepts, or if the guest  
> checks that an interrupt was actually received.  Of course no sane guest  
> does this.

We could just wait for an intercept if the guest does not intercept INTR
(which is unlikely). Problem is that this might allow the guest to
protect itself from migration. I'll check if there is another intercept
which could be used here.

	Joerg

Re: Nested SVM and migration

[Avi Kivity]

On 02/21/2010 02:10 PM, Joerg Roedel wrote:
> On Sat, Feb 20, 2010 at 01:26:49PM -1000, Zachary Amsden wrote:
>    
>> The infrastructure is already there to import / export and migrate MSR
>> settings.  MSRs are also 64-bit, and hold "model-specific" settings, so
>> if you don't mind thinking of the nested feature as a model-specific
>> feature of the KVM-SVM CPU, it's even somewhat well defined in terms of
>> the architecture.
>>      
> There is a lot of additional state to migrate if the vcpu is running
> nested. To be architecturally correct you need to transfer 6kb of data
> through MSRs only for the msr permission bitmap.

The msr permission bitmap is in guest memory, so it is already migrated.

> The rest comes down to
> the nested intercept masks

These are in the vmcb, which is in guest memory.

> and some small bits like the global interrupt
> flag and the nested vmcb address.

We need to migrate GIF regardless, it is toggled outside guest mode.

> It is doable but I still think its
> complicated to get this right. The simplest approach would be to
> disallow migration when the vcpu is running in guest mode.
>    

Agree, though I dislike the need to introduce a "force vmexit" ioctl.

-- 

error compiling committee.c: too many arguments to function

Re: Nested SVM and migration

[Joerg Roedel]

On Sun, Feb 21, 2010 at 02:24:02PM +0200, Avi Kivity wrote:
> On 02/21/2010 02:10 PM, Joerg Roedel wrote:
>> On Sat, Feb 20, 2010 at 01:26:49PM -1000, Zachary Amsden wrote:
>>    
>>> The infrastructure is already there to import / export and migrate MSR
>>> settings.  MSRs are also 64-bit, and hold "model-specific" settings, so
>>> if you don't mind thinking of the nested feature as a model-specific
>>> feature of the KVM-SVM CPU, it's even somewhat well defined in terms of
>>> the architecture.
>>>      
>> There is a lot of additional state to migrate if the vcpu is running
>> nested. To be architecturally correct you need to transfer 6kb of data
>> through MSRs only for the msr permission bitmap.
>
> The msr permission bitmap is in guest memory, so it is already migrated.

This will work almost always but its not architecturally correct
because the memory contents may have changed since the last vmrun
instruction. On the other hand we already have this problem with the
current nested msr intercept handling...

>> The rest comes down to
>> the nested intercept masks
>
> These are in the vmcb, which is in guest memory.

Same as with the MSR permission map here.

>> It is doable but I still think its
>> complicated to get this right. The simplest approach would be to
>> disallow migration when the vcpu is running in guest mode.
>>    
>
> Agree, though I dislike the need to introduce a "force vmexit" ioctl.

Yes, this has possible issues too. If we reconstruct the nested state from
the nested vmcb there is not much state left which needs migration. But
we should keep in mind that this is not how real hardware works.

	Joerg

Re: Nested SVM and migration

[Avi Kivity]

On 02/21/2010 02:41 PM, Joerg Roedel wrote:
> On Sun, Feb 21, 2010 at 02:24:02PM +0200, Avi Kivity wrote:
>    
>> On 02/21/2010 02:10 PM, Joerg Roedel wrote:
>>      
>>> On Sat, Feb 20, 2010 at 01:26:49PM -1000, Zachary Amsden wrote:
>>>
>>>        
>>>> The infrastructure is already there to import / export and migrate MSR
>>>> settings.  MSRs are also 64-bit, and hold "model-specific" settings, so
>>>> if you don't mind thinking of the nested feature as a model-specific
>>>> feature of the KVM-SVM CPU, it's even somewhat well defined in terms of
>>>> the architecture.
>>>>
>>>>          
>>> There is a lot of additional state to migrate if the vcpu is running
>>> nested. To be architecturally correct you need to transfer 6kb of data
>>> through MSRs only for the msr permission bitmap.
>>>        
>> The msr permission bitmap is in guest memory, so it is already migrated.
>>      
> This will work almost always but its not architecturally correct
> because the memory contents may have changed since the last vmrun
> instruction. On the other hand we already have this problem with the
> current nested msr intercept handling...
>    

So, if some other cpu (or the guest itself, with appropriate 
permissions) modifies the msr permission bitmap, svm will not notice 
this?  svm loads the bitmap during entry?

>>> The rest comes down to
>>> the nested intercept masks
>>>        
>> These are in the vmcb, which is in guest memory.
>>      
> Same as with the MSR permission map here.
>
>    

Yes (as with the msr permission bitmap pointers).

>>> It is doable but I still think its
>>> complicated to get this right. The simplest approach would be to
>>> disallow migration when the vcpu is running in guest mode.
>>>
>>>        
>> Agree, though I dislike the need to introduce a "force vmexit" ioctl.
>>      
> Yes, this has possible issues too. If we reconstruct the nested state from
> the nested vmcb there is not much state left which needs migration. But
> we should keep in mind that this is not how real hardware works.
>    

I don't think you can tell, unless the host cpu modifying the vmcb is 
synchronized with the guest (or the guest modifies its own vmcb).  But 
this is all academic.

-- 
error compiling committee.c: too many arguments to function

Re: Nested SVM and migration

[Joerg Roedel]

On Sun, Feb 21, 2010 at 02:54:01PM +0200, Avi Kivity wrote:
> So, if some other cpu (or the guest itself, with appropriate  
> permissions) modifies the msr permission bitmap, svm will not notice  
> this?  svm loads the bitmap during entry?

Yes.

> I don't think you can tell, unless the host cpu modifying the vmcb is  
> synchronized with the guest (or the guest modifies its own vmcb).  But  
> this is all academic.

Hmm, another thing comes to mind. We would need some redesign of the
nested_svm code to allow userspace to put a vcpu directly into nested
state. With the MSR approach, all userspace does is to write MSRs into
the vcpu before the first run?

	Joerg

Re: Nested SVM and migration

[Avi Kivity]

On 02/21/2010 03:09 PM, Joerg Roedel wrote:
>
>> I don't think you can tell, unless the host cpu modifying the vmcb is
>> synchronized with the guest (or the guest modifies its own vmcb).  But
>> this is all academic.
>>      
> Hmm, another thing comes to mind. We would need some redesign of the
> nested_svm code to allow userspace to put a vcpu directly into nested
> state. With the MSR approach, all userspace does is to write MSRs into
> the vcpu before the first run?
>    

(Either synthetic msrs, or an new state ioctl).  The state would contain 
a bit that says whether the guest is in guest or host mode.

However, since we're breaking the architecture one way or another, let's 
just go with the synthetic INTR intercept.

-- 
error compiling committee.c: too many arguments to function

Re: Nested SVM and migration

[Zachary Amsden]

On 02/21/2010 02:10 AM, Joerg Roedel wrote:
> On Sat, Feb 20, 2010 at 01:26:49PM -1000, Zachary Amsden wrote:
>    
>> The infrastructure is already there to import / export and migrate MSR
>> settings.  MSRs are also 64-bit, and hold "model-specific" settings, so
>> if you don't mind thinking of the nested feature as a model-specific
>> feature of the KVM-SVM CPU, it's even somewhat well defined in terms of
>> the architecture.
>>      
> There is a lot of additional state to migrate if the vcpu is running
> nested. To be architecturally correct you need to transfer 6kb of data
> through MSRs only for the msr permission bitmap. The rest comes down to
> the nested intercept masks and some small bits like the global interrupt
> flag and the nested vmcb address. It is doable but I still think its
> complicated to get this right. The simplest approach would be to
> disallow migration when the vcpu is running in guest mode.
>    

How is this a lot?  The guest may have multiple gigabytes of memory and 
potentially terabytes of storage.  An additional 4-8k of control state 
is not a lot.

>    
>> Mostly the problem is figuring out what chunk of MSR space to use.
>>      
> And hoping that this MSR space is not used by real hardware in the
> future ;-)
>    

There are places in MSR space that can easily be set aside for this, in 
fact, I think already are being used by Hyper-V.

And further, even if it is used by real hardware, how does it matter?  
These MSRs are to be used by control software to import / export data to 
the virtual CPU.  When being accessed by hardware virtualization, they 
can be re-aliased to any meaning at all, so there is no collision.

Zach

Re: Nested SVM and migration

[Zachary Amsden]

On 02/21/2010 02:24 AM, Avi Kivity wrote:
> On 02/21/2010 02:10 PM, Joerg Roedel wrote:
>
>> It is doable but I still think its
>> complicated to get this right. The simplest approach would be to
>> disallow migration when the vcpu is running in guest mode.
>
> Agree, though I dislike the need to introduce a "force vmexit" ioctl.
>

How can this possibly work for guests which never exit SVM?  They can 
never be migrated.

Zach

Re: Nested SVM and migration

[Avi Kivity]

On 02/22/2010 06:42 PM, Zachary Amsden wrote:
> On 02/21/2010 02:24 AM, Avi Kivity wrote:
>> On 02/21/2010 02:10 PM, Joerg Roedel wrote:
>>
>>> It is doable but I still think its
>>> complicated to get this right. The simplest approach would be to
>>> disallow migration when the vcpu is running in guest mode.
>>
>> Agree, though I dislike the need to introduce a "force vmexit" ioctl.
>>
>
> How can this possibly work for guests which never exit SVM?  They can 
> never be migrated.

The force vmexit would generate an INTR #vmexit even if the INTR 
intercept was disabled and even if no INTR is pending.  However this was 
shot down since there was no equivalent vmx exit reason that we can 
except the guest to reasonably handle.

-- 
error compiling committee.c: too many arguments to function

Re: Nested SVM and migration

[Zachary Amsden]

On 02/21/2010 03:09 AM, Joerg Roedel wrote:
> On Sun, Feb 21, 2010 at 02:54:01PM +0200, Avi Kivity wrote:
>    
>> So, if some other cpu (or the guest itself, with appropriate
>> permissions) modifies the msr permission bitmap, svm will not notice
>> this?  svm loads the bitmap during entry?
>>      
> Yes.
>    

Ugh.  So we have non-reversible architectural state all over again.  
There are ways around this problem, all ugly, but the easiest is 
shadowing the MSR permission bitmap.

>> I don't think you can tell, unless the host cpu modifying the vmcb is
>> synchronized with the guest (or the guest modifies its own vmcb).  But
>> this is all academic.
>>      
> Hmm, another thing comes to mind. We would need some redesign of the
> nested_svm code to allow userspace to put a vcpu directly into nested
> state. With the MSR approach, all userspace does is to write MSRs into
> the vcpu before the first run?
>    

How does MSR_KVM_NESTED_SVM_ACTIVE not solve this problem?

Re: Nested SVM and migration

[Zachary Amsden]

On 02/21/2010 04:43 AM, Joerg Roedel wrote:
> On Sun, Feb 21, 2010 at 03:40:55PM +0200, Avi Kivity wrote:
>    
>> On 02/21/2010 03:14 PM, Avi Kivity wrote:
>>      
>>> (Either synthetic msrs, or an new state ioctl).  The state would
>>> contain a bit that says whether the guest is in guest or host mode.
>>>
>>> However, since we're breaking the architecture one way or another,
>>> let's just go with the synthetic INTR intercept.
>>>
>>>        
>> On the other hand, there is no equivalent intercept in vmx.  The
>> "external interrupt" exit can be configured to run an INTACK cycle and
>> capture the vector in a vmcs field, and there's no vector we can insert
>> there (Xen for example uses this).
>>      
> Difficult. We could use an instruction intercept which has no side
> effect on guest state (invlpg for example). But thats a lot more
> dangerous than an INTR intercept. What about PENDING_INTERRUPT? Are
> there hypervisors that may get confused getting this intercept without
> asking for it?
>    

Yes, hypervisors which never mean to handle exits will break.  There are 
many possible reasons you might design such a hypervisor; security, 
non-trivial software lock-out of SVM, special hardware concern, research 
project, or perhaps just a shell hypervisor which has one module capable 
of handling exits, and one module which doesn't require it, used for 
high performance but still maintaining binary compatibility with the 
underlying OS.

Perhaps it is just a guest bug, but any design which disallows this is 
visibly and fundamentally architecturally broken.

On the other hand, architectural effects of a new state ioctl or MSR 
update which is properly implemented are completely invisible to the 
guest.  I don't have a strong preference which way that is done, but I 
do have a very strong preference not to break the architecture.

Zach

Re: Nested SVM and migration

[Zachary Amsden]

On 02/21/2010 05:56 AM, Joerg Roedel wrote:
> On Sun, Feb 21, 2010 at 05:07:45PM +0200, Avi Kivity wrote:
>    
>> On 02/21/2010 04:43 PM, Joerg Roedel wrote:
>>      
>>> Difficult. We could use an instruction intercept which has no side
>>> effect on guest state (invlpg for example).
>>>        
>> Especially as the guest might disable it and BUG() if npt is enabled.
>>
>>      
>>> But thats a lot more
>>> dangerous than an INTR intercept. What about PENDING_INTERRUPT? Are
>>> there hypervisors that may get confused getting this intercept without
>>> asking for it?
>>>
>>>        
>> That will likely confuse kvm, it means interrupts are not blocked so it
>> is okay to inject one, and if the guest is running with interrupts
>> disabled then the next entry will fail.
>>      
> Completly different idea:
>
> 1. Give userspace an ioctl to freeze the complete VM (all vcpus must be
>     stopped afterwards)
> 2. The freeze routine does the following:
>
> freeze_vm(...)
> {
> 	stop_all_vcpus();
> 	for_each_vcpu(vcpu)
> 		kvm_x86_ops->freeze_vcpu(vcpu);
> }
>
> The module specific freeze_vcpu function can save the nested state
> inside the guests memory (vmcs area for nested-vmx and hsave area for
> nested-svm). When no vcpu is running anymore this is save.
>
> After migration the information is restored from there before any vcpu
> is started again.
>
> unfreeze_vm(...)
> {
> 	for_each_vcpu(vcpu)
> 		kvm_x86_ops->unfreeze_vcpu(vcpu);
> 	restart_all_vcpus();
> }
>    

This is exactly the ioctl approach, yes?

> In the SVM case this still leaves the problem that the MSR bitmap must
> be read again from guests memory on unfreeze but that is not a real
> problem.
>    

This is easily solved by shadowing the MSR bitmap.  Too bad we have to 
do it, but at least the state isn't hidden in a non-extractable field 
inside the processor.

Zach

Re: Nested SVM and migration

[Avi Kivity]

On 02/22/2010 06:56 PM, Zachary Amsden wrote:
>
>> In the SVM case this still leaves the problem that the MSR bitmap must
>> be read again from guests memory on unfreeze but that is not a real
>> problem.
>
> This is easily solved by shadowing the MSR bitmap.  Too bad we have to 
> do it, but at least the state isn't hidden in a non-extractable field 
> inside the processor.
>

We could write-protect the bitmaps and shadow them lazily when changed 
(which should be never; also improves vmentry time).  Write protecting 
is bad since it breaks up large pages, but is necessary for exposing npt 
to the guest, which is needed for reasonable performance, so we may as 
well do it.

-- 
error compiling committee.c: too many arguments to function

Re: Nested SVM and migration

[Zachary Amsden]

On 02/22/2010 06:44 AM, Avi Kivity wrote:
> On 02/22/2010 06:42 PM, Zachary Amsden wrote:
>> On 02/21/2010 02:24 AM, Avi Kivity wrote:
>>> On 02/21/2010 02:10 PM, Joerg Roedel wrote:
>>>
>>>> It is doable but I still think its
>>>> complicated to get this right. The simplest approach would be to
>>>> disallow migration when the vcpu is running in guest mode.
>>>
>>> Agree, though I dislike the need to introduce a "force vmexit" ioctl.
>>>
>>
>> How can this possibly work for guests which never exit SVM?  They can 
>> never be migrated.
>
> The force vmexit would generate an INTR #vmexit even if the INTR 
> intercept was disabled and even if no INTR is pending.  However this 
> was shot down since there was no equivalent vmx exit reason that we 
> can except the guest to reasonably handle.

While true, my point is more precisely - how can this possibly work for 
guests which MUST never exit SVM?  As in, the hypervisor is broken or 
deliberately disabled from taking exits, and in fact, may no longer even 
exist in memory?

Zach

Re: Nested SVM and migration

[Avi Kivity]

On 02/22/2010 07:00 PM, Zachary Amsden wrote:
>> The force vmexit would generate an INTR #vmexit even if the INTR 
>> intercept was disabled and even if no INTR is pending.  However this 
>> was shot down since there was no equivalent vmx exit reason that we 
>> can except the guest to reasonably handle.
>
>
> While true, my point is more precisely - how can this possibly work 
> for guests which MUST never exit SVM?  As in, the hypervisor is broken 
> or deliberately disabled from taking exits, and in fact, may no longer 
> even exist in memory?

These guests will be broken.  My assumption was that only malicious 
guests will disable INTR intercepts (though I can imagine a 
Luvalley-like system that disables INTR intercepts when running dom0).

-- 
error compiling committee.c: too many arguments to function

Re: Nested SVM and migration

[Joerg Roedel]

On Mon, Feb 22, 2010 at 06:46:19AM -1000, Zachary Amsden wrote:
> On 02/21/2010 03:09 AM, Joerg Roedel wrote:
> >On Sun, Feb 21, 2010 at 02:54:01PM +0200, Avi Kivity wrote:
> >>So, if some other cpu (or the guest itself, with appropriate
> >>permissions) modifies the msr permission bitmap, svm will not notice
> >>this?  svm loads the bitmap during entry?
> >Yes.
> 
> Ugh.  So we have non-reversible architectural state all over again.
> There are ways around this problem, all ugly, but the easiest is
> shadowing the MSR permission bitmap.
> 
> >>I don't think you can tell, unless the host cpu modifying the vmcb is
> >>synchronized with the guest (or the guest modifies its own vmcb).  But
> >>this is all academic.
> >Hmm, another thing comes to mind. We would need some redesign of the
> >nested_svm code to allow userspace to put a vcpu directly into nested
> >state. With the MSR approach, all userspace does is to write MSRs into
> >the vcpu before the first run?
> 
> How does MSR_KVM_NESTED_SVM_ACTIVE not solve this problem?

Image migration from host1 -> host2

When you want to put a vcpu directly into nested state you need to
recalculate certain stuff like the intercept bitmaps (intercept bitmaps
from host1 and host2 might be different) or the tsc_offset. But this can all
be done in the vcpu_unfreeze ioctl.

	Joerg

Re: Nested SVM and migration

[Zachary Amsden]

On 02/22/2010 07:02 AM, Avi Kivity wrote:
> On 02/22/2010 07:00 PM, Zachary Amsden wrote:
>>> The force vmexit would generate an INTR #vmexit even if the INTR 
>>> intercept was disabled and even if no INTR is pending.  However this 
>>> was shot down since there was no equivalent vmx exit reason that we 
>>> can except the guest to reasonably handle.
>>
>>
>> While true, my point is more precisely - how can this possibly work 
>> for guests which MUST never exit SVM?  As in, the hypervisor is 
>> broken or deliberately disabled from taking exits, and in fact, may 
>> no longer even exist in memory?
>
> These guests will be broken.  My assumption was that only malicious 
> guests will disable INTR intercepts (though I can imagine a 
> Luvalley-like system that disables INTR intercepts when running dom0).

Not an SVM expert, but can't you pass through INTR in SVM and leave a 
fully functioning guest which technically runs under SVM but requires no 
hypervisor?  Is that what the Luvalley system does?

Zach

Re: Nested SVM and migration

[Avi Kivity]

On 02/22/2010 07:07 PM, Zachary Amsden wrote:
> On 02/22/2010 07:02 AM, Avi Kivity wrote:
>> On 02/22/2010 07:00 PM, Zachary Amsden wrote:
>>>> The force vmexit would generate an INTR #vmexit even if the INTR 
>>>> intercept was disabled and even if no INTR is pending.  However 
>>>> this was shot down since there was no equivalent vmx exit reason 
>>>> that we can except the guest to reasonably handle.
>>>
>>>
>>> While true, my point is more precisely - how can this possibly work 
>>> for guests which MUST never exit SVM?  As in, the hypervisor is 
>>> broken or deliberately disabled from taking exits, and in fact, may 
>>> no longer even exist in memory?
>>
>> These guests will be broken.  My assumption was that only malicious 
>> guests will disable INTR intercepts (though I can imagine a 
>> Luvalley-like system that disables INTR intercepts when running dom0).
>
> Not an SVM expert, but can't you pass through INTR in SVM and leave a 
> fully functioning guest which technically runs under SVM but requires 
> no hypervisor? 

You could, but without trapping INTR, you can't reliably multiplex 
guests or have a hypervisor-controlled network interface.  That means 
you're likely a blue pill thing.

> Is that what the Luvalley system does?

Luvalley is vmx only at the moment, but it certainly could let its dom0 
handle interrupts (since the scheduler and all device drivers are in 
dom0).  Once it switches to a different guest, it needs to enable INTR.

-- 
error compiling committee.c: too many arguments to function

Re: Nested SVM and migration

[Zachary Amsden]

On 02/22/2010 07:11 AM, Avi Kivity wrote:
> On 02/22/2010 07:07 PM, Zachary Amsden wrote:
>> On 02/22/2010 07:02 AM, Avi Kivity wrote:
>>> On 02/22/2010 07:00 PM, Zachary Amsden wrote:
>>>>> The force vmexit would generate an INTR #vmexit even if the INTR 
>>>>> intercept was disabled and even if no INTR is pending.  However 
>>>>> this was shot down since there was no equivalent vmx exit reason 
>>>>> that we can except the guest to reasonably handle.
>>>>
>>>>
>>>> While true, my point is more precisely - how can this possibly work 
>>>> for guests which MUST never exit SVM?  As in, the hypervisor is 
>>>> broken or deliberately disabled from taking exits, and in fact, may 
>>>> no longer even exist in memory?
>>>
>>> These guests will be broken.  My assumption was that only malicious 
>>> guests will disable INTR intercepts (though I can imagine a 
>>> Luvalley-like system that disables INTR intercepts when running dom0).
>>
>> Not an SVM expert, but can't you pass through INTR in SVM and leave a 
>> fully functioning guest which technically runs under SVM but requires 
>> no hypervisor? 
>
> You could, but without trapping INTR, you can't reliably multiplex 
> guests or have a hypervisor-controlled network interface.  That means 
> you're likely a blue pill thing.

Not necessarily; you could be a very subversive BIOS.  You only 
intercept #UD instruction and emulate SSE3 instructions in software.  
Your control structure you mark unavailable as reserved BIOS memory and 
you pass on interrupts and all exceptions to the booted OS.

You then implement nested VMRUN so as not to lock the OS out of the 
hardware SVM acceleration..

Quite reasonable actually, and not a blue pill.  Not 100% secure and it 
doesn't need to be, but it is 100% correct for a guest which obeys the 
standard reasonable rules of not messing with BIOS reserved memory.

>
>> Is that what the Luvalley system does?
>
> Luvalley is vmx only at the moment, but it certainly could let its 
> dom0 handle interrupts (since the scheduler and all device drivers are 
> in dom0).  Once it switches to a different guest, it needs to enable 
> INTR.

I checked it out, interesting stuff.

Zach

Re: Nested SVM and migration

[Joerg Roedel]

On Sun, Feb 21, 2010 at 02:09:15PM +0100, Joerg Roedel wrote:
> On Sun, Feb 21, 2010 at 02:54:01PM +0200, Avi Kivity wrote:
> > So, if some other cpu (or the guest itself, with appropriate  
> > permissions) modifies the msr permission bitmap, svm will not notice  
> > this?  svm loads the bitmap during entry?
> 
> Yes.

Talked again with a few people about it and the architectural answer is
a bit different. The right answer to that is:

The cpu is allowed to load the msr permission map to internal state, but
not required to do it by the architecture. The result of changing the
msr permission map of a running vmcb is undefined.

So we are fine with the way nested svm does it. And we don't need to
care about that situation at all and just migrate whats in guest memory.

Thanks,

	Joerg

Re: Nested SVM and migration

[Zachary Amsden]

On 02/24/2010 05:23 AM, Joerg Roedel wrote:
> On Sun, Feb 21, 2010 at 02:09:15PM +0100, Joerg Roedel wrote:
>    
>> On Sun, Feb 21, 2010 at 02:54:01PM +0200, Avi Kivity wrote:
>>      
>>> So, if some other cpu (or the guest itself, with appropriate
>>> permissions) modifies the msr permission bitmap, svm will not notice
>>> this?  svm loads the bitmap during entry?
>>>        
>> Yes.
>>      
> Talked again with a few people about it and the architectural answer is
> a bit different. The right answer to that is:
>
> The cpu is allowed to load the msr permission map to internal state, but
> not required to do it by the architecture. The result of changing the
> msr permission map of a running vmcb is undefined.
>
> So we are fine with the way nested svm does it. And we don't need to
> care about that situation at all and just migrate whats in guest memory.
>    

Awesome!  Thanks for checking it out.