State of current pv_ops backend for KVM?

State of current pv_ops backend for KVM?

[Anthony Liguori]

Howdy,

Does anyone know what the state of a pv_ops backend for KVM is?  I know 
Ingo has an implementation that implements CR3 caching but I don't see 
any branches in Avi's git tree.

Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as 
how the host infrastructure is there?  I'd be willing to do some leg 
work here...

Regards,

Anthony Liguori

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]


* Anthony Liguori <anthony-rdkfGonbjUSkNkDKm+mE6A@public.gmane.org> wrote:

> Howdy,
> 
> Does anyone know what the state of a pv_ops backend for KVM is?  I 
> know Ingo has an implementation that implements CR3 caching but I 
> don't see any branches in Avi's git tree.
> 
> Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as 
> how the host infrastructure is there?  I'd be willing to do some leg 
> work here...

feel free to pick up the stuff i did - i'm somewhat "distracted" by the 
CFS scheduler at the moment :) You can find it at:

   http://redhat.com/~mingo/kvm-paravirt-patches/kvm-patches.tar.gz

(it has some trivial cleanliness issues and a handful of -rt specific 
things, plus it's against an older kernel)

later on i'll join the fun too :)

	Ingo

[-- Attachment #2: kvm-patches.tar.gz --]
[-- Type: application/x-gzip, Size: 44262 bytes --]

[-- Attachment #3: Type: text/plain, Size: 286 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[-- Attachment #4: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: State of current pv_ops backend for KVM?

[Ingo Molnar]

* Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> wrote:

> feel free to pick up the stuff i did - i'm somewhat "distracted" by 
> the CFS scheduler at the moment :) You can find it at:
> 
>    http://redhat.com/~mingo/kvm-paravirt-patches/kvm-patches.tar.gz

and that's all under the GPL of course.

	Ingo

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Avi Kivity]

Anthony Liguori wrote:
> Howdy,
>
> Does anyone know what the state of a pv_ops backend for KVM is?  I 
> know Ingo has an implementation that implements CR3 caching but I 
> don't see any branches in Avi's git tree.
>
> Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as 
> how the host infrastructure is there?  I'd be willing to do some leg 
> work here... 

I think we should aim for a full direct paging implementation, where the 
cr3 cache is just the cherry on top that makes context switches proceed 
at native speed.  Jun has a paper on it for OLS and I hope patches too.

A minor drawback with direct paging is that you are more or less forced 
to use pae on the guest, but I think that's acceptable.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Anthony Liguori]

Avi Kivity wrote:
> Anthony Liguori wrote:
>> Howdy,
>>
>> Does anyone know what the state of a pv_ops backend for KVM is?  I 
>> know Ingo has an implementation that implements CR3 caching but I 
>> don't see any branches in Avi's git tree.
>>
>> Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing 
>> as how the host infrastructure is there?  I'd be willing to do some 
>> leg work here... 
>
> I think we should aim for a full direct paging implementation, where 
> the cr3 cache is just the cherry on top that makes context switches 
> proceed at native speed.  Jun has a paper on it for OLS and I hope 
> patches too.

That's a bit more ambitious than I was thinking.  I would be very happy 
with a pv timer, interrupt controller, and some basic CPU 
optimizations.  I'm still not convinced the perf benefits of direct 
paging justify the annoyances that come with it (guests being aware of 
pfns).  This is especially true with NPT/EPT right around the corner.

Regards,

Anthony Liguori

> A minor drawback with direct paging is that you are more or less 
> forced to use pae on the guest, but I think that's acceptable.
>


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Avi Kivity]

Anthony Liguori wrote:
> Avi Kivity wrote:
>> Anthony Liguori wrote:
>>> Howdy,
>>>
>>> Does anyone know what the state of a pv_ops backend for KVM is?  I 
>>> know Ingo has an implementation that implements CR3 caching but I 
>>> don't see any branches in Avi's git tree.
>>>
>>> Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing 
>>> as how the host infrastructure is there?  I'd be willing to do some 
>>> leg work here... 
>>
>> I think we should aim for a full direct paging implementation, where 
>> the cr3 cache is just the cherry on top that makes context switches 
>> proceed at native speed.  Jun has a paper on it for OLS and I hope 
>> patches too.
>
> That's a bit more ambitious than I was thinking.  I would be very 
> happy with a pv timer, interrupt controller, and some basic CPU 
> optimizations.  I'm still not convinced the perf benefits of direct 
> paging justify the annoyances that come with it (guests being aware of 
> pfns).  

I thought that with paravirt_ops, all the awareness is limited to the 
paravirt implementation, and nothing in the core kernel is actually 
aware of host pfns.

pv timer and interrupt controller make a lot of sense, especially if 
steal timer functionality is included.

> This is especially true with NPT/EPT right around the corner.

It depends on the workload.  It was advanced (by Ingo) that NPT/EPT 
increase the tlb miss latency, which is important for steady-state 
workloads, whereas shadow and direct paging do not.  It will be 
interesting to measure this.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Anthony Liguori]

Avi Kivity wrote:
> Anthony Liguori wrote:
>> Avi Kivity wrote:
>>> Anthony Liguori wrote:
>>>> Howdy,
>>>>
>>>> Does anyone know what the state of a pv_ops backend for KVM is?  I 
>>>> know Ingo has an implementation that implements CR3 caching but I 
>>>> don't see any branches in Avi's git tree.
>>>>
>>>> Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing 
>>>> as how the host infrastructure is there?  I'd be willing to do some 
>>>> leg work here... 
>>>
>>> I think we should aim for a full direct paging implementation, where 
>>> the cr3 cache is just the cherry on top that makes context switches 
>>> proceed at native speed.  Jun has a paper on it for OLS and I hope 
>>> patches too.
>>
>> That's a bit more ambitious than I was thinking.  I would be very 
>> happy with a pv timer, interrupt controller, and some basic CPU 
>> optimizations.  I'm still not convinced the perf benefits of direct 
>> paging justify the annoyances that come with it (guests being aware 
>> of pfns).  
>
> I thought that with paravirt_ops, all the awareness is limited to the 
> paravirt implementation, and nothing in the core kernel is actually 
> aware of host pfns.

The pv_ops implementation still has to deal with guest pfn => host pfn 
translation.  This means that a guest has to be put into a special mode 
for save/restore/migrate to work to ensure that no host pfn's are on the 
stack or in registers.  Changing save/restore/migration to from 
guest-transparent to guest-interactive is a big loss to me.

> pv timer and interrupt controller make a lot of sense, especially if 
> steal timer functionality is included.

Yup :-)

>> This is especially true with NPT/EPT right around the corner.
>
> It depends on the workload.  It was advanced (by Ingo) that NPT/EPT 
> increase the tlb miss latency, which is important for steady-state 
> workloads, whereas shadow and direct paging do not.  It will be 
> interesting to measure this.

There will most certainly be trade-offs, but I certainly don't expect 
direct paging to beat out NPT/EPT in certain areas.

Regards,

Anthony Liguori


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Avi Kivity]

Anthony Liguori wrote:
>>>
>>> That's a bit more ambitious than I was thinking.  I would be very 
>>> happy with a pv timer, interrupt controller, and some basic CPU 
>>> optimizations.  I'm still not convinced the perf benefits of direct 
>>> paging justify the annoyances that come with it (guests being aware 
>>> of pfns).  
>>
>> I thought that with paravirt_ops, all the awareness is limited to the 
>> paravirt implementation, and nothing in the core kernel is actually 
>> aware of host pfns.
>
> The pv_ops implementation still has to deal with guest pfn => host pfn 
> translation.  This means that a guest has to be put into a special 
> mode for save/restore/migrate to work to ensure that no host pfn's are 
> on the stack or in registers.  Changing save/restore/migration to from 
> guest-transparent to guest-interactive is a big loss to me.

I think it can be done without guest awareness, by having a special code 
section where the translation is done.  The hypervisor can choose not to 
transfer control when rip points there.

>>> This is especially true with NPT/EPT right around the corner.
>>
>> It depends on the workload.  It was advanced (by Ingo) that NPT/EPT 
>> increase the tlb miss latency, which is important for steady-state 
>> workloads, whereas shadow and direct paging do not.  It will be 
>> interesting to measure this.
>
> There will most certainly be trade-offs, but I certainly don't expect 
> direct paging to beat out NPT/EPT in certain areas.

Sure.  Like I said, we'll need measurements to decide this.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Rusty Russell]

On Mon, 2007-05-28 at 20:54 -0500, Anthony Liguori wrote:
> Howdy,
> 
> Does anyone know what the state of a pv_ops backend for KVM is?  I know 
> Ingo has an implementation that implements CR3 caching but I don't see 
> any branches in Avi's git tree.
> 
> Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as 
> how the host infrastructure is there?  I'd be willing to do some leg 
> work here...

"Me too".  It's at least worth toying with the mmu batching stuff to see
if that can shave some cycles there: it seems that cr3 caching is the
main win for context switch so batching there won't help.

IIRC Ingo used a magic MSR to detect kvm, and there was a question mark
over that.  VMI uses a different technique, which is probably worth
considering...

Cheers,
Rusty.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Rusty Russell]

On Tue, 2007-05-29 at 17:00 +0300, Avi Kivity wrote:
> Anthony Liguori wrote:
> > Avi Kivity wrote:
> >> I think we should aim for a full direct paging implementation, where 
> >> the cr3 cache is just the cherry on top that makes context switches 
> >> proceed at native speed.  Jun has a paper on it for OLS and I hope 
> >> patches too.
> >
> > That's a bit more ambitious than I was thinking.  I would be very 
> > happy with a pv timer, interrupt controller, and some basic CPU 
> > optimizations.  I'm still not convinced the perf benefits of direct 
> > paging justify the annoyances that come with it (guests being aware of 
> > pfns).  
> 
> I thought that with paravirt_ops, all the awareness is limited to the 
> paravirt implementation, and nothing in the core kernel is actually 
> aware of host pfns.

It is, but now you've broken the abstraction you get other messes.  You
can no longer simply swap out pages behind the guest's back, copy on
write or combine identical pages or other tricks without synchronizing
with the guest.

It can be done, but it's not clear to me that verifying PTEs is that
much faster than shadowing them if you're prepared to be that invasive
in the guest anyway.

Cheers,
Rusty.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Anthony Liguori]

Rusty Russell wrote:
> On Mon, 2007-05-28 at 20:54 -0500, Anthony Liguori wrote:
>   
>> Howdy,
>>
>> Does anyone know what the state of a pv_ops backend for KVM is?  I know 
>> Ingo has an implementation that implements CR3 caching but I don't see 
>> any branches in Avi's git tree.
>>
>> Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing as 
>> how the host infrastructure is there?  I'd be willing to do some leg 
>> work here...
>>     
>
> "Me too".  It's at least worth toying with the mmu batching stuff to see
> if that can shave some cycles there: it seems that cr3 caching is the
> main win for context switch so batching there won't help.
>   

I'm working on a simple patch that just adds the infrastructure.

> IIRC Ingo used a magic MSR to detect kvm, and there was a question mark
> over that.

I am not too worried about using an MSR.  There's really no fool proof 
method here.  The only thing to do IMHO is move the kvm pv_ops backend 
into a separate file and add appropriate Kconfig stuff.

>   VMI uses a different technique, which is probably worth
> considering...
>   

VMI can determine that it's under VMware since VMware has to publish an 
option ROM.  It's quite neat, but overkill for what we're doing (since 
we're not going to be using a ROM anyway).

Regards,

Anthony Liguori

> Cheers,
> Rusty.
>
>
>   


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Avi Kivity]

Rusty Russell wrote:
> On Tue, 2007-05-29 at 17:00 +0300, Avi Kivity wrote:
>   
>> Anthony Liguori wrote:
>>     
>>> Avi Kivity wrote:
>>>       
>>>> I think we should aim for a full direct paging implementation, where 
>>>> the cr3 cache is just the cherry on top that makes context switches 
>>>> proceed at native speed.  Jun has a paper on it for OLS and I hope 
>>>> patches too.
>>>>         
>>> That's a bit more ambitious than I was thinking.  I would be very 
>>> happy with a pv timer, interrupt controller, and some basic CPU 
>>> optimizations.  I'm still not convinced the perf benefits of direct 
>>> paging justify the annoyances that come with it (guests being aware of 
>>> pfns).  
>>>       
>> I thought that with paravirt_ops, all the awareness is limited to the 
>> paravirt implementation, and nothing in the core kernel is actually 
>> aware of host pfns.
>>     
>
> It is, but now you've broken the abstraction you get other messes.  You
> can no longer simply swap out pages behind the guest's back, copy on
> write or combine identical pages or other tricks without synchronizing
> with the guest.
>
>   

These are good points.

> It can be done, 

Maybe we'll get something from the s390 port.

> but it's not clear to me that verifying PTEs is that
> much faster than shadowing them if you're prepared to be that invasive
> in the guest anyway.
>   
This can probably be measured on Xen which can switch from direct to
shadow mode on runtime.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Rusty Russell]

On Wed, 2007-05-30 at 08:01 +0300, Avi Kivity wrote:
> Rusty Russell wrote:
> > but it's not clear to me that verifying PTEs is that
> > much faster than shadowing them if you're prepared to be that invasive
> > in the guest anyway.
> >   
> This can probably be measured on Xen which can switch from direct to
> shadow mode on runtime.

Sorry, I was a little oblique here.  The Xen shadow mode code is aimed
at supporting full virtualization, so it's not really a fair
comparison.  

Their direct page access code hooks deep into the guest; the outstanding
question is how efficient would a shadow implementation which made
similar guest changes be?

Unfortunately, there's a lot of coding between here and the answer 8)

Rusty.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Zachary Amsden]

Anthony Liguori wrote:
> Rusty Russell wrote:
>> On Mon, 2007-05-28 at 20:54 -0500, Anthony Liguori wrote:
>>  
>>> Howdy,
>>>
>>> Does anyone know what the state of a pv_ops backend for KVM is?  I 
>>> know Ingo has an implementation that implements CR3 caching but I 
>>> don't see any branches in Avi's git tree.
>>>
>>> Perhaps we should try for a simple pv_ops backend for 2.6.23 seeing 
>>> as how the host infrastructure is there?  I'd be willing to do some 
>>> leg work here...
>>>     
>>
>> "Me too".  It's at least worth toying with the mmu batching stuff to see
>> if that can shave some cycles there: it seems that cr3 caching is the
>> main win for context switch so batching there won't help.
>>   
>
> I'm working on a simple patch that just adds the infrastructure.
>
>> IIRC Ingo used a magic MSR to detect kvm, and there was a question mark
>> over that.
>
> I am not too worried about using an MSR.  There's really no fool proof 
> method here.  The only thing to do IMHO is move the kvm pv_ops backend 
> into a separate file and add appropriate Kconfig stuff.
>
>>   VMI uses a different technique, which is probably worth
>> considering...
>>   
>
> VMI can determine that it's under VMware since VMware has to publish 
> an option ROM.  It's quite neat, but overkill for what we're doing 
> (since we're not going to be using a ROM anyway).

The near to being blessed way to do this is to use a CPUID leaf (I 
believe it is 0x40000000) to identify the hypervisor.  I don't have the 
exact details, but making CPUID 0x40000000 return 'KVMKVMKVMKVM' sounds 
like a pretty safe detection technique.

We're probably dropping the ROM probing in favor of this for 64-bit.

Zach

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: State of current pv_ops backend for KVM?

[Anthony Liguori]

Zachary Amsden wrote:
>> VMI can determine that it's under VMware since VMware has to publish 
>> an option ROM.  It's quite neat, but overkill for what we're doing 
>> (since we're not going to be using a ROM anyway).
>
> The near to being blessed way to do this is to use a CPUID leaf (I 
> believe it is 0x40000000) to identify the hypervisor.  I don't have 
> the exact details, but making CPUID 0x40000000 return 'KVMKVMKVMKVM' 
> sounds like a pretty safe detection technique.

I apparently missed the memo :-)  I'll update the patch.

Regards,

Anthony Liguori

> We're probably dropping the ROM probing in favor of this for 64-bit.


> Zach
>


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/