Paravirt KVM capabilities

Paravirt KVM capabilities

[Hugo Mills]

[-- Attachment #1.1: Type: text/plain, Size: 747 bytes --]

   I'm a little unclear on the capabilities (current or planned) of
the new paravirtualisation feature in KVM.

   Does running paravirtual guests on KVM still require a
hardware-VM-capable processor, or will it eventually be possible to
run a paravirtualised Linux on an older machine (such as, say, a
first-generation Athlon 64)?

   As a follow-up question, would it be possible then to run a 32-bit
guest on a 64-bit host?

   Thanks,
   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 1C335860 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
  --- Anyone who says their system is completely secure understands ---  
                      neither systems nor security.                      

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: Paravirt KVM capabilities

[Avi Kivity]

Hugo Mills wrote:
>    I'm a little unclear on the capabilities (current or planned) of
> the new paravirtualisation feature in KVM.
>
>    Does running paravirtual guests on KVM still require a
> hardware-VM-capable processor, or will it eventually be possible to
> run a paravirtualised Linux on an older machine (such as, say, a
> first-generation Athlon 64)?
>   

If the code comes in to support it, and if it doesn't hurt full hardware 
virtualization, kvm can be extended to support it.  I have no plans to 
work on it myself but will review and accept patches.

It's also something for the Linux community in general to decide:  if we 
want separate interfaces for paravirtualization and full virtualization 
(lhype and kvm) or a merged interface.  I can see arguments in favor of 
both positions.

>    As a follow-up question, would it be possible then to run a 32-bit
> guest on a 64-bit host?

That is possible now, and hopefully will continue to be possible.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Rusty Russell]

On Tue, 2007-01-09 at 16:27 +0200, Avi Kivity wrote:
> Hugo Mills wrote:
> >    Does running paravirtual guests on KVM still require a
> > hardware-VM-capable processor, or will it eventually be possible to
> > run a paravirtualised Linux on an older machine (such as, say, a
> > first-generation Athlon 64)?
>
> It's also something for the Linux community in general to decide:  if we 
> want separate interfaces for paravirtualization and full virtualization 
> (lhype and kvm) or a merged interface.  I can see arguments in favor of 
> both positions.

Yeah, me too.

	Ignoring technical questions for a moment, the main issue is that lhype
doesn't have an ABI; you have to run the same(ish) kernel as guest and
host.  This is by design: I wanted it to be a platform for
experimentation and hacking, because I think we're still at that stage
with x86 Linux virtualization.

	That's not to say that I don't think we can share code, it's just
important to realize that our aims are different.

Cheers,
Rusty.



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Ingo Molnar]

* Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org> wrote:

> > It's also something for the Linux community in general to decide: if 
> > we want separate interfaces for paravirtualization and full 
> > virtualization (lhype and kvm) or a merged interface.  I can see 
> > arguments in favor of both positions.
> 
> Yeah, me too.
> 
> 	Ignoring technical questions for a moment, the main issue is 
> that lhype doesn't have an ABI; you have to run the same(ish) kernel 
> as guest and host.  This is by design: I wanted it to be a platform 
> for experimentation and hacking, because I think we're still at that 
> stage with x86 Linux virtualization.

i really really think KVM and lhype should merge, creating KVM/HVM 
(Hardware Virtual Machine) and KVM/LL (Linux on Linux). But that's 
really your business to decide, not mine :-) I guess it's not a pure 
technical decision :-/

( later on KVM/LL could be extended to also run unmodified guests 
  similar to how kqemu does it: by running guest user-space on ring 3 
  and letting Qemu do all of ring0 emulation. That is already leagues 
  faster than default Qemu, and it would also nicely test lhype's 
  cr3/pagetable management infrastructure. It can also run most 32-bit 
  OSs out there - largely extending the userbase even on legacy 
  hardware. )

regarding ABI: agreed that it's experimental right now and should stay 
so for some time, but i dont see a reason why the hypercall API that 
i've posted in the past few days couldnt be evolved in the way Linux 
syscalls evolve, by only adding to them. That doesnt really impact the 
flexibility of the virtualization subsystem. We could also possibly 
phase out older hypercalls, because the number of "applications" relying 
on it will be very low.

	Ingo

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Ulrich Drepper]

[-- Attachment #1.1: Type: text/plain, Size: 646 bytes --]

Ingo Molnar wrote:
> i really really think KVM and lhype should merge, creating KVM/HVM 
> (Hardware Virtual Machine) and KVM/LL (Linux on Linux).

This is only sufficient if either KVM with paravirt Linux kernels has no
performance penalty or lhype becomes able to execute paravirt Linux kernels.

I can certainly attest that there is a lot of demand for running Linux
domains with kernels other than the one running on the hardware.  This
is Xen's bread and butter and from the albeit old numbers I've seen so
far KVM has problems competing.

-- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: Paravirt KVM capabilities

[Ingo Molnar]

* Ulrich Drepper <drepper-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> Ingo Molnar wrote:
> > i really really think KVM and lhype should merge, creating KVM/HVM 
> > (Hardware Virtual Machine) and KVM/LL (Linux on Linux).
> 
> This is only sufficient if either KVM with paravirt Linux kernels has 
> no performance penalty or lhype becomes able to execute paravirt Linux 
> kernels.

yes, both would be the goal i think. Single kernel image can run as a 
KVM host, as a KVM/HVM guest [if CPU support] or as a KVM/LL guest [if 
no CPU support]. There's no hypercall overhead on native kernels, we 
have binary-patching infrastructure in place to turn them into NOPs. 
(which makes it quite close to zero-cost)

right now the KVM paravirtualization work we are doing is gradually 
transitioning KVM/HVM towards KVM/LL in essence, by eliminating all VM 
exit reasons from KVM/HVM and turning them into hypercalls. Once that 
has been achieved, KVM/LL could be implemented by an extra ll.c module 
in drivers/kvm/, which does the cr3 tricks and pagetable maintainance 
and CPU state save/restore, fault/trap/irq passback (and not much else).

So i see very nice short and long term synergy between native Linux, 
KVM/HVM guests and KVM/LL guests. What is an hypercall-accelerated 
driver under KVM/HVM is a paravirtual driver on KVM/LL. What is a 
hypercall-based speedup for KVM/HVM is a paravirtual facility for 
KVM/LL. One and the same thing serves both purposes.

> I can certainly attest that there is a lot of demand for running Linux 
> domains with kernels other than the one running on the hardware.  This 
> is Xen's bread and butter and from the albeit old numbers I've seen so 
> far KVM has problems competing.

yes. Running /any/ OS with maximum possible performance is the goal, and 
i'd like the hypercall API to become an ABI in essence, with as much 
backwards compatibility for older kernels as possible.

	Ingo

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Avi Kivity]

Ingo Molnar wrote:
> * Ulrich Drepper <drepper-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
>   
>> Ingo Molnar wrote:
>>     
>>> i really really think KVM and lhype should merge, creating KVM/HVM 
>>> (Hardware Virtual Machine) and KVM/LL (Linux on Linux).
>>>       
>> This is only sufficient if either KVM with paravirt Linux kernels has 
>> no performance penalty or lhype becomes able to execute paravirt Linux 
>> kernels.
>>     
>
> yes, both would be the goal i think. Single kernel image can run as a 
> KVM host, as a KVM/HVM guest [if CPU support] or as a KVM/LL guest [if 
> no CPU support]. There's no hypercall overhead on native kernels, we 
> have binary-patching infrastructure in place to turn them into NOPs. 
> (which makes it quite close to zero-cost)
>
> right now the KVM paravirtualization work we are doing is gradually 
> transitioning KVM/HVM towards KVM/LL in essence, by eliminating all VM 
> exit reasons from KVM/HVM and turning them into hypercalls. Once that 
> has been achieved, KVM/LL could be implemented by an extra ll.c module 
> in drivers/kvm/, which does the cr3 tricks and pagetable maintainance 
> and CPU state save/restore, fault/trap/irq passback (and not much else).
>
> So i see very nice short and long term synergy between native Linux, 
> KVM/HVM guests and KVM/LL guests. What is an hypercall-accelerated 
> driver under KVM/HVM is a paravirtual driver on KVM/LL. What is a 
> hypercall-based speedup for KVM/HVM is a paravirtual facility for 
> KVM/LL. One and the same thing serves both purposes.
>
>   

If you have a CONFIG_PARAVIRT guest, I believe it will always be faster 
to run it without hardware assisted virtualization:

- you cannot eliminate vmexits due to host interrupts
- a hypercall will (probably) keep being more expensive than a syscall; 
it simply has a lot more work to do
- cr3 switches for CONFIG_PARAVIRT syscalls (which are necessary on 
x86_64) will probably become very cheap with tagged tlbs

So, in my opinion, CONFIG_PARAVIRT on non-hvm will eventually win on 
pure performance.  Hardware assisted virtualization will be used where 
you wish to run unmodified guests, and where ABI stability is critical.

I won't mind being proved wrong, though.  And of course we should share 
drivers and APIs between the two technologies.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Rusty Russell]

On Wed, 2007-01-10 at 10:47 +0100, Ingo Molnar wrote:
> * Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org> wrote:
> regarding ABI: agreed that it's experimental right now and should stay 
> so for some time, but i dont see a reason why the hypercall API that 
> i've posted in the past few days couldnt be evolved in the way Linux 
> syscalls evolve, by only adding to them. That doesnt really impact the 
> flexibility of the virtualization subsystem. We could also possibly 
> phase out older hypercalls, because the number of "applications" relying 
> on it will be very low.

I suggest you continue adding hypercalls, and see where you end up.  It
will be very interesting, particularly when you get to the paravirt_ops
lazy_mode hooks.

However, I believe it is worthwhile to have a mini hypervisor which
totals less than 5000 lines, including userspace utilities.  I don't
believe it will change the world, but maybe some of the ideas will be
incorporated in "big serious" hypervisors.  But meanwhile, it's fun!

Cheers!
Rusty.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Ingo Molnar]

* Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:

> If you have a CONFIG_PARAVIRT guest, I believe it will always be 
> faster to run it without hardware assisted virtualization:
> 
> - you cannot eliminate vmexits due to host interrupts
> - a hypercall will (probably) keep being more expensive than a syscall; 
> it simply has a lot more work to do
> - cr3 switches for CONFIG_PARAVIRT syscalls (which are necessary on 
> x86_64) will probably become very cheap with tagged tlbs

but irq overhead is nothing in importance compared to basic syscall 
overhead. KVM/HVM already runs guest kernel syscalls at native speed. 
KVM/LL (or Xen) has to switch cr3s to enter guest kernel context, and 
has to switch it back to get back to guest user context. It might be
pretty fast with tagged TLBs, but not zero-cost.

	Ingo

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[sean-cz32UWRuVxBIf6P1QZMOBw]

Hello all,

hard and software virtualisation has been with us since the mid 1960's, as
I am sure you are all aware of.

See http://en.wikipedia.org/wiki/VM/CMS


JC


On Wed, January 10, 2007 11:46, Rusty Russell wrote:
> On Wed, 2007-01-10 at 10:47 +0100, Ingo Molnar wrote:
>
>> * Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org> wrote:
>> regarding ABI: agreed that it's experimental right now and should stay so
>> for some time, but i dont see a reason why the hypercall API that i've
>> posted in the past few days couldnt be evolved in the way Linux syscalls
>> evolve, by only adding to them. That doesnt really impact the
>> flexibility of the virtualization subsystem. We could also possibly
>> phase out older hypercalls, because the number of "applications"
>> relying on it will be very low.
>
> I suggest you continue adding hypercalls, and see where you end up.  It
> will be very interesting, particularly when you get to the paravirt_ops
> lazy_mode hooks.
>
> However, I believe it is worthwhile to have a mini hypervisor which
> totals less than 5000 lines, including userspace utilities.  I don't
> believe it will change the world, but maybe some of the ideas will be
> incorporated in "big serious" hypervisors.  But meanwhile, it's fun!
>
> Cheers!
> Rusty.
>
>
>
> -------------------------------------------------------------------------
>  Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your opinions on IT & business topics through brief surveys - and earn
> cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>  _______________________________________________
> kvm-devel mailing list kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>
>


John Cassidy (Dipl.-Ingr.)

Berninastrasse 9

8057 Zuerich

Europe

Telephone: +41 (0) 43 300 4602

Mobile:    +41 (0) 79 207 3268


E-Mail: john.cassidy-+vc80D69rgWBubSflsco8w@public.gmane.org

http://www.JDCassidy.net

http://www.europeunited.org

http://en.wikipedia.org/wiki/Europe_United







-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Avi Kivity]

Ingo Molnar wrote:
> * Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
>
>   
>> If you have a CONFIG_PARAVIRT guest, I believe it will always be 
>> faster to run it without hardware assisted virtualization:
>>
>> - you cannot eliminate vmexits due to host interrupts
>> - a hypercall will (probably) keep being more expensive than a syscall; 
>> it simply has a lot more work to do
>> - cr3 switches for CONFIG_PARAVIRT syscalls (which are necessary on 
>> x86_64) will probably become very cheap with tagged tlbs
>>     
>
> but irq overhead is nothing in importance compared to basic syscall 
> overhead. KVM/HVM already runs guest kernel syscalls at native speed. 
> KVM/LL (or Xen) has to switch cr3s to enter guest kernel context, and 
> has to switch it back to get back to guest user context. It might be
> pretty fast with tagged TLBs, but not zero-cost.
>   

For i386 Xen does not switch cr3 IIRC.  Perhaps even not for x86_64 if 
it can use the segment limits which AMD re-added (I think it does?)

I think for i386 Xen does not go through the hypervisor at all: it hacks 
int 0x80 to trap directly to ring 1.  So there's still the overhead of 
using int rather than sysenter, but not much more.  I don't know how the 
(many syscalls) x (smaller overhead) vs (fewer interrupts) x (greater 
overhead) stack up.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Ingo Molnar]

* Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:

> For i386 Xen does not switch cr3 IIRC. [...]

correct.

> [...] Perhaps even not for x86_64 if it can use the segment limits 
> which AMD re-added (I think it does?)

i'm not sure. Older ones definitely used cr3 switching and on Intel 
there's no segment limits on 64-bit AFAIK.

> I think for i386 Xen does not go through the hypervisor at all: it 
> hacks int 0x80 to trap directly to ring 1.  So there's still the 
> overhead of using int rather than sysenter, but not much more. [...]

that alone is already ~100-200 cycles overhead, it essentially doubles 
the null syscall overhead. It matters at millions of syscalls per second 
workloads, 100 cycles overhead at 1 million syscalls a second means 5% 
performance difference at 2GHz.

and longer-term, the cost of VM exit due to IRQs is expected to get 
cheaper. The cost of int $0x80 versus SYSENTER will stay constant, or 
will get worse.

	Ingo

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Avi Kivity]

Ingo Molnar wrote:
> * Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
>
>   
>> For i386 Xen does not switch cr3 IIRC. [...]
>>     
>
> correct.
>
>   
>> [...] Perhaps even not for x86_64 if it can use the segment limits 
>> which AMD re-added (I think it does?)
>>     
>
> i'm not sure. Older ones definitely used cr3 switching and on Intel 
> there's no segment limits on 64-bit AFAIK.
>
>   

I'd be surprised if that remained the case for too long.


>> I think for i386 Xen does not go through the hypervisor at all: it 
>> hacks int 0x80 to trap directly to ring 1.  So there's still the 
>> overhead of using int rather than sysenter, but not much more. [...]
>>     
>
> that alone is already ~100-200 cycles overhead, it essentially doubles 
> the null syscall overhead. It matters at millions of syscalls per second 
> workloads, 100 cycles overhead at 1 million syscalls a second means 5% 
> performance difference at 2GHz.
>
>   

That doesn't strike me as too bad.


> and longer-term, the cost of VM exit due to IRQs is expected to get 
> cheaper. The cost of int $0x80 versus SYSENTER will stay constant, or 
> will get worse.
>   

Agreed.  But I still don't know whether the lines on the graph will cross.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Anthony Liguori]

Avi Kivity wrote:
> Ingo Molnar wrote:
>   
>> * Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
>>
>>
>>     
>>> If you have a CONFIG_PARAVIRT guest, I believe it will always be
>>> faster to run it without hardware assisted virtualization:
>>>
>>> - you cannot eliminate vmexits due to host interrupts
>>> - a hypercall will (probably) keep being more expensive than a syscall;
>>> it simply has a lot more work to do
>>> - cr3 switches for CONFIG_PARAVIRT syscalls (which are necessary on
>>> x86_64) will probably become very cheap with tagged tlbs
>>>
>>>       
>> but irq overhead is nothing in importance compared to basic syscall
>> overhead. KVM/HVM already runs guest kernel syscalls at native speed.
>> KVM/LL (or Xen) has to switch cr3s to enter guest kernel context, and
>> has to switch it back to get back to guest user context. It might be
>> pretty fast with tagged TLBs, but not zero-cost.
>>
>>     
>
> For i386 Xen does not switch cr3 IIRC.  Perhaps even not for x86_64 if
> it can use the segment limits which AMD re-added (I think it does?)
>   

Xen setups the IDT to deliver directly to ring 1 for syscalls as you 
suggested.

At the moment, Xen doesn't make use of the segment limits on AMD on 
64bit.  Currently, the guest kernel runs in ring 3 and I presume there 
are a good deal of assumptions about that.

Xen does, however, use global pages for the kernel (and it's) memory so 
that should help a bit.

One thing to consider though is how things like hardware nested paging 
and CR3 caching come into play.  On a context switch, a Xen style 
paravirt has to use hypercalls to change CR3 (since it's privileged) 
whereas on VT, there's at least the chance of hitting the cache before 
taking a VMEXIT.

Also, nested paging should considerably change the performance 
characteristics of a FV guest.  While TLB lookups will end up being more 
expensive (since more memory accesses are required), the overall cost of 
page fault handling will go down significantly.  Recall, even in direct 
paged mode, Xen has to take page faults in the hypervisor first.  With 
nested paging, presumably page faults can be delivered directly to the 
guest[1].

[1] This assumes that the implementation will allow for physical memory 
holes which will result in VMEXITs (for MMIO emulation).

Regards,

Anthony Liguori

> I think for i386 Xen does not go through the hypervisor at all: it hacks
> int 0x80 to trap directly to ring 1.  So there's still the overhead of
> using int rather than sysenter, but not much more.  I don't know how the
> (many syscalls) x (smaller overhead) vs (fewer interrupts) x (greater
> overhead) stack up.
>
>
> --
> error compiling committee.c: too many arguments to function
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> kvm-devel mailing list
> kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>   


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Avi Kivity]

Anthony Liguori wrote:
> Avi Kivity wrote:
>> Ingo Molnar wrote:
>>  
>>> * Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
>>>
>>>
>>>    
>>>> If you have a CONFIG_PARAVIRT guest, I believe it will always be
>>>> faster to run it without hardware assisted virtualization:
>>>>
>>>> - you cannot eliminate vmexits due to host interrupts
>>>> - a hypercall will (probably) keep being more expensive than a 
>>>> syscall;
>>>> it simply has a lot more work to do
>>>> - cr3 switches for CONFIG_PARAVIRT syscalls (which are necessary on
>>>> x86_64) will probably become very cheap with tagged tlbs
>>>>
>>>>       
>>> but irq overhead is nothing in importance compared to basic syscall
>>> overhead. KVM/HVM already runs guest kernel syscalls at native speed.
>>> KVM/LL (or Xen) has to switch cr3s to enter guest kernel context, and
>>> has to switch it back to get back to guest user context. It might be
>>> pretty fast with tagged TLBs, but not zero-cost.
>>>
>>>     
>>
>> For i386 Xen does not switch cr3 IIRC.  Perhaps even not for x86_64 if
>> it can use the segment limits which AMD re-added (I think it does?)
>>   
>
> Xen setups the IDT to deliver directly to ring 1 for syscalls as you 
> suggested.
>
> At the moment, Xen doesn't make use of the segment limits on AMD on 
> 64bit.  Currently, the guest kernel runs in ring 3 and I presume there 
> are a good deal of assumptions about that.
>
> Xen does, however, use global pages for the kernel (and it's) memory 
> so that should help a bit.

That means it needs to flush the Xen tlb entries on domain switch.  I 
guess that's a good tradeoff since guest context switches are more 
common than domain switches.

>
> One thing to consider though is how things like hardware nested paging 
> and CR3 caching come into play.  On a context switch, a Xen style 
> paravirt has to use hypercalls to change CR3 (since it's privileged) 
> whereas on VT, there's at least the chance of hitting the cache before 
> taking a VMEXIT.
>
> Also, nested paging should considerably change the performance 
> characteristics of a FV guest.  While TLB lookups will end up being 
> more expensive (since more memory accesses are required), the overall 
> cost of page fault handling will go down significantly.  Recall, even 
> in direct paged mode, Xen has to take page faults in the hypervisor 
> first.  With nested paging, presumably page faults can be delivered 
> directly to the guest[1].

I agree that nested page tables changes things dramatically wrt. 
paging.  My comments apply only to non-nested paging.


>
> [1] This assumes that the implementation will allow for physical 
> memory holes which will result in VMEXITs (for MMIO emulation).

The assumption is correct according to my reading of the docs.  NPT can 
deliver a #PF or a #VMEXIT(NPF), with the expected semantics.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Ulrich Drepper]

[-- Attachment #1.1: Type: text/plain, Size: 534 bytes --]

Ingo Molnar wrote:
> that alone is already ~100-200 cycles overhead, it essentially doubles 
> the null syscall overhead. It matters at millions of syscalls per second 
> workloads, 100 cycles overhead at 1 million syscalls a second means 5% 
> performance difference at 2GHz.

Plus, x86-64 uses only syscall which doesn't have the opportunity to be
intercepted in ring 1, right?  That's the much more important target
going forward.

-- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: Paravirt KVM capabilities

[Avi Kivity]

Ulrich Drepper wrote:
> Ingo Molnar wrote:
>   
>> that alone is already ~100-200 cycles overhead, it essentially doubles 
>> the null syscall overhead. It matters at millions of syscalls per second 
>> workloads, 100 cycles overhead at 1 million syscalls a second means 5% 
>> performance difference at 2GHz.
>>     
>
> Plus, x86-64 uses only syscall which doesn't have the opportunity to be
> intercepted in ring 1, right?  That's the much more important target
> going forward.
>   

Agreed.  64-bit syscalls are much better with hardware assist.

[Everyone complains how x86 is loaded with 20+ years of cruft; AMD comes 
and cleans it up; then we discover how 4 rings and segmentation are 
actually useful]

[[Can't a paravirtualized kernel use a vdso to use int $0x80 instead of 
syscall?]]

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Ulrich Drepper]

[-- Attachment #1.1: Type: text/plain, Size: 314 bytes --]

Avi Kivity wrote:
> [[Can't a paravirtualized kernel use a vdso to use int $0x80 instead of
> syscall?]]

No.  The ABI is to inline syscall instructions.  That's possible since
it's not as limited/broken as sysenter.

-- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: Paravirt KVM capabilities

[Avi Kivity]

Ulrich Drepper wrote:
> Avi Kivity wrote:
>   
>> [[Can't a paravirtualized kernel use a vdso to use int $0x80 instead of
>> syscall?]]
>>     
>
> No.  The ABI is to inline syscall instructions.  That's possible since
> it's not as limited/broken as sysenter.
>   

Then maybe kvm+npt+paravirt can indeed be competitive with pure paravirt.

Although npt requires up to 16 accesses for a tlb fill, usually all but 
two will be cached on a moderately sized guest, so I don't expect the 
cost to be prohibitive.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Rusty Russell]

On Wed, 2007-01-10 at 11:52 +0100, Ingo Molnar wrote:
> * Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
> 
> > If you have a CONFIG_PARAVIRT guest, I believe it will always be 
> > faster to run it without hardware assisted virtualization:
> > 
> > - you cannot eliminate vmexits due to host interrupts
> > - a hypercall will (probably) keep being more expensive than a syscall; 
> > it simply has a lot more work to do
> > - cr3 switches for CONFIG_PARAVIRT syscalls (which are necessary on 
> > x86_64) will probably become very cheap with tagged tlbs
> 
> but irq overhead is nothing in importance compared to basic syscall 
> overhead. KVM/HVM already runs guest kernel syscalls at native speed. 
> KVM/LL (or Xen) has to switch cr3s to enter guest kernel context

Err no, this isn't true.  See Documentation/lhype.txt or various blog
entries on the subject 8)  Both Xen and lhype get native syscall speeds
(within measurement error).

Rusty.



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Ingo Molnar]

* Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org> wrote:

> > > - cr3 switches for CONFIG_PARAVIRT syscalls (which are necessary 
> > > on x86_64) will probably become very cheap with tagged tlbs
> > 
> > but irq overhead is nothing in importance compared to basic syscall 
> > overhead. KVM/HVM already runs guest kernel syscalls at native 
> > speed. KVM/LL (or Xen) has to switch cr3s to enter guest kernel 
> > context
> 
> Err no, this isn't true.  See Documentation/lhype.txt or various blog 
> entries on the subject 8) Both Xen and lhype get native syscall speeds 
> (within measurement error).

i was talking about 64-bit. (we dont really design for 32-bit anymore.) 
I know that lhype uses Xen's ring 1 trick, but that's a 32-bit-only 
thing. Also, can SYSENTER trap from guest userspace ring 3 into guest 
kernelspace ring 1 on lhype?

	Ingo

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Rusty Russell]

On Thu, 2007-01-11 at 01:49 +0100, Ingo Molnar wrote:
> * Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org> wrote:
> 
> > > > - cr3 switches for CONFIG_PARAVIRT syscalls (which are necessary 
> > > > on x86_64) will probably become very cheap with tagged tlbs
> > > 
> > > but irq overhead is nothing in importance compared to basic syscall 
> > > overhead. KVM/HVM already runs guest kernel syscalls at native 
> > > speed. KVM/LL (or Xen) has to switch cr3s to enter guest kernel 
> > > context
> > 
> > Err no, this isn't true.  See Documentation/lhype.txt or various blog 
> > entries on the subject 8) Both Xen and lhype get native syscall speeds 
> > (within measurement error).
> 
> i was talking about 64-bit. (we dont really design for 32-bit anymore.) 

Oh, ok, because there doesn't seem much point to avoiding using HVM if
you're assuming 64 bit, except as an optimization.  Full paravirt is
more useful for 32-bit, hence lhype started there...

> I know that lhype uses Xen's ring 1 trick, but that's a 32-bit-only 
> thing. Also, can SYSENTER trap from guest userspace ring 3 into guest 
> kernelspace ring 1 on lhype?

As I understand it, sysenter has to go to ring 0, and the reflection
cost is way greater than the saving.

Rusty.



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Ingo Molnar]

* Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org> wrote:

> > > Err no, this isn't true.  See Documentation/lhype.txt or various blog 
> > > entries on the subject 8) Both Xen and lhype get native syscall speeds 
> > > (within measurement error).
> > 
> > i was talking about 64-bit. (we dont really design for 32-bit anymore.) 
[...]
> 
> > I know that lhype uses Xen's ring 1 trick, but that's a 32-bit-only 
> > thing. Also, can SYSENTER trap from guest userspace ring 3 into guest 
> > kernelspace ring 1 on lhype?
> 
> As I understand it, sysenter has to go to ring 0, and the reflection 
> cost is way greater than the saving.

so how can both "Xen and lhype get native syscall speeds (within 
measurement error)", if it cannot use SYSENTER (it has to use int 
$0x80), while a HVM kernel can?

	Ingo

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Rusty Russell]

On Thu, 2007-01-11 at 03:51 +0100, Ingo Molnar wrote:
> * Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org> wrote:
> 
> > > > Err no, this isn't true.  See Documentation/lhype.txt or various blog 
> > > > entries on the subject 8) Both Xen and lhype get native syscall speeds 
> > > > (within measurement error).
> > > 
> > > i was talking about 64-bit. (we dont really design for 32-bit anymore.) 
> [...]
> > 
> > > I know that lhype uses Xen's ring 1 trick, but that's a 32-bit-only 
> > > thing. Also, can SYSENTER trap from guest userspace ring 3 into guest 
> > > kernelspace ring 1 on lhype?
> > 
> > As I understand it, sysenter has to go to ring 0, and the reflection 
> > cost is way greater than the saving.
> 
> so how can both "Xen and lhype get native syscall speeds (within 
> measurement error)", if it cannot use SYSENTER (it has to use int 
> $0x80), while a HVM kernel can?

Sorry, my mistake.  I was measuring with a statically linked binary,
which didn't use SYSENTER even when available 8(

I've fixed virtbench now, and reflects this correctly.

"syscalls almost as fast as poorly-implemented native syscalls"?
Rusty.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Paravirt KVM capabilities

[Ingo Molnar]

* Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org> wrote:

> > so how can both "Xen and lhype get native syscall speeds (within 
> > measurement error)", if it cannot use SYSENTER (it has to use int 
> > $0x80), while a HVM kernel can?
> 
> Sorry, my mistake.  I was measuring with a statically linked binary, 
> which didn't use SYSENTER even when available 8(
> 
> I've fixed virtbench now, and reflects this correctly.
> 
> "syscalls almost as fast as poorly-implemented native syscalls"?

hehe :-)

	Ingo

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV