KVM is type 1 hypervisor, but...

KVM is type 1 hypervisor, but...

[Sylvain Leroux]

[-- Attachment #1.1: Type: text/plain, Size: 1254 bytes --]

Dear all,

I certainly do not want to start a type 1 vs type 2 holly war. But I'm
working on some introductory course on virtualization and I have hard
time to conciliate the average user daily experience with the premise
KVM *is* type 1.

More precisely, I've read KVM turns the Linux kernel into a type 1
hypervisor. I can understand that. However, from the user perspective,
when running Linux with KVM, you can still launch process _outside_ any
guest. Something I could represent like that:

           +----------+----------+
           |   app B  |  app C   |
           +----------+----------+
           | guest OS | guest OS |
+----------+----------+----------+
|  app A   | QEMU/KVM | QEMU/KVM |
+--------------------------------+
|            Linux/KVM           |
+--------------------------------+
|                HW              |
+--------------------------------+


With that model, "app A" _seems_ to be running directly "on the host".
And to be honest, it furiously reminds me the typical model for a type 2
hypervisor.


Or are we considering "app A" is running in its own "hidden" VM,
something like the dom0 in Xen?



-- 
-- Sylvain Leroux
-- sylvain@chicoree.fr
-- http://www.chicoree.fr


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 862 bytes --]

Re: KVM is type 1 hypervisor, but...

[Christoph Hellwig]

On Sat, May 27, 2017 at 07:06:29PM +0200, Sylvain Leroux wrote:
> Dear all,
> 
> I certainly do not want to start a type 1 vs type 2 holly war. But I'm
> working on some introductory course on virtualization and I have hard
> time to conciliate the average user daily experience with the premise
> KVM *is* type 1.

I think your mail just very much eplained why this arbitrary type 1
vs type 2 classification doesn't make any sense at all.

Re: KVM is type 1 hypervisor, but...

[Sylvain Leroux]

[-- Attachment #1.1: Type: text/plain, Size: 1581 bytes --]

On 05/28/2017 08:34 AM, Christoph Hellwig wrote:
> I think your mail just very much eplained why this arbitrary type 1
> vs type 2 classification doesn't make any sense at all.
> 

As of myself, I would consider more the "type 1 vs type 2" thing as a
continuum rather than two distinct solutions. Most if not all modern
virtualization solutions probably lies somewhere between those two
extreme cases.


Unfortunately, per the course curriculum, I cannot avoid dealing with
that subject. I will probably explain why "type 1 vs type 2" is a rather
theoretical and mostly outdated concept (any reference to support that
point would be welcome ;) )

But remains the fact I don't feel confident enough yet to answer a
question like "Is KVM type 1 or type 2?"

It is somewhat more easy to answer that question for Xen or
Hyper-V--since the dom0/Parent partition *is* really handled as a VM by
those hypervisors. But I don't think there is a similar concept with
KVM. Or is there?

An other argument I read is KVM is "type 1" because the hypervisor runs
in kernel space. Whereas (for example) VirtualBox is type 2 because the
hypervisor runs in user space. But that feels like an oversimplification
to me. If tomorrow Oracle released VirtualBox as a kernel module rather
than a user space application, would that really made them consider it
as "type 1"? If so, that would certainly prove the "type 1 vs type 2"
thing as being nothing more than a marketing argument...





-- 
-- Sylvain Leroux
-- sylvain@chicoree.fr
-- http://www.chicoree.fr


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 862 bytes --]

Re: KVM is type 1 hypervisor, but...

[Christoph Hellwig]

On Sun, May 28, 2017 at 10:46:45AM +0200, Sylvain Leroux wrote:
> On 05/28/2017 08:34 AM, Christoph Hellwig wrote:
> > I think your mail just very much eplained why this arbitrary type 1
> > vs type 2 classification doesn't make any sense at all.
> > 
> 
> As of myself, I would consider more the "type 1 vs type 2" thing as a
> continuum rather than two distinct solutions. Most if not all modern
> virtualization solutions probably lies somewhere between those two
> extreme cases.

Not just all modern, but simply all, that's why the difference didn't
every make sense.

Each IBM's CP (aka VM) always allowed executing some code, and it's
been growing more and more.

Or VMWare ESXi supports running unmodified Linux binaries - it has
various limitations compared to modern Linux, but if you compare
it to early Linux or even early unix it's a fully featured OS.

At the same time if run Linux/KVM with a a totally cut down kernel
config and a root fs that contains nothing but kvmtool or qemu
it's about as useless for running general purpose programs on
the hypervisor kernel as it gets.

> It is somewhat more easy to answer that question for Xen or
> Hyper-V--since the dom0/Parent partition *is* really handled as a VM by
> those hypervisors. But I don't think there is a similar concept with
> KVM. Or is there?

There isn't.

> An other argument I read is KVM is "type 1" because the hypervisor runs
> in kernel space. Whereas (for example) VirtualBox is type 2 because the
> hypervisor runs in user space.

That's not the case.  Virtualbox has a (horrible) kernel module that
does all the low-level hypervisor work.  Think of it as a badly written
replacement for both kvm and kqemu.

Re: KVM is type 1 hypervisor, but...

[Sylvain Leroux]

Thank you Christoph for all those great comments.

You're very convincing in explaining your point. But how could you
explain there is such abundant literacy taking the type 1/type 2
dichotomy as a fundamental distinction between hypervisors? On the other
hand, I found very difficult to find papers debunking that myth. But
maybe you or the KVM team did write such an article and I missed it?

Re: KVM is type 1 hypervisor, but...

[Jan Kiszka]

On 2017-05-28 21:06, Sylvain Leroux wrote:
> Thank you Christoph for all those great comments.
> 
> You're very convincing in explaining your point. But how could you
> explain there is such abundant literacy taking the type 1/type 2
> dichotomy as a fundamental distinction between hypervisors? On the other
> hand, I found very difficult to find papers debunking that myth. But
> maybe you or the KVM team did write such an article and I missed it?
> 

It's more productive to develop hypervisors than to write about them. ;)

People like classifications. They suggest the world is simple and can be
abstracted. You need to look closer to understand that this is generally
not that simple. But that's a boring, often disliked story to tell...

Jan

Re: KVM is type 1 hypervisor, but...

[Christoph Hellwig]

I never understood why people cared about it too much,  the terms come
from a academic paper in the 70s and we're even fully correct back
then, and have gotten less an less useful.

To make some sense of them it might be better to understand them in
terms of use cases rather than implementation, e.g. in terms of
KVM someone running a VM or two in addition to a normal desktop Linux
(e.g. a kernel developer testing things, or a Windows VM for a certain
application) fits into the type 2 model, while a server whos primary
purpose is to host VMs it type 1.  But with todays cloud or
hyperconverged architectures a single host system very often runs
VM and actualy workloads (e.g. storage backends or databases) as well.

So in my eyes it's a very theoretical concept that doesn't have any
practical relevance.

Re: KVM is type 1 hypervisor, but...

[Sylvain Leroux]

On 05/29/2017 08:41 AM, Jan Kiszka wrote:
> It's more productive to develop hypervisors than to write about them. ;)

_I_ agree with that. My boss, not necessary.


To let you know at which point non-IT people may be stuck, in a
different area I'm still required to teach "classfull" IP networking.
Seriously, that was obsoleted by RFC1518/1519 more that 20 years ago!
How many of my students will encounter devices still using classfull
network?

> 
> People like classifications. They suggest the world is simple and can be
> abstracted. You need to look closer to understand that this is generally
> not that simple. But that's a boring, often disliked story to tell...

Yes, exactly. That's why I wanted to take time discussing with you,
"experts" about that topic. It would have been hundred times more easy
to just copy-paste and rehash the same story we can find in a plethora
of blog posts.

But my boss will _not_ do that. If he sees in my course outline that
"hypervisor type 1 type 2 classification is meaningless", he will type
that in his favorite search engine and will find pages and pages of
results saying the exact opposite.


Going back to your initial argument Jan, seconded by Christoph:
> It's more productive to develop hypervisors than to write about them. ;)

On 05/29/2017 10:13 AM, Christoph Hellwig wrote:
> I never understood why people cared about it too much,  the terms come
> from a academic paper in the 70s and we're even fully correct back
> then, and have gotten less an less useful.


If we want those arguments to be heard by a large audience, I think we
cannot avoid the need for some "authority" to publish about that. Call
that a manifesto or a paper, but the point is people like me would have
an "argument from authority" to quote and reference, so that will become
an idea to discuss, rather than just an opinion exchanged by some people
in a mailing lists.


-- 
-- Sylvain Leroux
-- sylvain@chicoree.fr
-- http://www.chicoree.fr

Re: KVM is type 1 hypervisor, but...

[Sylvain Leroux]

On 05/29/2017 10:13 AM, Christoph Hellwig wrote:
> To make some sense of them it might be better to understand them in
> terms of use cases rather than implementation, e.g. in terms of
> KVM someone running a VM or two in addition to a normal desktop Linux
> (e.g. a kernel developer testing things, or a Windows VM for a certain
> application) fits into the type 2 model, while a server whos primary
> purpose is to host VMs it type 1.  But with todays cloud or
> hyperconverged architectures a single host system very often runs
> VM and actualy workloads (e.g. storage backends or databases) as well.
I like the idea of thinking in terms of use cases.

Could you elaborate a little bit about that?

If we could find an *unbiased* way of comparing the behavior of various
"virtualization architecture" in some real world situation, I could add
that to my course. And probably record it so it could be published on
YouTube or elsewhere. That could be a way to help spreading the idea we
can compare VM solutions differently than using a 40 years old
classification.

-- 
-- Sylvain Leroux
-- sylvain@yesik.it
-- https://yesik.it

Re: KVM is type 1 hypervisor, but...

[Jan Kiszka]

On 2017-05-29 11:05, Sylvain Leroux wrote:
> 
> On 05/29/2017 08:41 AM, Jan Kiszka wrote:
>> It's more productive to develop hypervisors than to write about them. ;)
> 
> _I_ agree with that. My boss, not necessary.
> 
> 
> To let you know at which point non-IT people may be stuck, in a
> different area I'm still required to teach "classfull" IP networking.
> Seriously, that was obsoleted by RFC1518/1519 more that 20 years ago!
> How many of my students will encounter devices still using classfull
> network?
> 
>>
>> People like classifications. They suggest the world is simple and can be
>> abstracted. You need to look closer to understand that this is generally
>> not that simple. But that's a boring, often disliked story to tell...
> 
> Yes, exactly. That's why I wanted to take time discussing with you,
> "experts" about that topic. It would have been hundred times more easy
> to just copy-paste and rehash the same story we can find in a plethora
> of blog posts.
> 
> But my boss will _not_ do that. If he sees in my course outline that
> "hypervisor type 1 type 2 classification is meaningless", he will type
> that in his favorite search engine and will find pages and pages of
> results saying the exact opposite.
> >
> Going back to your initial argument Jan, seconded by Christoph:
>> It's more productive to develop hypervisors than to write about them. ;)
> 
> On 05/29/2017 10:13 AM, Christoph Hellwig wrote:
>> I never understood why people cared about it too much,  the terms come
>> from a academic paper in the 70s and we're even fully correct back
>> then, and have gotten less an less useful.
> 
> 
> If we want those arguments to be heard by a large audience, I think we
> cannot avoid the need for some "authority" to publish about that. Call
> that a manifesto or a paper, but the point is people like me would have
> an "argument from authority" to quote and reference, so that will become
> an idea to discuss, rather than just an opinion exchanged by some people
> in a mailing lists.

See, the classification attempt is an academic approach to an academic
problem. Most folks here are engineers. Their business is to find
architectures that create optimal combinations of what is technically
possible. That's what you may make them write or talk about.


But you may soon cite another example for how meaningless the
classification is in practice:

"[...] This scheme does not fit into the traditional classification of
hypervisors [7] – it can be seen as a mixture of Type-1 and Type-2
hypervisors: It runs on raw hardware like a bare-metal hypervisor
without an underlying system level, but still cannot operate without
Linux as a system aide to provide initialised hardware. Linux is used
as bootloader, but not for operation."

>From the accepted OSPERT17 paper of a student of us on the Jailhouse
hypervisor architecture. Maybe I could motivate him writing about the
classification nonsense alone as well, but he is also very good engineer
and code contributor... ;)


And if you want to do something good to your students: teach concepts by
practical examples, particularly the exceptions from textbook rules.

Jan

Re: KVM is type 1 hypervisor, but...

[Paolo Bonzini]

On 27/05/2017 19:06, Sylvain Leroux wrote:
> I certainly do not want to start a type 1 vs type 2 holly war. But I'm
> working on some introductory course on virtualization and I have hard
> time to conciliate the average user daily experience with the premise
> KVM *is* type 1.

Hi Sylvain,

it would help if you provided the original quote about type-1 and
type-2, because I have never been able to find it.  It is often cited
together with the above Popek-Goldberg paper but I cannot find it in there

If you really want to cut hypervisors in two, you could distinguish
"type-1" hypervisors that run in supervisor mode (x86 says ring 0) from
"type-2" hypervisors that run in user mode (x86 says ring 3).

It is debatable that this classification makes sense because, on a
virtualizable machine with paging, a hypervisor running in ring 3 would
likely need some help from the kernel in order to build page tables;
Popek and Goldberg's paper ("Formal Requirements for Virtualizable Third
Generation Architectures" paper, CACM vol. 17 n. 7 pages 412-421) only
mentions a simple segmented architecture.  But anyway, based on this
classification, type-2 hypervisors on x86 _must_ be doing dynamic
translation, because x86 with virtual machine extensions is not
virtualizable and virtual machine extensions are only accessible from
ring 0.  Anything that uses virtual machine extensions would be type-1.

You arrive to the same conclusion if you say that type-1 runs in VMX
root mode and type-2 runs in VMX non-root mode.

Another case where the distinction is substantially blurred by computers
and OSes newer than the 1970s is I/O devices.  In this case, VFIO allows
I/O devices to be used directly by the virtual machine with no overhead
for I/O calls, and together with KVM no overhead for interrupts either.

In other words, kernel modules like KVM or Apple's Hypervisor.framework
augment conventional OSes with the abilities of a VMM, but KVM and
Hypervisor.framework (and VirtualBox too) are definitely "bare metal".

I would just ignore it.  To some extent, the modern usage of the type-1
and type-2 terms is really more about VMware and Xen trying to bash KVM,
than anything else.

Paolo

Re: KVM is type 1 hypervisor, but...

[Paolo Bonzini]

On 31/05/2017 15:29, Konrad Rzeszutek Wilk wrote:
> 
>     I would just ignore it.  To some extent, the modern usage of the type-1
>     and type-2 terms is really more about VMware and Xen trying to bash KVM,
>     than anything else.
> 
> 
> Could you point me to where Xen is bashing KVM about the type? I am
> mighty curious where it has any mention of one type being better than
> the other.

Citrix definitely tried:

https://www.citrix.com/blogs/wp-content/uploads/2009/11/CitrixXenClient_SolutionBrief.pdf?version=1
"In the case of a Type-2 hypervisor, if the base OS is compromised, any
of the virtual machines running on top of it are subject to compromise.
A Type-1 solution provides a secure locked down hypervisor which owns
all of the hardware, providing a fundamentally trusted platform to build
multiple virtual machines with varying levels of privilege and trust.
Additionally, with a Type-1 solution the virtual machines are totally
isolated from each other so performance or security issues within one
environment will not affect the other desktop environments on the
system. In short, Type-1 provides greater control of security, a finer
degree of arbitration over system resources, and higher levels of
performance


I didn't want to accuse the open source project---sorry about that.

Paolo

Re: KVM is type 1 hypervisor, but...

[Sylvain Leroux]

[-- Attachment #1.1: Type: text/plain, Size: 2000 bytes --]

Hi Paolo,

First, thank you for the very detailed answer. Here are few comments:

On 05/30/2017 03:16 PM, Paolo Bonzini wrote:
> it would help if you provided the original quote about type-1 and
> type-2, because I have never been able to find it.  It is often cited
> together with the above Popek-Goldberg paper but I cannot find it in there
COMPUTER ARCHITECTURAL SYSTEMSPRINCIPLES FOR VIRTUAL
Robert P. Goldberg -- pp20 & following
http://www.dtic.mil/dtic/tr/fulltext/u2/772809.pdf


> If you really want to cut hypervisors in two, you could distinguish
> "type-1" hypervisors that run in supervisor mode (x86 says ring 0) from
> "type-2" hypervisors that run in user mode (x86 says ring 3).
I wonder if there is some tool allowing to measure the amount of time a
process spend in each ring?

> Another case where the distinction is substantially blurred by computers
> and OSes newer than the 1970s is I/O devices.  In this case, VFIO allows
> I/O devices to be used directly by the virtual machine with no overhead
> for I/O calls, and together with KVM no overhead for interrupts either.
>
> In other words, kernel modules like KVM or Apple's Hypervisor.framework
> augment conventional OSes with the abilities of a VMM, but KVM and
> Hypervisor.framework (and VirtualBox too) are definitely "bare metal".

This is a little bit outside of the scope of my initial question, but
isn't "augment[ing] conventional OSes with the abilities of a VMM"
actually increasing the thread surface of the system?

Let me make the devil's advocate here: let's imagine I run BSD & Linux
guests on x86 host. With VMWare ESX, a bug in the Linux Kernel would
thread *only* the Linux guests. But with KVM (and Xen, for what it
worth), it would thread *all* guests.

Or is there some way in KVM to protect the VMM sub-system from other
parts of the kernel (esp. from modules/device drivers)?



-- 
-- Sylvain Leroux
-- sylvain@chicoree.fr
-- http://www.chicoree.fr


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 862 bytes --]

Re: KVM is type 1 hypervisor, but...

[Sylvain Leroux]

Hi Jan,


On 05/29/2017 10:34 PM, Jan Kiszka wrote:
> But you may soon cite another example for how meaningless the
> classification is in practice:
> 
> "[...] This scheme does not fit into the traditional classification of
> hypervisors [7] – it can be seen as a mixture of Type-1 and Type-2
> hypervisors: It runs on raw hardware like a bare-metal hypervisor
> without an underlying system level, but still cannot operate without
> Linux as a system aide to provide initialised hardware. Linux is used
> as bootloader, but not for operation."
> 
> From the accepted OSPERT17 paper of a student of us on the Jailhouse
> hypervisor architecture. 
I'm looking forward to read that paper. I took a look at the Github
repository. Sounds very interesting project. Is there some draft paper
available somewhere?


> Maybe I could motivate him writing about the
> classification nonsense alone as well, but he is also very good engineer
> and code contributor... ;)
That would be nice. Really that type-1 type-2 thing is poisoning the
discussion since too long now.
The one that could debunk that myth would become famous and endlessly
praised for having put a tombstone on that non-sense (<-- is that
motivating enough?)



> And if you want to do something good to your students: teach concepts by
> practical examples, particularly the exceptions from textbook rules.
;)

-- 
-- Sylvain Leroux
-- sylvain@chicoree.fr
-- http://www.chicoree.fr

Re: KVM is type 1 hypervisor, but...

[Paolo Bonzini]

On 05/06/2017 16:39, Sylvain Leroux wrote:
> COMPUTER ARCHITECTURAL SYSTEMSPRINCIPLES FOR VIRTUAL
> Robert P. Goldberg -- pp20 & following
> http://www.dtic.mil/dtic/tr/fulltext/u2/772809.pdf

Thanks, that was useful.

The definition does not mention a conventional OS, but rather an
"extended host".  From page 22:

  Type I--The VMM runs on a bare machine.
  Type II--The VMM runs on an extended host [53,75], under the
  host operating system.

Page 99 extends this a bit: "Type II (extended machine host) virtual
machine organization is one in which the VMM runs, not on a bare machine
as the suoervisor, but rather on an extended host under the host
supervisor".

"Extended host" is not really an expression that is used anymore, but it
is defined in page 30 of the thesis:

  A pseudo-machine [99], also called an extended machine [53] or a user
  machine [75], is a composite machine produced through a combination of
  harware and software, in which the machine's apparent architcture has
  been changed slightly to make the machine more convenient to use.
  Typicallyg these architectural changes have taken the form of removing
  I/O channels and devices, and adding system calls to perform I/O and
  and other operations.

Note how [53,75] are the same citations at page 22; this is exactly what
I mentioned earlier: "type I" hypervisors (Goldberg says VMMs) run in
supervisor mode, "type II" VMMs run in user mode.

>> If you really want to cut hypervisors in two, you could distinguish
>> "type-1" hypervisors that run in supervisor mode (x86 says ring 0) from
>> "type-2" hypervisors that run in user mode (x86 says ring 3).
>
> I wonder if there is some tool allowing to measure the amount of time a
> process spend in each ring?

You don't need that.  You just need to know that x86 is not even
virtualizable from ring 3, all you can do is emulation (e.g. dynamic
binary translation).

So, QEMU running in dynamic binary translation mode is a type 2 VMM.

KVM runs on a bare machine (kernel mode, we'd say today), but it
delegates some services (I/O emulation) to a less privileged component
(QEMU).  This is not a type 2 VMM, it is a type 1 VMM that follows
security principles such as privilege separation.

>> Another case where the distinction is substantially blurred by computers
>> and OSes newer than the 1970s is I/O devices.  In this case, VFIO allows
>> I/O devices to be used directly by the virtual machine with no overhead
>> for I/O calls, and together with KVM no overhead for interrupts either.
>>
>> In other words, kernel modules like KVM or Apple's Hypervisor.framework
>> augment conventional OSes with the abilities of a VMM, but KVM and
>> Hypervisor.framework (and VirtualBox too) are definitely "bare metal".
> This is a little bit outside of the scope of my initial question, but
> isn't "augment[ing] conventional OSes with the abilities of a VMM"
> actually increasing the thread surface of the system?

s/thread/threat/g (or threaten depending on the grammar), I guess :)

> Let me make the devil's advocate here: let's imagine I run BSD & Linux
> guests on x86 host. With VMWare ESX, a bug in the Linux Kernel would
> thread *only* the Linux guests. But with KVM (and Xen, for what it
> worth), it would thread *all* guests.

No, only if the guests can jump out of the hypervisor.

If you have code execution in the hypervisor, the game is over.  Because
the hypervisor (Xen, KVM, ESX) runs in ring 0, there is no need to use a
Linux vulnerability to extend your privilege further.

If you have code execution in QEMU, you could reuse the same
vulnerability to gain code execution in the host and attack all guests.
Then we're in something like this scenario:

> Or is there some way in KVM to protect the VMM sub-system from other
> parts of the kernel (esp. from modules/device drivers)?

Not exactly, but there are ways in Linux to protect other parts of the
kernel from QEMU.  The most common one is "sVirt", which is basically a
set of SELinux policy and conventions that let you run virtual machines
in a highly confined process, that can only access resources destined to
the virtual machine.

Paolo