Hypervisor architecture?

Hypervisor architecture?

[Etienne Martineau]

Is there any documents around that describe the 'internal' architecture
of the Hypervisor?

So far my understanding is such that the Hypervisor is a 'strip-down'
Linux 2.6.(12/13) + lot's of customization specific to Xen. 

Because there is so much documentation around for the Linux kernel I
think it would be _nice_ to have a 'conceptual mapping' between the
Hypervisor and the Linux kernel....

For example:
-Under Linux, processes are described by 'struct task_struct'
-Under Hypervisor, VMs are described by 'struct domain'

What is common?
What is different?

For those who are already familiar with the Linux kernel there would be
a net advantage when trying to 'pickup' on how things are done within
the hypervisor

-Etienne

Re: Hypervisor architecture?

[Jeremy Fitzhardinge]

On 05/04/2010 12:25 PM, Etienne Martineau wrote:
> Is there any documents around that describe the 'internal' architecture
> of the Hypervisor?
>
> So far my understanding is such that the Hypervisor is a 'strip-down'
> Linux 2.6.(12/13) + lot's of customization specific to Xen. 
>   

Xen and Linux diverged a long time ago, early in the Linux 2.6 series,
with occasional feature-specific transfusions from Linux into Xen
since.  There are some similarities in how they deal with platform
issues (APICs, etc), but they're mostly different now.

> What is common?
> What is different?
>   

A vcpu is akin to a task; a domain is like a process (where multi-vcpu =
multi-threaded).  Events are a bit like signals (pending events are
checked for on return to guest mode and the return is redirected to the
event handler).  Hypercalls are very similar to syscalls.  Memory
management is largely different.

One of the big differences is that Xen doesn't have a per-vcpu
hypervisor (kernel) stack, and vcpus don't have a hypervisor context. 
While they're actually running in the hypervisor they use a pcpu stack,
but if it blocks/deschedules then it must always return to guest
context, saving away enough info to continue what it was doing when it
re-enters Xen.  Once you get your head around that, a lot of things
become much clearer.

    J

Re: Hypervisor architecture?

[Etienne Martineau]

On Tue, 2010-05-04 at 13:33 -0700, Jeremy Fitzhardinge wrote:
> On 05/04/2010 12:25 PM, Etienne Martineau wrote:
> > Is there any documents around that describe the 'internal' architecture
> > of the Hypervisor?
> >
> > So far my understanding is such that the Hypervisor is a 'strip-down'
> > Linux 2.6.(12/13) + lot's of customization specific to Xen. 
> >   
> 
> Xen and Linux diverged a long time ago, early in the Linux 2.6 series,
> with occasional feature-specific transfusions from Linux into Xen
> since.  There are some similarities in how they deal with platform
> issues (APICs, etc), but they're mostly different now.
> 
> > What is common?
> > What is different?
> >   
> 
> A vcpu is akin to a task; a domain is like a process (where multi-vcpu =
> multi-threaded).
Does the scheduler schedule 'vcpu' or 'domain'?

>   Events are a bit like signals (pending events are
> checked for on return to guest mode and the return is redirected to the
> event handler).  Hypercalls are very similar to syscalls.
Fair enough;

>   Memory
> management is largely different.
I see two modes of operation?
(A) HVM; it looks like the Hypervisor is doing 'paging' and maintain
sPTE.
(B) PV; Guest has access to %cr3 and it seems to me that the Hypervisor
is not involve on the fast path?

I still haven't figure out the 'page_fault' path all the way up to the
guest...

> 
> One of the big differences is that Xen doesn't have a per-vcpu
> hypervisor (kernel) stack, and vcpus don't have a hypervisor context. 
> While they're actually running in the hypervisor they use a pcpu stack,
> but if it blocks/deschedules then it must always return to guest
> context, saving away enough info to continue what it was doing when it
> re-enters Xen.
What is the rational behind the scene?

Is it a matter of optimization; Since Hypervisor doesn't 'execute' code
on the behalf of the Guest there is no requirement for per-vcpu
hypervisor stack? {How about system call then}

Or maybe a function of feature; It would be inconvenient to have an
Hypervisor context associated with a VM when trying to migrate to
another environment?

>   Once you get your head around that, a lot of things
> become much clearer.
> 
>     J

I see this as _valuable_ information for Xen newbies (like me). I think
it would be good to have a 'xen/Documentation' folder to capture
Hypervisor specific information?

Correct me if I'm wrong but it looks like the existing 'docs/*' is
geared toward 'operation' rather than internal implementation.

I volunteer to help if needed.


Thanks for your reply Jeremy.

-Etienne

Re: Hypervisor architecture?

[Jeremy Fitzhardinge]

On 05/04/2010 04:36 PM, Etienne Martineau wrote:
>> A vcpu is akin to a task; a domain is like a process (where multi-vcpu =
>> multi-threaded).
>>     
> Does the scheduler schedule 'vcpu' or 'domain'?
>   

vcpus; domains aren't schedulable entities.  They're akin to a Linux mm.

>>   Memory
>> management is largely different.
>>     
> I see two modes of operation?
> (A) HVM; it looks like the Hypervisor is doing 'paging' and maintain
> sPTE.
>   

Yes, though with hap it doesn't need to maintain a shadow pagetable.

> (B) PV; Guest has access to %cr3 and it seems to me that the Hypervisor
> is not involve on the fast path?
>   

Yes and no.  The guest-visible cr3 is the "real" cr3, but the hypervisor
controls what it's being loaded with, and mediates every pte update. 
But the hypervisor doesn't need to do anything to satisfy a tlb miss.

With hvm+shadow, pte updates could be simple memory writes, but the
hypervisor needs to use tlb flushes to sync the guest and shadow
pagetables, and possibly rebuild the shadow pagetable on "tlb miss".

hvm+hap needs much less hypervisor intervention overall.  The main
downside is that tlb misses can be much more expensive in the processor.

>> One of the big differences is that Xen doesn't have a per-vcpu
>> hypervisor (kernel) stack, and vcpus don't have a hypervisor context. 
>> While they're actually running in the hypervisor they use a pcpu stack,
>> but if it blocks/deschedules then it must always return to guest
>> context, saving away enough info to continue what it was doing when it
>> re-enters Xen.
>>     
> What is the rational behind the scene?
>
> Is it a matter of optimization; Since Hypervisor doesn't 'execute' code
> on the behalf of the Guest there is no requirement for per-vcpu
> hypervisor stack? {How about system call then}
>
> Or maybe a function of feature; It would be inconvenient to have an
> Hypervisor context associated with a VM when trying to migrate to
> another environment?
>   

I think the idea is that hypercalls should always have a deterministic
execution time; they always return to the guest's control within a
certain time period, though they may not have completed their job yet. 
It is another way of dealing with "slow syscalls" and how to deal with
syscall interrupt/restart.

I don't know how deterministic hypercall timing is these days, but they
definitely can't hang around in the hypervisor indefinitely without
tying up a pcpu.

> I see this as _valuable_ information for Xen newbies (like me). I think
> it would be good to have a 'xen/Documentation' folder to capture
> Hypervisor specific information?
>
> Correct me if I'm wrong but it looks like the existing 'docs/*' is
> geared toward 'operation' rather than internal implementation.
>
> I volunteer to help if needed.

I'm sure patches will be accepted.

    J

Re: Hypervisor architecture?

[TSFH]

On Tue, 2010-05-04 at 12:25 -0700, Etienne Martineau wrote:
>
> What is common?
> What is different?
> 

There is a book you may find valuable written by David Chisnall

The definitive guide to the Xen hypervisor
ISBN-10: 013234971X
ISBN-13: 978-0132349710

The book covers in great detail the inner-workings of Xen 3.0. Not sure
how much of it is still valid in today's Xen. I know David was working
on an errata for a second print, covering Xen 3.1, but he has postponed
the work on the book for the past year as the sales (unsurprisingly)
have marginally reached 3k copies.

It would also be useful if someone on this list who have read the book
could enlighten the community WRT possible caveats on the book or parts
that no longer resemble the current implementation, as it appears Xen
has changed considerably since the book was released.


-- 
TSFH <taskstructfromhell@googlemail.com>
Context Hell, Inc.

Re: Hypervisor architecture?

[George Dunlap]

On Tue, May 4, 2010 at 6:20 PM, TSFH <taskstructfromhell@googlemail.com> wrote:
> It would also be useful if someone on this list who have read the book
> could enlighten the community WRT possible caveats on the book or parts
> that no longer resemble the current implementation, as it appears Xen
> has changed considerably since the book was released.

I haven't read the book, but we have had several people come to the
list to try to get help getting the "pluggable scheduler" example from
the book working.  The two caveats are:
* The example code, as it's written in the book, doesn't actually work
* The scheduler is actually a lot harder than it looks.  It's at one
of the very lowest layers.  If there's a bug, the whole system hangs
(as opposed to just returning an error message).  Debugging it takes a
lot of skill.  So it's not a good place to start playing around with
Xen. :-)

 -George