Re: Hypervisor architecture?

[Etienne Martineau <etmartin@cisco.com>]

On Tue, 2010-05-04 at 13:33 -0700, Jeremy Fitzhardinge wrote:
> On 05/04/2010 12:25 PM, Etienne Martineau wrote:
> > Is there any documents around that describe the 'internal' architecture
> > of the Hypervisor?
> >
> > So far my understanding is such that the Hypervisor is a 'strip-down'
> > Linux 2.6.(12/13) + lot's of customization specific to Xen. 
> >   
> 
> Xen and Linux diverged a long time ago, early in the Linux 2.6 series,
> with occasional feature-specific transfusions from Linux into Xen
> since.  There are some similarities in how they deal with platform
> issues (APICs, etc), but they're mostly different now.
> 
> > What is common?
> > What is different?
> >   
> 
> A vcpu is akin to a task; a domain is like a process (where multi-vcpu =
> multi-threaded).
Does the scheduler schedule 'vcpu' or 'domain'?

>   Events are a bit like signals (pending events are
> checked for on return to guest mode and the return is redirected to the
> event handler).  Hypercalls are very similar to syscalls.
Fair enough;

>   Memory
> management is largely different.
I see two modes of operation?
(A) HVM; it looks like the Hypervisor is doing 'paging' and maintain
sPTE.
(B) PV; Guest has access to %cr3 and it seems to me that the Hypervisor
is not involve on the fast path?

I still haven't figure out the 'page_fault' path all the way up to the
guest...

> 
> One of the big differences is that Xen doesn't have a per-vcpu
> hypervisor (kernel) stack, and vcpus don't have a hypervisor context. 
> While they're actually running in the hypervisor they use a pcpu stack,
> but if it blocks/deschedules then it must always return to guest
> context, saving away enough info to continue what it was doing when it
> re-enters Xen.
What is the rational behind the scene?

Is it a matter of optimization; Since Hypervisor doesn't 'execute' code
on the behalf of the Guest there is no requirement for per-vcpu
hypervisor stack? {How about system call then}

Or maybe a function of feature; It would be inconvenient to have an
Hypervisor context associated with a VM when trying to migrate to
another environment?

>   Once you get your head around that, a lot of things
> become much clearer.
> 
>     J

I see this as _valuable_ information for Xen newbies (like me). I think
it would be good to have a 'xen/Documentation' folder to capture
Hypervisor specific information?

Correct me if I'm wrong but it looks like the existing 'docs/*' is
geared toward 'operation' rather than internal implementation.

I volunteer to help if needed.


Thanks for your reply Jeremy.

-Etienne