Re: Hypervisor architecture?

[Jeremy Fitzhardinge <jeremy@goop.org>]

On 05/04/2010 04:36 PM, Etienne Martineau wrote:
>> A vcpu is akin to a task; a domain is like a process (where multi-vcpu =
>> multi-threaded).
>>     
> Does the scheduler schedule 'vcpu' or 'domain'?
>   

vcpus; domains aren't schedulable entities.  They're akin to a Linux mm.

>>   Memory
>> management is largely different.
>>     
> I see two modes of operation?
> (A) HVM; it looks like the Hypervisor is doing 'paging' and maintain
> sPTE.
>   

Yes, though with hap it doesn't need to maintain a shadow pagetable.

> (B) PV; Guest has access to %cr3 and it seems to me that the Hypervisor
> is not involve on the fast path?
>   

Yes and no.  The guest-visible cr3 is the "real" cr3, but the hypervisor
controls what it's being loaded with, and mediates every pte update. 
But the hypervisor doesn't need to do anything to satisfy a tlb miss.

With hvm+shadow, pte updates could be simple memory writes, but the
hypervisor needs to use tlb flushes to sync the guest and shadow
pagetables, and possibly rebuild the shadow pagetable on "tlb miss".

hvm+hap needs much less hypervisor intervention overall.  The main
downside is that tlb misses can be much more expensive in the processor.

>> One of the big differences is that Xen doesn't have a per-vcpu
>> hypervisor (kernel) stack, and vcpus don't have a hypervisor context. 
>> While they're actually running in the hypervisor they use a pcpu stack,
>> but if it blocks/deschedules then it must always return to guest
>> context, saving away enough info to continue what it was doing when it
>> re-enters Xen.
>>     
> What is the rational behind the scene?
>
> Is it a matter of optimization; Since Hypervisor doesn't 'execute' code
> on the behalf of the Guest there is no requirement for per-vcpu
> hypervisor stack? {How about system call then}
>
> Or maybe a function of feature; It would be inconvenient to have an
> Hypervisor context associated with a VM when trying to migrate to
> another environment?
>   

I think the idea is that hypercalls should always have a deterministic
execution time; they always return to the guest's control within a
certain time period, though they may not have completed their job yet. 
It is another way of dealing with "slow syscalls" and how to deal with
syscall interrupt/restart.

I don't know how deterministic hypercall timing is these days, but they
definitely can't hang around in the hypervisor indefinitely without
tying up a pcpu.

> I see this as _valuable_ information for Xen newbies (like me). I think
> it would be good to have a 'xen/Documentation' folder to capture
> Hypervisor specific information?
>
> Correct me if I'm wrong but it looks like the existing 'docs/*' is
> geared toward 'operation' rather than internal implementation.
>
> I volunteer to help if needed.

I'm sure patches will be accepted.

    J