RE: Device model architecture (Was Re: Re: Are linkerscripts needed?)

RE: Device model architecture (Was Re: Re: Are linkerscripts needed?)

[Ian Pratt]

 > > When switching between the vmcs guest and the minios, we 
> just need to 
> > switch new values into the guest_table variable and the shadow_mode 
> > variable and then all the Xen logic will do the right 
> thing. (no cr3 
> > flush in incurred)
> 
> I guess the main logic behind your argument is that there is 
> no need to fully virtualize the device models, so no need to 
> run them within a non-root VMCS.

We can't run them in the same non-root VMCS as the guest since we need
some virtual address space. Since the hardware does a cr3 switch to the
monitor_table when doing a vmexit, wwe might as well make better use of
this and treat the device models as just another paravirtualized guest.
 
> This should give us some of the same management benefits that 
> running them inside a VMX domain would give us (memory 
> sharing, device models go away when the domain is destroyed etc).

Yep. It also makes them interact with backend drivers in an identical
way to normal paravirtualized front ends. Ideally, we'd pick devices to
emulate that use DMA and descriptor rings so that the only time a VMexit
would occur would when the virtual device is 'kicked' when making the tx
ring non-empty, or the receive interrupt. Performance should be pretty
good, and if you want maximum performance, install the paravirtualzed
drivers that are able to do page flipping.

> The only comment I have is: for the case where the device 
> models require the services of a backend driver, you might be 
> paying more than what we currently pay (one domain switch) 
> i.e. vmx domain -> mini os -> backend. But it should be 
> faster for everything else.

The vmx_domain to mini-OS switch happens as part of the vmexit. Work
will get queued just like a normal frontend talking to a backend, so I
don't think there's a performance issue here -- it will certainly be
faster than the current situation. 

Devices such as the APIC/IOAPIC/PIT etc can all be emulated without
calling out of the device model, and should work with just the same
performance as having them in Xen as we do today.

Best,
Ian 

Re: Device model architecture (Was Re: Re: Are linkerscripts needed?)

[Arun Sharma]

Ian Pratt wrote:
>>I guess the main logic behind your argument is that there is 
>>no need to fully virtualize the device models, so no need to 
>>run them within a non-root VMCS.
> 
> 
> We can't run them in the same non-root VMCS as the guest since we need
> some virtual address space. [...]

You can, with a world switch using a different CR3.

> 
>>The only comment I have is: for the case where the device 
>>models require the services of a backend driver, you might be 
>>paying more than what we currently pay (one domain switch) 
>>i.e. vmx domain -> mini os -> backend. But it should be 
>>faster for everything else.
> 
> 
> The vmx_domain to mini-OS switch happens as part of the vmexit.

It's actually the vmexit + cost of upcall to the mini-os + cost of 
domain switch to the backend (say dom0).

Right now, it's vmexit + cost of domain switch to dom0 + cost of dom0 
kernel to user mode device model.

So the difference is

cost of dom0 kernel to user - cost of frontend to backend switch

I suspect that the delta is small and not much to worry about.

> Devices such as the APIC/IOAPIC/PIT etc can all be emulated without
> calling out of the device model, and should work with just the same
> performance as having them in Xen as we do today.

True, it's definitely a win for devices that don't require communication 
with the backend.

Also, some customers might want to use the split I/O model, where the 
VMX domain directly wants to use a raw partition on real disk. I'd 
rather have drivers in the OS hosting the device-models instead of going 
through the backend.

	-Arun

Re: Device model architecture (Was Re: Re: Are linkerscripts needed?)

[Arun Sharma]

Ian Pratt wrote:
>  > > When switching between the vmcs guest and the minios, we 
> 
>>just need to 
>>
>>>switch new values into the guest_table variable and the shadow_mode 
>>>variable and then all the Xen logic will do the right 
>>
>>thing. (no cr3 
>>
>>>flush in incurred)
>>
>>I guess the main logic behind your argument is that there is 
>>no need to fully virtualize the device models, so no need to 
>>run them within a non-root VMCS.
> 
> 
> We can't run them in the same non-root VMCS as the guest since we need
> some virtual address space. Since the hardware does a cr3 switch to the
> monitor_table when doing a vmexit, wwe might as well make better use of
> this and treat the device models as just another paravirtualized guest.
>  

I thought some more about this. We can't think of device models as a 
bunch of code that run only during a vmexit. It needs to be able to 
handle asynchronous events such as keyboard/mouse input or other sources 
of events and inject interrupts into the VMX domain ASAP.

Specifically, on a dual CPU system or a HT system, we should be able to 
run device models on one CPU and the VMX domains on another CPU. This is 
possible in the current design.

	-Arun

RE: Device model architecture (Was Re: Re: Are linkerscripts needed?)

[Ian Pratt]

> > We can't run them in the same non-root VMCS as the guest 
> since we need 
> > some virtual address space. Since the hardware does a cr3 switch to 
> > the monitor_table when doing a vmexit, wwe might as well 
> make better 
> > use of this and treat the device models as just another 
> paravirtualized guest.
> >  
> 
> I thought some more about this. We can't think of device 
> models as a bunch of code that run only during a vmexit. It 
> needs to be able to handle asynchronous events such as 
> keyboard/mouse input or other sources of events and inject 
> interrupts into the VMX domain ASAP.

Sure, we need to generate interrupts into the VMX domain (e.g. as a
result of an event from a backend to a frontend indicating a network
packet has been received). The ipi of receiving the event will cause a
vmexit, execute the device model, and then inject an interrupt into the
vmx guest.
 
> Specifically, on a dual CPU system or a HT system, we should 
> be able to run device models on one CPU and the VMX domains 
> on another CPU. 

I'm not totally convinced about this. The frontend drivers are pretty
simple and take little in the way of CPU. Although its useful to overlap
backend execution with the domain, I'm not so sure its useful to overlap
frontend execution.

Ian