Re: [RFC, PATCH 3/24] i386 Vmi interface definition

[Zachary Amsden <zach@vmware.com>]

Chris Wright wrote:
> * Zachary Amsden (zach@vmware.com) wrote:
>   
>> Master definition of VMI interface, including calls, constants, and
>> interface version.
>>     
>
>   
>> +/* VROM call table definitions */
>> +#define VROM_CALL_LEN             32
>> +
>> +typedef struct VROMCallEntry {
>> +   char f[VROM_CALL_LEN];
>> +} VROMCallEntry;
>>     
>
> And the call entry is meant to be handled in whatever mechanism hypervisor
> prefers for its entry points (ABI constraints notwithstanding) as in,
> arbitrary software interrupt, or call gate, etc?  I guess for transparent
> it has to, since those would be local calls.   Quite similar to the
> hypercall entry point that Xen places on the hypercall_page, so easily
> compatible.

See below.

> The document is slightly more descriptive.  The above reserved slots
> are shown as:
>
> 	char		reserved[32];
> 	char		elfHeader[64];
>
> But that's only 3 (0-2).  I think I'm missing some small bit of magic.
>
>   
>> +typedef struct VROMCallTable {
>> +   VROMCallEntry    vromCall[128];           // @ 0x80: ROM calls 4-127
>> +} VROMCallTable;
>>     
>
> That comment eludes me.  Are 0-3 special somehow (IOW, I thought it was
> just 0-2 as per above), and is it suggesting int 0x80?
>   

Yeah, most of this is rather crufty - it is in transition.  We had a 
full blown ROM image with 32-byte aligned stub points at one point for 
all of the VMI calls.  In fact, it still is in that form, and it was 
required to overlap exactly with a native ROM that was built into 
Linux.  See patch 6, VMI magic fixes for the details.

The machinery is already in place to do this, and it is a very nice 
thing that Xen has decided to adopt a similar approach (to the ROM) of 
publishing hypervisor code.  I think they even use 32-byte alignment as 
well.  The power of an indirection at this layer is just too attractive, 
and once you decide on a single binary image, I think it is inevitable 
that everyone will converge on the same idea.  The same concept surfaces 
over and over - vsyscall being another example.  You're basically 
dynamically linking in code at runtime, which is a pretty common thing 
to do, and gives you a very powerful redirection interface.

But we discovered that we couldn't achieve native performance, even 
using directly linked calls into the ROM.  Calling out for sti / cli / 
popf / pushf on fast paths is just too expensive.  We had to inline the 
native code to match native performance.  And this leaves an opportunity 
for flexibility and performance in the hypervisor implementation.

You don't want direct calls to 32-byte stubs; in fact, as Joshua found, 
you can't get optimal performance in a hypervisor that way.  What you 
need is the ability to preferentially inline certain hot calls into the 
VMI layer by using NOP padding.  The non-hot calls can call out to the 
hypercall page.

Publishing code is the first step - inlining is the second, and it gets 
you back the hit you took when indirecting your fast paths.

Zach