Re: [RFC PATCH 0/3] generic hypercall support

[Avi Kivity <avi@redhat.com>]

Gregory Haskins wrote:
>> Consider nested virtualization where the host (H) runs a guest (G1)
>> which is itself a hypervisor, running a guest (G2).  The host exposes
>> a set of virtio (V1..Vn) devices for guest G1.  Guest G1, rather than
>> creating a new virtio devices and bridging it to one of V1..Vn,
>> assigns virtio device V1 to guest G2, and prays.
>>
>> Now guest G2 issues a hypercall.  Host H traps the hypercall, sees it
>> originated in G1 while in guest mode, so it injects it into G1.  G1
>> examines the parameters but can't make any sense of them, so it
>> returns an error to G2.
>>
>> If this were done using mmio or pio, it would have just worked.  With
>> pio, H would have reflected the pio into G1, G1 would have done the
>> conversion from G2's port number into G1's port number and reissued
>> the pio, finally trapped by H and used to issue the I/O. 
>>     
>
> I might be missing something, but I am not seeing the difference here. 
> We have an "address" (in this case the HC-id) and a context (in this
> case G1 running in non-root mode).   Whether the  trap to H is a HC or a
> PIO, the context tells us that it needs to re-inject the same trap to G1
> for proper handling.  So the "address" is re-injected from H to G1 as an
> emulated trap to G1s root-mode, and we continue (just like the PIO).
>   

So far, so good (though in fact mmio can short-circuit G2->H directly).

> And likewise, in both cases, G1 would (should?) know what to do with
> that "address" as it relates to G2, just as it would need to know what
> the PIO address is for.  Typically this would result in some kind of
> translation of that "address", but I suppose even this is completely
> arbitrary and only G1 knows for sure.  E.g. it might translate from
> hypercall vector X to Y similar to your PIO example, it might completely
> change transports, or it might terminate locally (e.g. emulated device
> in G1).   IOW: G2 might be using hypercalls to talk to G1, and G1 might
> be using MMIO to talk to H.  I don't think it matters from a topology
> perspective (though it might from a performance perspective).
>   

How can you translate a hypercall?  G1's and H's hypercall mechanisms 
can be completely different.


>> So the upshoot is that hypercalls for devices must not be the primary
>> method of communications; they're fine as an optimization, but we
>> should always be able to fall back on something else.  We also need to
>> figure out how G1 can stop V1 from advertising hypercall support.
>>     
> I agree it would be desirable to be able to control this exposure. 
> However, I am not currently convinced its strictly necessary because of
> the reason you mentioned above.  And also note that I am not currently
> convinced its even possible to control it.
>
> For instance, what if G1 is an old KVM, or (dare I say) a completely
> different hypervisor?  You could control things like whether G1 can see
> the VMX/SVM option at a coarse level, but once you expose VMX/SVM, who
> is to say what G1 will expose to G2?  G1 may very well advertise a HC
> feature bit to G2 which may allow G2 to try to make a VMCALL.  How do
> you stop that?
>   

I don't see any way.

If, instead of a hypercall we go through the pio hypercall route, then 
it all resolves itself.  G2 issues a pio hypercall, H bounces it to G1, 
G1 either issues a pio or a pio hypercall depending on what the H and G1 
negotiated.  Of course mmio is faster in this case since it traps directly.

btw, what's the hypercall rate you're seeing? at 10K hypercalls/sec, a 
0.4us difference will buy us 0.4% reduction in cpu load, so let's see 
what's the potential gain here.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.