Re: [PATCH v7 3/3] x86/hvm: Indicate avaliability of HW support of APIC virtualization to HVM guests

[Boris Ostrovsky <boris.ostrovsky@oracle.com>]

On 03/25/2014 09:03 PM, Zhang, Yang Z wrote:
> Boris Ostrovsky wrote on 2014-03-25:
>> On 03/25/2014 05:45 AM, Jan Beulich wrote:
>>>>>> On 25.03.14 at 00:18, <boris.ostrovsky@oracle.com> wrote:
>>>> +void vmx_hypervisor_cpuid_leaf(uint32_t sub_idx,
>>>> +                               uint32_t *eax, uint32_t *ebx,
>>>> +                               uint32_t *ecx, uint32_t *edx) {
>>>> +    if ( sub_idx != 0 )
>>>> +        return;
>>>> +
>>>> +    if ( cpu_has_vmx_apic_reg_virt )
>>>> +        *eax |= XEN_HVM_CPUID_APIC_ACCESS_VIRT;
>>>> +    if ( cpu_has_vmx_virtualize_x2apic_mode )
>>>> +        *eax |= XEN_HVM_CPUID_X2APIC_VIRT; }
>>> So did the two of you then settle on (a) needing to expose two bits
>>> rather than just one and (b) these being the two relevant features
>>> to expose?
>> My argument is that we can't know which APIC model a guest uses and so
>> both are needed. For PVHVM we default to APIC (MMIO accesses), I can't
>> remember what unenlightened HVM Linux would do. And then there are
>> other OSs.
>>
>> For (b) having either (or both) of these two seems to be sufficient to
>> bring down the number of VMEXITs when switching from pirqs to APIC.
>> It's more important to agree on (a) since for (b) we can always add another bit.
> In currently Xen, virtualize_x2apic_mode takes effect only when APICv is enabled. Without APICv, virtualize_x2apic_mode is never set.
> Per your request, you only want to drop the pirqs if guest is using x2apic. So, just check it inside guest OS is enough. NB: to use x2apic for guest doesn't require the virtualize_x2apic_mode been set.

Yang and I talked a bit off-list and I don't think there was an 
agreement on this.

Here is a simple experiment to demonstrate why exposing 
virtualize_x2apic_mode is important (with one correction to what I said 
earlier: PVHVM guest will actualy default to x2apic, at least on Intel 
CPUs):

With existing code (using pirqs, i.e. no APIC/x2apic accesses), VMEXIT 
stats look as follows:

14397 HLT
   22420 INJ_VIRQ
    8551 INTR
   29849 INTR_WINDOW
       4 MMIO_READ
       2 MMIO_WRITE
     628 TRAP
       2 unknown
   78157 VMENTRY
   78157 VMEXIT
   29299 VMMCALL

Without pirqs (i.e. guest using x2APIC), and with virtualized x2apic, 
virtualized APIC register accesses:

   15572 HLT
     164 INJ_VIRQ
    4218 INTR
     184 INTR_WINDOW
     624 TRAP
   23199 VMENTRY
   23198 VMEXIT
    2607 VMMCALL

Without pirqs (again, guest uses x2APIC), without virtualized x2apic 
support but with virtualized APIC register access (which can be 
simulated by having msr_high of MSR_IA32_VMX_PROCBASED_CTLS2 clear bit 4):

      53 cpu_change
   18674 HLT
     226 INJ_VIRQ
   11702 INTR
     294 INTR_WINDOW
   35186 MSR_WRITE
     791 TRAP
   70441 VMENTRY
   70440 VMEXIT
3823 VMMCALL

In other words, if the guest is unaware of the fact that x2apic is not 
virtualized, it will disable pirqs for no good reason.

-boris