Re: How does kvm support x2apic?

["Zhang Haoyu" <zhanghy@sangfor.com>]

>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> According to this:
>>>>>>>>
>>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>>>>>>
>>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>>>>>>> even earlier.
>>>>>>>>
>>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>>>>>>
>>>>>>>> However, this following patch:
>>>>>>>>
>>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>>>>>>
>>>>>>>> Also claims that adding support for x2apic. But this later patch was
>>>>>>>> for kernel 3.9.
>>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>>>>>>> which belongs to APIC virtualization from certain angle.
>>>>>> Thanks Haoyu, I am reading the Intel x2apic manual.
>>>>>>
>>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>>>>>
>>>>>> but I don't see the so called "virtual x2apic mode", it seems that the
>>>>>> manual does not mention anything about that. Do you mean that when the
>>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
>>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
>>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or,
>>>>>> "emulate x2apic for guest" is the foundation of "support virtual
>>>>>> x2apic mode"?
>>>>>>
>>>>>> -Jidong
>>>>>>
>>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in
>>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29
>>>>>- APIC virtualization and virtual interrupts. Ignore my questions
>>>>>please, but thank you anyway.
>>>>>
>>>> Yes, virtual x2apic mode is part of APICv.
>>>> Guest has no idea about it is running in a virtual machine,
>>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
>>>>
>>>
>>>Regarding this "VMM also prevent guest from enabling/disabling virtual
>>>x2apic mode", do you mean that guest is not allowed to do something
>>>like:
>>>
>>>Assuming we are in the guest, and we see the original value of the
>>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was
>>>enabled. And now, we try to set bit 10 (EXTD bit):
>>>
>>>wrmsr 0x1b 0xfee00d00.
>>>
>>>Do you mean this wrmsr operation is not allowed? If fact, I tried
>>>this, and this simply crashes the guest kernel. (I am using Linux
>>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And
>>>this is the exact reason that I initialized this whole discussion. My
>>>feeling is this might be a kvm bug, so I wish to figure out and try to
>>>fix it.
>>>
>> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode,
>> you are writing the wrong MSR address.
>> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.
>>
>Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR.
>In particular I was trying to write bit 10, which is the x2APIC
>enable/disable bit. As I said, I just wish to enable that x2APIC. The
>original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10
>is 0. According to SDM manual, this bit has to be 1 if we want to
>enable x2APIC.
>
Sorry, I misread it.
Please try
u64 val;
rdmsrl(0x0000001b, val);
if ( !(val & (1UL << 10)) )
    wrmsrl(0x0000001b, val | (1UL << 10));

If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure.
Please report the crash message.

>>>>>>>>
>>>>>>>> So I am very confused:
>>>>>>>>
>>>>>>>> First, what's the difference between these two patches? Or say, does
>>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>>>>>>
>>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>>>>>>
>>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>>>>>>has exposed this feature to guest.) The word "use" means something
>>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>>>>>>> IA32_APICBASE MSR (i.e. bit 10).
>>>>>>>>
>>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>>>>>>> please see enable_IR_x2apic().
>>>>>>>
>>>>>>>> -Jidong