How does kvm support x2apic?

How does kvm support x2apic?

[Jidong Xiao]

Hi,

According to this:

https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd

It looks like kvm have been supporting x2apic since kernel 2.6.32, or
even earlier.

However, this following patch:

https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5

Also claims that adding support for x2apic. But this later patch was
for kernel 3.9.

So I am very confused:

First, what's the difference between these two patches? Or say, does
kvm support x2apic since kernel 2.6.32, or since kernel 3.9?

Second, can guest use x2apic even if the host does not? (Assuming qemu
has exposed this feature to guest.) The word "use" means something
like, accessing x2apic registers, setting the x2apic enable bit in
IA32_APICBASE MSR (i.e. bit 10).

-Jidong

Re: How does kvm support x2apic?

[Zhang Haoyu]

> Hi,
>
> According to this:
> 
> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
> 
> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
> even earlier.
>
This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.

> However, this following patch:
> 
> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
> 
> Also claims that adding support for x2apic. But this later patch was
> for kernel 3.9.
This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
which belongs to APIC virtualization from certain angle.
>
> So I am very confused:
> 
> First, what's the difference between these two patches? Or say, does
> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
> 
kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.

> Second, can guest use x2apic even if the host does not? (Assuming qemu
>has exposed this feature to guest.) The word "use" means something
> like, accessing x2apic registers, setting the x2apic enable bit in
> IA32_APICBASE MSR (i.e. bit 10).
> 
Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
please see enable_IR_x2apic().

> -Jidong

Re: How does kvm support x2apic?

[Jidong Xiao]

On Tue, Jun 10, 2014 at 9:33 PM, Zhang Haoyu <zhanghy@sangfor.com> wrote:
>> Hi,
>>
>> According to this:
>>
>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>
>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>> even earlier.
>>
> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>
>> However, this following patch:
>>
>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>
>> Also claims that adding support for x2apic. But this later patch was
>> for kernel 3.9.
> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
> which belongs to APIC virtualization from certain angle.
Thanks Haoyu, I am reading the Intel x2apic manual.

http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf

but I don't see the so called "virtual x2apic mode", it seems that the
manual does not mention anything about that. Do you mean that when the
Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
this the same as the aforementioned "emulate x2apic for guest"? Or,
"emulate x2apic for guest" is the foundation of "support virtual
x2apic mode"?

-Jidong

>>
>> So I am very confused:
>>
>> First, what's the difference between these two patches? Or say, does
>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>
> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>
>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>has exposed this feature to guest.) The word "use" means something
>> like, accessing x2apic registers, setting the x2apic enable bit in
>> IA32_APICBASE MSR (i.e. bit 10).
>>
> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
> please see enable_IR_x2apic().
>
>> -Jidong
>

Re: How does kvm support x2apic?

[Jidong Xiao]

On Tue, Jun 10, 2014 at 11:54 PM, Jidong Xiao <jidong.xiao@gmail.com> wrote:
> On Tue, Jun 10, 2014 at 9:33 PM, Zhang Haoyu <zhanghy@sangfor.com> wrote:
>>> Hi,
>>>
>>> According to this:
>>>
>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>
>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>> even earlier.
>>>
>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>
>>> However, this following patch:
>>>
>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>
>>> Also claims that adding support for x2apic. But this later patch was
>>> for kernel 3.9.
>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>> which belongs to APIC virtualization from certain angle.
> Thanks Haoyu, I am reading the Intel x2apic manual.
>
> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>
> but I don't see the so called "virtual x2apic mode", it seems that the
> manual does not mention anything about that. Do you mean that when the
> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
> this the same as the aforementioned "emulate x2apic for guest"? Or,
> "emulate x2apic for guest" is the foundation of "support virtual
> x2apic mode"?
>
> -Jidong
>
Oh, I found the answer, the virtualized x2apic mode is not defined in
the x2apic manual, but is defined in the intel SDM manual, chapter 29
- APIC virtualization and virtual interrupts. Ignore my questions
please, but thank you anyway.

-Jidong

>>>
>>> So I am very confused:
>>>
>>> First, what's the difference between these two patches? Or say, does
>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>
>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>
>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>has exposed this feature to guest.) The word "use" means something
>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>> IA32_APICBASE MSR (i.e. bit 10).
>>>
>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>> please see enable_IR_x2apic().
>>
>>> -Jidong
>>

Re: How does kvm support x2apic?

[Zhang Haoyu]

>>>> Hi,
>>>>
>>>> According to this:
>>>>
>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>>
>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>>> even earlier.
>>>>
>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>>
>>>> However, this following patch:
>>>>
>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>>
>>>> Also claims that adding support for x2apic. But this later patch was
>>>> for kernel 3.9.
>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>>> which belongs to APIC virtualization from certain angle.
>> Thanks Haoyu, I am reading the Intel x2apic manual.
>>
>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>
>> but I don't see the so called "virtual x2apic mode", it seems that the
>> manual does not mention anything about that. Do you mean that when the
>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
>> this the same as the aforementioned "emulate x2apic for guest"? Or,
>> "emulate x2apic for guest" is the foundation of "support virtual
>> x2apic mode"?
>>
>> -Jidong
>>
>Oh, I found the answer, the virtualized x2apic mode is not defined in
>the x2apic manual, but is defined in the intel SDM manual, chapter 29
>- APIC virtualization and virtual interrupts. Ignore my questions
>please, but thank you anyway.
>
Yes, virtual x2apic mode is part of APICv.
Guest has no idea about it is running in a virtual machine, 
VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.

>-Jidong
>
>>>>
>>>> So I am very confused:
>>>>
>>>> First, what's the difference between these two patches? Or say, does
>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>>
>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>>
>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>>has exposed this feature to guest.) The word "use" means something
>>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>>> IA32_APICBASE MSR (i.e. bit 10).
>>>>
>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>>> please see enable_IR_x2apic().
>>>
>>>> -Jidong

Re: How does kvm support x2apic?

[Jidong Xiao]

On Wed, Jun 11, 2014 at 12:53 AM, Zhang Haoyu <zhanghy@sangfor.com> wrote:
>>>>> Hi,
>>>>>
>>>>> According to this:
>>>>>
>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>>>
>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>>>> even earlier.
>>>>>
>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>>>
>>>>> However, this following patch:
>>>>>
>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>>>
>>>>> Also claims that adding support for x2apic. But this later patch was
>>>>> for kernel 3.9.
>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>>>> which belongs to APIC virtualization from certain angle.
>>> Thanks Haoyu, I am reading the Intel x2apic manual.
>>>
>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>>
>>> but I don't see the so called "virtual x2apic mode", it seems that the
>>> manual does not mention anything about that. Do you mean that when the
>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
>>> this the same as the aforementioned "emulate x2apic for guest"? Or,
>>> "emulate x2apic for guest" is the foundation of "support virtual
>>> x2apic mode"?
>>>
>>> -Jidong
>>>
>>Oh, I found the answer, the virtualized x2apic mode is not defined in
>>the x2apic manual, but is defined in the intel SDM manual, chapter 29
>>- APIC virtualization and virtual interrupts. Ignore my questions
>>please, but thank you anyway.
>>
> Yes, virtual x2apic mode is part of APICv.
> Guest has no idea about it is running in a virtual machine,
> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
>

Regarding this "VMM also prevent guest from enabling/disabling virtual
x2apic mode", do you mean that guest is not allowed to do something
like:

Assuming we are in the guest, and we see the original value of the
IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was
enabled. And now, we try to set bit 10 (EXTD bit):

wrmsr 0x1b 0xfee00d00.

Do you mean this wrmsr operation is not allowed? If fact, I tried
this, and this simply crashes the guest kernel. (I am using Linux
2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And
this is the exact reason that I initialized this whole discussion. My
feeling is this might be a kvm bug, so I wish to figure out and try to
fix it.

>>>>>
>>>>> So I am very confused:
>>>>>
>>>>> First, what's the difference between these two patches? Or say, does
>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>>>
>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>>>
>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>>>has exposed this feature to guest.) The word "use" means something
>>>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>>>> IA32_APICBASE MSR (i.e. bit 10).
>>>>>
>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>>>> please see enable_IR_x2apic().
>>>>
>>>>> -Jidong
>

Re: How does kvm support x2apic?

[Jidong Xiao]

On Wed, Jun 11, 2014 at 1:08 AM, Jidong Xiao <jidong.xiao@gmail.com> wrote:
> On Wed, Jun 11, 2014 at 12:53 AM, Zhang Haoyu <zhanghy@sangfor.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> According to this:
>>>>>>
>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>>>>
>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>>>>> even earlier.
>>>>>>
>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>>>>
>>>>>> However, this following patch:
>>>>>>
>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>>>>
>>>>>> Also claims that adding support for x2apic. But this later patch was
>>>>>> for kernel 3.9.
>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>>>>> which belongs to APIC virtualization from certain angle.
>>>> Thanks Haoyu, I am reading the Intel x2apic manual.
>>>>
>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>>>
>>>> but I don't see the so called "virtual x2apic mode", it seems that the
>>>> manual does not mention anything about that. Do you mean that when the
>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
>>>> this the same as the aforementioned "emulate x2apic for guest"? Or,
>>>> "emulate x2apic for guest" is the foundation of "support virtual
>>>> x2apic mode"?
>>>>
>>>> -Jidong
>>>>
>>>Oh, I found the answer, the virtualized x2apic mode is not defined in
>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29
>>>- APIC virtualization and virtual interrupts. Ignore my questions
>>>please, but thank you anyway.
>>>
>> Yes, virtual x2apic mode is part of APICv.
>> Guest has no idea about it is running in a virtual machine,
>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
>>
>
> Regarding this "VMM also prevent guest from enabling/disabling virtual
> x2apic mode", do you mean that guest is not allowed to do something
> like:
>
> Assuming we are in the guest, and we see the original value of the
> IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was
> enabled. And now, we try to set bit 10 (EXTD bit):
>
> wrmsr 0x1b 0xfee00d00.
>
> Do you mean this wrmsr operation is not allowed? If fact, I tried
> this, and this simply crashes the guest kernel. (I am using Linux
> 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And
> this is the exact reason that I initialized this whole discussion. My
> feeling is this might be a kvm bug, so I wish to figure out and try to
> fix it.
>

Can someone test this also? Just set bit 10 of IA32_APICBASE MSR in
your guest, and see if the guest crashes. According to Intel SDM
manual, this is the approach to enable x2apic mode. And I think
normally it should not crash the guest kernel.

One can write that msr with the following tool:

https://01.org/msr-tools

And compile this tool and just run the aforementioned command:

wrmsr 0x1b 0xfee00d00 (rdmsr 0x1b to confirm the original value is 0xfee00900.)

I tried on several machines, and this command just crash the kernel.
And I am not sure why.

>>>>>>
>>>>>> So I am very confused:
>>>>>>
>>>>>> First, what's the difference between these two patches? Or say, does
>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>>>>
>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>>>>
>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>>>>has exposed this feature to guest.) The word "use" means something
>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>>>>> IA32_APICBASE MSR (i.e. bit 10).
>>>>>>
>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>>>>> please see enable_IR_x2apic().
>>>>>
>>>>>> -Jidong
>>

Re: How does kvm support x2apic?

[Zhang Haoyu]

>>>>>> Hi,
>>>>>>
>>>>>> According to this:
>>>>>>
>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>>>>
>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>>>>> even earlier.
>>>>>>
>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>>>>
>>>>>> However, this following patch:
>>>>>>
>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>>>>
>>>>>> Also claims that adding support for x2apic. But this later patch was
>>>>>> for kernel 3.9.
>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>>>>> which belongs to APIC virtualization from certain angle.
>>>> Thanks Haoyu, I am reading the Intel x2apic manual.
>>>>
>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>>>
>>>> but I don't see the so called "virtual x2apic mode", it seems that the
>>>> manual does not mention anything about that. Do you mean that when the
>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
>>>> this the same as the aforementioned "emulate x2apic for guest"? Or,
>>>> "emulate x2apic for guest" is the foundation of "support virtual
>>>> x2apic mode"?
>>>>
>>>> -Jidong
>>>>
>>>Oh, I found the answer, the virtualized x2apic mode is not defined in
>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29
>>>- APIC virtualization and virtual interrupts. Ignore my questions
>>>please, but thank you anyway.
>>>
>> Yes, virtual x2apic mode is part of APICv.
>> Guest has no idea about it is running in a virtual machine,
>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
>>
>
>Regarding this "VMM also prevent guest from enabling/disabling virtual
>x2apic mode", do you mean that guest is not allowed to do something
>like:
>
>Assuming we are in the guest, and we see the original value of the
>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was
>enabled. And now, we try to set bit 10 (EXTD bit):
>
>wrmsr 0x1b 0xfee00d00.
>
>Do you mean this wrmsr operation is not allowed? If fact, I tried
>this, and this simply crashes the guest kernel. (I am using Linux
>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And
>this is the exact reason that I initialized this whole discussion. My
>feeling is this might be a kvm bug, so I wish to figure out and try to
>fix it.
>
The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode,
you are writing the wrong MSR address.
Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.

>>>>>>
>>>>>> So I am very confused:
>>>>>>
>>>>>> First, what's the difference between these two patches? Or say, does
>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>>>>
>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>>>>
>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>>>>has exposed this feature to guest.) The word "use" means something
>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>>>>> IA32_APICBASE MSR (i.e. bit 10).
>>>>>>
>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>>>>> please see enable_IR_x2apic().
>>>>>
>>>>>> -Jidong

Re: How does kvm support x2apic?

[Jidong Xiao]

On Wed, Jun 11, 2014 at 2:27 AM, Zhang Haoyu <zhanghy@sangfor.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> According to this:
>>>>>>>
>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>>>>>
>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>>>>>> even earlier.
>>>>>>>
>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>>>>>
>>>>>>> However, this following patch:
>>>>>>>
>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>>>>>
>>>>>>> Also claims that adding support for x2apic. But this later patch was
>>>>>>> for kernel 3.9.
>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>>>>>> which belongs to APIC virtualization from certain angle.
>>>>> Thanks Haoyu, I am reading the Intel x2apic manual.
>>>>>
>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>>>>
>>>>> but I don't see the so called "virtual x2apic mode", it seems that the
>>>>> manual does not mention anything about that. Do you mean that when the
>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or,
>>>>> "emulate x2apic for guest" is the foundation of "support virtual
>>>>> x2apic mode"?
>>>>>
>>>>> -Jidong
>>>>>
>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in
>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29
>>>>- APIC virtualization and virtual interrupts. Ignore my questions
>>>>please, but thank you anyway.
>>>>
>>> Yes, virtual x2apic mode is part of APICv.
>>> Guest has no idea about it is running in a virtual machine,
>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
>>>
>>
>>Regarding this "VMM also prevent guest from enabling/disabling virtual
>>x2apic mode", do you mean that guest is not allowed to do something
>>like:
>>
>>Assuming we are in the guest, and we see the original value of the
>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was
>>enabled. And now, we try to set bit 10 (EXTD bit):
>>
>>wrmsr 0x1b 0xfee00d00.
>>
>>Do you mean this wrmsr operation is not allowed? If fact, I tried
>>this, and this simply crashes the guest kernel. (I am using Linux
>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And
>>this is the exact reason that I initialized this whole discussion. My
>>feeling is this might be a kvm bug, so I wish to figure out and try to
>>fix it.
>>
> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode,
> you are writing the wrong MSR address.
> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.
>
Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR.
In particular I was trying to write bit 10, which is the x2APIC
enable/disable bit. As I said, I just wish to enable that x2APIC. The
original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10
is 0. According to SDM manual, this bit has to be 1 if we want to
enable x2APIC.

>>>>>>>
>>>>>>> So I am very confused:
>>>>>>>
>>>>>>> First, what's the difference between these two patches? Or say, does
>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>>>>>
>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>>>>>
>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>>>>>has exposed this feature to guest.) The word "use" means something
>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>>>>>> IA32_APICBASE MSR (i.e. bit 10).
>>>>>>>
>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>>>>>> please see enable_IR_x2apic().
>>>>>>
>>>>>>> -Jidong
>

Re: How does kvm support x2apic?

[Zhang Haoyu]

>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> According to this:
>>>>>>>>
>>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>>>>>>
>>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>>>>>>> even earlier.
>>>>>>>>
>>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>>>>>>
>>>>>>>> However, this following patch:
>>>>>>>>
>>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>>>>>>
>>>>>>>> Also claims that adding support for x2apic. But this later patch was
>>>>>>>> for kernel 3.9.
>>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>>>>>>> which belongs to APIC virtualization from certain angle.
>>>>>> Thanks Haoyu, I am reading the Intel x2apic manual.
>>>>>>
>>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>>>>>
>>>>>> but I don't see the so called "virtual x2apic mode", it seems that the
>>>>>> manual does not mention anything about that. Do you mean that when the
>>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
>>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
>>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or,
>>>>>> "emulate x2apic for guest" is the foundation of "support virtual
>>>>>> x2apic mode"?
>>>>>>
>>>>>> -Jidong
>>>>>>
>>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in
>>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29
>>>>>- APIC virtualization and virtual interrupts. Ignore my questions
>>>>>please, but thank you anyway.
>>>>>
>>>> Yes, virtual x2apic mode is part of APICv.
>>>> Guest has no idea about it is running in a virtual machine,
>>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
>>>>
>>>
>>>Regarding this "VMM also prevent guest from enabling/disabling virtual
>>>x2apic mode", do you mean that guest is not allowed to do something
>>>like:
>>>
>>>Assuming we are in the guest, and we see the original value of the
>>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was
>>>enabled. And now, we try to set bit 10 (EXTD bit):
>>>
>>>wrmsr 0x1b 0xfee00d00.
>>>
>>>Do you mean this wrmsr operation is not allowed? If fact, I tried
>>>this, and this simply crashes the guest kernel. (I am using Linux
>>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And
>>>this is the exact reason that I initialized this whole discussion. My
>>>feeling is this might be a kvm bug, so I wish to figure out and try to
>>>fix it.
>>>
>> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode,
>> you are writing the wrong MSR address.
>> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.
>>
>Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR.
>In particular I was trying to write bit 10, which is the x2APIC
>enable/disable bit. As I said, I just wish to enable that x2APIC. The
>original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10
>is 0. According to SDM manual, this bit has to be 1 if we want to
>enable x2APIC.
>
Sorry, I misread it.
Please try
u64 val;
rdmsrl(0x0000001b, val);
if ( !(val & (1UL << 10)) )
    wrmsrl(0x0000001b, val | (1UL << 10));

If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure.
Please report the crash message.

>>>>>>>>
>>>>>>>> So I am very confused:
>>>>>>>>
>>>>>>>> First, what's the difference between these two patches? Or say, does
>>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>>>>>>
>>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>>>>>>
>>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>>>>>>has exposed this feature to guest.) The word "use" means something
>>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>>>>>>> IA32_APICBASE MSR (i.e. bit 10).
>>>>>>>>
>>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>>>>>>> please see enable_IR_x2apic().
>>>>>>>
>>>>>>>> -Jidong

Re: How does kvm support x2apic?

[Jidong Xiao]

On Wed, Jun 11, 2014 at 4:49 AM, Zhang Haoyu <zhanghy@sangfor.com> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> According to this:
>>>>>>>>>
>>>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>>>>>>>
>>>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>>>>>>>> even earlier.
>>>>>>>>>
>>>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>>>>>>>
>>>>>>>>> However, this following patch:
>>>>>>>>>
>>>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>>>>>>>
>>>>>>>>> Also claims that adding support for x2apic. But this later patch was
>>>>>>>>> for kernel 3.9.
>>>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>>>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>>>>>>>> which belongs to APIC virtualization from certain angle.
>>>>>>> Thanks Haoyu, I am reading the Intel x2apic manual.
>>>>>>>
>>>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>>>>>>
>>>>>>> but I don't see the so called "virtual x2apic mode", it seems that the
>>>>>>> manual does not mention anything about that. Do you mean that when the
>>>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
>>>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
>>>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or,
>>>>>>> "emulate x2apic for guest" is the foundation of "support virtual
>>>>>>> x2apic mode"?
>>>>>>>
>>>>>>> -Jidong
>>>>>>>
>>>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in
>>>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29
>>>>>>- APIC virtualization and virtual interrupts. Ignore my questions
>>>>>>please, but thank you anyway.
>>>>>>
>>>>> Yes, virtual x2apic mode is part of APICv.
>>>>> Guest has no idea about it is running in a virtual machine,
>>>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
>>>>>
>>>>
>>>>Regarding this "VMM also prevent guest from enabling/disabling virtual
>>>>x2apic mode", do you mean that guest is not allowed to do something
>>>>like:
>>>>
>>>>Assuming we are in the guest, and we see the original value of the
>>>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was
>>>>enabled. And now, we try to set bit 10 (EXTD bit):
>>>>
>>>>wrmsr 0x1b 0xfee00d00.
>>>>
>>>>Do you mean this wrmsr operation is not allowed? If fact, I tried
>>>>this, and this simply crashes the guest kernel. (I am using Linux
>>>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And
>>>>this is the exact reason that I initialized this whole discussion. My
>>>>feeling is this might be a kvm bug, so I wish to figure out and try to
>>>>fix it.
>>>>
>>> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode,
>>> you are writing the wrong MSR address.
>>> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.
>>>
>>Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR.
>>In particular I was trying to write bit 10, which is the x2APIC
>>enable/disable bit. As I said, I just wish to enable that x2APIC. The
>>original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10
>>is 0. According to SDM manual, this bit has to be 1 if we want to
>>enable x2APIC.
>>
> Sorry, I misread it.
> Please try
> u64 val;
> rdmsrl(0x0000001b, val);
> if ( !(val & (1UL << 10)) )
>     wrmsrl(0x0000001b, val | (1UL << 10));
>
> If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure.
> Please report the crash message.
>
Yes I wrote a simple kernel module which just includes the above four
lines in the module init function, and when I try to install the
module with insmod command, the crash still happens, the crash message
is:

[ 1025.424260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
task.ti=f58b2000)
[ 1025.425093] Stack:
[ 1025.425093] Call Trace:
[ 1025.425093] Leftover inexact backtrace:
[ 1025.425093]
[ 1025.425093] Code: 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0
8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00
00 00 89 f0 <e8> 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf
00 00

And this above message will be displayed repeatedly all the time and
the system will become unresponsive.
[ 1090.922260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
task.ti=f58b2000)
[ 1090.923089] Stack:
[ 1090.923089] Call Trace:
[ 1090.923089] Leftover inexact backtrace:
[ 1090.923089]
[ 1090.923089] Code: 83 c4 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba
94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba
80 00 00 00 <89> f0 e8 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00
8d bf



>>>>>>>>>
>>>>>>>>> So I am very confused:
>>>>>>>>>
>>>>>>>>> First, what's the difference between these two patches? Or say, does
>>>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>>>>>>>
>>>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>>>>>>>
>>>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>>>>>>>has exposed this feature to guest.) The word "use" means something
>>>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>>>>>>>> IA32_APICBASE MSR (i.e. bit 10).
>>>>>>>>>
>>>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>>>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>>>>>>>> please see enable_IR_x2apic().
>>>>>>>>
>>>>>>>>> -Jidong
>

Re: How does kvm support x2apic?

[Zhang Haoyu]

>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> According to this:
>>>>>>>>>>
>>>>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>>>>>>>>
>>>>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>>>>>>>>> even earlier.
>>>>>>>>>>
>>>>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>>>>>>>>
>>>>>>>>>> However, this following patch:
>>>>>>>>>>
>>>>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>>>>>>>>
>>>>>>>>>> Also claims that adding support for x2apic. But this later patch was
>>>>>>>>>> for kernel 3.9.
>>>>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>>>>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>>>>>>>>> which belongs to APIC virtualization from certain angle.
>>>>>>>> Thanks Haoyu, I am reading the Intel x2apic manual.
>>>>>>>>
>>>>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>>>>>>>
>>>>>>>> but I don't see the so called "virtual x2apic mode", it seems that the
>>>>>>>> manual does not mention anything about that. Do you mean that when the
>>>>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
>>>>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
>>>>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or,
>>>>>>>> "emulate x2apic for guest" is the foundation of "support virtual
>>>>>>>> x2apic mode"?
>>>>>>>>
>>>>>>>> -Jidong
>>>>>>>>
>>>>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in
>>>>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29
>>>>>>>- APIC virtualization and virtual interrupts. Ignore my questions
>>>>>>>please, but thank you anyway.
>>>>>>>
>>>>>> Yes, virtual x2apic mode is part of APICv.
>>>>>> Guest has no idea about it is running in a virtual machine,
>>>>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
>>>>>>
>>>>>
>>>>>Regarding this "VMM also prevent guest from enabling/disabling virtual
>>>>>x2apic mode", do you mean that guest is not allowed to do something
>>>>>like:
>>>>>
>>>>>Assuming we are in the guest, and we see the original value of the
>>>>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was
>>>>>enabled. And now, we try to set bit 10 (EXTD bit):
>>>>>
>>>>>wrmsr 0x1b 0xfee00d00.
>>>>>
>>>>>Do you mean this wrmsr operation is not allowed? If fact, I tried
>>>>>this, and this simply crashes the guest kernel. (I am using Linux
>>>>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And
>>>>>this is the exact reason that I initialized this whole discussion. My
>>>>>feeling is this might be a kvm bug, so I wish to figure out and try to
>>>>>fix it.
>>>>>
>>>> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode,
>>>> you are writing the wrong MSR address.
>>>> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.
>>>>
>>>Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR.
>>>In particular I was trying to write bit 10, which is the x2APIC
>>>enable/disable bit. As I said, I just wish to enable that x2APIC. The
>>>original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10
>>>is 0. According to SDM manual, this bit has to be 1 if we want to
>>>enable x2APIC.
>>>
>> Sorry, I misread it.
>> Please try
>> u64 val;
>> rdmsrl(0x0000001b, val);
>> if ( !(val & (1UL << 10)) )
>>     wrmsrl(0x0000001b, val | (1UL << 10));
>>
>> If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure.
>> Please report the crash message.
>>
>Yes I wrote a simple kernel module which just includes the above four
>lines in the module init function, and when I try to install the
>module with insmod command, the crash still happens, the crash message
>is:
>
>[ 1025.424260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
>task.ti=f58b2000)
>[ 1025.425093] Stack:
>[ 1025.425093] Call Trace:
>[ 1025.425093] Leftover inexact backtrace:
>[ 1025.425093]
>[ 1025.425093] Code: 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0
>8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00
>00 00 89 f0 <e8> 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf
>00 00
>
Does the vcpu have x2apic feature?
please make a confirmation by 'cat /proc/cpuinfo | grep -i x2apic'
Configure crashkernel to kdump more info, please.

Thanks,
Zhang Haoyu
>And this above message will be displayed repeatedly all the time and
>the system will become unresponsive.
>[ 1090.922260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
>task.ti=f58b2000)
>[ 1090.923089] Stack:
>[ 1090.923089] Call Trace:
>[ 1090.923089] Leftover inexact backtrace:
>[ 1090.923089]
>[ 1090.923089] Code: 83 c4 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba
>94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba
>80 00 00 00 <89> f0 e8 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00
>8d bf
>
>
>
>>>>>>>>>>
>>>>>>>>>> So I am very confused:
>>>>>>>>>>
>>>>>>>>>> First, what's the difference between these two patches? Or say, does
>>>>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>>>>>>>>
>>>>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>>>>>>>>
>>>>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>>>>>>>>has exposed this feature to guest.) The word "use" means something
>>>>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>>>>>>>>> IA32_APICBASE MSR (i.e. bit 10).
>>>>>>>>>>
>>>>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>>>>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>>>>>>>>> please see enable_IR_x2apic().
>>>>>>>>>
>>>>>>>>>> -Jidong

Re: How does kvm support x2apic?

[Jidong Xiao]

On Wed, Jun 11, 2014 at 8:42 PM, Zhang Haoyu <zhanghy@sangfor.com> wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> According to this:
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd
>>>>>>>>>>>
>>>>>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or
>>>>>>>>>>> even earlier.
>>>>>>>>>>>
>>>>>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio.
>>>>>>>>>>
>>>>>>>>>>> However, this following patch:
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5
>>>>>>>>>>>
>>>>>>>>>>> Also claims that adding support for x2apic. But this later patch was
>>>>>>>>>>> for kernel 3.9.
>>>>>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps,
>>>>>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit,
>>>>>>>>>> which belongs to APIC virtualization from certain angle.
>>>>>>>>> Thanks Haoyu, I am reading the Intel x2apic manual.
>>>>>>>>>
>>>>>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf
>>>>>>>>>
>>>>>>>>> but I don't see the so called "virtual x2apic mode", it seems that the
>>>>>>>>> manual does not mention anything about that. Do you mean that when the
>>>>>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized
>>>>>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't
>>>>>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or,
>>>>>>>>> "emulate x2apic for guest" is the foundation of "support virtual
>>>>>>>>> x2apic mode"?
>>>>>>>>>
>>>>>>>>> -Jidong
>>>>>>>>>
>>>>>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in
>>>>>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29
>>>>>>>>- APIC virtualization and virtual interrupts. Ignore my questions
>>>>>>>>please, but thank you anyway.
>>>>>>>>
>>>>>>> Yes, virtual x2apic mode is part of APICv.
>>>>>>> Guest has no idea about it is running in a virtual machine,
>>>>>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations.
>>>>>>>
>>>>>>
>>>>>>Regarding this "VMM also prevent guest from enabling/disabling virtual
>>>>>>x2apic mode", do you mean that guest is not allowed to do something
>>>>>>like:
>>>>>>
>>>>>>Assuming we are in the guest, and we see the original value of the
>>>>>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was
>>>>>>enabled. And now, we try to set bit 10 (EXTD bit):
>>>>>>
>>>>>>wrmsr 0x1b 0xfee00d00.
>>>>>>
>>>>>>Do you mean this wrmsr operation is not allowed? If fact, I tried
>>>>>>this, and this simply crashes the guest kernel. (I am using Linux
>>>>>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And
>>>>>>this is the exact reason that I initialized this whole discussion. My
>>>>>>feeling is this might be a kvm bug, so I wish to figure out and try to
>>>>>>fix it.
>>>>>>
>>>>> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode,
>>>>> you are writing the wrong MSR address.
>>>>> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer.
>>>>>
>>>>Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR.
>>>>In particular I was trying to write bit 10, which is the x2APIC
>>>>enable/disable bit. As I said, I just wish to enable that x2APIC. The
>>>>original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10
>>>>is 0. According to SDM manual, this bit has to be 1 if we want to
>>>>enable x2APIC.
>>>>
>>> Sorry, I misread it.
>>> Please try
>>> u64 val;
>>> rdmsrl(0x0000001b, val);
>>> if ( !(val & (1UL << 10)) )
>>>     wrmsrl(0x0000001b, val | (1UL << 10));
>>>
>>> If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure.
>>> Please report the crash message.
>>>
>>Yes I wrote a simple kernel module which just includes the above four
>>lines in the module init function, and when I try to install the
>>module with insmod command, the crash still happens, the crash message
>>is:
>>
>>[ 1025.424260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
>>task.ti=f58b2000)
>>[ 1025.425093] Stack:
>>[ 1025.425093] Call Trace:
>>[ 1025.425093] Leftover inexact backtrace:
>>[ 1025.425093]
>>[ 1025.425093] Code: 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0
>>8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00
>>00 00 89 f0 <e8> 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf
>>00 00
>>
> Does the vcpu have x2apic feature?
> please make a confirmation by 'cat /proc/cpuinfo | grep -i x2apic'
> Configure crashkernel to kdump more info, please.
>
> Thanks,
> Zhang Haoyu

Hi, Haoyu,

I think I have found the root cause. As I said I was using 2.6.34
kernel in the guest, and CONFIG_X86_X2APIC was not enabled by default
in my kernel. After I set CONFIG_X86_X2APIC and recompile the kernel,
the new kernel just works fine in x2apic mode. Thanks for your time
and help.

-Jidong

>>And this above message will be displayed repeatedly all the time and
>>the system will become unresponsive.
>>[ 1090.922260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0
>>task.ti=f58b2000)
>>[ 1090.923089] Stack:
>>[ 1090.923089] Call Trace:
>>[ 1090.923089] Leftover inexact backtrace:
>>[ 1090.923089]
>>[ 1090.923089] Code: 83 c4 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba
>>94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba
>>80 00 00 00 <89> f0 e8 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00
>>8d bf
>>
>>
>>
>>>>>>>>>>>
>>>>>>>>>>> So I am very confused:
>>>>>>>>>>>
>>>>>>>>>>> First, what's the difference between these two patches? Or say, does
>>>>>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9?
>>>>>>>>>>>
>>>>>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9.
>>>>>>>>>>
>>>>>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu
>>>>>>>>>>>has exposed this feature to guest.) The word "use" means something
>>>>>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in
>>>>>>>>>>> IA32_APICBASE MSR (i.e. bit 10).
>>>>>>>>>>>
>>>>>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO.
>>>>>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic,
>>>>>>>>>> please see enable_IR_x2apic().
>>>>>>>>>>
>>>>>>>>>>> -Jidong
>