* Re: How does kvm support x2apic? @ 2014-06-11 1:33 Zhang Haoyu 2014-06-11 3:54 ` Jidong Xiao 0 siblings, 1 reply; 13+ messages in thread From: Zhang Haoyu @ 2014-06-11 1:33 UTC (permalink / raw) To: Jidong Xiao, KVM > Hi, > > According to this: > > https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd > > It looks like kvm have been supporting x2apic since kernel 2.6.32, or > even earlier. > This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. > However, this following patch: > > https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 > > Also claims that adding support for x2apic. But this later patch was > for kernel 3.9. This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, which belongs to APIC virtualization from certain angle. > > So I am very confused: > > First, what's the difference between these two patches? Or say, does > kvm support x2apic since kernel 2.6.32, or since kernel 3.9? > kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. > Second, can guest use x2apic even if the host does not? (Assuming qemu >has exposed this feature to guest.) The word "use" means something > like, accessing x2apic registers, setting the x2apic enable bit in > IA32_APICBASE MSR (i.e. bit 10). > Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, please see enable_IR_x2apic(). > -Jidong ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-11 1:33 How does kvm support x2apic? Zhang Haoyu @ 2014-06-11 3:54 ` Jidong Xiao 2014-06-11 4:15 ` Jidong Xiao 0 siblings, 1 reply; 13+ messages in thread From: Jidong Xiao @ 2014-06-11 3:54 UTC (permalink / raw) To: Zhang Haoyu; +Cc: KVM On Tue, Jun 10, 2014 at 9:33 PM, Zhang Haoyu <zhanghy@sangfor.com> wrote: >> Hi, >> >> According to this: >> >> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >> >> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >> even earlier. >> > This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. > >> However, this following patch: >> >> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >> >> Also claims that adding support for x2apic. But this later patch was >> for kernel 3.9. > This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, > you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, > which belongs to APIC virtualization from certain angle. Thanks Haoyu, I am reading the Intel x2apic manual. http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf but I don't see the so called "virtual x2apic mode", it seems that the manual does not mention anything about that. Do you mean that when the Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't this the same as the aforementioned "emulate x2apic for guest"? Or, "emulate x2apic for guest" is the foundation of "support virtual x2apic mode"? -Jidong >> >> So I am very confused: >> >> First, what's the difference between these two patches? Or say, does >> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >> > kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. > >> Second, can guest use x2apic even if the host does not? (Assuming qemu >>has exposed this feature to guest.) The word "use" means something >> like, accessing x2apic registers, setting the x2apic enable bit in >> IA32_APICBASE MSR (i.e. bit 10). >> > Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. > But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, > please see enable_IR_x2apic(). > >> -Jidong > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-11 3:54 ` Jidong Xiao @ 2014-06-11 4:15 ` Jidong Xiao 2014-06-11 4:53 ` Zhang Haoyu 0 siblings, 1 reply; 13+ messages in thread From: Jidong Xiao @ 2014-06-11 4:15 UTC (permalink / raw) To: Zhang Haoyu; +Cc: KVM On Tue, Jun 10, 2014 at 11:54 PM, Jidong Xiao <jidong.xiao@gmail.com> wrote: > On Tue, Jun 10, 2014 at 9:33 PM, Zhang Haoyu <zhanghy@sangfor.com> wrote: >>> Hi, >>> >>> According to this: >>> >>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>> >>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>> even earlier. >>> >> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >> >>> However, this following patch: >>> >>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>> >>> Also claims that adding support for x2apic. But this later patch was >>> for kernel 3.9. >> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >> which belongs to APIC virtualization from certain angle. > Thanks Haoyu, I am reading the Intel x2apic manual. > > http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf > > but I don't see the so called "virtual x2apic mode", it seems that the > manual does not mention anything about that. Do you mean that when the > Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized > IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't > this the same as the aforementioned "emulate x2apic for guest"? Or, > "emulate x2apic for guest" is the foundation of "support virtual > x2apic mode"? > > -Jidong > Oh, I found the answer, the virtualized x2apic mode is not defined in the x2apic manual, but is defined in the intel SDM manual, chapter 29 - APIC virtualization and virtual interrupts. Ignore my questions please, but thank you anyway. -Jidong >>> >>> So I am very confused: >>> >>> First, what's the difference between these two patches? Or say, does >>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>> >> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >> >>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>has exposed this feature to guest.) The word "use" means something >>> like, accessing x2apic registers, setting the x2apic enable bit in >>> IA32_APICBASE MSR (i.e. bit 10). >>> >> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >> please see enable_IR_x2apic(). >> >>> -Jidong >> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-11 4:15 ` Jidong Xiao @ 2014-06-11 4:53 ` Zhang Haoyu 2014-06-11 5:08 ` Jidong Xiao 0 siblings, 1 reply; 13+ messages in thread From: Zhang Haoyu @ 2014-06-11 4:53 UTC (permalink / raw) To: Jidong Xiao; +Cc: KVM >>>> Hi, >>>> >>>> According to this: >>>> >>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>>> >>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>>> even earlier. >>>> >>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >>> >>>> However, this following patch: >>>> >>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>>> >>>> Also claims that adding support for x2apic. But this later patch was >>>> for kernel 3.9. >>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >>> which belongs to APIC virtualization from certain angle. >> Thanks Haoyu, I am reading the Intel x2apic manual. >> >> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >> >> but I don't see the so called "virtual x2apic mode", it seems that the >> manual does not mention anything about that. Do you mean that when the >> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized >> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't >> this the same as the aforementioned "emulate x2apic for guest"? Or, >> "emulate x2apic for guest" is the foundation of "support virtual >> x2apic mode"? >> >> -Jidong >> >Oh, I found the answer, the virtualized x2apic mode is not defined in >the x2apic manual, but is defined in the intel SDM manual, chapter 29 >- APIC virtualization and virtual interrupts. Ignore my questions >please, but thank you anyway. > Yes, virtual x2apic mode is part of APICv. Guest has no idea about it is running in a virtual machine, VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations. >-Jidong > >>>> >>>> So I am very confused: >>>> >>>> First, what's the difference between these two patches? Or say, does >>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>>> >>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >>> >>>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>>has exposed this feature to guest.) The word "use" means something >>>> like, accessing x2apic registers, setting the x2apic enable bit in >>>> IA32_APICBASE MSR (i.e. bit 10). >>>> >>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >>> please see enable_IR_x2apic(). >>> >>>> -Jidong ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-11 4:53 ` Zhang Haoyu @ 2014-06-11 5:08 ` Jidong Xiao 2014-06-11 5:29 ` Jidong Xiao 2014-06-11 6:27 ` Zhang Haoyu 0 siblings, 2 replies; 13+ messages in thread From: Jidong Xiao @ 2014-06-11 5:08 UTC (permalink / raw) To: Zhang Haoyu; +Cc: KVM On Wed, Jun 11, 2014 at 12:53 AM, Zhang Haoyu <zhanghy@sangfor.com> wrote: >>>>> Hi, >>>>> >>>>> According to this: >>>>> >>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>>>> >>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>>>> even earlier. >>>>> >>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >>>> >>>>> However, this following patch: >>>>> >>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>>>> >>>>> Also claims that adding support for x2apic. But this later patch was >>>>> for kernel 3.9. >>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >>>> which belongs to APIC virtualization from certain angle. >>> Thanks Haoyu, I am reading the Intel x2apic manual. >>> >>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >>> >>> but I don't see the so called "virtual x2apic mode", it seems that the >>> manual does not mention anything about that. Do you mean that when the >>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized >>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't >>> this the same as the aforementioned "emulate x2apic for guest"? Or, >>> "emulate x2apic for guest" is the foundation of "support virtual >>> x2apic mode"? >>> >>> -Jidong >>> >>Oh, I found the answer, the virtualized x2apic mode is not defined in >>the x2apic manual, but is defined in the intel SDM manual, chapter 29 >>- APIC virtualization and virtual interrupts. Ignore my questions >>please, but thank you anyway. >> > Yes, virtual x2apic mode is part of APICv. > Guest has no idea about it is running in a virtual machine, > VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations. > Regarding this "VMM also prevent guest from enabling/disabling virtual x2apic mode", do you mean that guest is not allowed to do something like: Assuming we are in the guest, and we see the original value of the IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was enabled. And now, we try to set bit 10 (EXTD bit): wrmsr 0x1b 0xfee00d00. Do you mean this wrmsr operation is not allowed? If fact, I tried this, and this simply crashes the guest kernel. (I am using Linux 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And this is the exact reason that I initialized this whole discussion. My feeling is this might be a kvm bug, so I wish to figure out and try to fix it. >>>>> >>>>> So I am very confused: >>>>> >>>>> First, what's the difference between these two patches? Or say, does >>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>>>> >>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >>>> >>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>>>has exposed this feature to guest.) The word "use" means something >>>>> like, accessing x2apic registers, setting the x2apic enable bit in >>>>> IA32_APICBASE MSR (i.e. bit 10). >>>>> >>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >>>> please see enable_IR_x2apic(). >>>> >>>>> -Jidong > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-11 5:08 ` Jidong Xiao @ 2014-06-11 5:29 ` Jidong Xiao 2014-06-11 6:27 ` Zhang Haoyu 1 sibling, 0 replies; 13+ messages in thread From: Jidong Xiao @ 2014-06-11 5:29 UTC (permalink / raw) To: Zhang Haoyu; +Cc: KVM On Wed, Jun 11, 2014 at 1:08 AM, Jidong Xiao <jidong.xiao@gmail.com> wrote: > On Wed, Jun 11, 2014 at 12:53 AM, Zhang Haoyu <zhanghy@sangfor.com> wrote: >>>>>> Hi, >>>>>> >>>>>> According to this: >>>>>> >>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>>>>> >>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>>>>> even earlier. >>>>>> >>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >>>>> >>>>>> However, this following patch: >>>>>> >>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>>>>> >>>>>> Also claims that adding support for x2apic. But this later patch was >>>>>> for kernel 3.9. >>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >>>>> which belongs to APIC virtualization from certain angle. >>>> Thanks Haoyu, I am reading the Intel x2apic manual. >>>> >>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >>>> >>>> but I don't see the so called "virtual x2apic mode", it seems that the >>>> manual does not mention anything about that. Do you mean that when the >>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized >>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't >>>> this the same as the aforementioned "emulate x2apic for guest"? Or, >>>> "emulate x2apic for guest" is the foundation of "support virtual >>>> x2apic mode"? >>>> >>>> -Jidong >>>> >>>Oh, I found the answer, the virtualized x2apic mode is not defined in >>>the x2apic manual, but is defined in the intel SDM manual, chapter 29 >>>- APIC virtualization and virtual interrupts. Ignore my questions >>>please, but thank you anyway. >>> >> Yes, virtual x2apic mode is part of APICv. >> Guest has no idea about it is running in a virtual machine, >> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations. >> > > Regarding this "VMM also prevent guest from enabling/disabling virtual > x2apic mode", do you mean that guest is not allowed to do something > like: > > Assuming we are in the guest, and we see the original value of the > IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was > enabled. And now, we try to set bit 10 (EXTD bit): > > wrmsr 0x1b 0xfee00d00. > > Do you mean this wrmsr operation is not allowed? If fact, I tried > this, and this simply crashes the guest kernel. (I am using Linux > 2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And > this is the exact reason that I initialized this whole discussion. My > feeling is this might be a kvm bug, so I wish to figure out and try to > fix it. > Can someone test this also? Just set bit 10 of IA32_APICBASE MSR in your guest, and see if the guest crashes. According to Intel SDM manual, this is the approach to enable x2apic mode. And I think normally it should not crash the guest kernel. One can write that msr with the following tool: https://01.org/msr-tools And compile this tool and just run the aforementioned command: wrmsr 0x1b 0xfee00d00 (rdmsr 0x1b to confirm the original value is 0xfee00900.) I tried on several machines, and this command just crash the kernel. And I am not sure why. >>>>>> >>>>>> So I am very confused: >>>>>> >>>>>> First, what's the difference between these two patches? Or say, does >>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>>>>> >>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >>>>> >>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>>>>has exposed this feature to guest.) The word "use" means something >>>>>> like, accessing x2apic registers, setting the x2apic enable bit in >>>>>> IA32_APICBASE MSR (i.e. bit 10). >>>>>> >>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >>>>> please see enable_IR_x2apic(). >>>>> >>>>>> -Jidong >> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-11 5:08 ` Jidong Xiao 2014-06-11 5:29 ` Jidong Xiao @ 2014-06-11 6:27 ` Zhang Haoyu 2014-06-11 6:46 ` Jidong Xiao 1 sibling, 1 reply; 13+ messages in thread From: Zhang Haoyu @ 2014-06-11 6:27 UTC (permalink / raw) To: Jidong Xiao; +Cc: KVM >>>>>> Hi, >>>>>> >>>>>> According to this: >>>>>> >>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>>>>> >>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>>>>> even earlier. >>>>>> >>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >>>>> >>>>>> However, this following patch: >>>>>> >>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>>>>> >>>>>> Also claims that adding support for x2apic. But this later patch was >>>>>> for kernel 3.9. >>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >>>>> which belongs to APIC virtualization from certain angle. >>>> Thanks Haoyu, I am reading the Intel x2apic manual. >>>> >>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >>>> >>>> but I don't see the so called "virtual x2apic mode", it seems that the >>>> manual does not mention anything about that. Do you mean that when the >>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized >>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't >>>> this the same as the aforementioned "emulate x2apic for guest"? Or, >>>> "emulate x2apic for guest" is the foundation of "support virtual >>>> x2apic mode"? >>>> >>>> -Jidong >>>> >>>Oh, I found the answer, the virtualized x2apic mode is not defined in >>>the x2apic manual, but is defined in the intel SDM manual, chapter 29 >>>- APIC virtualization and virtual interrupts. Ignore my questions >>>please, but thank you anyway. >>> >> Yes, virtual x2apic mode is part of APICv. >> Guest has no idea about it is running in a virtual machine, >> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations. >> > >Regarding this "VMM also prevent guest from enabling/disabling virtual >x2apic mode", do you mean that guest is not allowed to do something >like: > >Assuming we are in the guest, and we see the original value of the >IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was >enabled. And now, we try to set bit 10 (EXTD bit): > >wrmsr 0x1b 0xfee00d00. > >Do you mean this wrmsr operation is not allowed? If fact, I tried >this, and this simply crashes the guest kernel. (I am using Linux >2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And >this is the exact reason that I initialized this whole discussion. My >feeling is this might be a kvm bug, so I wish to figure out and try to >fix it. > The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, you are writing the wrong MSR address. Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer. >>>>>> >>>>>> So I am very confused: >>>>>> >>>>>> First, what's the difference between these two patches? Or say, does >>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>>>>> >>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >>>>> >>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>>>>has exposed this feature to guest.) The word "use" means something >>>>>> like, accessing x2apic registers, setting the x2apic enable bit in >>>>>> IA32_APICBASE MSR (i.e. bit 10). >>>>>> >>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >>>>> please see enable_IR_x2apic(). >>>>> >>>>>> -Jidong ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-11 6:27 ` Zhang Haoyu @ 2014-06-11 6:46 ` Jidong Xiao 2014-06-11 8:49 ` Zhang Haoyu 0 siblings, 1 reply; 13+ messages in thread From: Jidong Xiao @ 2014-06-11 6:46 UTC (permalink / raw) To: Zhang Haoyu; +Cc: KVM On Wed, Jun 11, 2014 at 2:27 AM, Zhang Haoyu <zhanghy@sangfor.com> wrote: >>>>>>> Hi, >>>>>>> >>>>>>> According to this: >>>>>>> >>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>>>>>> >>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>>>>>> even earlier. >>>>>>> >>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >>>>>> >>>>>>> However, this following patch: >>>>>>> >>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>>>>>> >>>>>>> Also claims that adding support for x2apic. But this later patch was >>>>>>> for kernel 3.9. >>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >>>>>> which belongs to APIC virtualization from certain angle. >>>>> Thanks Haoyu, I am reading the Intel x2apic manual. >>>>> >>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >>>>> >>>>> but I don't see the so called "virtual x2apic mode", it seems that the >>>>> manual does not mention anything about that. Do you mean that when the >>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized >>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't >>>>> this the same as the aforementioned "emulate x2apic for guest"? Or, >>>>> "emulate x2apic for guest" is the foundation of "support virtual >>>>> x2apic mode"? >>>>> >>>>> -Jidong >>>>> >>>>Oh, I found the answer, the virtualized x2apic mode is not defined in >>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29 >>>>- APIC virtualization and virtual interrupts. Ignore my questions >>>>please, but thank you anyway. >>>> >>> Yes, virtual x2apic mode is part of APICv. >>> Guest has no idea about it is running in a virtual machine, >>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations. >>> >> >>Regarding this "VMM also prevent guest from enabling/disabling virtual >>x2apic mode", do you mean that guest is not allowed to do something >>like: >> >>Assuming we are in the guest, and we see the original value of the >>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was >>enabled. And now, we try to set bit 10 (EXTD bit): >> >>wrmsr 0x1b 0xfee00d00. >> >>Do you mean this wrmsr operation is not allowed? If fact, I tried >>this, and this simply crashes the guest kernel. (I am using Linux >>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And >>this is the exact reason that I initialized this whole discussion. My >>feeling is this might be a kvm bug, so I wish to figure out and try to >>fix it. >> > The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, > you are writing the wrong MSR address. > Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer. > Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. In particular I was trying to write bit 10, which is the x2APIC enable/disable bit. As I said, I just wish to enable that x2APIC. The original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 is 0. According to SDM manual, this bit has to be 1 if we want to enable x2APIC. >>>>>>> >>>>>>> So I am very confused: >>>>>>> >>>>>>> First, what's the difference between these two patches? Or say, does >>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>>>>>> >>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >>>>>> >>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>>>>>has exposed this feature to guest.) The word "use" means something >>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in >>>>>>> IA32_APICBASE MSR (i.e. bit 10). >>>>>>> >>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >>>>>> please see enable_IR_x2apic(). >>>>>> >>>>>>> -Jidong > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-11 6:46 ` Jidong Xiao @ 2014-06-11 8:49 ` Zhang Haoyu 2014-06-11 15:12 ` Jidong Xiao 0 siblings, 1 reply; 13+ messages in thread From: Zhang Haoyu @ 2014-06-11 8:49 UTC (permalink / raw) To: Jidong Xiao; +Cc: KVM >>>>>>>> Hi, >>>>>>>> >>>>>>>> According to this: >>>>>>>> >>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>>>>>>> >>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>>>>>>> even earlier. >>>>>>>> >>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >>>>>>> >>>>>>>> However, this following patch: >>>>>>>> >>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>>>>>>> >>>>>>>> Also claims that adding support for x2apic. But this later patch was >>>>>>>> for kernel 3.9. >>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >>>>>>> which belongs to APIC virtualization from certain angle. >>>>>> Thanks Haoyu, I am reading the Intel x2apic manual. >>>>>> >>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >>>>>> >>>>>> but I don't see the so called "virtual x2apic mode", it seems that the >>>>>> manual does not mention anything about that. Do you mean that when the >>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized >>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't >>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or, >>>>>> "emulate x2apic for guest" is the foundation of "support virtual >>>>>> x2apic mode"? >>>>>> >>>>>> -Jidong >>>>>> >>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in >>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29 >>>>>- APIC virtualization and virtual interrupts. Ignore my questions >>>>>please, but thank you anyway. >>>>> >>>> Yes, virtual x2apic mode is part of APICv. >>>> Guest has no idea about it is running in a virtual machine, >>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations. >>>> >>> >>>Regarding this "VMM also prevent guest from enabling/disabling virtual >>>x2apic mode", do you mean that guest is not allowed to do something >>>like: >>> >>>Assuming we are in the guest, and we see the original value of the >>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was >>>enabled. And now, we try to set bit 10 (EXTD bit): >>> >>>wrmsr 0x1b 0xfee00d00. >>> >>>Do you mean this wrmsr operation is not allowed? If fact, I tried >>>this, and this simply crashes the guest kernel. (I am using Linux >>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And >>>this is the exact reason that I initialized this whole discussion. My >>>feeling is this might be a kvm bug, so I wish to figure out and try to >>>fix it. >>> >> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, >> you are writing the wrong MSR address. >> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer. >> >Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. >In particular I was trying to write bit 10, which is the x2APIC >enable/disable bit. As I said, I just wish to enable that x2APIC. The >original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 >is 0. According to SDM manual, this bit has to be 1 if we want to >enable x2APIC. > Sorry, I misread it. Please try u64 val; rdmsrl(0x0000001b, val); if ( !(val & (1UL << 10)) ) wrmsrl(0x0000001b, val | (1UL << 10)); If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure. Please report the crash message. >>>>>>>> >>>>>>>> So I am very confused: >>>>>>>> >>>>>>>> First, what's the difference between these two patches? Or say, does >>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>>>>>>> >>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >>>>>>> >>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>>>>>>has exposed this feature to guest.) The word "use" means something >>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in >>>>>>>> IA32_APICBASE MSR (i.e. bit 10). >>>>>>>> >>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >>>>>>> please see enable_IR_x2apic(). >>>>>>> >>>>>>>> -Jidong ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-11 8:49 ` Zhang Haoyu @ 2014-06-11 15:12 ` Jidong Xiao 2014-06-12 0:42 ` Zhang Haoyu 0 siblings, 1 reply; 13+ messages in thread From: Jidong Xiao @ 2014-06-11 15:12 UTC (permalink / raw) To: Zhang Haoyu; +Cc: KVM On Wed, Jun 11, 2014 at 4:49 AM, Zhang Haoyu <zhanghy@sangfor.com> wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> According to this: >>>>>>>>> >>>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>>>>>>>> >>>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>>>>>>>> even earlier. >>>>>>>>> >>>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >>>>>>>> >>>>>>>>> However, this following patch: >>>>>>>>> >>>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>>>>>>>> >>>>>>>>> Also claims that adding support for x2apic. But this later patch was >>>>>>>>> for kernel 3.9. >>>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >>>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >>>>>>>> which belongs to APIC virtualization from certain angle. >>>>>>> Thanks Haoyu, I am reading the Intel x2apic manual. >>>>>>> >>>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >>>>>>> >>>>>>> but I don't see the so called "virtual x2apic mode", it seems that the >>>>>>> manual does not mention anything about that. Do you mean that when the >>>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized >>>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't >>>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or, >>>>>>> "emulate x2apic for guest" is the foundation of "support virtual >>>>>>> x2apic mode"? >>>>>>> >>>>>>> -Jidong >>>>>>> >>>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in >>>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29 >>>>>>- APIC virtualization and virtual interrupts. Ignore my questions >>>>>>please, but thank you anyway. >>>>>> >>>>> Yes, virtual x2apic mode is part of APICv. >>>>> Guest has no idea about it is running in a virtual machine, >>>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations. >>>>> >>>> >>>>Regarding this "VMM also prevent guest from enabling/disabling virtual >>>>x2apic mode", do you mean that guest is not allowed to do something >>>>like: >>>> >>>>Assuming we are in the guest, and we see the original value of the >>>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was >>>>enabled. And now, we try to set bit 10 (EXTD bit): >>>> >>>>wrmsr 0x1b 0xfee00d00. >>>> >>>>Do you mean this wrmsr operation is not allowed? If fact, I tried >>>>this, and this simply crashes the guest kernel. (I am using Linux >>>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And >>>>this is the exact reason that I initialized this whole discussion. My >>>>feeling is this might be a kvm bug, so I wish to figure out and try to >>>>fix it. >>>> >>> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, >>> you are writing the wrong MSR address. >>> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer. >>> >>Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. >>In particular I was trying to write bit 10, which is the x2APIC >>enable/disable bit. As I said, I just wish to enable that x2APIC. The >>original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 >>is 0. According to SDM manual, this bit has to be 1 if we want to >>enable x2APIC. >> > Sorry, I misread it. > Please try > u64 val; > rdmsrl(0x0000001b, val); > if ( !(val & (1UL << 10)) ) > wrmsrl(0x0000001b, val | (1UL << 10)); > > If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure. > Please report the crash message. > Yes I wrote a simple kernel module which just includes the above four lines in the module init function, and when I try to install the module with insmod command, the crash still happens, the crash message is: [ 1025.424260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0 task.ti=f58b2000) [ 1025.425093] Stack: [ 1025.425093] Call Trace: [ 1025.425093] Leftover inexact backtrace: [ 1025.425093] [ 1025.425093] Code: 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 00 00 89 f0 <e8> 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf 00 00 And this above message will be displayed repeatedly all the time and the system will become unresponsive. [ 1090.922260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0 task.ti=f58b2000) [ 1090.923089] Stack: [ 1090.923089] Call Trace: [ 1090.923089] Leftover inexact backtrace: [ 1090.923089] [ 1090.923089] Code: 83 c4 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 00 00 <89> f0 e8 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf >>>>>>>>> >>>>>>>>> So I am very confused: >>>>>>>>> >>>>>>>>> First, what's the difference between these two patches? Or say, does >>>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>>>>>>>> >>>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >>>>>>>> >>>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>>>>>>>has exposed this feature to guest.) The word "use" means something >>>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in >>>>>>>>> IA32_APICBASE MSR (i.e. bit 10). >>>>>>>>> >>>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >>>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >>>>>>>> please see enable_IR_x2apic(). >>>>>>>> >>>>>>>>> -Jidong > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-11 15:12 ` Jidong Xiao @ 2014-06-12 0:42 ` Zhang Haoyu 2014-06-12 3:24 ` Jidong Xiao 0 siblings, 1 reply; 13+ messages in thread From: Zhang Haoyu @ 2014-06-12 0:42 UTC (permalink / raw) To: Jidong Xiao; +Cc: KVM >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> According to this: >>>>>>>>>> >>>>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>>>>>>>>> >>>>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>>>>>>>>> even earlier. >>>>>>>>>> >>>>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >>>>>>>>> >>>>>>>>>> However, this following patch: >>>>>>>>>> >>>>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>>>>>>>>> >>>>>>>>>> Also claims that adding support for x2apic. But this later patch was >>>>>>>>>> for kernel 3.9. >>>>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >>>>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >>>>>>>>> which belongs to APIC virtualization from certain angle. >>>>>>>> Thanks Haoyu, I am reading the Intel x2apic manual. >>>>>>>> >>>>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >>>>>>>> >>>>>>>> but I don't see the so called "virtual x2apic mode", it seems that the >>>>>>>> manual does not mention anything about that. Do you mean that when the >>>>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized >>>>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't >>>>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or, >>>>>>>> "emulate x2apic for guest" is the foundation of "support virtual >>>>>>>> x2apic mode"? >>>>>>>> >>>>>>>> -Jidong >>>>>>>> >>>>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in >>>>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29 >>>>>>>- APIC virtualization and virtual interrupts. Ignore my questions >>>>>>>please, but thank you anyway. >>>>>>> >>>>>> Yes, virtual x2apic mode is part of APICv. >>>>>> Guest has no idea about it is running in a virtual machine, >>>>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations. >>>>>> >>>>> >>>>>Regarding this "VMM also prevent guest from enabling/disabling virtual >>>>>x2apic mode", do you mean that guest is not allowed to do something >>>>>like: >>>>> >>>>>Assuming we are in the guest, and we see the original value of the >>>>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was >>>>>enabled. And now, we try to set bit 10 (EXTD bit): >>>>> >>>>>wrmsr 0x1b 0xfee00d00. >>>>> >>>>>Do you mean this wrmsr operation is not allowed? If fact, I tried >>>>>this, and this simply crashes the guest kernel. (I am using Linux >>>>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And >>>>>this is the exact reason that I initialized this whole discussion. My >>>>>feeling is this might be a kvm bug, so I wish to figure out and try to >>>>>fix it. >>>>> >>>> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, >>>> you are writing the wrong MSR address. >>>> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer. >>>> >>>Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. >>>In particular I was trying to write bit 10, which is the x2APIC >>>enable/disable bit. As I said, I just wish to enable that x2APIC. The >>>original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 >>>is 0. According to SDM manual, this bit has to be 1 if we want to >>>enable x2APIC. >>> >> Sorry, I misread it. >> Please try >> u64 val; >> rdmsrl(0x0000001b, val); >> if ( !(val & (1UL << 10)) ) >> wrmsrl(0x0000001b, val | (1UL << 10)); >> >> If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure. >> Please report the crash message. >> >Yes I wrote a simple kernel module which just includes the above four >lines in the module init function, and when I try to install the >module with insmod command, the crash still happens, the crash message >is: > >[ 1025.424260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0 >task.ti=f58b2000) >[ 1025.425093] Stack: >[ 1025.425093] Call Trace: >[ 1025.425093] Leftover inexact backtrace: >[ 1025.425093] >[ 1025.425093] Code: 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0 >8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 >00 00 89 f0 <e8> 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf >00 00 > Does the vcpu have x2apic feature? please make a confirmation by 'cat /proc/cpuinfo | grep -i x2apic' Configure crashkernel to kdump more info, please. Thanks, Zhang Haoyu >And this above message will be displayed repeatedly all the time and >the system will become unresponsive. >[ 1090.922260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0 >task.ti=f58b2000) >[ 1090.923089] Stack: >[ 1090.923089] Call Trace: >[ 1090.923089] Leftover inexact backtrace: >[ 1090.923089] >[ 1090.923089] Code: 83 c4 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba >94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba >80 00 00 00 <89> f0 e8 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 >8d bf > > > >>>>>>>>>> >>>>>>>>>> So I am very confused: >>>>>>>>>> >>>>>>>>>> First, what's the difference between these two patches? Or say, does >>>>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>>>>>>>>> >>>>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >>>>>>>>> >>>>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>>>>>>>>has exposed this feature to guest.) The word "use" means something >>>>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in >>>>>>>>>> IA32_APICBASE MSR (i.e. bit 10). >>>>>>>>>> >>>>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >>>>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >>>>>>>>> please see enable_IR_x2apic(). >>>>>>>>> >>>>>>>>>> -Jidong ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: How does kvm support x2apic? 2014-06-12 0:42 ` Zhang Haoyu @ 2014-06-12 3:24 ` Jidong Xiao 0 siblings, 0 replies; 13+ messages in thread From: Jidong Xiao @ 2014-06-12 3:24 UTC (permalink / raw) To: Zhang Haoyu; +Cc: KVM On Wed, Jun 11, 2014 at 8:42 PM, Zhang Haoyu <zhanghy@sangfor.com> wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> According to this: >>>>>>>>>>> >>>>>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>>>>>>>>>> >>>>>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>>>>>>>>>> even earlier. >>>>>>>>>>> >>>>>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >>>>>>>>>> >>>>>>>>>>> However, this following patch: >>>>>>>>>>> >>>>>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>>>>>>>>>> >>>>>>>>>>> Also claims that adding support for x2apic. But this later patch was >>>>>>>>>>> for kernel 3.9. >>>>>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >>>>>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >>>>>>>>>> which belongs to APIC virtualization from certain angle. >>>>>>>>> Thanks Haoyu, I am reading the Intel x2apic manual. >>>>>>>>> >>>>>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >>>>>>>>> >>>>>>>>> but I don't see the so called "virtual x2apic mode", it seems that the >>>>>>>>> manual does not mention anything about that. Do you mean that when the >>>>>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized >>>>>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't >>>>>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or, >>>>>>>>> "emulate x2apic for guest" is the foundation of "support virtual >>>>>>>>> x2apic mode"? >>>>>>>>> >>>>>>>>> -Jidong >>>>>>>>> >>>>>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in >>>>>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29 >>>>>>>>- APIC virtualization and virtual interrupts. Ignore my questions >>>>>>>>please, but thank you anyway. >>>>>>>> >>>>>>> Yes, virtual x2apic mode is part of APICv. >>>>>>> Guest has no idea about it is running in a virtual machine, >>>>>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations. >>>>>>> >>>>>> >>>>>>Regarding this "VMM also prevent guest from enabling/disabling virtual >>>>>>x2apic mode", do you mean that guest is not allowed to do something >>>>>>like: >>>>>> >>>>>>Assuming we are in the guest, and we see the original value of the >>>>>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was >>>>>>enabled. And now, we try to set bit 10 (EXTD bit): >>>>>> >>>>>>wrmsr 0x1b 0xfee00d00. >>>>>> >>>>>>Do you mean this wrmsr operation is not allowed? If fact, I tried >>>>>>this, and this simply crashes the guest kernel. (I am using Linux >>>>>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And >>>>>>this is the exact reason that I initialized this whole discussion. My >>>>>>feeling is this might be a kvm bug, so I wish to figure out and try to >>>>>>fix it. >>>>>> >>>>> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, >>>>> you are writing the wrong MSR address. >>>>> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer. >>>>> >>>>Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. >>>>In particular I was trying to write bit 10, which is the x2APIC >>>>enable/disable bit. As I said, I just wish to enable that x2APIC. The >>>>original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 >>>>is 0. According to SDM manual, this bit has to be 1 if we want to >>>>enable x2APIC. >>>> >>> Sorry, I misread it. >>> Please try >>> u64 val; >>> rdmsrl(0x0000001b, val); >>> if ( !(val & (1UL << 10)) ) >>> wrmsrl(0x0000001b, val | (1UL << 10)); >>> >>> If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure. >>> Please report the crash message. >>> >>Yes I wrote a simple kernel module which just includes the above four >>lines in the module init function, and when I try to install the >>module with insmod command, the crash still happens, the crash message >>is: >> >>[ 1025.424260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0 >>task.ti=f58b2000) >>[ 1025.425093] Stack: >>[ 1025.425093] Call Trace: >>[ 1025.425093] Leftover inexact backtrace: >>[ 1025.425093] >>[ 1025.425093] Code: 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0 >>8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 >>00 00 89 f0 <e8> 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf >>00 00 >> > Does the vcpu have x2apic feature? > please make a confirmation by 'cat /proc/cpuinfo | grep -i x2apic' > Configure crashkernel to kdump more info, please. > > Thanks, > Zhang Haoyu Hi, Haoyu, I think I have found the root cause. As I said I was using 2.6.34 kernel in the guest, and CONFIG_X86_X2APIC was not enabled by default in my kernel. After I set CONFIG_X86_X2APIC and recompile the kernel, the new kernel just works fine in x2apic mode. Thanks for your time and help. -Jidong >>And this above message will be displayed repeatedly all the time and >>the system will become unresponsive. >>[ 1090.922260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0 >>task.ti=f58b2000) >>[ 1090.923089] Stack: >>[ 1090.923089] Call Trace: >>[ 1090.923089] Leftover inexact backtrace: >>[ 1090.923089] >>[ 1090.923089] Code: 83 c4 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba >>94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba >>80 00 00 00 <89> f0 e8 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 >>8d bf >> >> >> >>>>>>>>>>> >>>>>>>>>>> So I am very confused: >>>>>>>>>>> >>>>>>>>>>> First, what's the difference between these two patches? Or say, does >>>>>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>>>>>>>>>> >>>>>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >>>>>>>>>> >>>>>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>>>>>>>>>has exposed this feature to guest.) The word "use" means something >>>>>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in >>>>>>>>>>> IA32_APICBASE MSR (i.e. bit 10). >>>>>>>>>>> >>>>>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >>>>>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >>>>>>>>>> please see enable_IR_x2apic(). >>>>>>>>>> >>>>>>>>>>> -Jidong > ^ permalink raw reply [flat|nested] 13+ messages in thread
* How does kvm support x2apic? @ 2014-06-10 17:30 Jidong Xiao 0 siblings, 0 replies; 13+ messages in thread From: Jidong Xiao @ 2014-06-10 17:30 UTC (permalink / raw) To: KVM Hi, According to this: https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd It looks like kvm have been supporting x2apic since kernel 2.6.32, or even earlier. However, this following patch: https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 Also claims that adding support for x2apic. But this later patch was for kernel 3.9. So I am very confused: First, what's the difference between these two patches? Or say, does kvm support x2apic since kernel 2.6.32, or since kernel 3.9? Second, can guest use x2apic even if the host does not? (Assuming qemu has exposed this feature to guest.) The word "use" means something like, accessing x2apic registers, setting the x2apic enable bit in IA32_APICBASE MSR (i.e. bit 10). -Jidong ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2014-06-12 3:24 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-06-11 1:33 How does kvm support x2apic? Zhang Haoyu 2014-06-11 3:54 ` Jidong Xiao 2014-06-11 4:15 ` Jidong Xiao 2014-06-11 4:53 ` Zhang Haoyu 2014-06-11 5:08 ` Jidong Xiao 2014-06-11 5:29 ` Jidong Xiao 2014-06-11 6:27 ` Zhang Haoyu 2014-06-11 6:46 ` Jidong Xiao 2014-06-11 8:49 ` Zhang Haoyu 2014-06-11 15:12 ` Jidong Xiao 2014-06-12 0:42 ` Zhang Haoyu 2014-06-12 3:24 ` Jidong Xiao -- strict thread matches above, loose matches on Subject: below -- 2014-06-10 17:30 Jidong Xiao
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox