The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM

The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM

[yi li]

Hi Guys,

there have a issue about smp_affinity cannot work correctly on guest
os when PCI passthrough device using msi/msi-x with KVM.

My reason:
pcpu will occur a lot of ipi interrupt to find the vcpu to handle the
irq.  so the guest os will VM_EXIT frequelty. right?

if smp_affinity can work correctly on guest os,  the best way is that
the vcpu handle the irq is cputune at the pcpu which handle the
kvm:pci-bus irq on the host.but  unfortunly, i find that smp_affinity
can not work correctly on guest os when msi/msi-x.

how to reproduce:
1: passthrough a netcard (Brodcom BCM5716S) to the guest os

2: ifup the netcard, the card will use msi-x interrupt default, and close the
irqbalance service

3:  echo 4 > cat /proc/irq/NETCARDIRQ/smp_affinity, so we assume the vcpu2
handle the irq.

4: we have set <vcpupin vcpu='2' cpuset='1'/> and set the irq kvm:pci-bus to
the pcpu1 on the host.

we think this configure will reduce the ipi interrupt when inject interrupt to
the guest os. but this irq is not only handle on vcpu2. maybe it is
not our expect。

YiLi
Thanks

Re: The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM

[Alex Williamson]

On Fri, 2012-11-23 at 11:06 +0800, yi li wrote:
> Hi Guys,
> 
> there have a issue about smp_affinity cannot work correctly on guest
> os when PCI passthrough device using msi/msi-x with KVM.
> 
> My reason:
> pcpu will occur a lot of ipi interrupt to find the vcpu to handle the
> irq.  so the guest os will VM_EXIT frequelty. right?
> 
> if smp_affinity can work correctly on guest os,  the best way is that
> the vcpu handle the irq is cputune at the pcpu which handle the
> kvm:pci-bus irq on the host.but  unfortunly, i find that smp_affinity
> can not work correctly on guest os when msi/msi-x.
> 
> how to reproduce:
> 1: passthrough a netcard (Brodcom BCM5716S) to the guest os
> 
> 2: ifup the netcard, the card will use msi-x interrupt default, and close the
> irqbalance service
> 
> 3:  echo 4 > cat /proc/irq/NETCARDIRQ/smp_affinity, so we assume the vcpu2
> handle the irq.
> 
> 4: we have set <vcpupin vcpu='2' cpuset='1'/> and set the irq kvm:pci-bus to
> the pcpu1 on the host.
> 
> we think this configure will reduce the ipi interrupt when inject interrupt to
> the guest os. but this irq is not only handle on vcpu2. maybe it is
> not our expect。

What version of qemu-kvm/qemu are you using?  There's been some work
recently specifically to enable this.  Thanks,

Alex

Re: The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM

[yi li]

hi Alex,

the qemu-kvm version 1.2.

Thanks.
YiLi

2012/11/26 Alex Williamson <alex.williamson@redhat.com>:
> On Fri, 2012-11-23 at 11:06 +0800, yi li wrote:
>> Hi Guys,
>>
>> there have a issue about smp_affinity cannot work correctly on guest
>> os when PCI passthrough device using msi/msi-x with KVM.
>>
>> My reason:
>> pcpu will occur a lot of ipi interrupt to find the vcpu to handle the
>> irq.  so the guest os will VM_EXIT frequelty. right?
>>
>> if smp_affinity can work correctly on guest os,  the best way is that
>> the vcpu handle the irq is cputune at the pcpu which handle the
>> kvm:pci-bus irq on the host.but  unfortunly, i find that smp_affinity
>> can not work correctly on guest os when msi/msi-x.
>>
>> how to reproduce:
>> 1: passthrough a netcard (Brodcom BCM5716S) to the guest os
>>
>> 2: ifup the netcard, the card will use msi-x interrupt default, and close the
>> irqbalance service
>>
>> 3:  echo 4 > cat /proc/irq/NETCARDIRQ/smp_affinity, so we assume the vcpu2
>> handle the irq.
>>
>> 4: we have set <vcpupin vcpu='2' cpuset='1'/> and set the irq kvm:pci-bus to
>> the pcpu1 on the host.
>>
>> we think this configure will reduce the ipi interrupt when inject interrupt to
>> the guest os. but this irq is not only handle on vcpu2. maybe it is
>> not our expect。
>
> What version of qemu-kvm/qemu are you using?  There's been some work
> recently specifically to enable this.  Thanks,
>
> Alex
>

Re: The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM

[Alex Williamson]

On Tue, 2012-11-27 at 00:47 +0800, yi li wrote:
> hi Alex,
> 
> the qemu-kvm version 1.2.

And is the device making use of MSI-X or MSI interrupts.  MSI-X should
work on 1.2, MSI does not yet support vector updates for affinity, but
patches are welcome.  Thanks,

Alex

> 2012/11/26 Alex Williamson <alex.williamson@redhat.com>:
> > On Fri, 2012-11-23 at 11:06 +0800, yi li wrote:
> >> Hi Guys,
> >>
> >> there have a issue about smp_affinity cannot work correctly on guest
> >> os when PCI passthrough device using msi/msi-x with KVM.
> >>
> >> My reason:
> >> pcpu will occur a lot of ipi interrupt to find the vcpu to handle the
> >> irq.  so the guest os will VM_EXIT frequelty. right?
> >>
> >> if smp_affinity can work correctly on guest os,  the best way is that
> >> the vcpu handle the irq is cputune at the pcpu which handle the
> >> kvm:pci-bus irq on the host.but  unfortunly, i find that smp_affinity
> >> can not work correctly on guest os when msi/msi-x.
> >>
> >> how to reproduce:
> >> 1: passthrough a netcard (Brodcom BCM5716S) to the guest os
> >>
> >> 2: ifup the netcard, the card will use msi-x interrupt default, and close the
> >> irqbalance service
> >>
> >> 3:  echo 4 > cat /proc/irq/NETCARDIRQ/smp_affinity, so we assume the vcpu2
> >> handle the irq.
> >>
> >> 4: we have set <vcpupin vcpu='2' cpuset='1'/> and set the irq kvm:pci-bus to
> >> the pcpu1 on the host.
> >>
> >> we think this configure will reduce the ipi interrupt when inject interrupt to
> >> the guest os. but this irq is not only handle on vcpu2. maybe it is
> >> not our expect。
> >
> > What version of qemu-kvm/qemu are you using?  There's been some work
> > recently specifically to enable this.  Thanks,
> >
> > Alex
> >

Re: The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM

[yi li]

Alex,

Thanks for your reply, and i will check it agiain with msi-x.

YiLi

2012/11/27 Alex Williamson <alex.williamson@redhat.com>:
> On Tue, 2012-11-27 at 00:47 +0800, yi li wrote:
>> hi Alex,
>>
>> the qemu-kvm version 1.2.
>
> And is the device making use of MSI-X or MSI interrupts.  MSI-X should
> work on 1.2, MSI does not yet support vector updates for affinity, but
> patches are welcome.  Thanks,
>
> Alex
>
>> 2012/11/26 Alex Williamson <alex.williamson@redhat.com>:
>> > On Fri, 2012-11-23 at 11:06 +0800, yi li wrote:
>> >> Hi Guys,
>> >>
>> >> there have a issue about smp_affinity cannot work correctly on guest
>> >> os when PCI passthrough device using msi/msi-x with KVM.
>> >>
>> >> My reason:
>> >> pcpu will occur a lot of ipi interrupt to find the vcpu to handle the
>> >> irq.  so the guest os will VM_EXIT frequelty. right?
>> >>
>> >> if smp_affinity can work correctly on guest os,  the best way is that
>> >> the vcpu handle the irq is cputune at the pcpu which handle the
>> >> kvm:pci-bus irq on the host.but  unfortunly, i find that smp_affinity
>> >> can not work correctly on guest os when msi/msi-x.
>> >>
>> >> how to reproduce:
>> >> 1: passthrough a netcard (Brodcom BCM5716S) to the guest os
>> >>
>> >> 2: ifup the netcard, the card will use msi-x interrupt default, and close the
>> >> irqbalance service
>> >>
>> >> 3:  echo 4 > cat /proc/irq/NETCARDIRQ/smp_affinity, so we assume the vcpu2
>> >> handle the irq.
>> >>
>> >> 4: we have set <vcpupin vcpu='2' cpuset='1'/> and set the irq kvm:pci-bus to
>> >> the pcpu1 on the host.
>> >>
>> >> we think this configure will reduce the ipi interrupt when inject interrupt to
>> >> the guest os. but this irq is not only handle on vcpu2. maybe it is
>> >> not our expect。
>> >
>> > What version of qemu-kvm/qemu are you using?  There's been some work
>> > recently specifically to enable this.  Thanks,
>> >
>> > Alex
>> >
>
>
>