Re: Performance test result between virtio_pci MSI-X disable and enable

["Michael S. Tsirkin" <mst@redhat.com>]

On Tue, Nov 23, 2010 at 10:53:10AM +0800, lidong chen wrote:
> Test method:
> Send the same traffic load between virtio_pci MSI-X disable and
> enable,and compare the cpu rate of host os.
> I used the same version of virtio driver, only modify the msi-x option.
> the host os version is 2.6.32.
> the virtio dirver is from rhel6.
> the guest version  os is 2.6.16.
> 
> Test result:
> with msi-x disable, the cpu rate of host os is 110%.
> with msi-x enable, the cpu rate of host os is 140%.
> 
> the /proc/interrupt with msi-x disable is below:
>            CPU0       CPU1
>   0:   12326706          0    IO-APIC-edge  timer
>   1:          8          0    IO-APIC-edge  i8042
>   8:          0          0    IO-APIC-edge  rtc
>   9:          0          0   IO-APIC-level  acpi
>  10:    4783008          0   IO-APIC-level  virtio2, virtio3
>  11:    5363828          0   IO-APIC-level  virtio1, virtio4, virtio5
>  12:        104          0    IO-APIC-edge  i8042
> NMI:    2857871    2650796
> LOC:   12324952   12325609
> ERR:          0
> MIS:          0
> 
> the /proc/interrupt with msi-x enable is below:
>           CPU0       CPU1
>  0:    1896802          0    IO-APIC-edge  timer
>  1:          8          0    IO-APIC-edge  i8042
>  4:         14          0    IO-APIC-edge  serial
>  8:          0          0    IO-APIC-edge  rtc
>  9:          0          0   IO-APIC-level  acpi
>  10:          0          0   IO-APIC-level  virtio1, virtio2, virtio5
>  11:          1          0   IO-APIC-level  virtio0, virtio3, virtio4

This one probably means there's a bug: when msix
is enabled there should not be any level interrupts.

>  12:        104          0    IO-APIC-edge  i8042
>  50:          1          0       PCI-MSI-X  virtio2-output
>  58:          0          0       PCI-MSI-X  virtio3-config
>  66:    2046985          0       PCI-MSI-X  virtio3-input
>  74:          2          0       PCI-MSI-X  virtio3-output
>  82:          0          0       PCI-MSI-X  virtio4-config
>  90:        217          0       PCI-MSI-X  virtio4-input
>  98:          0          0       PCI-MSI-X  virtio4-output
> 177:          0          0       PCI-MSI-X  virtio0-config
> 185:     341831          0       PCI-MSI-X  virtio0-input
> 193:          1          0       PCI-MSI-X  virtio0-output
> 201:          0          0       PCI-MSI-X  virtio1-config
> 209:     188747          0       PCI-MSI-X  virtio1-input
> 217:          1          0       PCI-MSI-X  virtio1-output
> 225:          0          0       PCI-MSI-X  virtio2-config
> 233:    2204149          0       PCI-MSI-X  virtio2-input
> NMI:    1455767    1426226
> LOC:    1896099    1896637
> ERR:          0
> MIS:          0

I just noticed that above msi-x shows 4M interrupts
and 1.5M NMI but non-MSI shows 10M and 3M.

> Code difference:
> I disalbe msi-x by modify the function  vp_find_vqs like this:

You can simply supply nvectors=0 in qemu.

> static int vp_find_vqs(struct virtio_device *vdev, unsigned nvqs,
>                       struct virtqueue *vqs[],
>                       vq_callback_t *callbacks[],
>                       const char *names[])
> {
> 
> #if 0
>        int err;
> 
>        /* Try MSI-X with one vector per queue. */
>        err = vp_try_to_find_vqs(vdev, nvqs, vqs, callbacks, names, true, true);
>        if (!err)
>                return 0;
>        /* Fallback: MSI-X with one vector for config, one shared for queues. */
>        err = vp_try_to_find_vqs(vdev, nvqs, vqs, callbacks, names,
>                                 true, false);
>        if (!err)
>                return 0;
>        /* Finally fall back to regular interrupts. */
> #endif
> 
>        return vp_try_to_find_vqs(vdev, nvqs, vqs, callbacks, names,
>                                  false, false);
> }
> 
> Conclusion:
> msi-x enable waste more cpu resource is caused by MSIX mask bit. In
> older kernels program this bit twice
> on every interrupt. and caused ept violation.

Wait a second, older kernels don't have msix support in virtio,
do they?

> So I think we should add a param to control this.with older kernels,
> we should disable MSIX.
> And I think this should deal by qemu.

I would like to see a comparison of msix enabled and disabled
with a guest that supports msix natively.

-- 
MST