Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores

[Alexandre DERUMIER <aderumier@odiso.com>]

I don't think it's fixed in 1.3 or 1.4, some proxmox users have reported again this bug with guest kernel 2.6.32. (proxmox host is rhel 6.3 kernel + qemu 1.4)



----- Mail original -----

De: "Davide Guerri" <d.guerri@unidata.it>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "Peter Lieven" <pl@dlhnet.de>, "Michael S. Tsirkin" <mst@redhat.com>, "Stefan Hajnoczi" <stefanha@gmail.com>, qemu-devel@nongnu.org, "Jan Kiszka" <jan.kiszka@web.de>, "Peter Lieven" <lieven-lists@dlhnet.de>, "Dietmar Maurer" <dietmar@proxmox.com>
Envoyé: Jeudi 14 Mars 2013 10:22:22
Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores

I'd like to reopen this thread because this problem is still here and it's really annoying.
Another possible work-around is to pin the virtio nic irq to one virtual CPU (with /proc/smp_affinity) but (of course) this is still sub-optimal.

Here are some graphs showing the performance of a heavy loaded machine after and before a live migration from KVM-1.0 to KVM-1.2.
Host is a Ubuntu 12.10 Kernel 3.5, guest is a Ubuntu 10.04.4 LTS Kernel 2.6.32

History
-------
CPU time: http://s1299.beta.photobucket.com/user/dguerri/media/CPU-10days_zps4a088ff0.png.html
CPU Load: http://s1299.beta.photobucket.com/user/dguerri/media/Load-10days_zpsff27f212.png.html
NET pps: http://s1299.beta.photobucket.com/user/dguerri/media/pps-10days_zps003dd039.png.html
NET Mbps: http://s1299.beta.photobucket.com/user/dguerri/media/Mbps-10days_zpsfc3cba8c.png.html

Current
-------
CPU time: http://s1299.beta.photobucket.com/user/dguerri/media/CPU-2days_zpsd362cac6.png.html
CPU Load: http://s1299.beta.photobucket.com/user/dguerri/media/load-2days_zpsd4b7b50d.png.html
NET pps: http://s1299.beta.photobucket.com/user/dguerri/media/pps-2days_zps8f5458c9.png.html
NET Mbps: http://s1299.beta.photobucket.com/user/dguerri/media/Mbps-2days_zps299338b9.png.html

Red arrows indicate the kvm version change.
Black arrows indicate when I "pinned" the virtio NIC IRQ to only one CPU (that machine has 2 virtual cores).
As can be seen the performance penalty is rather high: that machine was almost unusable!

Does version 1.3 fixes this issue?

Could someone with the required knowledge look into this, please?
Please, this is a very nasty bug because I guess I'm not the only one who is unable to upgrade all the machines with a (not-so) old kernel... :)

Thanks!

Davide Guerri.




On 09/dic/2012, at 19:38, Alexandre DERUMIER <aderumier@odiso.com> wrote:

>>> Have you had any further progress on this regression/problem?
>
> Hi Peter,
> I didn't re-tested myself,
> but a proxmox user who's have the problem with qemu-kvm 1.2, with windows guest and linux guest,
> don't have the problem anymore with qemu 1.3.
>
> http://forum.proxmox.com/threads/12157-Win2003R2-in-KVM-VM-is-slow-in-PVE-2-2-when-multiply-CPU-cores-allowed
>
> I'll try to redone test myself this week
>
> Regards,
>
> Alexandre
>
> ----- Mail original -----
>
> De: "Peter Lieven" <pl@dlhnet.de>
> À: "Alexandre DERUMIER" <aderumier@odiso.com>
> Cc: "Dietmar Maurer" <dietmar@proxmox.com>, "Stefan Hajnoczi" <stefanha@gmail.com>, "Jan Kiszka" <jan.kiszka@web.de>, qemu-devel@nongnu.org, "Michael S. Tsirkin" <mst@redhat.com>, "Peter Lieven" <lieven-lists@dlhnet.de>
> Envoyé: Lundi 3 Décembre 2012 12:23:11
> Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores
>
>
> Am 16.11.2012 um 12:00 schrieb Alexandre DERUMIER <aderumier@odiso.com>: 
>
>>>> While trying to reproduce the bug, we just detected that it depends on the hardware (mainboard) you run on.
>>>>
>>>> Sigh :-/
>>
>> Hi,
>>
>> I can reproduce the bug on all my dell servers,differents generation (R710 (intel),R815 (amd), 2950 (intel).
>>
>> They all use broadcom bnx2 network card (don't know if it can be related)
>>
>> host kernel : rhel 63 with 2.6.32 kernel
>>
>> guest kernel : 2.6.32 (debian squeeze, ubuntu).
>>
>> No problem with guest kernel 3.2
>
> Have you had any further progress on this regression/problem?
>
> Thanks,
> Peter
>
>>
>>
>>
>>
>> ----- Mail original -----
>>
>> De: "Dietmar Maurer" <dietmar@proxmox.com>
>> À: "Peter Lieven" <lieven-lists@dlhnet.de>
>> Cc: "Stefan Hajnoczi" <stefanha@gmail.com>, "Peter Lieven" <pl@dlhnet.de>, "Jan Kiszka" <jan.kiszka@web.de>, qemu-devel@nongnu.org, "Michael S. Tsirkin" <mst@redhat.com>
>> Envoyé: Vendredi 16 Novembre 2012 11:44:26
>> Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores
>>
>>>> I only tested with RHEL6.3 kernel on host.
>>>
>>> can you check if there is a difference on interrupt delivery between those
>>> two?
>>>
>>> cat /proc/interrupts should be sufficient after some traffic has flown.
>>
>> While trying to reproduce the bug, we just detected that it depends on the hardware (mainboard) you run on.
>>
>> Sigh :-/
>
>