Re: [Qemu-devel] vhost-scsi port to v1.1.0 + MSI-X performance regression

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2012-07-24 22:20, Nicholas A. Bellinger wrote:
> On Tue, 2012-07-24 at 09:57 +0200, Jan Kiszka wrote:
>> On 2012-07-24 09:42, Nicholas A. Bellinger wrote:
>>> Hi Anthony, Stefan & QEMU folks,
>>>
> 
> <SNIP>
> 
>>> However, thus far I've not been able to get virtio-scsi <-> tcm_vhost
>>> I/O to actually work against the latest qemu.git/master..  
>>>
>>> So while doing a (manual) bisection w/ this series to track down the
>>> issue with qemu/master, I managed to run across something else..  With
>>> the vhost-scsi series applied, everything is working as expected up
>>> until the following commit:
>>>
>>> commit 1523ed9e1d46b0b54540049d491475ccac7e6421
>>> Author: Jan Kiszka <jan.kiszka@siemens.com>
>>> Date:   Thu May 17 10:32:39 2012 -0300
>>>
>>>     virtio/vhost: Add support for KVM in-kernel MSI injection
>>>
>>>
>>> This commit ends up triggering the following assert immediately after
>>> starting qemu with virtio-scsi <-> tcm_vhost:
>>>
>>> qemu-system-x86_64: /usr/src/qemu.git/hw/msix.c:515:
>>>        msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier &&
>>>                                     dev->msix_vector_release_notifier' failed.
>>>
>>> OK, so adding the following hack allows me to boot:
>>>
>>> diff --git a/hw/msix.c b/hw/msix.c
>>> index 59c7a83..6036909 100644
>>> --- a/hw/msix.c
>>> +++ b/hw/msix.c
>>> @@ -511,6 +511,11 @@ void msix_unset_vector_notifiers(PCIDevice *dev)
>>>  {
>>>      int vector;
>>>  
>>> +    if (!dev->msix_vector_use_notifier && !dev->msix_vector_release_notifier) {
>>> +        printf("Hit NULL msix_unset_vector_notifiers for: %s\n", dev->name);
>>> +        return;
>>> +    }
>>> +
>>>      assert(dev->msix_vector_use_notifier &&
>>>             dev->msix_vector_release_notifier);
>>>  
>>> --
>>
>> Can you post a backtrace from gdb?
>>
> 
> Sure, w/o the above patch the backtrace with commit 1523ed9e1d looks
> like the following:
> 
> (gdb) run
> Starting program: /usr/src/qemu.git/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 2 -m 2048 -serial file:/tmp/vhost-serial.txt -hda /usr/src/qemu-vhost.git/debian_squeeze_amd64_standard-old.qcow2 -vhost-scsi id=vhost-scsi0,wwpn=naa.600140579ad21088,tpgt=1 -device virtio-scsi-pci,vhost-scsi=vhost-scsi0,event_idx=off
> [Thread debugging using libthread_db enabled]
> wwpn = "vhost-scsi0" tpgt = "1"
> [New Thread 0x7ffff45f8700 (LWP 26508)]
> [New Thread 0x7ffff3bf6700 (LWP 26509)]
> [New Thread 0x7ffff33f5700 (LWP 26510)]
> vhost_scsi_stop
> Failed to clear endpoint
> qemu-system-x86_64: /usr/src/qemu.git/hw/msix.c:515: msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier && dev->msix_vector_release_notifier' failed.
> 
> Program received signal SIGABRT, Aborted.
> 0x00007ffff5e8b165 in raise () from /lib/libc.so.6
> (gdb) bt
> #0  0x00007ffff5e8b165 in raise () from /lib/libc.so.6
> #1  0x00007ffff5e8df70 in abort () from /lib/libc.so.6
> #2  0x00007ffff5e842b1 in __assert_fail () from /lib/libc.so.6
> #3  0x00000000004a84a1 in msix_unset_vector_notifiers (dev=0x1463a70) at /usr/src/qemu.git/hw/msix.c:514
> #4  0x00000000004d2865 in virtio_pci_set_guest_notifiers (opaque=0x6788, assign=136)

There are obviously random parameter values submitted. This is broken.

>     at /usr/src/qemu.git/hw/virtio-pci.c:703
> #5  0x000000000062955f in vhost_dev_stop (hdev=0x126c8a8, vdev=0x1465220) at /usr/src/qemu.git/hw/vhost.c:954
> #6  0x0000000000628989 in vhost_scsi_stop (vs=0x126c890, vdev=0x1465220) at /usr/src/qemu.git/hw/vhost-scsi.c:115

And I suspect this is the origin: A stop on an unstarted vhost device.
Am I right?

But such a bug may not yet explain a performance regression - unless the
spurious stop creates a state that slows things down later on.

Did you check that direct MSI-X vector injection is actually in use
later on?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux