current qemu-kvm doesn't work with vhost

current qemu-kvm doesn't work with vhost

[Amos Kong]

host kernel: 2.6.39-rc2+
qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349

(gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2 -snapshot -device virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he

(qemu) [New Thread 0x7fffbe7dd700 (LWP 10410)]
[New Thread 0x7fffbdfdc700 (LWP 10411)]
qemu-kvm: /project/rh/qemu-kvm/hw/msix.c:616: msix_unset_mask_notifier: Assertion `dev->msix_mask_notifier' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff3233700 (LWP 10405)]
0x00007ffff6593ba5 in raise () from /lib/libc.so.6
(gdb) bt
#0  0x00007ffff6593ba5 in raise () from /lib/libc.so.6
#1  0x00007ffff65976b0 in abort () from /lib/libc.so.6
#2  0x00007ffff658ca71 in __assert_fail () from /lib/libc.so.6
#3  0x00000000005754eb in msix_unset_mask_notifier (dev=0x1e56c50) at /project/rh/qemu-kvm/hw/msix.c:616
#4  0x0000000000571ea4 in virtio_pci_set_guest_notifiers (opaque=0x1e56c50, assign=true) at /project/rh/qemu-kvm/hw/virtio-pci.c:674
#5  0x000000000042c279 in vhost_dev_start (hdev=0x11c0420, vdev=0x1c40f10) at /project/rh/qemu-kvm/hw/vhost.c:710
#6  0x0000000000429fe9 in vhost_net_start (net=0x11c0420, dev=0x1c40f10) at /project/rh/qemu-kvm/hw/vhost_net.c:142
#7  0x0000000000425644 in virtio_net_vhost_status (n=0x1c40f10, status=7 '\a') at /project/rh/qemu-kvm/hw/virtio-net.c:127
#8  0x00000000004256ec in virtio_net_set_status (vdev=0x1c40f10, status=7 '\a') at /project/rh/qemu-kvm/hw/virtio-net.c:144
#9  0x00000000005709e6 in virtio_set_status (vdev=0x1c40f10, val=7 '\a') at /project/rh/qemu-kvm/hw/virtio.h:136
#10 0x000000000057136f in virtio_ioport_write (opaque=0x1e56c50, addr=18, val=7) at /project/rh/qemu-kvm/hw/virtio-pci.c:338
#11 0x00000000005717db in virtio_pci_config_writeb (opaque=0x1e56c50, addr=18, val=7) at /project/rh/qemu-kvm/hw/virtio-pci.c:462
#12 0x000000000048409d in ioport_write (index=0, address=49202, data=7) at ioport.c:81
#13 0x0000000000484756 in cpu_outb (addr=49202, val=7 '\a') at ioport.c:266
#14 0x0000000000432dd2 in kvm_handle_io (port=49202, data=0x7ffff7ff3000, direction=1, size=1, count=1) at /project/rh/qemu-kvm/kvm-all.c:864
#15 0x0000000000433266 in kvm_cpu_exec (env=0x188e5b0) at /project/rh/qemu-kvm/kvm-all.c:1008
#16 0x00000000004091ef in qemu_kvm_cpu_thread_fn (arg=0x188e5b0) at /project/rh/qemu-kvm/cpus.c:808
#17 0x00007ffff79be971 in start_thread () from /lib/libpthread.so.0
#18 0x00007ffff664692d in clone () from /lib/libc.so.6
#19 0x0000000000000000 in ?? ()

Re: current qemu-kvm doesn't work with vhost

[Jan Kiszka]

On 2011-06-10 05:08, Amos Kong wrote:
> host kernel: 2.6.39-rc2+
> qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349
> 
> (gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2 -snapshot -device virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he
> 
> (qemu) [New Thread 0x7fffbe7dd700 (LWP 10410)]
> [New Thread 0x7fffbdfdc700 (LWP 10411)]
> qemu-kvm: /project/rh/qemu-kvm/hw/msix.c:616: msix_unset_mask_notifier: Assertion `dev->msix_mask_notifier' failed.
> 
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x7ffff3233700 (LWP 10405)]
> 0x00007ffff6593ba5 in raise () from /lib/libc.so.6
> (gdb) bt
> #0  0x00007ffff6593ba5 in raise () from /lib/libc.so.6
> #1  0x00007ffff65976b0 in abort () from /lib/libc.so.6
> #2  0x00007ffff658ca71 in __assert_fail () from /lib/libc.so.6
> #3  0x00000000005754eb in msix_unset_mask_notifier (dev=0x1e56c50) at /project/rh/qemu-kvm/hw/msix.c:616
> #4  0x0000000000571ea4 in virtio_pci_set_guest_notifiers (opaque=0x1e56c50, assign=true) at /project/rh/qemu-kvm/hw/virtio-pci.c:674
> #5  0x000000000042c279 in vhost_dev_start (hdev=0x11c0420, vdev=0x1c40f10) at /project/rh/qemu-kvm/hw/vhost.c:710
> #6  0x0000000000429fe9 in vhost_net_start (net=0x11c0420, dev=0x1c40f10) at /project/rh/qemu-kvm/hw/vhost_net.c:142
> #7  0x0000000000425644 in virtio_net_vhost_status (n=0x1c40f10, status=7 '\a') at /project/rh/qemu-kvm/hw/virtio-net.c:127
> #8  0x00000000004256ec in virtio_net_set_status (vdev=0x1c40f10, status=7 '\a') at /project/rh/qemu-kvm/hw/virtio-net.c:144
> #9  0x00000000005709e6 in virtio_set_status (vdev=0x1c40f10, val=7 '\a') at /project/rh/qemu-kvm/hw/virtio.h:136
> #10 0x000000000057136f in virtio_ioport_write (opaque=0x1e56c50, addr=18, val=7) at /project/rh/qemu-kvm/hw/virtio-pci.c:338
> #11 0x00000000005717db in virtio_pci_config_writeb (opaque=0x1e56c50, addr=18, val=7) at /project/rh/qemu-kvm/hw/virtio-pci.c:462
> #12 0x000000000048409d in ioport_write (index=0, address=49202, data=7) at ioport.c:81
> #13 0x0000000000484756 in cpu_outb (addr=49202, val=7 '\a') at ioport.c:266
> #14 0x0000000000432dd2 in kvm_handle_io (port=49202, data=0x7ffff7ff3000, direction=1, size=1, count=1) at /project/rh/qemu-kvm/kvm-all.c:864
> #15 0x0000000000433266 in kvm_cpu_exec (env=0x188e5b0) at /project/rh/qemu-kvm/kvm-all.c:1008
> #16 0x00000000004091ef in qemu_kvm_cpu_thread_fn (arg=0x188e5b0) at /project/rh/qemu-kvm/cpus.c:808
> #17 0x00007ffff79be971 in start_thread () from /lib/libpthread.so.0
> #18 0x00007ffff664692d in clone () from /lib/libc.so.6
> #19 0x0000000000000000 in ?? ()

I already came across that symptom in a different context. Fixed by the
patch below. 

However, the real issue is related to an upstream cleanup of the
virtio-pci build. That reveals some unneeded build dependencies in
qemu-kvm. Will post a fix.

Jan

-----8<-----

From: Jan Kiszka <jan.kiszka@siemens.com>
qemu-kvm: Fix error path of virtio_pci_set_guest_notifiers

The mask notifier is never installed on error, so this deinstallation
will just trigger an assert in msix_unset_mask_notifier.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 hw/virtio-pci.c |    4 ----
 1 files changed, 0 insertions(+), 4 deletions(-)

diff --git a/hw/virtio-pci.c b/hw/virtio-pci.c
index c4735c5..311c47b 100644
--- a/hw/virtio-pci.c
+++ b/hw/virtio-pci.c
@@ -688,10 +688,6 @@ static int virtio_pci_set_guest_notifiers(void *opaque, bool assign)
 
 assign_error:
     /* We get here on assignment failure. Recover by undoing for VQs 0 .. n. */
-    if (assign) {
-        msix_unset_mask_notifier(&proxy->pci_dev);
-    }
-
     while (--n >= 0) {
         virtio_pci_set_guest_notifier(opaque, n, !assign);
     }

Re: current qemu-kvm doesn't work with vhost

[Georg Hopp]

Jan Kiszka <jan.kiszka <at> web.de> writes:
> 
> On 2011-06-10 05:08, Amos Kong wrote:
> > host kernel: 2.6.39-rc2+
> > qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349
> > 
> > (gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2 -snapshot
-device
> virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he
> > 
> 
> I already came across that symptom in a different context. Fixed by the
> patch below. 
> 
> However, the real issue is related to an upstream cleanup of the
> virtio-pci build. That reveals some unneeded build dependencies in
> qemu-kvm. Will post a fix.
> 
> Jan

FYI

I encountered the same problem and applied the patch.

Well this results in the following error while starting the guest:

 qemu-system-x86_64: unable to start vhost net: 38: 
  falling back on userspace virtio

and i have no network at all. I will disable vhost=on for now.

Greetz Georg

Re: current qemu-kvm doesn't work with vhost

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1104 bytes --]

On 2011-06-17 09:10, Georg Hopp wrote:
> Jan Kiszka <jan.kiszka <at> web.de> writes:
>>
>> On 2011-06-10 05:08, Amos Kong wrote:
>>> host kernel: 2.6.39-rc2+
>>> qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349
>>>
>>> (gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2 -snapshot
> -device
>> virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he
>>>
>>
>> I already came across that symptom in a different context. Fixed by the
>> patch below. 
>>
>> However, the real issue is related to an upstream cleanup of the
>> virtio-pci build. That reveals some unneeded build dependencies in
>> qemu-kvm. Will post a fix.
>>
>> Jan
> 
> FYI
> 
> I encountered the same problem and applied the patch.
> 
> Well this results in the following error while starting the guest:
> 
>  qemu-system-x86_64: unable to start vhost net: 38: 
>   falling back on userspace virtio
> 
> and i have no network at all. I will disable vhost=on for now.

Hmm, works fine for me. The vhost-net module is loaded (though I got a
different message when I forgot to load it)?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

AW: current qemu-kvm doesn't work with vhost

[Georg Hopp]

Am 17.06.2011 09:29:41 schrieb(en) Jan Kiszka:
> On 2011-06-17 09:10, Georg Hopp wrote:
> > Jan Kiszka <jan.kiszka <at> web.de> writes:
> >>
> >> On 2011-06-10 05:08, Amos Kong wrote:
> >>> host kernel: 2.6.39-rc2+
> >>> qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349
> >>>
> >>> (gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2  
> -snapshot
> > -device
> >> virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he
> >>>
> >>
> >> I already came across that symptom in a different context. Fixed  
> by the
> >> patch below.
> >>
> >> However, the real issue is related to an upstream cleanup of the
> >> virtio-pci build. That reveals some unneeded build dependencies in
> >> qemu-kvm. Will post a fix.
> >>
> >> Jan
> >
> > FYI
> >
> > I encountered the same problem and applied the patch.
> >
> > Well this results in the following error while starting the guest:
> >
> >  qemu-system-x86_64: unable to start vhost net: 38:
> >   falling back on userspace virtio
> >
> > and i have no network at all. I will disable vhost=on for now.
> 
> Hmm, works fine for me. The vhost-net module is loaded (though I got a
> different message when I forgot to load it)?
> 
> Jan
> 

Generally it works for me until git revision b2146d8bd.

I have compiled vhost-net directly in my kernel so a have definetly not  
forgotten to load it...
As i use gentoo i made an ebuild that installes exactly this revision.
If i find the time to do some debugging i will do so, but actually i am  
very busy with my job
family and the use of kvm is just a sparetime thing. :D

If it would be of some help i can set a breakpoint just before the path  
and see what
causes the message.

Greetz
    Georg

Re: current qemu-kvm doesn't work with vhost

[Avi Kivity]

On 06/10/2011 09:12 AM, Jan Kiszka wrote:
> From: Jan Kiszka<jan.kiszka@siemens.com>
> qemu-kvm: Fix error path of virtio_pci_set_guest_notifiers
>
> The mask notifier is never installed on error, so this deinstallation
> will just trigger an assert in msix_unset_mask_notifier.

Applied, thanks.

-- 
error compiling committee.c: too many arguments to function

Re: current qemu-kvm doesn't work with vhost

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]

On 2011-06-17 22:31, Georg Hopp wrote:
> Am 17.06.2011 09:29:41 schrieb(en) Jan Kiszka:
>> On 2011-06-17 09:10, Georg Hopp wrote:
>> > Jan Kiszka <jan.kiszka <at> web.de> writes:
>> >>
>> >> On 2011-06-10 05:08, Amos Kong wrote:
>> >>> host kernel: 2.6.39-rc2+
>> >>> qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349
>> >>>
>> >>> (gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2
>> -snapshot
>> > -device
>> >> virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he
>> >>>
>> >>
>> >> I already came across that symptom in a different context. Fixed by
>> the
>> >> patch below.
>> >>
>> >> However, the real issue is related to an upstream cleanup of the
>> >> virtio-pci build. That reveals some unneeded build dependencies in
>> >> qemu-kvm. Will post a fix.
>> >>
>> >> Jan
>> >
>> > FYI
>> >
>> > I encountered the same problem and applied the patch.
>> >
>> > Well this results in the following error while starting the guest:
>> >
>> >  qemu-system-x86_64: unable to start vhost net: 38:
>> >   falling back on userspace virtio
>> >
>> > and i have no network at all. I will disable vhost=on for now.
>>
>> Hmm, works fine for me. The vhost-net module is loaded (though I got a
>> different message when I forgot to load it)?
>>
>> Jan
>>
> 
> Generally it works for me until git revision b2146d8bd.

You mean including that commit, right?

> 
> I have compiled vhost-net directly in my kernel so a have definetly not
> forgotten to load it...
> As i use gentoo i made an ebuild that installes exactly this revision.
> If i find the time to do some debugging i will do so, but actually i am
> very busy with my job
> family and the use of kvm is just a sparetime thing. :D
> 
> If it would be of some help i can set a breakpoint just before the path
> and see what
> causes the message.

Let's start with double-checking that you are on ce5f0a588b, did a make
clean && make, and then actually used that result. I suspect an
inconsistent build as ce5f0a588b makes the difference between ENOSYS
(38) and working vhost support here.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

AW: current qemu-kvm doesn't work with vhost

[Georg Hopp]

Am 19.06.2011 10:58:20 schrieb(en) Jan Kiszka:
> On 2011-06-17 22:31, Georg Hopp wrote:
> > Am 17.06.2011 09:29:41 schrieb(en) Jan Kiszka:
> >> On 2011-06-17 09:10, Georg Hopp wrote:
> >> > Jan Kiszka <jan.kiszka <at> web.de> writes:
> >> >>
> >> >> On 2011-06-10 05:08, Amos Kong wrote:
> >> >>> host kernel: 2.6.39-rc2+
> >> >>> qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349
> >> >>>
> >> >>> (gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2
> >> -snapshot
> >> > -device
> >> >> virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he
> >> >>>
> >> >>
> >> >> I already came across that symptom in a different context.  
> Fixed by
> >> the
> >> >> patch below.
> >> >>
> >> >> However, the real issue is related to an upstream cleanup of the
> >> >> virtio-pci build. That reveals some unneeded build dependencies  
> in
> >> >> qemu-kvm. Will post a fix.
> >> >>
> >> >> Jan
> >> >
> >> > FYI
> >> >
> >> > I encountered the same problem and applied the patch.
> >> >
> >> > Well this results in the following error while starting the  
> guest:
> >> >
> >> >  qemu-system-x86_64: unable to start vhost net: 38:
> >> >   falling back on userspace virtio
> >> >
> >> > and i have no network at all. I will disable vhost=on for now.
> >>
> >> Hmm, works fine for me. The vhost-net module is loaded (though I  
> got a
> >> different message when I forgot to load it)?
> >>
> >> Jan
> >>
> >
> > Generally it works for me until git revision b2146d8bd.
> 
> You mean including that commit, right?
> 
> >
> > I have compiled vhost-net directly in my kernel so a have definetly  
> not
> > forgotten to load it...
> > As i use gentoo i made an ebuild that installes exactly this  
> revision.
> > If i find the time to do some debugging i will do so, but actually  
> i am
> > very busy with my job
> > family and the use of kvm is just a sparetime thing. :D
> >
> > If it would be of some help i can set a breakpoint just before the  
> path
> > and see what
> > causes the message.
> 
> Let's start with double-checking that you are on ce5f0a588b, did a  
> make
> clean && make, and then actually used that result. I suspect an
> inconsistent build as ce5f0a588b makes the difference between ENOSYS
> (38) and working vhost support here.
> 
> Jan
> 
> 

Hi,

sorry for the long wait. Had a hard weekend. ;)

at revision: ce5f0a588b     - check
applied patch               - check
clean make                  - check

And now everything works as expected...well, at least i got no error at  
all.
Thanks! Anything else i can do?

Georg

AW: current qemu-kvm doesn't work with vhost

[Georg Hopp]

Am 19.06.2011 10:58:20 schrieb(en) Jan Kiszka:
> On 2011-06-17 22:31, Georg Hopp wrote:
> > Am 17.06.2011 09:29:41 schrieb(en) Jan Kiszka:
> >> On 2011-06-17 09:10, Georg Hopp wrote:
> >> > Jan Kiszka <jan.kiszka <at> web.de> writes:
> >> >>
> >> >> On 2011-06-10 05:08, Amos Kong wrote:
> >> >>> host kernel: 2.6.39-rc2+
> >> >>> qemu-kvm : 05f1737582ab6c075476bde931c5eafbc62a9349
> >> >>>
> >> >>> (gdb) r -monitor stdio -m 800 ~/RHEL-Server-6.0-64-virtio.qcow2
> >> -snapshot
> >> > -device
> >> >> virtio-net-pci,netdev=he -netdev tap,vhost=on,id=he
> >> >>>
> >> >>
> >> >> I already came across that symptom in a different context.  
> Fixed by
> >> the
> >> >> patch below.
> >> >>
> >> >> However, the real issue is related to an upstream cleanup of the
> >> >> virtio-pci build. That reveals some unneeded build dependencies  
> in
> >> >> qemu-kvm. Will post a fix.
> >> >>
> >> >> Jan
> >> >
> >> > FYI
> >> >
> >> > I encountered the same problem and applied the patch.
> >> >
> >> > Well this results in the following error while starting the  
> guest:
> >> >
> >> >  qemu-system-x86_64: unable to start vhost net: 38:
> >> >   falling back on userspace virtio
> >> >
> >> > and i have no network at all. I will disable vhost=on for now.
> >>
> >> Hmm, works fine for me. The vhost-net module is loaded (though I  
> got a
> >> different message when I forgot to load it)?
> >>
> >> Jan
> >>
> >
> > Generally it works for me until git revision b2146d8bd.
> 
> You mean including that commit, right?
> 
> >
> > I have compiled vhost-net directly in my kernel so a have definetly  
> not
> > forgotten to load it...
> > As i use gentoo i made an ebuild that installes exactly this  
> revision.
> > If i find the time to do some debugging i will do so, but actually  
> i am
> > very busy with my job
> > family and the use of kvm is just a sparetime thing. :D
> >
> > If it would be of some help i can set a breakpoint just before the  
> path
> > and see what
> > causes the message.
> 
> Let's start with double-checking that you are on ce5f0a588b, did a  
> make
> clean && make, and then actually used that result. I suspect an
> inconsistent build as ce5f0a588b makes the difference between ENOSYS
> (38) and working vhost support here.
> 
> Jan
> 
> 

Hi again,

tried the current HEAD without any patches and vhost works again for me
and with much better performance than before.

------------------------------------------------------------
Client connecting to host, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.100.4 port 48053 connected with 192.168.100.1 port  
5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  7.63 GBytes  6.55 Gbits/sec

Thanks for the great work!

Greets
    Georg