[Qemu-devel] assert hit on master with q35 machine type

[Qemu-devel] assert hit on master with q35 machine type

[Doug Goldstein]

[-- Attachment #1: Type: text/plain, Size: 1371 bytes --]

I'm currently hitting an assert on master. I've bisected the change down
to aff8fd18f1786fc5af259a9bc0077727222f51ca but obviously that's a false
positive because that adds the assert for the condition. So obviously
the issue was introduced prior to this. As another data point the issue
only occurs with q35 machines and will only happen with pc-q35-2.7 and
pc-q35-2.8 (and obviously the aliased q35). If I use pc-q35-2.6 it does
not hit.

The message is: qemu-system-x86_64:
/home/doug/work/qemu/hw/virtio/virtio.c:214:
virtio_queue_set_notification: Assertion `vq->notification_disabled > 0'
failed.

But that clearly just shows its the assert that was added in that
commit. To create the machine I use the following command line:

./work/qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -M pc-q35-2.8
-device intel-iommu -cpu host -m 2048 -smp 2 -drive
if=pflash,format=raw,file=/tmp/tmp.Sno2BP3saP -global
isa-debugcon.iobase=0x402 -debugcon file:/tmp/tmp.KCrwM98Mus -monitor
stdio -chardev socket,host=127.0.0.1,port=4455,id=S0,server,nowait
-device isa-serial,chardev=S0 -device piix3-usb-uhci -device usb-tablet
-netdev id=net0,type=tap -device virtio-net-pci,netdev=net0 -boot
order=n -device qxl-vga -gdb tcp::1234

It happens almost right away. Let me know how I can help track this down
further.

Thanks.
-- 
Doug Goldstein


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

Re: [Qemu-devel] assert hit on master with q35 machine type

[Dr. David Alan Gilbert]

copying in Stefan since it's his commit.

* Doug Goldstein (cardoe@cardoe.com) wrote:
> I'm currently hitting an assert on master. I've bisected the change down
> to aff8fd18f1786fc5af259a9bc0077727222f51ca but obviously that's a false
> positive because that adds the assert for the condition. So obviously
> the issue was introduced prior to this. As another data point the issue
> only occurs with q35 machines and will only happen with pc-q35-2.7 and
> pc-q35-2.8 (and obviously the aliased q35). If I use pc-q35-2.6 it does
> not hit.
> 
> The message is: qemu-system-x86_64:
> /home/doug/work/qemu/hw/virtio/virtio.c:214:
> virtio_queue_set_notification: Assertion `vq->notification_disabled > 0'
> failed.
> 
> But that clearly just shows its the assert that was added in that
> commit. To create the machine I use the following command line:
> 
> ./work/qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -M pc-q35-2.8
> -device intel-iommu -cpu host -m 2048 -smp 2 -drive
> if=pflash,format=raw,file=/tmp/tmp.Sno2BP3saP -global
> isa-debugcon.iobase=0x402 -debugcon file:/tmp/tmp.KCrwM98Mus -monitor
> stdio -chardev socket,host=127.0.0.1,port=4455,id=S0,server,nowait
> -device isa-serial,chardev=S0 -device piix3-usb-uhci -device usb-tablet
> -netdev id=net0,type=tap -device virtio-net-pci,netdev=net0 -boot
> order=n -device qxl-vga -gdb tcp::1234
> 
> It happens almost right away. Let me know how I can help track this down
> further.
> 
> Thanks.
> -- 
> Doug Goldstein
> 


--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] assert hit on master with q35 machine type

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1814 bytes --]

On Mon, Jan 09, 2017 at 01:08:53PM +0000, Dr. David Alan Gilbert wrote:
> copying in Stefan since it's his commit.
> 
> * Doug Goldstein (cardoe@cardoe.com) wrote:
> > I'm currently hitting an assert on master. I've bisected the change down
> > to aff8fd18f1786fc5af259a9bc0077727222f51ca but obviously that's a false
> > positive because that adds the assert for the condition. So obviously
> > the issue was introduced prior to this. As another data point the issue
> > only occurs with q35 machines and will only happen with pc-q35-2.7 and
> > pc-q35-2.8 (and obviously the aliased q35). If I use pc-q35-2.6 it does
> > not hit.
> > 
> > The message is: qemu-system-x86_64:
> > /home/doug/work/qemu/hw/virtio/virtio.c:214:
> > virtio_queue_set_notification: Assertion `vq->notification_disabled > 0'
> > failed.
> > 
> > But that clearly just shows its the assert that was added in that
> > commit. To create the machine I use the following command line:
> > 
> > ./work/qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -M pc-q35-2.8
> > -device intel-iommu -cpu host -m 2048 -smp 2 -drive
> > if=pflash,format=raw,file=/tmp/tmp.Sno2BP3saP -global
> > isa-debugcon.iobase=0x402 -debugcon file:/tmp/tmp.KCrwM98Mus -monitor
> > stdio -chardev socket,host=127.0.0.1,port=4455,id=S0,server,nowait
> > -device isa-serial,chardev=S0 -device piix3-usb-uhci -device usb-tablet
> > -netdev id=net0,type=tap -device virtio-net-pci,netdev=net0 -boot
> > order=n -device qxl-vga -gdb tcp::1234
> > 
> > It happens almost right away. Let me know how I can help track this down
> > further.

I can't reproduce the assertion failure.

Are you able to provide a backtrace (e.g. launch QEMU under gdb and then
use the 'bt' command after the assertion failure is raised)?

Thanks,
Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] assert hit on master with q35 machine type

[Doug Goldstein]

[-- Attachment #1: Type: text/plain, Size: 4395 bytes --]

On 1/10/17 10:01 AM, Stefan Hajnoczi wrote:
> On Mon, Jan 09, 2017 at 01:08:53PM +0000, Dr. David Alan Gilbert wrote:
>> copying in Stefan since it's his commit.
> 
> I can't reproduce the assertion failure.
> 
> Are you able to provide a backtrace (e.g. launch QEMU under gdb and then
> use the 'bt' command after the assertion failure is raised)?
> 
> Thanks,
> Stefan
> 

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
#1  0x00007ffff44db3ea in __GI_abort () at abort.c:89
#2  0x00007ffff44d1bb7 in __assert_fail_base (fmt=<optimized out>,
    assertion=assertion@entry=0x555555ae8e8a "vq->notification_disabled
> 0",
    file=file@entry=0x555555ae89c0
"/home/doug/work/qemu/hw/virtio/virtio.c",
    line=line@entry=215,
    function=function@entry=0x555555ae9630 <__PRETTY_FUNCTION__.43707>
"virtio_queue_set_notification") at assert.c:92
#3  0x00007ffff44d1c62 in __GI___assert_fail (
    assertion=assertion@entry=0x555555ae8e8a "vq->notification_disabled
> 0",
    file=file@entry=0x555555ae89c0
"/home/doug/work/qemu/hw/virtio/virtio.c",
    line=line@entry=215,
    function=function@entry=0x555555ae9630 <__PRETTY_FUNCTION__.43707>
"virtio_queue_set_notification") at assert.c:101
#4  0x00005555557f25d6 in virtio_queue_set_notification (vq=0x55555666aa90,
    enable=enable@entry=1) at /home/doug/work/qemu/hw/virtio/virtio.c:215
#5  0x00005555557dc311 in virtio_net_has_buffers (q=<optimized out>,
    q=<optimized out>, bufsize=102)
    at /home/doug/work/qemu/hw/net/virtio-net.c:1008
#6  virtio_net_receive (nc=<optimized out>, buf=0x555557386b88 "", size=102)
    at /home/doug/work/qemu/hw/net/virtio-net.c:1148
#7  0x00005555559cad33 in nc_sendv_compat (flags=<optimized out>, iovcnt=1,
    iov=0x7fffead746d0, nc=0x55555788b340) at net/net.c:705
#8  qemu_deliver_packet_iov (sender=<optimized out>, flags=<optimized out>,
    iov=0x7fffead746d0, iovcnt=1, opaque=0x55555788b340) at net/net.c:732
#9  0x00005555559cd929 in qemu_net_queue_deliver (size=<optimized out>,
    data=<optimized out>, flags=<optimized out>, sender=<optimized out>,
    queue=0x55555788b550) at net/queue.c:164
#10 qemu_net_queue_flush (queue=0x55555788b550) at net/queue.c:261
#11 0x00005555559cab0c in qemu_flush_or_purge_queued_packets (
    nc=0x55555788b340, purge=<optimized out>) at net/net.c:611
#12 0x00005555557f1eff in virtio_queue_notify_vq (vq=0x55555666aa90)
    at /home/doug/work/qemu/hw/virtio/virtio.c:1318
#13 0x00005555557adec8 in memory_region_write_accessor (mr=0x555557756d30,
    addr=0, value=<optimized out>, size=2, shift=<optimized out>,
    mask=<optimized out>, attrs=...) at /home/doug/work/qemu/memory.c:526
#14 0x00005555557ab25d in access_with_adjusted_size (addr=addr@entry=0,
    value=value@entry=0x7fffead74888, size=size@entry=2,
    access_size_min=<optimized out>, access_size_max=<optimized out>,
    access=0x5555557ade50 <memory_region_write_accessor>, mr=0x555557756d30,
    attrs=...) at /home/doug/work/qemu/memory.c:592
#15 0x00005555557af28c in memory_region_dispatch_write (
    mr=mr@entry=0x555557756d30, addr=0, data=<optimized out>,
    size=size@entry=2, attrs=attrs@entry=...)
    at /home/doug/work/qemu/memory.c:1336
#16 0x0000555555764d8a in address_space_write_continue (mr=0x555557756d30,
    l=<optimized out>, addr1=<optimized out>, len=2, buf=0x7ffff7fed028 "",
    attrs=..., addr=34359750656, as=0x5555560aa6a0 <address_space_memory>)
    at /home/doug/work/qemu/exec.c:2635
#17 address_space_write (as=0x5555560aa6a0 <address_space_memory>,
    addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized
out>)
    at /home/doug/work/qemu/exec.c:2686
#18 0x000055555576532d in address_space_rw (as=<optimized out>,
    addr=<optimized out>, attrs=..., attrs@entry=...,
    buf=buf@entry=0x7ffff7fed028 "", len=<optimized out>,
    is_write=<optimized out>) at /home/doug/work/qemu/exec.c:2788
#19 0x00005555557aa2a8 in kvm_cpu_exec (cpu=cpu@entry=0x5555565bc7f0)
    at /home/doug/work/qemu/kvm-all.c:1968
#20 0x0000555555797d48 in qemu_kvm_cpu_thread_fn (arg=0x5555565bc7f0)
    at /home/doug/work/qemu/cpus.c:998
#21 0x00007ffff48726ca in start_thread (arg=0x7fffead75700)
    at pthread_create.c:333
#22 0x00007ffff45ac0af in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

-- 
Doug Goldstein


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

Re: [Qemu-devel] assert hit on master with q35 machine type

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2101 bytes --]

On Tue, Jan 10, 2017 at 07:22:24PM -0600, Doug Goldstein wrote:
> On 1/10/17 10:01 AM, Stefan Hajnoczi wrote:
> > On Mon, Jan 09, 2017 at 01:08:53PM +0000, Dr. David Alan Gilbert wrote:
> >> copying in Stefan since it's his commit.
> > 
> > I can't reproduce the assertion failure.
> > 
> > Are you able to provide a backtrace (e.g. launch QEMU under gdb and then
> > use the 'bt' command after the assertion failure is raised)?
> > 
> > Thanks,
> > Stefan
> > 
> 
> #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
> #1  0x00007ffff44db3ea in __GI_abort () at abort.c:89
> #2  0x00007ffff44d1bb7 in __assert_fail_base (fmt=<optimized out>,
>     assertion=assertion@entry=0x555555ae8e8a "vq->notification_disabled
> > 0",
>     file=file@entry=0x555555ae89c0
> "/home/doug/work/qemu/hw/virtio/virtio.c",
>     line=line@entry=215,
>     function=function@entry=0x555555ae9630 <__PRETTY_FUNCTION__.43707>
> "virtio_queue_set_notification") at assert.c:92
> #3  0x00007ffff44d1c62 in __GI___assert_fail (
>     assertion=assertion@entry=0x555555ae8e8a "vq->notification_disabled
> > 0",
>     file=file@entry=0x555555ae89c0
> "/home/doug/work/qemu/hw/virtio/virtio.c",
>     line=line@entry=215,
>     function=function@entry=0x555555ae9630 <__PRETTY_FUNCTION__.43707>
> "virtio_queue_set_notification") at assert.c:101
> #4  0x00005555557f25d6 in virtio_queue_set_notification (vq=0x55555666aa90,
>     enable=enable@entry=1) at /home/doug/work/qemu/hw/virtio/virtio.c:215
> #5  0x00005555557dc311 in virtio_net_has_buffers (q=<optimized out>,
>     q=<optimized out>, bufsize=102)
>     at /home/doug/work/qemu/hw/net/virtio-net.c:1008
> #6  virtio_net_receive (nc=<optimized out>, buf=0x555557386b88 "", size=102)
>     at /home/doug/work/qemu/hw/net/virtio-net.c:1148

Thanks, it's a legitimate bug.  This code still assumes
virtio_queue_set_notification() calls are idempotent.  It needs to be
updated to support nested callers (e.g. virtqueue polling and
virtio-net).

I'll send a patch to address this and CC you.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]