[Qemu-devel] [BUG] QEMU crashes with dpdk virtio pmd

[Qemu-devel] [BUG] QEMU crashes with dpdk virtio pmd

[wangyunjian]

Qemu crashes, with pre-condition:
vm xml config with multiqueue, and the vm's driver virtio-net support multi-queue

reproduce steps:
i. start dpdk testpmd in VM with the virtio nic
ii. stop testpmd
iii. reboot the VM

This commit "f9d6dbf0  remove virtio queues if the guest doesn't support multiqueue" is introduced.

Qemu version: QEMU emulator version 2.9.50 (v2.9.0-137-g32c7e0a)
VM DPDK version:  DPDK-1.6.1

Call Trace:
#0  0x00007f60881fe5d7 in raise () from /usr/lib64/libc.so.6
#1  0x00007f60881ffcc8 in abort () from /usr/lib64/libc.so.6
#2  0x00007f608823e2f7 in __libc_message () from /usr/lib64/libc.so.6
#3  0x00007f60882456d3 in _int_free () from /usr/lib64/libc.so.6
#4  0x00007f608900158f in g_free () from /usr/lib64/libglib-2.0.so.0
#5  0x00007f6088fea32c in iter_remove_or_steal () from /usr/lib64/libglib-2.0.so.0
#6  0x00007f608edc0986 in object_property_del_all (obj=0x7f6091e74800) at qom/object.c:410
#7  object_finalize (data=0x7f6091e74800) at qom/object.c:467
#8  object_unref (obj=obj@entry=0x7f6091e74800) at qom/object.c:903
#9  0x00007f608eaf1fd3 in phys_section_destroy (mr=0x7f6091e74800) at git/qemu/exec.c:1154
#10 phys_sections_free (map=0x7f6090b72bb0) at git/qemu/exec.c:1163
#11 address_space_dispatch_free (d=0x7f6090b72b90) at git/qemu/exec.c:2514
#12 0x00007f608ee91ace in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:272
#13 0x00007f6089b0ddc5 in start_thread () from /usr/lib64/libpthread.so.0
#14 0x00007f60882bf71d in clone () from /usr/lib64/libc.so.6

Call Trace:
#0  0x00007fdccaeb9790 in ?? ()
#1  0x00007fdcd82d09fc in object_property_del_all (obj=0x7fdcdb8acf60) at qom/object.c:405
#2  object_finalize (data=0x7fdcdb8acf60) at qom/object.c:467
#3  object_unref (obj=obj@entry=0x7fdcdb8acf60) at qom/object.c:903
#4  0x00007fdcd8001fd3 in phys_section_destroy (mr=0x7fdcdb8acf60) at git/qemu/exec.c:1154
#5  phys_sections_free (map=0x7fdcdc86aa00) at git/qemu/exec.c:1163
#6  address_space_dispatch_free (d=0x7fdcdc86a9e0) at git/qemu/exec.c:2514
#7  0x00007fdcd83a1ace in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:272
#8  0x00007fdcd301ddc5 in start_thread () from /usr/lib64/libpthread.so.0
#9  0x00007fdcd17cf71d in clone () from /usr/lib64/libc.so.6

Re: [Qemu-devel] [BUG] QEMU crashes with dpdk virtio pmd

[wangyunjian]

The q->tx_bh will free in virtio_net_del_queue() function, when remove virtio queues 
if the guest doesn't support multiqueue. But it might be still referenced by others (eg . virtio_net_set_status()),
which need so set NULL.

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 7d091c9..98bd683 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -1522,9 +1522,12 @@ static void virtio_net_del_queue(VirtIONet *n, int index)
     if (q->tx_timer) {
         timer_del(q->tx_timer);
         timer_free(q->tx_timer);
+        q->tx_timer = NULL;
     } else {
         qemu_bh_delete(q->tx_bh);
+        q->tx_bh = NULL;
     }
+    q->tx_waiting = 0;
     virtio_del_queue(vdev, index * 2 + 1);
 }

From: wangyunjian 
Sent: Monday, April 24, 2017 6:10 PM
To: qemu-devel@nongnu.org; Michael S. Tsirkin <mst@redhat.com>; 'Jason Wang' <jasowang@redhat.com>
Cc: wangyunjian <wangyunjian@huawei.com>; caihe <caihe@huawei.com>
Subject: [Qemu-devel][BUG] QEMU crashes with dpdk virtio pmd 

Qemu crashes, with pre-condition:
vm xml config with multiqueue, and the vm's driver virtio-net support multi-queue

reproduce steps:
i. start dpdk testpmd in VM with the virtio nic
ii. stop testpmd
iii. reboot the VM

This commit "f9d6dbf0  remove virtio queues if the guest doesn't support multiqueue" is introduced.

Qemu version: QEMU emulator version 2.9.50 (v2.9.0-137-g32c7e0a)
VM DPDK version:  DPDK-1.6.1

Call Trace:
#0  0x00007f60881fe5d7 in raise () from /usr/lib64/libc.so.6
#1  0x00007f60881ffcc8 in abort () from /usr/lib64/libc.so.6
#2  0x00007f608823e2f7 in __libc_message () from /usr/lib64/libc.so.6
#3  0x00007f60882456d3 in _int_free () from /usr/lib64/libc.so.6
#4  0x00007f608900158f in g_free () from /usr/lib64/libglib-2.0.so.0
#5  0x00007f6088fea32c in iter_remove_or_steal () from /usr/lib64/libglib-2.0.so.0
#6  0x00007f608edc0986 in object_property_del_all (obj=0x7f6091e74800) at qom/object.c:410
#7  object_finalize (data=0x7f6091e74800) at qom/object.c:467
#8  object_unref (obj=obj@entry=0x7f6091e74800) at qom/object.c:903
#9  0x00007f608eaf1fd3 in phys_section_destroy (mr=0x7f6091e74800) at git/qemu/exec.c:1154
#10 phys_sections_free (map=0x7f6090b72bb0) at git/qemu/exec.c:1163
#11 address_space_dispatch_free (d=0x7f6090b72b90) at git/qemu/exec.c:2514
#12 0x00007f608ee91ace in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:272
#13 0x00007f6089b0ddc5 in start_thread () from /usr/lib64/libpthread.so.0
#14 0x00007f60882bf71d in clone () from /usr/lib64/libc.so.6

Call Trace:
#0  0x00007fdccaeb9790 in ?? ()
#1  0x00007fdcd82d09fc in object_property_del_all (obj=0x7fdcdb8acf60) at qom/object.c:405
#2  object_finalize (data=0x7fdcdb8acf60) at qom/object.c:467
#3  object_unref (obj=obj@entry=0x7fdcdb8acf60) at qom/object.c:903
#4  0x00007fdcd8001fd3 in phys_section_destroy (mr=0x7fdcdb8acf60) at git/qemu/exec.c:1154
#5  phys_sections_free (map=0x7fdcdc86aa00) at git/qemu/exec.c:1163
#6  address_space_dispatch_free (d=0x7fdcdc86a9e0) at git/qemu/exec.c:2514
#7  0x00007fdcd83a1ace in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:272
#8  0x00007fdcd301ddc5 in start_thread () from /usr/lib64/libpthread.so.0
#9  0x00007fdcd17cf71d in clone () from /usr/lib64/libc.so.6

Re: [Qemu-devel] [BUG] QEMU crashes with dpdk virtio pmd

[Jason Wang]

On 2017年04月25日 19:37, wangyunjian wrote:
> The q->tx_bh will free in virtio_net_del_queue() function, when remove virtio queues
> if the guest doesn't support multiqueue. But it might be still referenced by others (eg . virtio_net_set_status()),
> which need so set NULL.
>
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 7d091c9..98bd683 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -1522,9 +1522,12 @@ static void virtio_net_del_queue(VirtIONet *n, int index)
>       if (q->tx_timer) {
>           timer_del(q->tx_timer);
>           timer_free(q->tx_timer);
> +        q->tx_timer = NULL;
>       } else {
>           qemu_bh_delete(q->tx_bh);
> +        q->tx_bh = NULL;
>       }
> +    q->tx_waiting = 0;
>       virtio_del_queue(vdev, index * 2 + 1);
>   }

Thanks a lot for the fix.

Two questions:

- If virtio_net_set_status() is the only function that may access tx_bh, 
it looks like setting tx_waiting to zero is sufficient?
- Can you post a formal patch for this?

Thanks

> From: wangyunjian
> Sent: Monday, April 24, 2017 6:10 PM
> To: qemu-devel@nongnu.org; Michael S. Tsirkin <mst@redhat.com>; 'Jason Wang' <jasowang@redhat.com>
> Cc: wangyunjian <wangyunjian@huawei.com>; caihe <caihe@huawei.com>
> Subject: [Qemu-devel][BUG] QEMU crashes with dpdk virtio pmd
>
> Qemu crashes, with pre-condition:
> vm xml config with multiqueue, and the vm's driver virtio-net support multi-queue
>
> reproduce steps:
> i. start dpdk testpmd in VM with the virtio nic
> ii. stop testpmd
> iii. reboot the VM
>
> This commit "f9d6dbf0  remove virtio queues if the guest doesn't support multiqueue" is introduced.
>
> Qemu version: QEMU emulator version 2.9.50 (v2.9.0-137-g32c7e0a)
> VM DPDK version:  DPDK-1.6.1
>
> Call Trace:
> #0  0x00007f60881fe5d7 in raise () from /usr/lib64/libc.so.6
> #1  0x00007f60881ffcc8 in abort () from /usr/lib64/libc.so.6
> #2  0x00007f608823e2f7 in __libc_message () from /usr/lib64/libc.so.6
> #3  0x00007f60882456d3 in _int_free () from /usr/lib64/libc.so.6
> #4  0x00007f608900158f in g_free () from /usr/lib64/libglib-2.0.so.0
> #5  0x00007f6088fea32c in iter_remove_or_steal () from /usr/lib64/libglib-2.0.so.0
> #6  0x00007f608edc0986 in object_property_del_all (obj=0x7f6091e74800) at qom/object.c:410
> #7  object_finalize (data=0x7f6091e74800) at qom/object.c:467
> #8  object_unref (obj=obj@entry=0x7f6091e74800) at qom/object.c:903
> #9  0x00007f608eaf1fd3 in phys_section_destroy (mr=0x7f6091e74800) at git/qemu/exec.c:1154
> #10 phys_sections_free (map=0x7f6090b72bb0) at git/qemu/exec.c:1163
> #11 address_space_dispatch_free (d=0x7f6090b72b90) at git/qemu/exec.c:2514
> #12 0x00007f608ee91ace in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:272
> #13 0x00007f6089b0ddc5 in start_thread () from /usr/lib64/libpthread.so.0
> #14 0x00007f60882bf71d in clone () from /usr/lib64/libc.so.6
>
> Call Trace:
> #0  0x00007fdccaeb9790 in ?? ()
> #1  0x00007fdcd82d09fc in object_property_del_all (obj=0x7fdcdb8acf60) at qom/object.c:405
> #2  object_finalize (data=0x7fdcdb8acf60) at qom/object.c:467
> #3  object_unref (obj=obj@entry=0x7fdcdb8acf60) at qom/object.c:903
> #4  0x00007fdcd8001fd3 in phys_section_destroy (mr=0x7fdcdb8acf60) at git/qemu/exec.c:1154
> #5  phys_sections_free (map=0x7fdcdc86aa00) at git/qemu/exec.c:1163
> #6  address_space_dispatch_free (d=0x7fdcdc86a9e0) at git/qemu/exec.c:2514
> #7  0x00007fdcd83a1ace in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:272
> #8  0x00007fdcd301ddc5 in start_thread () from /usr/lib64/libpthread.so.0
> #9  0x00007fdcd17cf71d in clone () from /usr/lib64/libc.so.6
>
>

Re: [Qemu-devel] [BUG] QEMU crashes with dpdk virtio pmd

[Gonglei (Arei)]

CCing Paolo and Stefan, since it has a relationship with bh in Qemu.

> -----Original Message-----
> From: Jason Wang [mailto:jasowang@redhat.com]
> 
> 
> On 2017年04月25日 19:37, wangyunjian wrote:
> > The q->tx_bh will free in virtio_net_del_queue() function, when remove virtio
> queues
> > if the guest doesn't support multiqueue. But it might be still referenced by
> others (eg . virtio_net_set_status()),
> > which need so set NULL.
> >
> > diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> > index 7d091c9..98bd683 100644
> > --- a/hw/net/virtio-net.c
> > +++ b/hw/net/virtio-net.c
> > @@ -1522,9 +1522,12 @@ static void virtio_net_del_queue(VirtIONet *n,
> int index)
> >       if (q->tx_timer) {
> >           timer_del(q->tx_timer);
> >           timer_free(q->tx_timer);
> > +        q->tx_timer = NULL;
> >       } else {
> >           qemu_bh_delete(q->tx_bh);
> > +        q->tx_bh = NULL;
> >       }
> > +    q->tx_waiting = 0;
> >       virtio_del_queue(vdev, index * 2 + 1);
> >   }
> 
> Thanks a lot for the fix.
> 
> Two questions:
> 
> - If virtio_net_set_status() is the only function that may access tx_bh,
> it looks like setting tx_waiting to zero is sufficient?

Currently yes, but we don't assure that it works for all scenarios, so
we set the tx_bh and tx_timer to NULL to avoid to possibly access wild pointer,
which is the common method for usage of bh in Qemu.

I have another question about the root cause of this issure.

This below trace is the path of setting tx_waiting to one in virtio_net_handle_tx_bh() :

Breakpoint 1, virtio_net_handle_tx_bh (vdev=0x0, vq=0x7f335ad13900) at /data/wyj/git/qemu/hw/net/virtio-net.c:1398
1398    {
(gdb) bt
#0  virtio_net_handle_tx_bh (vdev=0x0, vq=0x7f335ad13900) at /data/wyj/git/qemu/hw/net/virtio-net.c:1398
#1  0x00007f3357bddf9c in virtio_bus_set_host_notifier (bus=<optimized out>, n=n@entry=1, assign=assign@entry=false) at hw/virtio/virtio-bus.c:297
#2  0x00007f3357a0055d in vhost_dev_disable_notifiers (hdev=hdev@entry=0x7f335ad84dc0, vdev=vdev@entry=0x7f335c6f5f90) at /data/wyj/git/qemu/hw/virtio/vhost.c:1422
#3  0x00007f33579e3373 in vhost_net_stop_one (net=0x7f335ad84dc0, dev=0x7f335c6f5f90) at /data/wyj/git/qemu/hw/net/vhost_net.c:289
#4  0x00007f33579e385b in vhost_net_stop (dev=dev@entry=0x7f335c6f5f90, ncs=<optimized out>, total_queues=total_queues@entry=8) at /data/wyj/git/qemu/hw/net/vhost_net.c:367
#5  0x00007f33579e15de in virtio_net_vhost_status (status=<optimized out>, n=0x7f335c6f5f90) at /data/wyj/git/qemu/hw/net/virtio-net.c:176
#6  virtio_net_set_status (vdev=0x7f335c6f5f90, status=0 '\000') at /data/wyj/git/qemu/hw/net/virtio-net.c:250
#7  0x00007f33579f8dc6 in virtio_set_status (vdev=vdev@entry=0x7f335c6f5f90, val=val@entry=0 '\000') at /data/wyj/git/qemu/hw/virtio/virtio.c:1146
#8  0x00007f3357bdd3cc in virtio_ioport_write (val=0, addr=18, opaque=0x7f335c6eda80) at hw/virtio/virtio-pci.c:387
#9  virtio_pci_config_write (opaque=0x7f335c6eda80, addr=18, val=0, size=<optimized out>) at hw/virtio/virtio-pci.c:511
#10 0x00007f33579b2155 in memory_region_write_accessor (mr=0x7f335c6ee470, addr=18, value=<optimized out>, size=1, shift=<optimized out>, mask=<optimized out>, attrs=...) at /data/wyj/git/qemu/memory.c:526
#11 0x00007f33579af2e9 in access_with_adjusted_size (addr=addr@entry=18, value=value@entry=0x7f32c97fc888, size=size@entry=1, access_size_min=<optimized out>, access_size_max=<optimized out>, access=access@entry=
    0x7f33579b20f0 <memory_region_write_accessor>, mr=mr@entry=0x7f335c6ee470, attrs=attrs@entry=...) at /data/wyj/git/qemu/memory.c:592
#12 0x00007f33579b2e15 in memory_region_dispatch_write (mr=mr@entry=0x7f335c6ee470, addr=addr@entry=18, data=0, size=size@entry=1, attrs=attrs@entry=...) at /data/wyj/git/qemu/memory.c:1319
#13 0x00007f335796cd93 in address_space_write_continue (mr=0x7f335c6ee470, l=1, addr1=18, len=1, buf=0x7f335773d000 "", attrs=..., addr=49170, as=0x7f3358317060 <address_space_io>) at /data/wyj/git/qemu/exec.c:2834
#14 address_space_write (as=<optimized out>, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>) at /data/wyj/git/qemu/exec.c:2879
#15 0x00007f335796d3ad in address_space_rw (as=<optimized out>, addr=addr@entry=49170, attrs=..., attrs@entry=..., buf=<optimized out>, len=len@entry=1, is_write=is_write@entry=true) at /data/wyj/git/qemu/exec.c:2981
#16 0x00007f33579ae226 in kvm_handle_io (count=1, size=1, direction=<optimized out>, data=<optimized out>, attrs=..., port=49170) at /data/wyj/git/qemu/kvm-all.c:1803
#17 kvm_cpu_exec (cpu=cpu@entry=0x7f335ae82070) at /data/wyj/git/qemu/kvm-all.c:2032
#18 0x00007f335799b632 in qemu_kvm_cpu_thread_fn (arg=0x7f335ae82070) at /data/wyj/git/qemu/cpus.c:1118
#19 0x00007f3352983dc5 in start_thread () from /usr/lib64/libpthread.so.0
#20 0x00007f335113571d in clone () from /usr/lib64/libc.so.6

It calls qemu_bh_schedule(q->tx_bh) at the bottom of virtio_net_handle_tx_bh(),
I don't know why virtio_net_tx_bh() doesn't be invoked, so that the q->tx_waiting is not zero.
[ps: we added logs in virtio_net_tx_bh() to verify that]

Some other information: 

It won't crash if we don't use vhost-net.


Thanks,
-Gonglei

> - Can you post a formal patch for this?
> 
> Thanks
> 
> > From: wangyunjian
> > Sent: Monday, April 24, 2017 6:10 PM
> > To: qemu-devel@nongnu.org; Michael S. Tsirkin <mst@redhat.com>; 'Jason
> Wang' <jasowang@redhat.com>
> > Cc: wangyunjian <wangyunjian@huawei.com>; caihe <caihe@huawei.com>
> > Subject: [Qemu-devel][BUG] QEMU crashes with dpdk virtio pmd
> >
> > Qemu crashes, with pre-condition:
> > vm xml config with multiqueue, and the vm's driver virtio-net support
> multi-queue
> >
> > reproduce steps:
> > i. start dpdk testpmd in VM with the virtio nic
> > ii. stop testpmd
> > iii. reboot the VM
> >
> > This commit "f9d6dbf0  remove virtio queues if the guest doesn't support
> multiqueue" is introduced.
> >
> > Qemu version: QEMU emulator version 2.9.50 (v2.9.0-137-g32c7e0a)
> > VM DPDK version:  DPDK-1.6.1
> >
> > Call Trace:
> > #0  0x00007f60881fe5d7 in raise () from /usr/lib64/libc.so.6
> > #1  0x00007f60881ffcc8 in abort () from /usr/lib64/libc.so.6
> > #2  0x00007f608823e2f7 in __libc_message () from /usr/lib64/libc.so.6
> > #3  0x00007f60882456d3 in _int_free () from /usr/lib64/libc.so.6
> > #4  0x00007f608900158f in g_free () from /usr/lib64/libglib-2.0.so.0
> > #5  0x00007f6088fea32c in iter_remove_or_steal () from
> /usr/lib64/libglib-2.0.so.0
> > #6  0x00007f608edc0986 in object_property_del_all (obj=0x7f6091e74800)
> at qom/object.c:410
> > #7  object_finalize (data=0x7f6091e74800) at qom/object.c:467
> > #8  object_unref (obj=obj@entry=0x7f6091e74800) at qom/object.c:903
> > #9  0x00007f608eaf1fd3 in phys_section_destroy (mr=0x7f6091e74800) at
> git/qemu/exec.c:1154
> > #10 phys_sections_free (map=0x7f6090b72bb0) at git/qemu/exec.c:1163
> > #11 address_space_dispatch_free (d=0x7f6090b72b90) at
> git/qemu/exec.c:2514
> > #12 0x00007f608ee91ace in call_rcu_thread (opaque=<optimized out>) at
> util/rcu.c:272
> > #13 0x00007f6089b0ddc5 in start_thread () from /usr/lib64/libpthread.so.0
> > #14 0x00007f60882bf71d in clone () from /usr/lib64/libc.so.6
> >
> > Call Trace:
> > #0  0x00007fdccaeb9790 in ?? ()
> > #1  0x00007fdcd82d09fc in object_property_del_all (obj=0x7fdcdb8acf60) at
> qom/object.c:405
> > #2  object_finalize (data=0x7fdcdb8acf60) at qom/object.c:467
> > #3  object_unref (obj=obj@entry=0x7fdcdb8acf60) at qom/object.c:903
> > #4  0x00007fdcd8001fd3 in phys_section_destroy (mr=0x7fdcdb8acf60) at
> git/qemu/exec.c:1154
> > #5  phys_sections_free (map=0x7fdcdc86aa00) at git/qemu/exec.c:1163
> > #6  address_space_dispatch_free (d=0x7fdcdc86a9e0) at
> git/qemu/exec.c:2514
> > #7  0x00007fdcd83a1ace in call_rcu_thread (opaque=<optimized out>) at
> util/rcu.c:272
> > #8  0x00007fdcd301ddc5 in start_thread () from /usr/lib64/libpthread.so.0
> > #9  0x00007fdcd17cf71d in clone () from /usr/lib64/libc.so.6
> >
> >

Re: [Qemu-devel] [BUG] QEMU crashes with dpdk virtio pmd

[Paolo Bonzini]

On 25/04/2017 14:02, Jason Wang wrote:
> 
> Thanks a lot for the fix.
> 
> Two questions:
> 
> - If virtio_net_set_status() is the only function that may access tx_bh,
> it looks like setting tx_waiting to zero is sufficient?

I think clearing tx_bh is better anyway, as leaving a dangling pointer
is not very hygienic.

Paolo

> - Can you post a formal patch for this?