From: Christian Borntraeger <borntraeger@de.ibm.com>
To: Paolo Bonzini <pbonzini@redhat.com>,
tu bo <tubo@linux.vnet.ibm.com>, Fam Zheng <famz@redhat.com>,
qemu-devel@nongnu.org
Cc: Kevin Wolf <kwolf@redhat.com>,
cornelia.huck@de.ibm.com, Stefan Hajnoczi <stefanha@redhat.com>,
qemu-block@nongnu.org, "Michael S. Tsirkin" <mst@redhat.com>
Subject: Re: [Qemu-devel] [PATCH 0/4] Tweaks around virtio-blk start/stop
Date: Thu, 17 Mar 2016 16:15:46 +0100 [thread overview]
Message-ID: <56EACA22.2020505@de.ibm.com> (raw)
In-Reply-To: <56EAC706.2040006@redhat.com>
On 03/17/2016 04:02 PM, Paolo Bonzini wrote:
>
>
> On 17/03/2016 13:39, Christian Borntraeger wrote:
>> As an interesting side note, I updated my system from F20 to F23 some days ago
>> (after the initial report). While To Bo is still on a F20 system. I was not able
>> to reproduce the original crash on f23. but going back to F20 made this
>> problem re-appear.
>>
>> Stack trace of thread 26429:
>> #0 0x00000000802008aa tracked_request_begin (qemu-system-s390x)
>> #1 0x0000000080203f3c bdrv_co_do_preadv (qemu-system-s390x)
>> #2 0x000000008020567c bdrv_co_do_readv (qemu-system-s390x)
>> #3 0x000000008025d0f4 coroutine_trampoline (qemu-system-s390x)
>> #4 0x000003ff943d150a __makecontext_ret (libc.so.6)
>>
>> this is with patch 2-4 plus the removal of virtio_queue_host_notifier_read.
>>
>> Without removing virtio_queue_host_notifier_read, I get the same mutex lockup (as expected).
>>
>> Maybe we have two independent issues here and this is some old bug in glibc or
>> whatever?
>
> I'm happy to try and reproduce on x86 if you give me some instruction
> (RHEL7 should be close enough to Fedora 20).
>
> Can you add an assert in virtio_blk_handle_output to catch reentrancy, like
that was quick (let me know if I should recompile with debugging)
(gdb) thread apply all bt
Thread 5 (Thread 0x3ff7b8ff910 (LWP 236419)):
#0 0x000003ff7cdfcf56 in syscall () from /lib64/libc.so.6
#1 0x000000001022452e in futex_wait (val=<optimized out>, ev=<optimized out>) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:292
#2 qemu_event_wait (ev=ev@entry=0x1082b5c4 <rcu_call_ready_event>) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:399
#3 0x000000001023353a in call_rcu_thread (opaque=<optimized out>) at /home/cborntra/REPOS/qemu/util/rcu.c:250
#4 0x000003ff7cf084c6 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff7ce02ec2 in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x3ff78eca910 (LWP 236426)):
#0 0x000003ff7cdf819a in ioctl () from /lib64/libc.so.6
#1 0x000000001005ddf8 in kvm_vcpu_ioctl (cpu=cpu@entry=0x10c27d40, type=type@entry=44672) at /home/cborntra/REPOS/qemu/kvm-all.c:1984
#2 0x000000001005df1c in kvm_cpu_exec (cpu=cpu@entry=0x10c27d40) at /home/cborntra/REPOS/qemu/kvm-all.c:1834
#3 0x000000001004b1be in qemu_kvm_cpu_thread_fn (arg=0x10c27d40) at /home/cborntra/REPOS/qemu/cpus.c:1050
#4 0x000003ff7cf084c6 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff7ce02ec2 in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x3ff7e8dcbb0 (LWP 236395)):
#0 0x000003ff7cdf66e6 in ppoll () from /lib64/libc.so.6
#1 0x00000000101a5e08 in ppoll (__ss=0x0, __timeout=0x3ffd6afe8a0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
#2 qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=1034000000) at /home/cborntra/REPOS/qemu/qemu-timer.c:325
#3 0x00000000101a56f2 in os_host_main_loop_wait (timeout=1034000000) at /home/cborntra/REPOS/qemu/main-loop.c:251
#4 main_loop_wait (nonblocking=<optimized out>) at /home/cborntra/REPOS/qemu/main-loop.c:505
#5 0x00000000100136d6 in main_loop () at /home/cborntra/REPOS/qemu/vl.c:1933
#6 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /home/cborntra/REPOS/qemu/vl.c:4656
Thread 2 (Thread 0x3ff7b0ff910 (LWP 236421)):
#0 0x000003ff7cdf66e6 in ppoll () from /lib64/libc.so.6
#1 0x00000000101a5e28 in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
#2 qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:313
#3 0x00000000101a727c in aio_poll (ctx=0x10880560, blocking=<optimized out>) at /home/cborntra/REPOS/qemu/aio-posix.c:453
#4 0x00000000100d39f0 in iothread_run (opaque=0x10880020) at /home/cborntra/REPOS/qemu/iothread.c:46
#5 0x000003ff7cf084c6 in start_thread () from /lib64/libpthread.so.0
#6 0x000003ff7ce02ec2 in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x3ff57fff910 (LWP 236427)):
#0 0x000003ff7cd3b650 in raise () from /lib64/libc.so.6
#1 0x000003ff7cd3ced8 in abort () from /lib64/libc.so.6
#2 0x000003ff7cd33666 in __assert_fail_base () from /lib64/libc.so.6
#3 0x000003ff7cd336f4 in __assert_fail () from /lib64/libc.so.6
#4 0x000000001007a3c4 in virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /home/cborntra/REPOS/qemu/hw/block/virtio-blk.c:595
#5 0x000000001009390e in virtio_queue_notify_vq (vq=0x10d77c70) at /home/cborntra/REPOS/qemu/hw/virtio/virtio.c:1095
#6 0x0000000010095894 in virtio_queue_notify_vq (vq=<optimized out>) at /home/cborntra/REPOS/qemu/hw/virtio/virtio.c:1091
#7 virtio_queue_notify (vdev=<optimized out>, n=n@entry=0) at /home/cborntra/REPOS/qemu/hw/virtio/virtio.c:1101
#8 0x00000000100a17c8 in virtio_ccw_hcall_notify (args=<optimized out>) at /home/cborntra/REPOS/qemu/hw/s390x/s390-virtio-ccw.c:66
#9 0x000000001009c210 in s390_virtio_hypercall (env=env@entry=0x10c75aa0) at /home/cborntra/REPOS/qemu/hw/s390x/s390-virtio-hcall.c:35
#10 0x00000000100cb4e8 in handle_hypercall (run=<optimized out>, cpu=0x10c6d7d0) at /home/cborntra/REPOS/qemu/target-s390x/kvm.c:1283
#11 handle_diag (ipb=<optimized out>, run=0x3ff78680000, cpu=0x10c6d7d0) at /home/cborntra/REPOS/qemu/target-s390x/kvm.c:1352
#12 handle_instruction (run=0x3ff78680000, cpu=0x10c6d7d0) at /home/cborntra/REPOS/qemu/target-s390x/kvm.c:1799
#13 handle_intercept (cpu=0x10c6d7d0) at /home/cborntra/REPOS/qemu/target-s390x/kvm.c:1842
#14 kvm_arch_handle_exit (cs=cs@entry=0x10c6d7d0, run=run@entry=0x3ff78680000) at /home/cborntra/REPOS/qemu/target-s390x/kvm.c:2028
#15 0x000000001005df70 in kvm_cpu_exec (cpu=cpu@entry=0x10c6d7d0) at /home/cborntra/REPOS/qemu/kvm-all.c:1921
#16 0x000000001004b1be in qemu_kvm_cpu_thread_fn (arg=0x10c6d7d0) at /home/cborntra/REPOS/qemu/cpus.c:1050
#17 0x000003ff7cf084c6 in start_thread () from /lib64/libpthread.so.0
#18 0x000003ff7ce02ec2 in thread_start () from /lib64/libc.so.6
next prev parent reply other threads:[~2016-03-17 15:15 UTC|newest]
Thread overview: 70+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-03-16 10:10 [Qemu-devel] [PATCH 0/4] Tweaks around virtio-blk start/stop Fam Zheng
2016-03-16 10:10 ` [Qemu-devel] [PATCH 1/4] block: Use drained section in bdrv_set_aio_context Fam Zheng
2016-03-16 10:27 ` Paolo Bonzini
2016-03-16 10:51 ` Fam Zheng
2016-03-16 10:10 ` [Qemu-devel] [PATCH 2/4] block-backend: Introduce blk_drained_begin/end Fam Zheng
2016-03-16 10:10 ` [Qemu-devel] [PATCH 3/4] virtio-blk: Use blk_drained_begin/end around dataplane stop Fam Zheng
2016-03-16 10:10 ` [Qemu-devel] [PATCH 4/4] virtio-blk: Clean up start/stop with mutex and BH Fam Zheng
2016-03-17 15:00 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2016-03-17 15:07 ` Paolo Bonzini
2016-03-22 12:52 ` Fam Zheng
2016-03-22 18:05 ` Paolo Bonzini
2016-03-23 8:10 ` Cornelia Huck
2016-03-23 9:08 ` Paolo Bonzini
2016-03-23 9:12 ` Christian Borntraeger
2016-03-24 8:19 ` tu bo
2016-03-24 8:32 ` Cornelia Huck
2016-03-24 8:47 ` Cornelia Huck
2016-03-24 9:31 ` Cornelia Huck
2016-03-16 10:28 ` [Qemu-devel] [PATCH 0/4] Tweaks around virtio-blk start/stop Paolo Bonzini
2016-03-16 10:49 ` Christian Borntraeger
2016-03-16 11:09 ` Paolo Bonzini
2016-03-16 11:24 ` Christian Borntraeger
2016-03-16 12:55 ` Paolo Bonzini
2016-03-16 13:38 ` Christian Borntraeger
2016-03-16 13:45 ` Paolo Bonzini
2016-03-17 0:39 ` Fam Zheng
2016-03-17 11:03 ` tu bo
2016-03-21 10:57 ` Fam Zheng
2016-03-21 11:15 ` Cornelia Huck
2016-03-21 12:45 ` Fam Zheng
2016-03-21 13:02 ` Cornelia Huck
2016-03-21 23:45 ` Fam Zheng
2016-03-22 8:06 ` Cornelia Huck
2016-03-22 7:10 ` tu bo
2016-03-22 7:18 ` Fam Zheng
2016-03-22 9:07 ` Cornelia Huck
2016-03-22 9:46 ` Paolo Bonzini
2016-03-22 11:59 ` Cornelia Huck
2016-03-22 12:11 ` Paolo Bonzini
2016-03-22 12:54 ` Cornelia Huck
2016-03-17 12:22 ` tu bo
2016-03-17 12:39 ` Christian Borntraeger
2016-03-17 13:02 ` Cornelia Huck
2016-03-17 15:02 ` Paolo Bonzini
2016-03-17 15:07 ` Christian Borntraeger
2016-03-17 15:15 ` Christian Borntraeger [this message]
2016-03-17 15:16 ` Christian Borntraeger
2016-03-17 16:08 ` Christian Borntraeger
2016-03-18 15:03 ` Paolo Bonzini
2016-03-21 9:42 ` Fam Zheng
2016-03-21 11:10 ` Christian Borntraeger
2016-03-21 12:17 ` Cornelia Huck
2016-03-21 13:47 ` TU BO
2016-03-21 13:54 ` Paolo Bonzini
2016-03-21 14:19 ` Cornelia Huck
2016-03-22 0:31 ` Fam Zheng
2016-03-16 11:32 ` Cornelia Huck
2016-03-16 11:48 ` Paolo Bonzini
2016-03-16 11:56 ` Cornelia Huck
2016-03-16 11:59 ` Paolo Bonzini
2016-03-16 12:22 ` Cornelia Huck
2016-03-16 12:32 ` Paolo Bonzini
2016-03-16 12:42 ` Cornelia Huck
2016-03-16 12:49 ` Paolo Bonzini
2016-03-16 13:04 ` Cornelia Huck
2016-03-16 13:10 ` Paolo Bonzini
2016-03-16 13:14 ` Cornelia Huck
2016-03-16 13:15 ` Paolo Bonzini
2016-03-16 11:52 ` Cornelia Huck
2016-03-16 11:54 ` Paolo Bonzini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56EACA22.2020505@de.ibm.com \
--to=borntraeger@de.ibm.com \
--cc=cornelia.huck@de.ibm.com \
--cc=famz@redhat.com \
--cc=kwolf@redhat.com \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@redhat.com \
--cc=tubo@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).