From: Coiby Xu <coiby.xu@gmail.com>
To: Peter Maydell <peter.maydell@linaro.org>
Cc: Kevin Wolf <kwolf@redhat.com>,
QEMU Developers <qemu-devel@nongnu.org>,
Raphael Norwitz <raphael.norwitz@nutanix.com>
Subject: Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
Date: Sun, 11 Jul 2021 20:08:54 +0800 [thread overview]
Message-ID: <20210711120854.andcppzuxo6ztykd@Rk> (raw)
In-Reply-To: <CAFEAcA-L5kztvCiS-Y+_LDiaFgWzuCpPYN1-cHNY9Q8qpi538Q@mail.gmail.com>
Hi Peter,
On Sat, Jul 10, 2021 at 02:30:36PM +0100, Peter Maydell wrote:
>I've noticed recently that intermittently 'make check' will hang on
>my aarch32 test system (really an aarch64 box with an aarch32 chroot).
I have a newbie question. How do you do an aarch32 chroot on an aarch64
box? At least, this issue seems to be not reproducible on an aarch64 box
directly. I specifically ran the qos-test for 5 consecutive times and
each time the test could finish successfully,
$ MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} \
QTEST_QEMU_BINARY=build/i386-softmmu/qemu-system-i386 QTEST_QEMU_IMG=build/qemu-img \
QTEST_QEMU_STORAGE_DAEMON_BINARY=build/storage-daemon/qemu-storage-daemon \
build/tests/qtest/qos-test
>
>I think from grep that this must be the vhost-user-blk test.
>
>Here's the process tree:
>
>pmaydell 13126 0.0 0.0 8988 6416 ? S Jul09 0:01 make
>-C build/all-a32 check V=1 GCC_COLORS= -j9
>pmaydell 19632 0.0 0.0 4432 2096 ? S Jul09 0:00 \_
>bash -o pipefail -c echo 'MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((
>${RANDOM:-0} % 255 + 1))} QTEST_QEMU_IMG=./qemu-img
>G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
>QTEST_QEMU_BINARY=./qemu-system-i386
>QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
>tests/qtest/qos-test --tap -k' &&
>MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
>QTEST_QEMU_IMG=./qemu-img
>G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
>QTEST_QEMU_BINARY=./qemu-system-i386
>QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
>tests/qtest/qos-test --tap -k -m quick < /dev/null |
>./scripts/tap-driver.pl --test-name="qtest-i386/qos-test"
>pmaydell 19634 0.0 0.0 13608 3076 ? Sl Jul09 0:02
>\_ tests/qtest/qos-test --tap -k -m quick
>pmaydell 20679 0.0 0.0 109076 16100 ? Sl Jul09 0:00
>| \_ ./storage-daemon/qemu-storage-daemon --blockdev
>driver=file,node-name=disk0,filename=qtest.X7RL2X --export
>type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-19634-sock.9LJoHn,node-name=disk0,writable=on,num-queues=1
>pmaydell 20681 0.0 0.2 447828 46544 ? Sl Jul09 0:00
>| \_ ./qemu-system-i386 -qtest unix:/tmp/qtest-19634.sock -qtest-log
>/dev/null -chardev socket,path=/tmp/qtest-19634.qmp,id=char0 -mon
>chardev=char0,mode=control -display none -M pc -device
>vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
>memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
>-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-sock.9LJoHn
>-accel qtest
>pmaydell 19635 0.0 0.0 10256 7176 ? S Jul09 0:00
>\_ perl ./scripts/tap-driver.pl --test-name=qtest-i386/qos-test
>
>
>Backtrace from tests/qtest/qos-test (not as helpful as it could
>be since this is an optimized build):
>
>(gdb) thread apply all bt
>
>Thread 2 (Thread 0xf76ff240 (LWP 19636)):
>#0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
>#1 0x005206de in qemu_futex_wait (val=<optimised out>, f=<optimised
>out>) at /home/peter.maydell/qemu/include/qemu/futex.h:29
>#2 qemu_event_wait (ev=ev@entry=0x5816fc <rcu_call_ready_event>) at
>../../util/qemu-thread-posix.c:480
>#3 0x005469c2 in call_rcu_thread (opaque=<optimised out>) at
>../../util/rcu.c:258
>#4 0x0051fbc2 in qemu_thread_start (args=<optimised out>) at
>../../util/qemu-thread-posix.c:541
>#5 0xf785a614 in start_thread (arg=0xf6ce711c) at pthread_create.c:463
>#6 0xf77f57ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>from /lib/arm-linux-gnueabihf/libc.so.6
>Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
>Thread 1 (Thread 0xf7a04010 (LWP 19634)):
>#0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
>#1 0xf7861d8c in __libc_read (fd=12, buf=buf@entry=0xff9ce8e4,
>nbytes=nbytes@entry=1024) at ../sysdeps/unix/sysv/linux/read.c:27
>#2 0x004ebc5a in read (__nbytes=1024, __buf=0xff9ce8e4,
>__fd=<optimised out>) at
>/usr/include/arm-linux-gnueabihf/bits/unistd.h:44
>#3 qtest_client_socket_recv_line (s=0x1a46cb8) at
>../../tests/qtest/libqtest.c:494
>#4 0x004ebd4e in qtest_rsp_args (s=s@entry=0x1a46cb8,
>expected_args=expected_args@entry=1) at
>../../tests/qtest/libqtest.c:521
>#5 0x004ec1ee in qtest_query_target_endianness (s=0x1a46cb8) at
>../../tests/qtest/libqtest.c:570
>#6 0x004ec94a in qtest_init_without_qmp_handshake
>(extra_args=<optimised out>) at ../../tests/qtest/libqtest.c:332
>#7 0x004ecd7a in qtest_init (extra_args=<optimised out>) at
>../../tests/qtest/libqtest.c:339
>#8 0x004ded10 in qtest_start (
> args=0x1a63710 "-M pc -device
>vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
>memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
>-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
>../../tests/qtest/libqtest-single.h:29
>#9 restart_qemu_or_continue (
> path=0x1a63710 "-M pc -device
>vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
>memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
>-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
>../../tests/qtest/qos-test.c:105
>#10 run_one_test (arg=<optimised out>) at ../../tests/qtest/qos-test.c:178
>#11 0xf794ee74 in ?? () from /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
>Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
>
>Backtrace from qemu-system-i386:
>
>(gdb) thread apply all bt
>
>Thread 4 (Thread 0xdfd0cb90 (LWP 20734)):
>#0 0xf6f85206 in __libc_do_syscall () at
>../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
>#1 0xf6f93492 in __GI___sigtimedwait (set=<optimised out>,
> set@entry=0xdfd0c3c4, info=info@entry=0xdfd0c324,
>timeout=timeout@entry=0x0) at
>../sysdeps/unix/sysv/linux/sigtimedwait.c:42
>#2 0xf7073e6c in __sigwait (set=set@entry=0xdfd0c3c4,
>sig=sig@entry=0xdfd0c3c0) at ../sysdeps/unix/sysv/linux/sigwait.c:28
>#3 0x00c999a6 in dummy_cpu_thread_fn (arg=0x24d1168) at
>../../accel/dummy-cpus.c:46
>#4 0x00e18ba2 in qemu_thread_start (args=<optimised out>) at
>../../util/qemu-thread-posix.c:541
>#5 0xf706b614 in start_thread (arg=0xa8db8843) at pthread_create.c:463
>#6 0xf70067ec in () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>
>Thread 3 (Thread 0xe07fcb90 (LWP 20710)):
>#0 0xf6f85204 in __libc_do_syscall () at
>../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
>#1 0xf6fff948 in __GI___poll (fds=0x2409b90, nfds=3, timeout=-1) at
>../sysdeps/unix/sysv/linux/poll.c:29
>#2 0xf781692e in () at /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
>
>Thread 2 (Thread 0xf1201b90 (LWP 20703)):
>#0 0xf7074f04 in __libc_do_syscall () at
>../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
>#1 0xf7072ad2 in __lll_lock_wait (futex=futex@entry=0x14f2e68
><qemu_global_mutex>, private=0) at lowlevellock.c:43
>#2 0xf706d3e2 in __GI___pthread_mutex_lock
>(mutex=mutex@entry=0x14f2e68 <qemu_global_mutex>) at
>pthread_mutex_lock.c:78
>#3 0x00e18c9c in qemu_mutex_lock_impl (mutex=0x14f2e68
><qemu_global_mutex>, file=0x1355bd4 "../../util/rcu.c", line=266)
> at ../../util/qemu-thread-posix.c:79
>#4 0x00c1403a in qemu_mutex_lock_iothread_impl (file=0x1355bd4
>"../../util/rcu.c", line=line@entry=266) at ../../softmmu/cpus.c:493
>#5 0x00e15ad4 in call_rcu_thread (opaque=<optimised out>) at
>../../util/rcu.c:266
>#6 0x00e18ba2 in qemu_thread_start (args=<optimised out>) at
>../../util/qemu-thread-posix.c:541
>#7 0xf706b614 in start_thread (arg=0xa8db8843) at pthread_create.c:463
>#8 0xf70067ec in () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>
>Thread 1 (Thread 0xf12f9010 (LWP 20681)):
>#0 0xf6f85204 in __libc_do_syscall () at
>../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
>#1 0xf6fff948 in __GI___poll (fds=0x3248040, nfds=2, timeout=-1) at
>../sysdeps/unix/sysv/linux/poll.c:29
>#2 0xf781692e in () at /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
>
>Backtrace of qemu-storage-daemon:
>
>(gdb) thread apply all bt
>
>Thread 2 (Thread 0xf21ffdd0 (LWP 20722)):
>#0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
>#1 0x009a9986 in qemu_futex_wait (val=<optimised out>, f=<optimised
>out>) at /home/peter.maydell/qemu/include/qemu/futex.h:29
>#2 qemu_event_wait (ev=ev@entry=0xa64958 <rcu_call_ready_event>) at
>../../util/qemu-thread-posix.c:480
>#3 0x009a5d66 in call_rcu_thread (opaque=<optimised out>) at
>../../util/rcu.c:258
>#4 0x009a8e6a in qemu_thread_start (args=<optimised out>) at
>../../util/qemu-thread-posix.c:541
>#5 0xf79dd614 in start_thread (arg=0xa310c11f) at pthread_create.c:463
>#6 0xf79787ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>from /lib/arm-linux-gnueabihf/libc.so.6
>Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
>Thread 1 (Thread 0xf22f7010 (LWP 20679)):
>#0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
>#1 0xf7971a04 in __GI_ppoll (fds=0x117c578, nfds=5,
>timeout=<optimised out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
> at ../sysdeps/unix/sysv/linux/ppoll.c:39
>#2 0x009a68fc in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimised
>out>, __fds=<optimised out>)
> at /usr/include/arm-linux-gnueabihf/bits/poll2.h:77
>#3 qemu_poll_ns (fds=<optimised out>, nfds=<optimised out>,
>timeout=<optimised out>) at ../../util/qemu-timer.c:336
>#4 0x009c0222 in os_host_main_loop_wait (timeout=-1) at
>../../util/main-loop.c:250
>#5 main_loop_wait (nonblocking=<optimised out>) at ../../util/main-loop.c:531
>#6 0x008ee92a in main (argc=5, argv=0xffdce5c4) at
>../../storage-daemon/qemu-storage-daemon.c:345
>
>
>Any ideas ?
>
>thanks
>-- PMM
--
Best regards,
Coiby
next prev parent reply other threads:[~2021-07-11 12:11 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-10 13:30 intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host Peter Maydell
2021-07-11 12:08 ` Coiby Xu [this message]
2021-07-11 12:16 ` Peter Maydell
2021-07-11 13:23 ` Richard Henderson
2021-07-11 14:21 ` Peter Maydell
2021-07-11 14:27 ` Richard Henderson
2021-07-11 22:19 ` Coiby Xu
2021-07-11 15:53 ` Peter Maydell
2021-07-11 22:20 ` Coiby Xu
2021-07-11 22:50 ` Coiby Xu
2021-07-12 9:39 ` Peter Maydell
2021-07-16 12:30 ` Coiby Xu
2021-07-16 14:31 ` Kevin Wolf
2021-07-22 8:45 ` Claudio Fontana
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210711120854.andcppzuxo6ztykd@Rk \
--to=coiby.xu@gmail.com \
--cc=kwolf@redhat.com \
--cc=peter.maydell@linaro.org \
--cc=qemu-devel@nongnu.org \
--cc=raphael.norwitz@nutanix.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).