* Block Migration Assertion in qemu-kvm 1.2.0 @ 2012-09-16 10:13 ` Peter Lieven 0 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-16 10:13 UTC (permalink / raw) To: qemu-devel@nongnu.org, kvm@vger.kernel.org Hi, when trying to block migrate a VM from one node to another, the source VM crashed with the following assertion: block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. Is this sth already addresses/known? Thanks, Peter ^ permalink raw reply [flat|nested] 17+ messages in thread
* [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 @ 2012-09-16 10:13 ` Peter Lieven 0 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-16 10:13 UTC (permalink / raw) To: qemu-devel@nongnu.org, kvm@vger.kernel.org Hi, when trying to block migrate a VM from one node to another, the source VM crashed with the following assertion: block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. Is this sth already addresses/known? Thanks, Peter ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 2012-09-16 10:13 ` [Qemu-devel] " Peter Lieven @ 2012-09-17 8:41 ` Kevin Wolf -1 siblings, 0 replies; 17+ messages in thread From: Kevin Wolf @ 2012-09-17 8:41 UTC (permalink / raw) To: Peter Lieven; +Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, Paolo Bonzini Am 16.09.2012 12:13, schrieb Peter Lieven: > Hi, > > when trying to block migrate a VM from one node to another, the source > VM crashed with the following assertion: > block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. > > Is this sth already addresses/known? Not that I'm aware of, at least. Block migration doesn't seem to check whether the device is already in use, maybe this is the problem. Not sure why it would be in use, though, and in my quick test it didn't crash. So we need some more information: What's you command line, did you do anything specific in the monitor with block devices, what does the stacktrace look like, etc.? Kevin ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 @ 2012-09-17 8:41 ` Kevin Wolf 0 siblings, 0 replies; 17+ messages in thread From: Kevin Wolf @ 2012-09-17 8:41 UTC (permalink / raw) To: Peter Lieven; +Cc: Paolo Bonzini, qemu-devel@nongnu.org, kvm@vger.kernel.org Am 16.09.2012 12:13, schrieb Peter Lieven: > Hi, > > when trying to block migrate a VM from one node to another, the source > VM crashed with the following assertion: > block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. > > Is this sth already addresses/known? Not that I'm aware of, at least. Block migration doesn't seem to check whether the device is already in use, maybe this is the problem. Not sure why it would be in use, though, and in my quick test it didn't crash. So we need some more information: What's you command line, did you do anything specific in the monitor with block devices, what does the stacktrace look like, etc.? Kevin ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 2012-09-17 8:41 ` Kevin Wolf @ 2012-09-17 14:35 ` Peter Lieven -1 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-17 14:35 UTC (permalink / raw) To: Kevin Wolf; +Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, Paolo Bonzini On 09/17/12 10:41, Kevin Wolf wrote: > Am 16.09.2012 12:13, schrieb Peter Lieven: >> Hi, >> >> when trying to block migrate a VM from one node to another, the source >> VM crashed with the following assertion: >> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >> >> Is this sth already addresses/known? > Not that I'm aware of, at least. > > Block migration doesn't seem to check whether the device is already in > use, maybe this is the problem. Not sure why it would be in use, though, > and in my quick test it didn't crash. It seems that it only happens if a vServer that has been block migrated earlier is block migrated the next time. > So we need some more information: What's you command line, did you do > anything specific in the monitor with block devices, what does the > stacktrace look like, etc.? Here is my cmdline: /usr/bin/qemu-kvm-1.2.0 -net tap,vlan=164,script=no,downscript=no,ifname=tap0 -net nic,vlan =164,model=e1000,macaddr=52:54:00:ff:01:19 -drive format=host_device,file=/dev/7cf58855099771c2/lieven-storage-migration-t-hd0,if=virtio,cache=none,aio=nat ive -m 2048 -smp 2,sockets=1,cores=2,threads=1 -monitor tcp:0:4001,server,nowait -vnc :1 -qmp tcp:0:3001,server,nowait -name 'lieven-storage-migration-test' -boot or der=dc,menu=off -k de -incoming tcp:172.21.55.34:5001 -pidfile /var/run/qemu/vm-254.pid -mem-path /hugepages -mem-prealloc -rtc base=utc -usb -usbdevice tablet -no -hpet -vga cirrus -cpu host,+x2apic,model_id='Intel(R) Xeon(R) CPU L5640 @ 2.27GHz',-tsc I have seen other errors as well in the meantime: block-migration.c:471: flush_blks: Assertion `block_mig_state.read_done >= 0' failed. qemu-kvm-1.2.0[27851]: segfault at 7f00746e78d7 ip 00007f67eca6226d sp 00007fff56ae3340 error 4 in qemu-system-x86_64[7f67ec9e9000+418000] I will now try to catch the situation in the debugger. Thanks, Peter > Kevin ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 @ 2012-09-17 14:35 ` Peter Lieven 0 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-17 14:35 UTC (permalink / raw) To: Kevin Wolf; +Cc: Paolo Bonzini, qemu-devel@nongnu.org, kvm@vger.kernel.org On 09/17/12 10:41, Kevin Wolf wrote: > Am 16.09.2012 12:13, schrieb Peter Lieven: >> Hi, >> >> when trying to block migrate a VM from one node to another, the source >> VM crashed with the following assertion: >> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >> >> Is this sth already addresses/known? > Not that I'm aware of, at least. > > Block migration doesn't seem to check whether the device is already in > use, maybe this is the problem. Not sure why it would be in use, though, > and in my quick test it didn't crash. It seems that it only happens if a vServer that has been block migrated earlier is block migrated the next time. > So we need some more information: What's you command line, did you do > anything specific in the monitor with block devices, what does the > stacktrace look like, etc.? Here is my cmdline: /usr/bin/qemu-kvm-1.2.0 -net tap,vlan=164,script=no,downscript=no,ifname=tap0 -net nic,vlan =164,model=e1000,macaddr=52:54:00:ff:01:19 -drive format=host_device,file=/dev/7cf58855099771c2/lieven-storage-migration-t-hd0,if=virtio,cache=none,aio=nat ive -m 2048 -smp 2,sockets=1,cores=2,threads=1 -monitor tcp:0:4001,server,nowait -vnc :1 -qmp tcp:0:3001,server,nowait -name 'lieven-storage-migration-test' -boot or der=dc,menu=off -k de -incoming tcp:172.21.55.34:5001 -pidfile /var/run/qemu/vm-254.pid -mem-path /hugepages -mem-prealloc -rtc base=utc -usb -usbdevice tablet -no -hpet -vga cirrus -cpu host,+x2apic,model_id='Intel(R) Xeon(R) CPU L5640 @ 2.27GHz',-tsc I have seen other errors as well in the meantime: block-migration.c:471: flush_blks: Assertion `block_mig_state.read_done >= 0' failed. qemu-kvm-1.2.0[27851]: segfault at 7f00746e78d7 ip 00007f67eca6226d sp 00007fff56ae3340 error 4 in qemu-system-x86_64[7f67ec9e9000+418000] I will now try to catch the situation in the debugger. Thanks, Peter > Kevin ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 2012-09-17 8:41 ` Kevin Wolf @ 2012-09-17 20:12 ` Peter Lieven -1 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-17 20:12 UTC (permalink / raw) To: Kevin Wolf; +Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, Paolo Bonzini On 09/17/12 10:41, Kevin Wolf wrote: > Am 16.09.2012 12:13, schrieb Peter Lieven: >> Hi, >> >> when trying to block migrate a VM from one node to another, the source >> VM crashed with the following assertion: >> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >> >> Is this sth already addresses/known? > Not that I'm aware of, at least. > > Block migration doesn't seem to check whether the device is already in > use, maybe this is the problem. Not sure why it would be in use, though, > and in my quick test it didn't crash. > > So we need some more information: What's you command line, did you do > anything specific in the monitor with block devices, what does the > stacktrace look like, etc.? kevin, it seems that i can very easily force a crash if I cancel a running block migration. here is a stack trace. i was not yet able to reproduce the asserts. please note that all the new threads where created at the moment i cancelled the migration. GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://bugs.launchpad.net/gdb-linaro/>... Reading symbols from /usr/bin/nice...(no debugging symbols found)...done. Starting program: /usr/bin/nice -n 1 /usr/bin/qemu-kvm-1.2.0 -net tap,vlan=164,script=no,downscript=no,ifname=tap0 -net nic,vlan=164,model=e1000,macaddr=52:54:00:ff:01:19 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-fbef4e007-f0f0017cdc25055a-lieven-storage-migration-t-hd0,if=virtio,cache=none,aio=native -m 2048 -smp 2,sockets=1,cores=2,threads=1 -monitor tcp:0:4001,server,nowait -vnc :1 -qmp tcp:0:3001,server,nowait -name lieven-storage-migration-test -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-254.pid -mem-path /hugepages -mem-prealloc -cpu host,+x2apic,model_id=Intel\(R\)\ Xeon\(R\)\ CPU\ \ \ \ \ \ \ \ \ \ \ L5640\ \ @\ 2.27GHz,-tsc -rtc base=utc -usb -usbdevice tablet -no-hpet -vga cirrus process 29659 is executing new program: /usr/local/qemu-kvm-1.2.0/bin/qemu-system-x86_64 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 165 block-migration.c: No such file or directory. [New Thread 0x7ffff6163700 (LWP 29663)] [New Thread 0x7ffff5962700 (LWP 29664)] [New Thread 0x7fffefe97700 (LWP 29673)] [New Thread 0x7fffef696700 (LWP 29674)] [Thread 0x7fffefe97700 (LWP 29673) exited] [New Thread 0x7fffefe97700 (LWP 29825)] [Thread 0x7fffefe97700 (LWP 29825) exited] [New Thread 0x7fffefe97700 (LWP 29969)] [Thread 0x7fffefe97700 (LWP 29969) exited] [New Thread 0x7fffefe97700 (LWP 30091)] [New Thread 0x7fffee15c700 (LWP 30092)] [New Thread 0x7fffecc47700 (LWP 30093)] [New Thread 0x7fffdbfff700 (LWP 30094)] [New Thread 0x7fffe33f3700 (LWP 30095)] [New Thread 0x7fffe20e0700 (LWP 30096)] [New Thread 0x7fffe18df700 (LWP 30097)] [New Thread 0x7fffe10de700 (LWP 30098)] [New Thread 0x7fffe08dd700 (LWP 30099)] [New Thread 0x7fffdb7fe700 (LWP 30100)] [New Thread 0x7fffdaffd700 (LWP 30101)] [New Thread 0x7fffda7fc700 (LWP 30102)] [New Thread 0x7fffd9ffb700 (LWP 30103)] [New Thread 0x7fffd97fa700 (LWP 30104)] [New Thread 0x7fffd8ff9700 (LWP 30105)] [New Thread 0x7fffa7fff700 (LWP 30106)] [New Thread 0x7fffa77fe700 (LWP 30107)] [New Thread 0x7fffa6ffd700 (LWP 30108)] [New Thread 0x7fffed853700 (LWP 30109)] [New Thread 0x7fffa67fc700 (LWP 30110)] [New Thread 0x7fffa5ffb700 (LWP 30111)] [New Thread 0x7fffa57fa700 (LWP 30112)] [New Thread 0x7fffa4ff9700 (LWP 30113)] [New Thread 0x7fff83fff700 (LWP 30114)] [New Thread 0x7fff837fe700 (LWP 30115)] [New Thread 0x7fff82ffd700 (LWP 30116)] [New Thread 0x7fff827fc700 (LWP 30117)] [New Thread 0x7fff81ffb700 (LWP 30118)] [New Thread 0x7fff817fa700 (LWP 30119)] [New Thread 0x7fff80ff9700 (LWP 30120)] [New Thread 0x7fff5ffff700 (LWP 30121)] [New Thread 0x7fff5f7fe700 (LWP 30122)] [New Thread 0x7fff5effd700 (LWP 30123)] [New Thread 0x7fff5e7fc700 (LWP 30124)] [New Thread 0x7fff5dffb700 (LWP 30125)] [New Thread 0x7fff5d7fa700 (LWP 30126)] [New Thread 0x7fff5cff9700 (LWP 30127)] [New Thread 0x7fff3ffff700 (LWP 30128)] [New Thread 0x7fff3f7fe700 (LWP 30129)] [New Thread 0x7fff3effd700 (LWP 30130)] [New Thread 0x7fff3e7fc700 (LWP 30131)] [New Thread 0x7fff3dffb700 (LWP 30132)] [New Thread 0x7fff3d7fa700 (LWP 30133)] [New Thread 0x7fff3cff9700 (LWP 30134)] [New Thread 0x7fff1ffff700 (LWP 30135)] [New Thread 0x7fff1f7fe700 (LWP 30136)] [New Thread 0x7fff1effd700 (LWP 30137)] [New Thread 0x7fff1e7fc700 (LWP 30138)] [New Thread 0x7fff1dffb700 (LWP 30139)] [New Thread 0x7fff1d7fa700 (LWP 30140)] [New Thread 0x7fff1cff9700 (LWP 30141)] [New Thread 0x7ffefffff700 (LWP 30142)] [New Thread 0x7ffeff7fe700 (LWP 30143)] [New Thread 0x7ffefeffd700 (LWP 30144)] [New Thread 0x7ffefe7fc700 (LWP 30145)] [New Thread 0x7ffefdffb700 (LWP 30146)] [New Thread 0x7ffefd7fa700 (LWP 30147)] [New Thread 0x7ffefcff9700 (LWP 30148)] [New Thread 0x7ffedffff700 (LWP 30149)] [New Thread 0x7ffedf7fe700 (LWP 30150)] [New Thread 0x7ffedeffd700 (LWP 30151)] [New Thread 0x7ffede7fc700 (LWP 30152)] [New Thread 0x7ffeddffb700 (LWP 30153)] [New Thread 0x7ffedd7fa700 (LWP 30154)] Program received signal SIGSEGV, Segmentation fault. 0x00005555555cd5f6 in bmds_set_aio_inflight (bmds=0x555556568280, sector_num=2396160, nb_sectors=2048, set=0) at block-migration.c:165 (gdb) \a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\athread apply all bt full Thread 71 (Thread 0x7ffedd7fa700 (LWP 30154)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffedd7f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 65877} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 70 (Thread 0x7ffeddffb700 (LWP 30153)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffeddffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 627866} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 69 (Thread 0x7ffede7fc700 (LWP 30152)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffede7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 759753} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 68 (Thread 0x7ffedeffd700 (LWP 30151)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffedeffceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 644535} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 67 (Thread 0x7ffedf7fe700 (LWP 30150)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffedf7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 712004} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 66 (Thread 0x7ffedffff700 (LWP 30149)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffedfffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 743448} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 65 (Thread 0x7ffefcff9700 (LWP 30148)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffefcff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555a4074a0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 754495} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 64 (Thread 0x7ffefd7fa700 (LWP 30147)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffefd7f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 762992} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 63 (Thread 0x7ffefdffb700 (LWP 30146)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffefdffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 28676} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 62 (Thread 0x7ffefe7fc700 (LWP 30145)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffefe7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555a4074a0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 809680} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 61 (Thread 0x7ffefeffd700 (LWP 30144)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffefeffceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 810945} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 60 (Thread 0x7ffeff7fe700 (LWP 30143)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffeff7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 33778} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 59 (Thread 0x7ffefffff700 (LWP 30142)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffeffffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 98618} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 58 (Thread 0x7fff1cff9700 (LWP 30141)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1cff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 816786} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 57 (Thread 0x7fff1d7fa700 (LWP 30140)): #0 0x00007ffff6971653 in pread64 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573b0dd in handle_aiocb_rw_linear (aiocb=0x55555bc7eb20, buf=0x7fff04000c00 "\326\246\002") at posix-aio-compat.c:224 offset = 0 len = 1048576 #2 0x000055555573b2c4 in handle_aiocb_rw (aiocb=0x55555bc7eb20) at posix-aio-compat.c:293 nbytes = 93825003889184 buf = 0x7fff04000c00 "\326\246\002" #3 0x000055555573b54c in aio_thread (unused=0x0) at posix-aio-compat.c:351 aiocb = 0x55555bc7eb20 ret = 0 tv = {tv_sec = 1347912395, tv_usec = 983439} ts = {tv_sec = 1347912405, tv_nsec = 0} #4 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #5 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #6 0x0000000000000000 in ?? () No symbol table info available. Thread 56 (Thread 0x7fff1dffb700 (LWP 30139)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1dffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7f150 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 32930} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 55 (Thread 0x7fff1e7fc700 (LWP 30138)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1e7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 816746} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 54 (Thread 0x7fff1effd700 (LWP 30137)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1effceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 42141} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 53 (Thread 0x7fff1f7fe700 (LWP 30136)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1f7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 121256} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 52 (Thread 0x7fff1ffff700 (LWP 30135)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1fffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 48151} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 51 (Thread 0x7fff3cff9700 (LWP 30134)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3cff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 207541} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 50 (Thread 0x7fff3d7fa700 (LWP 30133)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3d7f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 73822} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 49 (Thread 0x7fff3dffb700 (LWP 30132)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3dffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 128860} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 48 (Thread 0x7fff3e7fc700 (LWP 30131)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3e7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 175835} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 47 (Thread 0x7fff3effd700 (LWP 30130)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3effceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 254947} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 46 (Thread 0x7fff3f7fe700 (LWP 30129)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3f7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 195999} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 45 (Thread 0x7fff3ffff700 (LWP 30128)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3fffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 239779} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 44 (Thread 0x7fff5cff9700 (LWP 30127)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5cff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 180411} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 43 (Thread 0x7fff5d7fa700 (LWP 30126)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5d7f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 173028} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 42 (Thread 0x7fff5dffb700 (LWP 30125)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5dffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 325936} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 41 (Thread 0x7fff5e7fc700 (LWP 30124)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5e7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 185622} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 40 (Thread 0x7fff5effd700 (LWP 30123)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5effceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 221321} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 39 (Thread 0x7fff5f7fe700 (LWP 30122)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5f7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 258187} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 38 (Thread 0x7fff5ffff700 (LWP 30121)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5fffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 346934} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 37 (Thread 0x7fff80ff9700 (LWP 30120)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff80ff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 282933} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 36 (Thread 0x7fff817fa700 (LWP 30119)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff817f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 330640} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 35 (Thread 0x7fff81ffb700 (LWP 30118)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff81ffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 330090} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 34 (Thread 0x7fff827fc700 (LWP 30117)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff827fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7f150 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 373531} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 33 (Thread 0x7fff82ffd700 (LWP 30116)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff82ffceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 324347} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 32 (Thread 0x7fff837fe700 (LWP 30115)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff837fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 362610} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 31 (Thread 0x7fff83fff700 (LWP 30114)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff83ffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 415042} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 30 (Thread 0x7fffa4ff9700 (LWP 30113)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa4ff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7f150 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 342087} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 29 (Thread 0x7fffa57fa700 (LWP 30112)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa57f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7f150 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 518047} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 28 (Thread 0x7fffa5ffb700 (LWP 30111)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa5ffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 514193} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 27 (Thread 0x7fffa67fc700 (LWP 30110)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa67fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 526408} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 26 (Thread 0x7fffed853700 (LWP 30109)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffed852eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 382861} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 25 (Thread 0x7fffa6ffd700 (LWP 30108)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa6ffceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 398159} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 24 (Thread 0x7fffa77fe700 (LWP 30107)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa77fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 519253} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 23 (Thread 0x7fffa7fff700 (LWP 30106)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa7ffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 624913} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 22 (Thread 0x7fffd8ff9700 (LWP 30105)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffd8ff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 560397} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 21 (Thread 0x7fffd97fa700 (LWP 30104)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffd97f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 522208} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 20 (Thread 0x7fffd9ffb700 (LWP 30103)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffd9ffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7eb20 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 531203} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 19 (Thread 0x7fffda7fc700 (LWP 30102)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffda7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 542959} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 18 (Thread 0x7fffdaffd700 (LWP 30101)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffdaffceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555a4074a0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 534592} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 17 (Thread 0x7fffdb7fe700 (LWP 30100)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffdb7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 536674} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 16 (Thread 0x7fffe08dd700 (LWP 30099)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffe08dceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555a4074a0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 629188} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 15 (Thread 0x7fffe10de700 (LWP 30098)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffe10ddeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 40152} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 14 (Thread 0x7fffe18df700 (LWP 30097)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffe18deeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 650586} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 13 (Thread 0x7fffe20e0700 (LWP 30096)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffe20dfeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 565139} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 12 (Thread 0x7fffe33f3700 (LWP 30095)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffe33f2eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 629929} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 11 (Thread 0x7fffdbfff700 (LWP 30094)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffdbffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 592997} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 10 (Thread 0x7fffecc47700 (LWP 30093)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffecc46eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 691042} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 9 (Thread 0x7fffee15c700 (LWP 30092)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffee15beb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555a4074a0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 710341} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 8 (Thread 0x7fffefe97700 (LWP 30091)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffefe96eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 707609} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 5 (Thread 0x7fffef696700 (LWP 29674)): #0 0x00007ffff696dd84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555575f7ae in qemu_cond_wait (cond=0x5555565b6c90, mutex=0x5555565b6cc0) at qemu-thread-posix.c:113 err = 0 __func__ = "qemu_cond_wait" #2 0x0000555555797c69 in vnc_worker_thread_loop (queue=0x5555565b6c90) at ui/vnc-jobs.c:222 job = 0x5555565a8980 entry = 0x0 tmp = 0x0 vs = {csock = -1, ds = 0x55555653faf0, dirty = {{0, 0, 0} <repeats 2048 times>}, lossy_rect = 0x55555bc7ebb0, vd = 0x7fffeffba010, need_update = 0, force_update = 0, features = 0, absolute = 0, last_x = 0, last_y = 0, client_width = 0, client_height = 0, share_mode = 0, vnc_encoding = 0, major = 0, minor = 0, auth = 0, challenge = '\000' <repeats 15 times>, info = 0x0, output = { capacity = 3149825, offset = 3145744, buffer = 0x7fffee360010 ""}, input = {capacity = 0, offset = 0, buffer = 0x0}, write_pixels = 0x55555579a63d <vnc_write_pixels_copy>, clientds = { flags = 0 '\000', width = 1024, height = 768, linesize = 4096, data = 0x7fffee967010 "\032\023*", pf = {bits_per_pixel = 32 ' ', bytes_per_pixel = 4 '\004', depth = 24 '\030', rmask = 16711680, gmask = 65280, bmask = 255, amask = 0, rshift = 16 '\020', gshift = 8 '\b', bshift = 0 '\000', ashift = 24 '\030', rmax = 255 '\377', gmax = 255 '\377', bmax = 255 '\377', amax = 255 '\377', rbits = 8 '\b', gbits = 8 '\b', bbits = 8 '\b', abits = 8 '\b'}}, audio_cap = 0x0, as = { freq = 0, nchannels = 0, fmt = AUD_FMT_U8, endianness = 0}, read_handler = 0, read_handler_expect = 0, modifiers_state = '\000' <repeats 255 times>, led = 0x0, abort = false, output_mutex = {lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, bh = 0x0, jobs_buffer = {capacity = 0, offset = 0, buffer = 0x0}, tight = { type = 0, quality = 255 '\377', compression = 9 '\t', pixel24 = 0 '\000', tight = {capacity = 0, offset = 0, buffer = 0x0}, tmp = {capacity = 0, offset = 0, buffer = 0x0}, zlib = {capacity = 0, offset = 0, buffer = 0x0}, gradient = { capacity = 0, offset = 0, buffer = 0x0}, levels = {0, 0, 0, 0}, stream = {{next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, { next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}}}, zlib = {zlib = { capacity = 0, offset = 0, buffer = 0x0}, tmp = {capacity = 0, offset = 0, buffer = 0x0}, stream = {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, level = 0}, hextile = { send_tile = 0x5555557849a0 <send_hextile_tile_32>}, zrle = { type = 0, fb = {capacity = 0, offset = 0, buffer = 0x0}, zrle = { capacity = 0, offset = 0, buffer = 0x0}, tmp = {capacity = 0, offset = 0, buffer = 0x0}, zlib = {capacity = 0, offset = 0, buffer = 0x0}, stream = {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, palette = {pool = {{ idx = 0, color = 0, next = {le_next = 0x0, le_prev = 0x0}} <repeats 256 times>}, size = 0, max = 0, bpp = 0, table = {{lh_first = 0x0} <repeats 256 times>}}}, zywrle = {buf = {0 <repeats 4090 times>, 128, 0, -157897112, 32767, 0, 0}}, mouse_mode_notifier = {notify = 0, node = { le_next = 0x0, le_prev = 0x55555575f948}}, next = { tqe_next = 0x0, tqe_prev = 0x5555565b6ce8}} n_rectangles = 1 saved_offset = 2 #3 0x0000555555798192 in vnc_worker_thread (arg=0x5555565b6c90) at ui/vnc-jobs.c:318 queue = 0x5555565b6c90 #4 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #5 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #6 0x0000000000000000 in ?? () No symbol table info available. Thread 3 (Thread 0x7ffff5962700 (LWP 29664)): #0 0x00007ffff668fd27 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x000055555580ebfa in kvm_vcpu_ioctl (env=0x555556543470, type=44672) at /usr/src/qemu-kvm-1.2.0/kvm-all.c:1693 ret = -321675584 arg = 0x0 ap = {{gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x7ffff5961e70, reg_save_area = 0x7ffff5961db0}} #2 0x000055555580e653 in kvm_cpu_exec (env=0x555556543470) at /usr/src/qemu-kvm-1.2.0/kvm-all.c:1580 run = 0x7ffff7fc3000 ret = 0 run_ret = 0 #3 0x00005555557a715b in qemu_kvm_cpu_thread_fn (arg=0x555556543470) at /usr/src/qemu-kvm-1.2.0/cpus.c:757 env = 0x555556543470 cpu = 0x555556543430 r = 65536 #4 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #5 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #6 0x0000000000000000 in ?? () No symbol table info available. Thread 2 (Thread 0x7ffff6163700 (LWP 29663)): #0 0x00007ffff668fd27 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x000055555580ebfa in kvm_vcpu_ioctl (env=0x5555565267a0, type=44672) at /usr/src/qemu-kvm-1.2.0/kvm-all.c:1693 ret = 32767 arg = 0x0 ap = {{gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x7ffff6162e70, reg_save_area = 0x7ffff6162db0}} #2 0x000055555580e653 in kvm_cpu_exec (env=0x5555565267a0) at /usr/src/qemu-kvm-1.2.0/kvm-all.c:1580 run = 0x7ffff7fc6000 ret = 0 run_ret = 0 #3 0x00005555557a715b in qemu_kvm_cpu_thread_fn (arg=0x5555565267a0) at /usr/src/qemu-kvm-1.2.0/cpus.c:757 env = 0x5555565267a0 cpu = 0x555556526760 r = 65536 #4 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #5 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #6 0x0000000000000000 in ?? () No symbol table info available. Thread 1 (Thread 0x7ffff7fef700 (LWP 29659)): #0 0x00005555555cd5f6 in bmds_set_aio_inflight (bmds=0x555556568280, sector_num=2396160, nb_sectors=2048, set=0) at block-migration.c:165 start = 1170 end = 1170 val = 93824994379674 idx = 18 bit = 18 #1 0x00005555555cd7dc in blk_mig_read_cb (opaque=0x55555868e510, ret=0) at block-migration.c:199 curr_time = 221018445336304 blk = 0x55555868e510 __PRETTY_FUNCTION__ = "blk_mig_read_cb" #2 0x00005555555d755a in bdrv_co_em_bh (opaque=0x5555565fee50) at block.c:3320 acb = 0x5555565fee50 #3 0x00005555555be8f7 in qemu_bh_poll () at async.c:71 bh = 0x5555565b6ea0 bhp = 0x7fffffffe170 next = 0x5555565a8980 ret = 1 nesting = 1 #4 0x00005555557280e6 in main_loop_wait (nonblocking=0) at main-loop.c:506 ret = 1 timeout = 4294967295 #5 0x000055555571ff9e in main_loop () at /usr/src/qemu-kvm-1.2.0/vl.c:1643 nonblocking = false last_io = 1 #6 0x0000555555726b93 in main (argc=38, argv=0x7fffffffe5f8, envp=0x7fffffffe730) at /usr/src/qemu-kvm-1.2.0/vl.c:3790 i = 64 snapshot = 0 linux_boot = 0 icount_option = 0x0 initrd_filename = 0x0 kernel_filename = 0x0 kernel_cmdline = 0x5555558d032a "" boot_devices = "dc", '\000' <repeats 30 times> ds = 0x55555653faf0 dcl = 0x0 cyls = 0 heads = 0 secs = 0 translation = 0 hda_opts = 0x0 opts = 0x55555650a120 machine_opts = 0x55555650a920 olist = 0x5780f65ad2e0 optind = 38 optarg = 0x7fffffffebd8 "cirrus" loadvm = 0x0 machine = 0x555555c615c0 cpu_model = 0x7fffffffeb5a "host,+x2apic,model_id=Intel(R) Xeon(R) CPU", ' ' <repeats 11 times>, "L5640 @ 2.27GHz,-tsc" vga_model = 0x7fffffffebd8 "cirrus" pid_file = 0x7fffffffeb19 "/var/run/qemu/vm-254.pid" incoming = 0x0 show_vnc_port = 0 defconfig = true userconfig = true log_mask = 0x0 log_file = 0x0 mem_trace = {malloc = 0x5555557231fa <malloc_and_trace>, realloc = 0x555555723252 <realloc_and_trace>, free = 0x5555557232b9 <free_and_trace>, calloc = 0, try_malloc = 0, try_realloc = 0} trace_events = 0x0 trace_file = 0x0 (gdb) q > Kevin ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 @ 2012-09-17 20:12 ` Peter Lieven 0 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-17 20:12 UTC (permalink / raw) To: Kevin Wolf; +Cc: Paolo Bonzini, qemu-devel@nongnu.org, kvm@vger.kernel.org On 09/17/12 10:41, Kevin Wolf wrote: > Am 16.09.2012 12:13, schrieb Peter Lieven: >> Hi, >> >> when trying to block migrate a VM from one node to another, the source >> VM crashed with the following assertion: >> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >> >> Is this sth already addresses/known? > Not that I'm aware of, at least. > > Block migration doesn't seem to check whether the device is already in > use, maybe this is the problem. Not sure why it would be in use, though, > and in my quick test it didn't crash. > > So we need some more information: What's you command line, did you do > anything specific in the monitor with block devices, what does the > stacktrace look like, etc.? kevin, it seems that i can very easily force a crash if I cancel a running block migration. here is a stack trace. i was not yet able to reproduce the asserts. please note that all the new threads where created at the moment i cancelled the migration. GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://bugs.launchpad.net/gdb-linaro/>... Reading symbols from /usr/bin/nice...(no debugging symbols found)...done. Starting program: /usr/bin/nice -n 1 /usr/bin/qemu-kvm-1.2.0 -net tap,vlan=164,script=no,downscript=no,ifname=tap0 -net nic,vlan=164,model=e1000,macaddr=52:54:00:ff:01:19 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-fbef4e007-f0f0017cdc25055a-lieven-storage-migration-t-hd0,if=virtio,cache=none,aio=native -m 2048 -smp 2,sockets=1,cores=2,threads=1 -monitor tcp:0:4001,server,nowait -vnc :1 -qmp tcp:0:3001,server,nowait -name lieven-storage-migration-test -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-254.pid -mem-path /hugepages -mem-prealloc -cpu host,+x2apic,model_id=Intel\(R\)\ Xeon\(R\)\ CPU\ \ \ \ \ \ \ \ \ \ \ L5640\ \ @\ 2.27GHz,-tsc -rtc base=utc -usb -usbdevice tablet -no-hpet -vga cirrus process 29659 is executing new program: /usr/local/qemu-kvm-1.2.0/bin/qemu-system-x86_64 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 165 block-migration.c: No such file or directory. [New Thread 0x7ffff6163700 (LWP 29663)] [New Thread 0x7ffff5962700 (LWP 29664)] [New Thread 0x7fffefe97700 (LWP 29673)] [New Thread 0x7fffef696700 (LWP 29674)] [Thread 0x7fffefe97700 (LWP 29673) exited] [New Thread 0x7fffefe97700 (LWP 29825)] [Thread 0x7fffefe97700 (LWP 29825) exited] [New Thread 0x7fffefe97700 (LWP 29969)] [Thread 0x7fffefe97700 (LWP 29969) exited] [New Thread 0x7fffefe97700 (LWP 30091)] [New Thread 0x7fffee15c700 (LWP 30092)] [New Thread 0x7fffecc47700 (LWP 30093)] [New Thread 0x7fffdbfff700 (LWP 30094)] [New Thread 0x7fffe33f3700 (LWP 30095)] [New Thread 0x7fffe20e0700 (LWP 30096)] [New Thread 0x7fffe18df700 (LWP 30097)] [New Thread 0x7fffe10de700 (LWP 30098)] [New Thread 0x7fffe08dd700 (LWP 30099)] [New Thread 0x7fffdb7fe700 (LWP 30100)] [New Thread 0x7fffdaffd700 (LWP 30101)] [New Thread 0x7fffda7fc700 (LWP 30102)] [New Thread 0x7fffd9ffb700 (LWP 30103)] [New Thread 0x7fffd97fa700 (LWP 30104)] [New Thread 0x7fffd8ff9700 (LWP 30105)] [New Thread 0x7fffa7fff700 (LWP 30106)] [New Thread 0x7fffa77fe700 (LWP 30107)] [New Thread 0x7fffa6ffd700 (LWP 30108)] [New Thread 0x7fffed853700 (LWP 30109)] [New Thread 0x7fffa67fc700 (LWP 30110)] [New Thread 0x7fffa5ffb700 (LWP 30111)] [New Thread 0x7fffa57fa700 (LWP 30112)] [New Thread 0x7fffa4ff9700 (LWP 30113)] [New Thread 0x7fff83fff700 (LWP 30114)] [New Thread 0x7fff837fe700 (LWP 30115)] [New Thread 0x7fff82ffd700 (LWP 30116)] [New Thread 0x7fff827fc700 (LWP 30117)] [New Thread 0x7fff81ffb700 (LWP 30118)] [New Thread 0x7fff817fa700 (LWP 30119)] [New Thread 0x7fff80ff9700 (LWP 30120)] [New Thread 0x7fff5ffff700 (LWP 30121)] [New Thread 0x7fff5f7fe700 (LWP 30122)] [New Thread 0x7fff5effd700 (LWP 30123)] [New Thread 0x7fff5e7fc700 (LWP 30124)] [New Thread 0x7fff5dffb700 (LWP 30125)] [New Thread 0x7fff5d7fa700 (LWP 30126)] [New Thread 0x7fff5cff9700 (LWP 30127)] [New Thread 0x7fff3ffff700 (LWP 30128)] [New Thread 0x7fff3f7fe700 (LWP 30129)] [New Thread 0x7fff3effd700 (LWP 30130)] [New Thread 0x7fff3e7fc700 (LWP 30131)] [New Thread 0x7fff3dffb700 (LWP 30132)] [New Thread 0x7fff3d7fa700 (LWP 30133)] [New Thread 0x7fff3cff9700 (LWP 30134)] [New Thread 0x7fff1ffff700 (LWP 30135)] [New Thread 0x7fff1f7fe700 (LWP 30136)] [New Thread 0x7fff1effd700 (LWP 30137)] [New Thread 0x7fff1e7fc700 (LWP 30138)] [New Thread 0x7fff1dffb700 (LWP 30139)] [New Thread 0x7fff1d7fa700 (LWP 30140)] [New Thread 0x7fff1cff9700 (LWP 30141)] [New Thread 0x7ffefffff700 (LWP 30142)] [New Thread 0x7ffeff7fe700 (LWP 30143)] [New Thread 0x7ffefeffd700 (LWP 30144)] [New Thread 0x7ffefe7fc700 (LWP 30145)] [New Thread 0x7ffefdffb700 (LWP 30146)] [New Thread 0x7ffefd7fa700 (LWP 30147)] [New Thread 0x7ffefcff9700 (LWP 30148)] [New Thread 0x7ffedffff700 (LWP 30149)] [New Thread 0x7ffedf7fe700 (LWP 30150)] [New Thread 0x7ffedeffd700 (LWP 30151)] [New Thread 0x7ffede7fc700 (LWP 30152)] [New Thread 0x7ffeddffb700 (LWP 30153)] [New Thread 0x7ffedd7fa700 (LWP 30154)] Program received signal SIGSEGV, Segmentation fault. 0x00005555555cd5f6 in bmds_set_aio_inflight (bmds=0x555556568280, sector_num=2396160, nb_sectors=2048, set=0) at block-migration.c:165 (gdb) \a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\athread apply all bt full Thread 71 (Thread 0x7ffedd7fa700 (LWP 30154)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffedd7f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 65877} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 70 (Thread 0x7ffeddffb700 (LWP 30153)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffeddffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 627866} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 69 (Thread 0x7ffede7fc700 (LWP 30152)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffede7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 759753} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 68 (Thread 0x7ffedeffd700 (LWP 30151)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffedeffceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 644535} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 67 (Thread 0x7ffedf7fe700 (LWP 30150)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffedf7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 712004} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 66 (Thread 0x7ffedffff700 (LWP 30149)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffedfffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 743448} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 65 (Thread 0x7ffefcff9700 (LWP 30148)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffefcff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555a4074a0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 754495} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 64 (Thread 0x7ffefd7fa700 (LWP 30147)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffefd7f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 762992} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 63 (Thread 0x7ffefdffb700 (LWP 30146)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffefdffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 28676} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 62 (Thread 0x7ffefe7fc700 (LWP 30145)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffefe7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555a4074a0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 809680} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 61 (Thread 0x7ffefeffd700 (LWP 30144)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffefeffceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 810945} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 60 (Thread 0x7ffeff7fe700 (LWP 30143)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffeff7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 33778} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 59 (Thread 0x7ffefffff700 (LWP 30142)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7ffeffffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 98618} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 58 (Thread 0x7fff1cff9700 (LWP 30141)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1cff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 816786} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 57 (Thread 0x7fff1d7fa700 (LWP 30140)): #0 0x00007ffff6971653 in pread64 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573b0dd in handle_aiocb_rw_linear (aiocb=0x55555bc7eb20, buf=0x7fff04000c00 "\326\246\002") at posix-aio-compat.c:224 offset = 0 len = 1048576 #2 0x000055555573b2c4 in handle_aiocb_rw (aiocb=0x55555bc7eb20) at posix-aio-compat.c:293 nbytes = 93825003889184 buf = 0x7fff04000c00 "\326\246\002" #3 0x000055555573b54c in aio_thread (unused=0x0) at posix-aio-compat.c:351 aiocb = 0x55555bc7eb20 ret = 0 tv = {tv_sec = 1347912395, tv_usec = 983439} ts = {tv_sec = 1347912405, tv_nsec = 0} #4 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #5 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #6 0x0000000000000000 in ?? () No symbol table info available. Thread 56 (Thread 0x7fff1dffb700 (LWP 30139)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1dffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7f150 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 32930} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 55 (Thread 0x7fff1e7fc700 (LWP 30138)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1e7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 816746} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 54 (Thread 0x7fff1effd700 (LWP 30137)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1effceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 42141} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 53 (Thread 0x7fff1f7fe700 (LWP 30136)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1f7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 121256} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 52 (Thread 0x7fff1ffff700 (LWP 30135)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff1fffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 48151} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 51 (Thread 0x7fff3cff9700 (LWP 30134)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3cff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 207541} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 50 (Thread 0x7fff3d7fa700 (LWP 30133)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3d7f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 73822} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 49 (Thread 0x7fff3dffb700 (LWP 30132)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3dffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 128860} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 48 (Thread 0x7fff3e7fc700 (LWP 30131)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3e7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 175835} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 47 (Thread 0x7fff3effd700 (LWP 30130)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3effceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 254947} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 46 (Thread 0x7fff3f7fe700 (LWP 30129)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3f7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 195999} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 45 (Thread 0x7fff3ffff700 (LWP 30128)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff3fffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 239779} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 44 (Thread 0x7fff5cff9700 (LWP 30127)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5cff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 180411} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 43 (Thread 0x7fff5d7fa700 (LWP 30126)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5d7f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 173028} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 42 (Thread 0x7fff5dffb700 (LWP 30125)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5dffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 325936} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 41 (Thread 0x7fff5e7fc700 (LWP 30124)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5e7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 185622} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 40 (Thread 0x7fff5effd700 (LWP 30123)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5effceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 221321} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 39 (Thread 0x7fff5f7fe700 (LWP 30122)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5f7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 258187} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 38 (Thread 0x7fff5ffff700 (LWP 30121)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff5fffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 346934} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 37 (Thread 0x7fff80ff9700 (LWP 30120)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff80ff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 282933} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 36 (Thread 0x7fff817fa700 (LWP 30119)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff817f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 330640} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 35 (Thread 0x7fff81ffb700 (LWP 30118)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff81ffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 330090} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 34 (Thread 0x7fff827fc700 (LWP 30117)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff827fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7f150 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 373531} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 33 (Thread 0x7fff82ffd700 (LWP 30116)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff82ffceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 324347} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 32 (Thread 0x7fff837fe700 (LWP 30115)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff837fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 362610} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 31 (Thread 0x7fff83fff700 (LWP 30114)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fff83ffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 415042} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 30 (Thread 0x7fffa4ff9700 (LWP 30113)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa4ff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7f150 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 342087} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 29 (Thread 0x7fffa57fa700 (LWP 30112)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa57f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7f150 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 518047} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 28 (Thread 0x7fffa5ffb700 (LWP 30111)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa5ffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 514193} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 27 (Thread 0x7fffa67fc700 (LWP 30110)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa67fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 526408} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 26 (Thread 0x7fffed853700 (LWP 30109)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffed852eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 382861} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 25 (Thread 0x7fffa6ffd700 (LWP 30108)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa6ffceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 398159} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 24 (Thread 0x7fffa77fe700 (LWP 30107)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa77fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 519253} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 23 (Thread 0x7fffa7fff700 (LWP 30106)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffa7ffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 624913} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 22 (Thread 0x7fffd8ff9700 (LWP 30105)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffd8ff8eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 560397} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 21 (Thread 0x7fffd97fa700 (LWP 30104)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffd97f9eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x5555565fe600 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 522208} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 20 (Thread 0x7fffd9ffb700 (LWP 30103)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffd9ffaeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7eb20 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 531203} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 19 (Thread 0x7fffda7fc700 (LWP 30102)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffda7fbeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 542959} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 18 (Thread 0x7fffdaffd700 (LWP 30101)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffdaffceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555a4074a0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 534592} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 17 (Thread 0x7fffdb7fe700 (LWP 30100)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffdb7fdeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 536674} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 16 (Thread 0x7fffe08dd700 (LWP 30099)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffe08dceb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555a4074a0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 629188} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 15 (Thread 0x7fffe10de700 (LWP 30098)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffe10ddeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7e370 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 40152} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 14 (Thread 0x7fffe18df700 (LWP 30097)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffe18deeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 650586} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 13 (Thread 0x7fffe20e0700 (LWP 30096)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffe20dfeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ed30 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 565139} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 12 (Thread 0x7fffe33f3700 (LWP 30095)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffe33f2eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 629929} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 11 (Thread 0x7fffdbfff700 (LWP 30094)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffdbffeeb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 592997} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 10 (Thread 0x7fffecc47700 (LWP 30093)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffecc46eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555bc7ef40 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 691042} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 9 (Thread 0x7fffee15c700 (LWP 30092)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffee15beb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x55555a4074a0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 710341} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 8 (Thread 0x7fffefe97700 (LWP 30091)): #0 0x00007ffff696e0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555573acf1 in cond_timedwait (cond=0x555556072260, mutex=0x555556072220, ts=0x7fffefe96eb0) at posix-aio-compat.c:107 ret = 0 #2 0x000055555573b468 in aio_thread (unused=0x0) at posix-aio-compat.c:337 aiocb = 0x555559d064e0 ret = 0 tv = {tv_sec = 1347912396, tv_usec = 707609} ts = {tv_sec = 1347912406, tv_nsec = 0} #3 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 5 (Thread 0x7fffef696700 (LWP 29674)): #0 0x00007ffff696dd84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000055555575f7ae in qemu_cond_wait (cond=0x5555565b6c90, mutex=0x5555565b6cc0) at qemu-thread-posix.c:113 err = 0 __func__ = "qemu_cond_wait" #2 0x0000555555797c69 in vnc_worker_thread_loop (queue=0x5555565b6c90) at ui/vnc-jobs.c:222 job = 0x5555565a8980 entry = 0x0 tmp = 0x0 vs = {csock = -1, ds = 0x55555653faf0, dirty = {{0, 0, 0} <repeats 2048 times>}, lossy_rect = 0x55555bc7ebb0, vd = 0x7fffeffba010, need_update = 0, force_update = 0, features = 0, absolute = 0, last_x = 0, last_y = 0, client_width = 0, client_height = 0, share_mode = 0, vnc_encoding = 0, major = 0, minor = 0, auth = 0, challenge = '\000' <repeats 15 times>, info = 0x0, output = { capacity = 3149825, offset = 3145744, buffer = 0x7fffee360010 ""}, input = {capacity = 0, offset = 0, buffer = 0x0}, write_pixels = 0x55555579a63d <vnc_write_pixels_copy>, clientds = { flags = 0 '\000', width = 1024, height = 768, linesize = 4096, data = 0x7fffee967010 "\032\023*", pf = {bits_per_pixel = 32 ' ', bytes_per_pixel = 4 '\004', depth = 24 '\030', rmask = 16711680, gmask = 65280, bmask = 255, amask = 0, rshift = 16 '\020', gshift = 8 '\b', bshift = 0 '\000', ashift = 24 '\030', rmax = 255 '\377', gmax = 255 '\377', bmax = 255 '\377', amax = 255 '\377', rbits = 8 '\b', gbits = 8 '\b', bbits = 8 '\b', abits = 8 '\b'}}, audio_cap = 0x0, as = { freq = 0, nchannels = 0, fmt = AUD_FMT_U8, endianness = 0}, read_handler = 0, read_handler_expect = 0, modifiers_state = '\000' <repeats 255 times>, led = 0x0, abort = false, output_mutex = {lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, bh = 0x0, jobs_buffer = {capacity = 0, offset = 0, buffer = 0x0}, tight = { type = 0, quality = 255 '\377', compression = 9 '\t', pixel24 = 0 '\000', tight = {capacity = 0, offset = 0, buffer = 0x0}, tmp = {capacity = 0, offset = 0, buffer = 0x0}, zlib = {capacity = 0, offset = 0, buffer = 0x0}, gradient = { capacity = 0, offset = 0, buffer = 0x0}, levels = {0, 0, 0, 0}, stream = {{next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, { next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}}}, zlib = {zlib = { capacity = 0, offset = 0, buffer = 0x0}, tmp = {capacity = 0, offset = 0, buffer = 0x0}, stream = {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, level = 0}, hextile = { send_tile = 0x5555557849a0 <send_hextile_tile_32>}, zrle = { type = 0, fb = {capacity = 0, offset = 0, buffer = 0x0}, zrle = { capacity = 0, offset = 0, buffer = 0x0}, tmp = {capacity = 0, offset = 0, buffer = 0x0}, zlib = {capacity = 0, offset = 0, buffer = 0x0}, stream = {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0, zfree = 0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, palette = {pool = {{ idx = 0, color = 0, next = {le_next = 0x0, le_prev = 0x0}} <repeats 256 times>}, size = 0, max = 0, bpp = 0, table = {{lh_first = 0x0} <repeats 256 times>}}}, zywrle = {buf = {0 <repeats 4090 times>, 128, 0, -157897112, 32767, 0, 0}}, mouse_mode_notifier = {notify = 0, node = { le_next = 0x0, le_prev = 0x55555575f948}}, next = { tqe_next = 0x0, tqe_prev = 0x5555565b6ce8}} n_rectangles = 1 saved_offset = 2 #3 0x0000555555798192 in vnc_worker_thread (arg=0x5555565b6c90) at ui/vnc-jobs.c:318 queue = 0x5555565b6c90 #4 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #5 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #6 0x0000000000000000 in ?? () No symbol table info available. Thread 3 (Thread 0x7ffff5962700 (LWP 29664)): #0 0x00007ffff668fd27 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x000055555580ebfa in kvm_vcpu_ioctl (env=0x555556543470, type=44672) at /usr/src/qemu-kvm-1.2.0/kvm-all.c:1693 ret = -321675584 arg = 0x0 ap = {{gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x7ffff5961e70, reg_save_area = 0x7ffff5961db0}} #2 0x000055555580e653 in kvm_cpu_exec (env=0x555556543470) at /usr/src/qemu-kvm-1.2.0/kvm-all.c:1580 run = 0x7ffff7fc3000 ret = 0 run_ret = 0 #3 0x00005555557a715b in qemu_kvm_cpu_thread_fn (arg=0x555556543470) at /usr/src/qemu-kvm-1.2.0/cpus.c:757 env = 0x555556543470 cpu = 0x555556543430 r = 65536 #4 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #5 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #6 0x0000000000000000 in ?? () No symbol table info available. Thread 2 (Thread 0x7ffff6163700 (LWP 29663)): #0 0x00007ffff668fd27 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x000055555580ebfa in kvm_vcpu_ioctl (env=0x5555565267a0, type=44672) at /usr/src/qemu-kvm-1.2.0/kvm-all.c:1693 ret = 32767 arg = 0x0 ap = {{gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x7ffff6162e70, reg_save_area = 0x7ffff6162db0}} #2 0x000055555580e653 in kvm_cpu_exec (env=0x5555565267a0) at /usr/src/qemu-kvm-1.2.0/kvm-all.c:1580 run = 0x7ffff7fc6000 ret = 0 run_ret = 0 #3 0x00005555557a715b in qemu_kvm_cpu_thread_fn (arg=0x5555565267a0) at /usr/src/qemu-kvm-1.2.0/cpus.c:757 env = 0x5555565267a0 cpu = 0x555556526760 r = 65536 #4 0x00007ffff6969e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #5 0x00007ffff66974bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #6 0x0000000000000000 in ?? () No symbol table info available. Thread 1 (Thread 0x7ffff7fef700 (LWP 29659)): #0 0x00005555555cd5f6 in bmds_set_aio_inflight (bmds=0x555556568280, sector_num=2396160, nb_sectors=2048, set=0) at block-migration.c:165 start = 1170 end = 1170 val = 93824994379674 idx = 18 bit = 18 #1 0x00005555555cd7dc in blk_mig_read_cb (opaque=0x55555868e510, ret=0) at block-migration.c:199 curr_time = 221018445336304 blk = 0x55555868e510 __PRETTY_FUNCTION__ = "blk_mig_read_cb" #2 0x00005555555d755a in bdrv_co_em_bh (opaque=0x5555565fee50) at block.c:3320 acb = 0x5555565fee50 #3 0x00005555555be8f7 in qemu_bh_poll () at async.c:71 bh = 0x5555565b6ea0 bhp = 0x7fffffffe170 next = 0x5555565a8980 ret = 1 nesting = 1 #4 0x00005555557280e6 in main_loop_wait (nonblocking=0) at main-loop.c:506 ret = 1 timeout = 4294967295 #5 0x000055555571ff9e in main_loop () at /usr/src/qemu-kvm-1.2.0/vl.c:1643 nonblocking = false last_io = 1 #6 0x0000555555726b93 in main (argc=38, argv=0x7fffffffe5f8, envp=0x7fffffffe730) at /usr/src/qemu-kvm-1.2.0/vl.c:3790 i = 64 snapshot = 0 linux_boot = 0 icount_option = 0x0 initrd_filename = 0x0 kernel_filename = 0x0 kernel_cmdline = 0x5555558d032a "" boot_devices = "dc", '\000' <repeats 30 times> ds = 0x55555653faf0 dcl = 0x0 cyls = 0 heads = 0 secs = 0 translation = 0 hda_opts = 0x0 opts = 0x55555650a120 machine_opts = 0x55555650a920 olist = 0x5780f65ad2e0 optind = 38 optarg = 0x7fffffffebd8 "cirrus" loadvm = 0x0 machine = 0x555555c615c0 cpu_model = 0x7fffffffeb5a "host,+x2apic,model_id=Intel(R) Xeon(R) CPU", ' ' <repeats 11 times>, "L5640 @ 2.27GHz,-tsc" vga_model = 0x7fffffffebd8 "cirrus" pid_file = 0x7fffffffeb19 "/var/run/qemu/vm-254.pid" incoming = 0x0 show_vnc_port = 0 defconfig = true userconfig = true log_mask = 0x0 log_file = 0x0 mem_trace = {malloc = 0x5555557231fa <malloc_and_trace>, realloc = 0x555555723252 <realloc_and_trace>, free = 0x5555557232b9 <free_and_trace>, calloc = 0, try_malloc = 0, try_realloc = 0} trace_events = 0x0 trace_file = 0x0 (gdb) q > Kevin ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 2012-09-17 20:12 ` Peter Lieven @ 2012-09-18 10:28 ` Peter Lieven -1 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-18 10:28 UTC (permalink / raw) To: Kevin Wolf; +Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, Paolo Bonzini On 09/17/12 22:12, Peter Lieven wrote: > On 09/17/12 10:41, Kevin Wolf wrote: >> Am 16.09.2012 12:13, schrieb Peter Lieven: >>> Hi, >>> >>> when trying to block migrate a VM from one node to another, the source >>> VM crashed with the following assertion: >>> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >>> >>> Is this sth already addresses/known? >> Not that I'm aware of, at least. >> >> Block migration doesn't seem to check whether the device is already in >> use, maybe this is the problem. Not sure why it would be in use, though, >> and in my quick test it didn't crash. >> >> So we need some more information: What's you command line, did you do >> anything specific in the monitor with block devices, what does the >> stacktrace look like, etc.? > kevin, it seems that i can very easily force a crash if I cancel a > running block migration. if I understand correctly what happens there are aio callbacks coming in after blk_mig_cleanup() has been called. what is the proper way to detect this in blk_mig_read_cb()? Thanks, Peter ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 @ 2012-09-18 10:28 ` Peter Lieven 0 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-18 10:28 UTC (permalink / raw) To: Kevin Wolf; +Cc: Paolo Bonzini, qemu-devel@nongnu.org, kvm@vger.kernel.org On 09/17/12 22:12, Peter Lieven wrote: > On 09/17/12 10:41, Kevin Wolf wrote: >> Am 16.09.2012 12:13, schrieb Peter Lieven: >>> Hi, >>> >>> when trying to block migrate a VM from one node to another, the source >>> VM crashed with the following assertion: >>> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >>> >>> Is this sth already addresses/known? >> Not that I'm aware of, at least. >> >> Block migration doesn't seem to check whether the device is already in >> use, maybe this is the problem. Not sure why it would be in use, though, >> and in my quick test it didn't crash. >> >> So we need some more information: What's you command line, did you do >> anything specific in the monitor with block devices, what does the >> stacktrace look like, etc.? > kevin, it seems that i can very easily force a crash if I cancel a > running block migration. if I understand correctly what happens there are aio callbacks coming in after blk_mig_cleanup() has been called. what is the proper way to detect this in blk_mig_read_cb()? Thanks, Peter ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 2012-09-18 10:28 ` Peter Lieven @ 2012-09-18 10:31 ` Kevin Wolf -1 siblings, 0 replies; 17+ messages in thread From: Kevin Wolf @ 2012-09-18 10:31 UTC (permalink / raw) To: Peter Lieven; +Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, Paolo Bonzini Am 18.09.2012 12:28, schrieb Peter Lieven: > On 09/17/12 22:12, Peter Lieven wrote: >> On 09/17/12 10:41, Kevin Wolf wrote: >>> Am 16.09.2012 12:13, schrieb Peter Lieven: >>>> Hi, >>>> >>>> when trying to block migrate a VM from one node to another, the source >>>> VM crashed with the following assertion: >>>> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >>>> >>>> Is this sth already addresses/known? >>> Not that I'm aware of, at least. >>> >>> Block migration doesn't seem to check whether the device is already in >>> use, maybe this is the problem. Not sure why it would be in use, though, >>> and in my quick test it didn't crash. >>> >>> So we need some more information: What's you command line, did you do >>> anything specific in the monitor with block devices, what does the >>> stacktrace look like, etc.? >> kevin, it seems that i can very easily force a crash if I cancel a >> running block migration. > if I understand correctly what happens there are aio callbacks coming in > after > blk_mig_cleanup() has been called. > > what is the proper way to detect this in blk_mig_read_cb()? You could try this, it doesn't detect the situation in blk_mig_read_cb(), but ensures that all callbacks happen before we do the actual cleanup (completely untested): diff --git a/block-migration.c b/block-migration.c index 7def8ab..ed93301 100644 --- a/block-migration.c +++ b/block-migration.c @@ -519,6 +519,8 @@ static void blk_mig_cleanup(void) BlkMigDevState *bmds; BlkMigBlock *blk; + bdrv_drain_all(); + set_dirty_tracking(0); while ((bmds = QSIMPLEQ_FIRST(&block_mig_state.bmds_list)) != NULL) { ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 @ 2012-09-18 10:31 ` Kevin Wolf 0 siblings, 0 replies; 17+ messages in thread From: Kevin Wolf @ 2012-09-18 10:31 UTC (permalink / raw) To: Peter Lieven; +Cc: Paolo Bonzini, qemu-devel@nongnu.org, kvm@vger.kernel.org Am 18.09.2012 12:28, schrieb Peter Lieven: > On 09/17/12 22:12, Peter Lieven wrote: >> On 09/17/12 10:41, Kevin Wolf wrote: >>> Am 16.09.2012 12:13, schrieb Peter Lieven: >>>> Hi, >>>> >>>> when trying to block migrate a VM from one node to another, the source >>>> VM crashed with the following assertion: >>>> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >>>> >>>> Is this sth already addresses/known? >>> Not that I'm aware of, at least. >>> >>> Block migration doesn't seem to check whether the device is already in >>> use, maybe this is the problem. Not sure why it would be in use, though, >>> and in my quick test it didn't crash. >>> >>> So we need some more information: What's you command line, did you do >>> anything specific in the monitor with block devices, what does the >>> stacktrace look like, etc.? >> kevin, it seems that i can very easily force a crash if I cancel a >> running block migration. > if I understand correctly what happens there are aio callbacks coming in > after > blk_mig_cleanup() has been called. > > what is the proper way to detect this in blk_mig_read_cb()? You could try this, it doesn't detect the situation in blk_mig_read_cb(), but ensures that all callbacks happen before we do the actual cleanup (completely untested): diff --git a/block-migration.c b/block-migration.c index 7def8ab..ed93301 100644 --- a/block-migration.c +++ b/block-migration.c @@ -519,6 +519,8 @@ static void blk_mig_cleanup(void) BlkMigDevState *bmds; BlkMigBlock *blk; + bdrv_drain_all(); + set_dirty_tracking(0); while ((bmds = QSIMPLEQ_FIRST(&block_mig_state.bmds_list)) != NULL) { ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 2012-09-18 10:31 ` Kevin Wolf @ 2012-09-19 5:49 ` Peter Lieven -1 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-19 5:49 UTC (permalink / raw) To: Kevin Wolf; +Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, Paolo Bonzini On 09/18/12 12:31, Kevin Wolf wrote: > Am 18.09.2012 12:28, schrieb Peter Lieven: >> On 09/17/12 22:12, Peter Lieven wrote: >>> On 09/17/12 10:41, Kevin Wolf wrote: >>>> Am 16.09.2012 12:13, schrieb Peter Lieven: >>>>> Hi, >>>>> >>>>> when trying to block migrate a VM from one node to another, the source >>>>> VM crashed with the following assertion: >>>>> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >>>>> >>>>> Is this sth already addresses/known? >>>> Not that I'm aware of, at least. >>>> >>>> Block migration doesn't seem to check whether the device is already in >>>> use, maybe this is the problem. Not sure why it would be in use, though, >>>> and in my quick test it didn't crash. >>>> >>>> So we need some more information: What's you command line, did you do >>>> anything specific in the monitor with block devices, what does the >>>> stacktrace look like, etc.? >>> kevin, it seems that i can very easily force a crash if I cancel a >>> running block migration. >> if I understand correctly what happens there are aio callbacks coming in >> after >> blk_mig_cleanup() has been called. >> >> what is the proper way to detect this in blk_mig_read_cb()? > You could try this, it doesn't detect the situation in > blk_mig_read_cb(), but ensures that all callbacks happen before we do > the actual cleanup (completely untested): after testing it for half an hour i can say, it seems to fix the problem. no segfaults and also no other assertions. while searching I have seen that the queses blk_list and bmds_list are initialized at qemu startup. wouldn't it be better to initialize them at init_blk_migration or at least check that they are really empty? i have also seen that prev_time_offset is not initialized. thank you, peter sth like this: --- qemu-kvm-1.2.0/block-migration.c.orig 2012-09-17 21:14:44.458429855 +0200 +++ qemu-kvm-1.2.0/block-migration.c 2012-09-17 21:15:40.599736962 +0200 @@ -311,8 +311,12 @@ static void init_blk_migration(QEMUFile block_mig_state.prev_progress = -1; block_mig_state.bulk_completed = 0; block_mig_state.total_time = 0; + block_mig_state.prev_time_offset = 0; block_mig_state.reads = 0; + QSIMPLEQ_INIT(&block_mig_state.bmds_list); + QSIMPLEQ_INIT(&block_mig_state.blk_list); + bdrv_iterate(init_blk_migration_it, NULL); } @@ -760,9 +764,6 @@ SaveVMHandlers savevm_block_handlers = { void blk_mig_init(void) { - QSIMPLEQ_INIT(&block_mig_state.bmds_list); - QSIMPLEQ_INIT(&block_mig_state.blk_list); - register_savevm_live(NULL, "block", 0, 1, &savevm_block_handlers, &block_mig_state); } > diff --git a/block-migration.c b/block-migration.c > index 7def8ab..ed93301 100644 > --- a/block-migration.c > +++ b/block-migration.c > @@ -519,6 +519,8 @@ static void blk_mig_cleanup(void) > BlkMigDevState *bmds; > BlkMigBlock *blk; > > + bdrv_drain_all(); > + > set_dirty_tracking(0); > > while ((bmds = QSIMPLEQ_FIRST(&block_mig_state.bmds_list)) != NULL) { ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 @ 2012-09-19 5:49 ` Peter Lieven 0 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-19 5:49 UTC (permalink / raw) To: Kevin Wolf; +Cc: Paolo Bonzini, qemu-devel@nongnu.org, kvm@vger.kernel.org On 09/18/12 12:31, Kevin Wolf wrote: > Am 18.09.2012 12:28, schrieb Peter Lieven: >> On 09/17/12 22:12, Peter Lieven wrote: >>> On 09/17/12 10:41, Kevin Wolf wrote: >>>> Am 16.09.2012 12:13, schrieb Peter Lieven: >>>>> Hi, >>>>> >>>>> when trying to block migrate a VM from one node to another, the source >>>>> VM crashed with the following assertion: >>>>> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >>>>> >>>>> Is this sth already addresses/known? >>>> Not that I'm aware of, at least. >>>> >>>> Block migration doesn't seem to check whether the device is already in >>>> use, maybe this is the problem. Not sure why it would be in use, though, >>>> and in my quick test it didn't crash. >>>> >>>> So we need some more information: What's you command line, did you do >>>> anything specific in the monitor with block devices, what does the >>>> stacktrace look like, etc.? >>> kevin, it seems that i can very easily force a crash if I cancel a >>> running block migration. >> if I understand correctly what happens there are aio callbacks coming in >> after >> blk_mig_cleanup() has been called. >> >> what is the proper way to detect this in blk_mig_read_cb()? > You could try this, it doesn't detect the situation in > blk_mig_read_cb(), but ensures that all callbacks happen before we do > the actual cleanup (completely untested): after testing it for half an hour i can say, it seems to fix the problem. no segfaults and also no other assertions. while searching I have seen that the queses blk_list and bmds_list are initialized at qemu startup. wouldn't it be better to initialize them at init_blk_migration or at least check that they are really empty? i have also seen that prev_time_offset is not initialized. thank you, peter sth like this: --- qemu-kvm-1.2.0/block-migration.c.orig 2012-09-17 21:14:44.458429855 +0200 +++ qemu-kvm-1.2.0/block-migration.c 2012-09-17 21:15:40.599736962 +0200 @@ -311,8 +311,12 @@ static void init_blk_migration(QEMUFile block_mig_state.prev_progress = -1; block_mig_state.bulk_completed = 0; block_mig_state.total_time = 0; + block_mig_state.prev_time_offset = 0; block_mig_state.reads = 0; + QSIMPLEQ_INIT(&block_mig_state.bmds_list); + QSIMPLEQ_INIT(&block_mig_state.blk_list); + bdrv_iterate(init_blk_migration_it, NULL); } @@ -760,9 +764,6 @@ SaveVMHandlers savevm_block_handlers = { void blk_mig_init(void) { - QSIMPLEQ_INIT(&block_mig_state.bmds_list); - QSIMPLEQ_INIT(&block_mig_state.blk_list); - register_savevm_live(NULL, "block", 0, 1, &savevm_block_handlers, &block_mig_state); } > diff --git a/block-migration.c b/block-migration.c > index 7def8ab..ed93301 100644 > --- a/block-migration.c > +++ b/block-migration.c > @@ -519,6 +519,8 @@ static void blk_mig_cleanup(void) > BlkMigDevState *bmds; > BlkMigBlock *blk; > > + bdrv_drain_all(); > + > set_dirty_tracking(0); > > while ((bmds = QSIMPLEQ_FIRST(&block_mig_state.bmds_list)) != NULL) { ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 2012-09-19 5:49 ` Peter Lieven (?) @ 2012-09-25 14:00 ` Kevin Wolf -1 siblings, 0 replies; 17+ messages in thread From: Kevin Wolf @ 2012-09-25 14:00 UTC (permalink / raw) To: Peter Lieven; +Cc: Paolo Bonzini, qemu-devel@nongnu.org, kvm@vger.kernel.org Am 19.09.2012 07:49, schrieb Peter Lieven: > On 09/18/12 12:31, Kevin Wolf wrote: >> Am 18.09.2012 12:28, schrieb Peter Lieven: >>> On 09/17/12 22:12, Peter Lieven wrote: >>>> On 09/17/12 10:41, Kevin Wolf wrote: >>>>> Am 16.09.2012 12:13, schrieb Peter Lieven: >>>>>> Hi, >>>>>> >>>>>> when trying to block migrate a VM from one node to another, the source >>>>>> VM crashed with the following assertion: >>>>>> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >>>>>> >>>>>> Is this sth already addresses/known? >>>>> Not that I'm aware of, at least. >>>>> >>>>> Block migration doesn't seem to check whether the device is already in >>>>> use, maybe this is the problem. Not sure why it would be in use, though, >>>>> and in my quick test it didn't crash. >>>>> >>>>> So we need some more information: What's you command line, did you do >>>>> anything specific in the monitor with block devices, what does the >>>>> stacktrace look like, etc.? >>>> kevin, it seems that i can very easily force a crash if I cancel a >>>> running block migration. >>> if I understand correctly what happens there are aio callbacks coming in >>> after >>> blk_mig_cleanup() has been called. >>> >>> what is the proper way to detect this in blk_mig_read_cb()? >> You could try this, it doesn't detect the situation in >> blk_mig_read_cb(), but ensures that all callbacks happen before we do >> the actual cleanup (completely untested): > after testing it for half an hour i can say, it seems to fix the problem. > no segfaults and also no other assertions. > > while searching I have seen that the queses blk_list and bmds_list are > initialized at > qemu startup. wouldn't it be better to initialize them at init_blk_migration > or at least check that they are really empty? i have also seen that > prev_time_offset > is not initialized. Probably. If you sent this as a proper patch with a SoB, I wouldn't reject it, but considering that block migration is deprecated anyway, I won't bother myself as long as there's no real bug. Kevin ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 2012-09-17 8:41 ` Kevin Wolf @ 2012-09-17 20:20 ` Peter Lieven -1 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-17 20:20 UTC (permalink / raw) To: Kevin Wolf; +Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, Paolo Bonzini On 09/17/12 10:41, Kevin Wolf wrote: > Am 16.09.2012 12:13, schrieb Peter Lieven: >> Hi, >> >> when trying to block migrate a VM from one node to another, the source >> VM crashed with the following assertion: >> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >> >> Is this sth already addresses/known? > Not that I'm aware of, at least. > > Block migration doesn't seem to check whether the device is already in > use, maybe this is the problem. Not sure why it would be in use, though, > and in my quick test it didn't crash. > > So we need some more information: What's you command line, did you do > anything specific in the monitor with block devices, what does the > stacktrace look like, etc.? i was also able to reproduce a "flush_blks: Assertion `block_mig_state.read_done >= 0' failed." by cancelling a block migration and restarting it afterwards. however, how can I grep a stack trace after an assert? thanks, peter > Kevin ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] Block Migration Assertion in qemu-kvm 1.2.0 @ 2012-09-17 20:20 ` Peter Lieven 0 siblings, 0 replies; 17+ messages in thread From: Peter Lieven @ 2012-09-17 20:20 UTC (permalink / raw) To: Kevin Wolf; +Cc: Paolo Bonzini, qemu-devel@nongnu.org, kvm@vger.kernel.org On 09/17/12 10:41, Kevin Wolf wrote: > Am 16.09.2012 12:13, schrieb Peter Lieven: >> Hi, >> >> when trying to block migrate a VM from one node to another, the source >> VM crashed with the following assertion: >> block.c:3829: bdrv_set_in_use: Assertion `bs->in_use != in_use' failed. >> >> Is this sth already addresses/known? > Not that I'm aware of, at least. > > Block migration doesn't seem to check whether the device is already in > use, maybe this is the problem. Not sure why it would be in use, though, > and in my quick test it didn't crash. > > So we need some more information: What's you command line, did you do > anything specific in the monitor with block devices, what does the > stacktrace look like, etc.? i was also able to reproduce a "flush_blks: Assertion `block_mig_state.read_done >= 0' failed." by cancelling a block migration and restarting it afterwards. however, how can I grep a stack trace after an assert? thanks, peter > Kevin ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2012-09-25 14:00 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-09-16 10:13 Block Migration Assertion in qemu-kvm 1.2.0 Peter Lieven 2012-09-16 10:13 ` [Qemu-devel] " Peter Lieven 2012-09-17 8:41 ` Kevin Wolf 2012-09-17 8:41 ` Kevin Wolf 2012-09-17 14:35 ` Peter Lieven 2012-09-17 14:35 ` Peter Lieven 2012-09-17 20:12 ` Peter Lieven 2012-09-17 20:12 ` Peter Lieven 2012-09-18 10:28 ` Peter Lieven 2012-09-18 10:28 ` Peter Lieven 2012-09-18 10:31 ` Kevin Wolf 2012-09-18 10:31 ` Kevin Wolf 2012-09-19 5:49 ` Peter Lieven 2012-09-19 5:49 ` Peter Lieven 2012-09-25 14:00 ` Kevin Wolf 2012-09-17 20:20 ` Peter Lieven 2012-09-17 20:20 ` Peter Lieven
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.