From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48579) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XTrT5-0004zn-4F for qemu-devel@nongnu.org; Tue, 16 Sep 2014 08:02:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1XTrSx-0007wX-Cf for qemu-devel@nongnu.org; Tue, 16 Sep 2014 08:02:47 -0400 Received: from mail-pa0-f45.google.com ([209.85.220.45]:47232) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XTrSx-0007vx-58 for qemu-devel@nongnu.org; Tue, 16 Sep 2014 08:02:39 -0400 Received: by mail-pa0-f45.google.com with SMTP id rd3so8728388pab.18 for ; Tue, 16 Sep 2014 05:02:33 -0700 (PDT) Message-ID: <541826CA.7050607@ozlabs.ru> Date: Tue, 16 Sep 2014 22:02:18 +1000 From: Alexey Kardashevskiy MIME-Version: 1.0 References: <5416C46D.7040105@ozlabs.ru> In-Reply-To: <5416C46D.7040105@ozlabs.ru> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] migration: qemu-coroutine-lock.c:141: qemu_co_mutex_unlock: Assertion `mutex->locked == 1' failed List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "qemu-devel@nongnu.org" , Paolo Bonzini , Eric Blake Cc: "Dr. David Alan Gilbert" , Stefan Hajnoczi Hi! I am having problems when migrate a guest via libvirt like this: virsh migrate --live --persistent --undefinesource --copy-storage-all --verbose --desturi qemu+ssh://legkvm/system --domain chig1 The XML used to create the guest is at the end of this mail. I see NBD FLUSH command after the destination QEMU received EOF for migration stream and this produces a crash in qcow2_co_flush_to_os() as s->lock is false or s->l2_table_cache is NULL. There are 2 scenarios I observe. First one is assert when qemu_co_mutex_unlock(&s->lock) right before "return 0": process_incoming_migration_co() qcow2_invalidate_cache() qcow2_close() qcow2_cache_flush() bdrv_flush() bdrv_co_flush() qemu_coroutine_yield() nbd_trip() NBD_CMD_FLUSH bdrv_co_flush() qcow2_co_flush_to_os() Second one is EOF is completely handled and next FLUSH crashes as s->l2_table_cache == NULL. Please help to understand what is going on here. Thanks! Probably this is enough :) If it is not, I pushed debug branch to git@github.com:aik/qemu.git * [new branch] mig-dbg-legkvm -> mig-dbg I added some traces and here is the log from the destination: /home/alexey/p/qemu/nbd.c:nbd_trip():L1293: Request/Reply complete +++Q+++ spapr_tce_pre_load 107 liobn 80000000 nb=262144 bus_off=0, shift=12, table=0x3fff915d0010 +++Q+++ (44846) qemu_loadvm_state 1000 EOF Received! section_type=0 (EOF=0) +++Q+++ (44846) qemu_loadvm_state 1007 +++Q+++ (44846) qemu_loadvm_state 1016 +++Q+++ (44846) qcow2_close 1399 START +++Q+++ (44846) qcow2_close 1404 _qcow2_cache_flush 0x10002b24030 0x100028e1390 - 0x100028df300 0x100028e1390 +++Q+++ (44846) bdrv_flush 5099 +++Q+++ (44846) bdrv_co_flush 4978 +++Q+++ (44846) bdrv_co_flush 4995 +++Q+++ (44846) bdrv_co_flush 4997 +++Q+++ (44846) bdrv_co_flush 5001 +++Q+++ (44846) bdrv_co_flush 5003 +++Q+++ (44846) bdrv_co_flush 5028 +++Q+++ (44846) bdrv_co_flush 4965 +++Q+++ (44846) bdrv_flush 5101 _qcow2_cache_flush 0x10002b24030 0x100028e13b0 - 0x100028df300 0x100028e1390 +++Q+++ (44846) bdrv_flush 5099 +++Q+++ (44846) bdrv_co_flush 4978 +++Q+++ (44846) bdrv_co_flush 4995 +++Q+++ (44846) bdrv_co_flush 4997 +++Q+++ (44846) bdrv_co_flush 5001 +++Q+++ (44846) bdrv_co_flush 5003 +++Q+++ (44846) bdrv_co_flush 5028 +++Q+++ (44846) bdrv_co_flush 4965 +++Q+++ (44846) bdrv_flush 5101 +++Q+++ (44846) qcow2_close 1409 +++Q+++ (44846) qcow2_close 1414 +++Q+++ (44846) qcow2_close 1422 +++Q+++ (44846) qcow2_close 1425 DONE! +++Q+++ qcow2_invalidate_cache 1459 /home/alexey/p/qemu/nbd.c:nbd_trip():L1164: Reading request. /home/alexey/p/qemu/nbd.c:nbd_receive_request():L785: Got request: { magic = 0x25609513, .type = 65539, from = 0 , len = 0 } /home/alexey/p/qemu/nbd.c:nbd_co_receive_request():L1130: Decoding type /home/alexey/p/qemu/nbd.c:nbd_trip():L1257: Request type is FLUSH +++Q+++ (44846) nbd_trip 1258 bs=0x100028db420 START +++Q+++ (44846) qcow2_co_flush_to_os 2171 _qcow2_cache_flush 0x10002e5f030 (nil) - 0x100028df300 (nil) 2014-09-16 11:34:36.731+0000: shutting down as first (nil) is referenced by "c->size". This is the sender: +++Q+++ (67154) qemu_savevm_state_complete 747 +++Q+++ (67154) qemu_savevm_state_complete 749 +++Q+++ (67154) qemu_savevm_state_complete 751 +++Q+++ (67154) migration_thread 617 +++Q+++ (67154) migration_thread 628 +++Q+++ (67154) migration_thread 667 +++Q+++ (67154) migration_thread 684 +++Q+++ (67154) migration_thread 686 +++Q+++ (67154) migration_thread 688 +++Q+++ (67154) bdrv_flush 5099 +++Q+++ (67154) bdrv_co_flush 4978 +++Q+++ (67154) bdrv_co_flush 5028 +++Q+++ (67154) nbd_client_session_co_flush 305 /home/alexey/p/qemu/nbd.c:nbd_send_request():L739: Sending request to client: { .from = 0, .le n = 0, .handle = 1099744563680, .type=65539} /home/alexey/p/qemu/nbd.c:nbd_receive_reply():L806: read failed This is the XML: [root@chikvm ~]# cat chig1-aik.xml chig1 bbf91237-3c78-489e-b426-ab593806c78b 4194304 4194304 1 /machine hvm destroy restart restart /usr/bin/qemu-system-ppc64.aik