From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45089) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X5GAO-0001OZ-Ib for qemu-devel@nongnu.org; Thu, 10 Jul 2014 11:21:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1X5GAI-0001We-04 for qemu-devel@nongnu.org; Thu, 10 Jul 2014 11:21:48 -0400 Received: from mx1.redhat.com ([209.132.183.28]:17452) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X5GAH-0001WH-Na for qemu-devel@nongnu.org; Thu, 10 Jul 2014 11:21:41 -0400 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s6AFLeTV025171 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 10 Jul 2014 11:21:40 -0400 Message-ID: <53BEAF80.1060308@redhat.com> Date: Thu, 10 Jul 2014 17:21:36 +0200 From: Paolo Bonzini MIME-Version: 1.0 References: <20140710151157.GM23577@redhat.com> In-Reply-To: <20140710151157.GM23577@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Help on possible hang in drive-mirror / query-block-jobs List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Daniel P. Berrange" , qemu-devel@nongnu.org Cc: Kevin Wolf , Fam Zheng , Stefan Hajnoczi Il 10/07/2014 17:11, Daniel P. Berrange ha scritto: > I've spent the last week debugging an issue that is hitting OpenStack > with drive-mirror/block job usage. > > Specifically we are seeing that a monitor command for 'query-block-jobs' > never replies to libvirt. After 3 minutes of waiting the test harness > times out and kills the VM. When working normally the entire test will > complete in just a couple of seconds, so we don't think the 3 minute > timeout is hitting a false positive. > > [...] The rate failure of this problem is only > around 1 in 400 uses of drive-mirror. [...] No one has ever > managed to reproduce the problem outside of our automated test > system environment, even when running the same tests locally and > we can't log into the test systems to get GDB traces or install > custom QEMU builds. > > The best I can do is to collect debug logs from libvirtd, and get > stdio from QEMU. The QEMU stderr/stdout shows nothing at all. The > libvirtd log shows the following sequence of monitor interactions > with QEMU: > > [...] > > 5. Libvirt waits for cleanup to complete: > > msg={"execute":"query-block-jobs","id":"libvirt-15"} > reply={"return": [{"io-status": "ok", "device": "drive-virtio-disk0", "busy": true, "len": 25165824, "offset": 25165824, "paused": false, "speed": 0, "type": "mirror"}], "id": "libvirt-15"} > > msg={"execute":"query-block-jobs","id":"libvirt-16"} > reply={"return": [{"io-status": "ok", "device": "drive-virtio-disk0", "busy": true, "len": 25165824, "offset": 25165824, "paused": false, "speed": 0, "type": "mirror"}], "id": "libvirt-16"} > > msg={"execute":"query-block-jobs","id":"libvirt-17"} > <...hang...> > > So we can see this last 'query-block-jobs' command hangs. I've looked at > the code for handling this monitor command and struggling to come up with > any ideas of why this might hang. My best idea was the bdrv_iterate() > call it does might be happening at the same time as another thread modifies > the list, but debugging on a local QEMU shows no changes to the list at > all due to drive-mirror/block jobs, so that doesn't seem to be the cause. No, all these modifications are anyway done with the big QEMU lock. All multitasking happens in coroutines, and the preemption points only happen when you have AIO. Can you install a custom QEMU? How many megabytes of stdout can your test rig tolerate? Any chance you can collect other files (traces)? Paolo