From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58813) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZcBar-0006ke-Uv for qemu-devel@nongnu.org; Wed, 16 Sep 2015 08:13:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZcBao-0005TK-Lh for qemu-devel@nongnu.org; Wed, 16 Sep 2015 08:13:45 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53412) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZcBao-0005TF-Be for qemu-devel@nongnu.org; Wed, 16 Sep 2015 08:13:42 -0400 From: Laszlo Ersek Message-ID: <55F95CF2.3000401@redhat.com> Date: Wed, 16 Sep 2015 14:13:38 +0200 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------040400080604060006080309" Subject: [Qemu-devel] qemu <-> libvirt communication regressed in QEMU commit 5243722376 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Emilio G. Cota" Cc: =?UTF-8?Q?Alex_Benn=c3=a9e?= , Paolo Bonzini , qemu devel list , Cole Robinson This is a multi-part message in MIME format. --------------040400080604060006080309 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi Emilio, I've arrived at your patch, noted in the subject, with bisection (please see the bisection log attached). I'm on RHEL-7.1. Sometimes I have to work with upstream QEMU, and then I use it with my preexistent libvirt guests, pulling QEMU somewhat infrequently. My libvirt-related version numbers are: libvirtd: 1.2.8-16.el7_1.3.x86_64 libvirt-python: 1.2.8-7.el7_1.1.x86_64 libvirt-g*: 0.1.7-3.el7.x86_64 virt-manager: 1.1.0-12.el7.noarch The symptom is that when your patch is built into QEMU, then QEMU starts, but hangs as soon as I click the specific VM's entry in virt-manager's list. In the process list ("ps"), I can then see two qemu processes, parent and child. I saved backtraces for both of them, while they were hung. The command lines are also visible in the attached text files. The line numbers (ie. the QEMU binary) matches the tree when checked out and built at exactly your patch. (I double checked: if I build at 5243722376^, then it works.) The configure command was: ./configure \ --audio-drv-list=alsa \ --target-list=x86_64-softmmu,i386-softmmu,aarch64-softmmu \ --disable-vde \ --enable-werror \ --enable-spice \ --disable-stack-protector \ --prefix=/opt/qemu-installed \ --disable-gtk \ --enable-debug \ --enable-trace-backends=stderr I don't think libvirt, or for that matter, any QMP interfaces, have anything to do with this. I rather believe that libvirt invokes QEMU for retrieving the capabilities in a way that exposes a possible problem in your patch. (Hence I provided my libvirt version numbers just to be sure.) ... In fact I'm confused about your patch. rcu_init() makes sure that at fork(), the parent will first acquire both "rcu_sync_lock" and "rcu_registry_lock". Meaning, no other thread in the parent can hold those mutexen when the parent thread calling fork() actually forks. Then, in the parent, the original thread simply releases both mutexen, in rcu_init_unlock(). In the child, only the one thread exists that called fork() in the parent. However, that one child thread does own the copies of both mutexen. So it is prudent for the child to release both copies. Your patch causes "rcu_registry_lock" to be reinitialized in the child, rather than released, plus "rcu_sync_lock" remains untouched (ie. locked by the one thread that exists in the child). Why is that correct? (Side note: we're talking process-private, not process-shared mutexen.) I can be easily wrong, but I don't understand the commit message, and why the patch is correct. ... Hm, I can see the discussion here: http://thread.gmane.org/gmane.comp.emulators.qemu/356765/focus=360421 Okay... let me see 24fa90499f... "The problem is that releasing error-checking locks in the child fails under glibc with EPERM". <-- That is a striking surprise to me, but still, the removal of PTHREAD_MUTEX_ERRORCHECK only justifies why your patch would *not* be necessary. The last paragraph of your email that I linked above talks about a "possibility of corruption". Maybe I've managed to trigger that. If so, I hope it won't be hard to fix up. ... Hm, apparently Alex had mentioned the same concern as I did now, about ignoring "rcu_sync_lock" in the child, in message . Was that concern cleared up eventually? Thanks! Laszlo --------------040400080604060006080309 Content-Type: text/x-log; name="bisect.log" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="bisect.log" git bisect start # bad: [619622424dba749feef752d76d79ef2569f7f250] Merge remote-tracking branch 'remotes/berrange/tags/vnc-crypto-v9-for-upstream' into staging git bisect bad 619622424dba749feef752d76d79ef2569f7f250 # good: [2b750d9d261bda7f75b39dfc1e1e5f22502929d5] Merge remote-tracking branch 'remotes/aurel/tags/pull-sh4-next-20150913' into staging git bisect good 2b750d9d261bda7f75b39dfc1e1e5f22502929d5 # bad: [a2aa09e18186801931763fbd40a751fa39971b18] Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging git bisect bad a2aa09e18186801931763fbd40a751fa39971b18 # bad: [0c71d41e2aa3c7356500ae624166f3bb8c201aee] scripts/dump-guest-memory.py: fix after RAMBlock change git bisect bad 0c71d41e2aa3c7356500ae624166f3bb8c201aee # good: [3c9589e180d98cdadb143bd2a792fb9d19d9aec6] Move RAMBlock and ram_list to ram_addr.h git bisect good 3c9589e180d98cdadb143bd2a792fb9d19d9aec6 # bad: [3904e6bf042391abc749d717465022e96e276fc7] cutils: Add qemu_strtoull() wrapper git bisect bad 3904e6bf042391abc749d717465022e96e276fc7 # bad: [709037636992e9289ce9147e59d56fb35d90b140] linux-user: call rcu_(un)register_thread on pthread_(exit|create) git bisect bad 709037636992e9289ce9147e59d56fb35d90b140 # bad: [5243722376873a48e9852a58b91f4d4101ee66e4] rcu: init rcu_registry_lock after fork git bisect bad 5243722376873a48e9852a58b91f4d4101ee66e4 # good: [12a1ddc160cb6a73e8a6c319f3962a20da2cd22f] Makefile.target: include top level build dir in vpath git bisect good 12a1ddc160cb6a73e8a6c319f3962a20da2cd22f # first bad commit: [5243722376873a48e9852a58b91f4d4101ee66e4] rcu: init rcu_registry_lock after fork --------------040400080604060006080309 Content-Type: text/plain; name="parent.txt" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="parent.txt" VUlEICAgICAgICBQSUQgIFBQSUQgIEMgU1RJTUUgVFRZICAgICAgICAgIFRJTUUgQ01ECnFl bXUgICAgIDE3MzA1ICAxNzUyICAwIDEzOjI0ID8gICAgICAgIDAwOjAwOjAwIC9vcHQvcWVt dS1pbnN0YWxsZWQvYmluL3FlbXUtc3lzdGVtLWkzODYgLVMgLW5vLXVzZXItY29uZmlnIC1u b2RlZmF1bHRzIC1ub2dyYXBoaWMgLU0gbm9uZSAtcW1wIHVuaXg6L3Zhci9saWIvbGlidmly dC9xZW11L2NhcGFiaWxpdGllcy5tb25pdG9yLnNvY2ssc2VydmVyLG5vd2FpdCAtcGlkZmls ZSAvdmFyL2xpYi9saWJ2aXJ0L3FlbXUvY2FwYWJpbGl0aWVzLnBpZGZpbGUgLWRhZW1vbml6 ZQoKKGdkYikgdGhyZWFkIGFwcGx5IGFsbCBidCBmdWxsCgpUaHJlYWQgMiAoVGhyZWFkIDB4 N2ZhOWMzZGI3NzAwIChMV1AgMTczMDYpKToKIzAgIDB4MDAwMDdmYTljN2RkYTk0OSBpbiBz eXNjYWxsICgpIGZyb20gL2xpYjY0L2xpYmMuc28uNgpObyBzeW1ib2wgdGFibGUgaW5mbyBh dmFpbGFibGUuCiMxICAweDAwMDA3ZmE5Y2ViYzBmNzMgaW4gZnV0ZXhfd2FpdCAoZXY9MHg3 ZmE5Y2Y1MjQ1YTQgPHJjdV9jYWxsX3JlYWR5X2V2ZW50PiwgdmFsPTQyOTQ5NjcyOTUpIGF0 IHV0aWwvcWVtdS10aHJlYWQtcG9zaXguYzozMDEKTm8gbG9jYWxzLgojMiAgMHgwMDAwN2Zh OWNlYmMxMDZhIGluIHFlbXVfZXZlbnRfd2FpdCAoZXY9MHg3ZmE5Y2Y1MjQ1YTQgPHJjdV9j YWxsX3JlYWR5X2V2ZW50PikgYXQgdXRpbC9xZW11LXRocmVhZC1wb3NpeC5jOjQwOAogICAg ICAgIHZhbHVlID0gMQojMyAgMHgwMDAwN2ZhOWNlYmQ0NjY2IGluIGNhbGxfcmN1X3RocmVh ZCAob3BhcXVlPTB4MCkgYXQgdXRpbC9yY3UuYzoyNTQKICAgICAgICB0cmllcyA9IDAKICAg ICAgICBuID0gMAogICAgICAgIG5vZGUgPSAweDdmYTljZTcxMjk5MAojNCAgMHgwMDAwN2Zh OWNkMmZlZGY1IGluIHN0YXJ0X3RocmVhZCAoKSBmcm9tIC9saWI2NC9saWJwdGhyZWFkLnNv LjAKTm8gc3ltYm9sIHRhYmxlIGluZm8gYXZhaWxhYmxlLgojNSAgMHgwMDAwN2ZhOWM3ZGUw MWFkIGluIGNsb25lICgpIGZyb20gL2xpYjY0L2xpYmMuc28uNgpObyBzeW1ib2wgdGFibGUg aW5mbyBhdmFpbGFibGUuCgpUaHJlYWQgMSAoVGhyZWFkIDB4N2ZhOWNlNmYyYmMwIChMV1Ag MTczMDUpKToKIzAgIDB4MDAwMDdmYTljZDMwNTI1ZCBpbiByZWFkICgpIGZyb20gL2xpYjY0 L2xpYnB0aHJlYWQuc28uMApObyBzeW1ib2wgdGFibGUgaW5mbyBhdmFpbGFibGUuCiMxICAw eDAwMDA3ZmE5Y2U5MTVjN2MgaW4gb3NfZGFlbW9uaXplICgpIGF0IG9zLXBvc2l4LmM6MjIz CiAgICAgICAgc3RhdHVzID0gMCAnXDAwMCcKICAgICAgICBsZW4gPSAxNDA3MzMyNjAwMzI5 MTIKICAgICAgICBwaWQgPSAxNzMwNwogICAgICAgIGZkcyA9IHs0LCA1fQojMiAgMHgwMDAw N2ZhOWNlOTJhODAzIGluIG1haW4gKGFyZ2M9MTIsIGFyZ3Y9MHg3ZmZmMDNmOGVmZDgsIGVu dnA9MHg3ZmZmMDNmOGYwNDApIGF0IHZsLmM6NDAzNAogICAgICAgIGkgPSAwCiAgICAgICAg c25hcHNob3QgPSAwCiAgICAgICAgbGludXhfYm9vdCA9IDAKICAgICAgICBpbml0cmRfZmls ZW5hbWUgPSAweDdmYTlkMDc0OWViMCAi7mvFzqlcMTc3IgogICAgICAgIGtlcm5lbF9maWxl bmFtZSA9IDB4N2ZhOWQwNzQ5ZWEwICIiCiAgICAgICAga2VybmVsX2NtZGxpbmUgPSAweDdm YTljZWJkNGUyMCA8X19saWJjX2NzdV9pbml0PiAiQVdBXDIxMf9BVklcMjEx9kFVSVwyMTHV QVRMXDIxNSUiCiAgICAgICAgYm9vdF9vcmRlciA9IDB4MAogICAgICAgIGJvb3Rfb25jZSA9 IDB4MAogICAgICAgIGRzID0gMHg3ZmE5Y2VjNTZkMzgKICAgICAgICBjeWxzID0gMAogICAg ICAgIGhlYWRzID0gMAogICAgICAgIHNlY3MgPSAwCiAgICAgICAgdHJhbnNsYXRpb24gPSAw CiAgICAgICAgaGRhX29wdHMgPSAweDAKICAgICAgICBvcHRzID0gMHg3ZmE5ZDA3OTBlOTAK ICAgICAgICBtYWNoaW5lX29wdHMgPSAweGZmZmZmZmZlN2ZmZmZmZmYKICAgICAgICBpY291 bnRfb3B0cyA9IDB4MAogICAgICAgIG9saXN0ID0gMHg3ZmE5Y2YwM2IxNDAgPHFlbXVfbWFj aGluZV9vcHRzPgogICAgICAgIG9wdGluZCA9IDEyCiAgICAgICAgb3B0YXJnID0gMHg3ZmE5 ZDA3OTBmNDAgIm5vbmUiCiAgICAgICAgbG9hZHZtID0gMHgwCiAgICAgICAgbWFjaGluZV9j bGFzcyA9IDB4N2ZhOWQwNzdhMTYwCiAgICAgICAgY3B1X21vZGVsID0gMHgwCiAgICAgICAg dmdhX21vZGVsID0gMHgwCiAgICAgICAgcXRlc3RfY2hyZGV2ID0gMHgwCiAgICAgICAgcXRl c3RfbG9nID0gMHgwCiAgICAgICAgcGlkX2ZpbGUgPSAweDdmZmYwM2Y4ZmY1OSAiL3Zhci9s aWIvbGlidmlydC9xZW11L2NhcGFiaWxpdGllcy5waWRmaWxlIgogICAgICAgIGluY29taW5n ID0gMHgwCiAgICAgICAgc2hvd192bmNfcG9ydCA9IDAKICAgICAgICBkZWZjb25maWcgPSB0 cnVlCiAgICAgICAgdXNlcmNvbmZpZyA9IGZhbHNlCiAgICAgICAgbG9nX21hc2sgPSAweDAK ICAgICAgICBsb2dfZmlsZSA9IDB4MAogICAgICAgIG1lbV90cmFjZSA9IHttYWxsb2MgPSAw eDdmYTljZTkyNzZhMiA8bWFsbG9jX2FuZF90cmFjZT4sIHJlYWxsb2MgPSAweDdmYTljZTky NzZkNyA8cmVhbGxvY19hbmRfdHJhY2U+LCBmcmVlID0gMHg3ZmE5Y2U5Mjc3MWIgPGZyZWVf YW5kX3RyYWNlPiwgY2FsbG9jID0gMHgwLCB0cnlfbWFsbG9jID0gMHgwLCB0cnlfcmVhbGxv YyA9IDB4MH0KICAgICAgICB0cmFjZV9ldmVudHMgPSAweDAKICAgICAgICB0cmFjZV9maWxl ID0gMHgwCiAgICAgICAgbWF4cmFtX3NpemUgPSAxMzQyMTc3MjgKICAgICAgICByYW1fc2xv dHMgPSAwCiAgICAgICAgdm1zdGF0ZV9kdW1wX2ZpbGUgPSAweDAKICAgICAgICBtYWluX2xv b3BfZXJyID0gMHgwCiAgICAgICAgZXJyID0gMHgwCiAgICAgICAgX19mdW5jX18gPSAibWFp biIKCg== --------------040400080604060006080309 Content-Type: text/plain; name="child.txt" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="child.txt" VUlEICAgICAgICBQSUQgIFBQSUQgIEMgU1RJTUUgVFRZICAgICAgICAgIFRJTUUgQ01ECnFl bXUgICAgIDE3MzA3IDE3MzA1ICAwIDEzOjI0ID8gICAgICAgIDAwOjAwOjAwIC9vcHQvcWVt dS1pbnN0YWxsZWQvYmluL3FlbXUtc3lzdGVtLWkzODYgLVMgLW5vLXVzZXItY29uZmlnIC1u b2RlZmF1bHRzIC1ub2dyYXBoaWMgLU0gbm9uZSAtcW1wIHVuaXg6L3Zhci9saWIvbGlidmly dC9xZW11L2NhcGFiaWxpdGllcy5tb25pdG9yLnNvY2ssc2VydmVyLG5vd2FpdCAtcGlkZmls ZSAvdmFyL2xpYi9saWJ2aXJ0L3FlbXUvY2FwYWJpbGl0aWVzLnBpZGZpbGUgLWRhZW1vbml6 ZQoKKGdkYikgdGhyZWFkIGFwcGx5IGFsbCBidCBmdWxsCgpUaHJlYWQgMSAoVGhyZWFkIDB4 N2ZhOWNlNmYyYmMwIChMV1AgMTczMDcpKToKIzAgIDB4MDAwMDdmYTljZDMwNGY3ZCBpbiBf X2xsbF9sb2NrX3dhaXQgKCkgZnJvbSAvbGliNjQvbGlicHRocmVhZC5zby4wCk5vIHN5bWJv bCB0YWJsZSBpbmZvIGF2YWlsYWJsZS4KIzEgIDB4MDAwMDdmYTljZDMwMGQzMiBpbiBfTF9s b2NrXzc5MSAoKSBmcm9tIC9saWI2NC9saWJwdGhyZWFkLnNvLjAKTm8gc3ltYm9sIHRhYmxl IGluZm8gYXZhaWxhYmxlLgojMiAgMHgwMDAwN2ZhOWNkMzAwYzM4IGluIHB0aHJlYWRfbXV0 ZXhfbG9jayAoKSBmcm9tIC9saWI2NC9saWJwdGhyZWFkLnNvLjAKTm8gc3ltYm9sIHRhYmxl IGluZm8gYXZhaWxhYmxlLgojMyAgMHgwMDAwN2ZhOWNlYmMwYWQxIGluIHFlbXVfbXV0ZXhf bG9jayAobXV0ZXg9MHg3ZmE5Y2Y1MjQ1NjAgPHJjdV9zeW5jX2xvY2s+KSBhdCB1dGlsL3Fl bXUtdGhyZWFkLXBvc2l4LmM6NzMKICAgICAgICBlcnIgPSAwCiAgICAgICAgX19mdW5jX18g PSAicWVtdV9tdXRleF9sb2NrIgojNCAgMHgwMDAwN2ZhOWNlYmQ0OTFhIGluIHJjdV9pbml0 X2xvY2sgKCkgYXQgdXRpbC9yY3UuYzozMjkKTm8gbG9jYWxzLgojNSAgMHgwMDAwN2ZhOWM3 ZGE3NTEyIGluIGZvcmsgKCkgZnJvbSAvbGliNjQvbGliYy5zby42Ck5vIHN5bWJvbCB0YWJs ZSBpbmZvIGF2YWlsYWJsZS4KIzYgIDB4MDAwMDdmYTljZTkxNWNlZiBpbiBvc19kYWVtb25p emUgKCkgYXQgb3MtcG9zaXguYzoyNDAKICAgICAgICBwaWQgPSAwCiAgICAgICAgZmRzID0g ezQsIDV9CiM3ICAweDAwMDA3ZmE5Y2U5MmE4MDMgaW4gbWFpbiAoYXJnYz0xMiwgYXJndj0w eDdmZmYwM2Y4ZWZkOCwgZW52cD0weDdmZmYwM2Y4ZjA0MCkgYXQgdmwuYzo0MDM0CiAgICAg ICAgaSA9IDAKICAgICAgICBzbmFwc2hvdCA9IDAKICAgICAgICBsaW51eF9ib290ID0gMAog ICAgICAgIGluaXRyZF9maWxlbmFtZSA9IDB4N2ZhOWQwNzQ5ZWIwICLua8XOqVwxNzciCiAg ICAgICAga2VybmVsX2ZpbGVuYW1lID0gMHg3ZmE5ZDA3NDllYTAgIiIKICAgICAgICBrZXJu ZWxfY21kbGluZSA9IDB4N2ZhOWNlYmQ0ZTIwIDxfX2xpYmNfY3N1X2luaXQ+ICJBV0FcMjEx /0FWSVwyMTH2QVVJXDIxMdVBVExcMjE1JSIKICAgICAgICBib290X29yZGVyID0gMHgwCiAg ICAgICAgYm9vdF9vbmNlID0gMHgwCiAgICAgICAgZHMgPSAweDdmYTljZWM1NmQzOAogICAg ICAgIGN5bHMgPSAwCiAgICAgICAgaGVhZHMgPSAwCiAgICAgICAgc2VjcyA9IDAKICAgICAg ICB0cmFuc2xhdGlvbiA9IDAKICAgICAgICBoZGFfb3B0cyA9IDB4MAogICAgICAgIG9wdHMg PSAweDdmYTlkMDc5MGU5MAogICAgICAgIG1hY2hpbmVfb3B0cyA9IDB4ZmZmZmZmZmU3ZmZm ZmZmZgogICAgICAgIGljb3VudF9vcHRzID0gMHgwCiAgICAgICAgb2xpc3QgPSAweDdmYTlj ZjAzYjE0MCA8cWVtdV9tYWNoaW5lX29wdHM+CiAgICAgICAgb3B0aW5kID0gMTIKICAgICAg ICBvcHRhcmcgPSAweDdmYTlkMDc5MGY0MCAibm9uZSIKICAgICAgICBsb2Fkdm0gPSAweDAK ICAgICAgICBtYWNoaW5lX2NsYXNzID0gMHg3ZmE5ZDA3N2ExNjAKICAgICAgICBjcHVfbW9k ZWwgPSAweDAKICAgICAgICB2Z2FfbW9kZWwgPSAweDAKICAgICAgICBxdGVzdF9jaHJkZXYg PSAweDAKICAgICAgICBxdGVzdF9sb2cgPSAweDAKICAgICAgICBwaWRfZmlsZSA9IDB4N2Zm ZjAzZjhmZjU5ICIvdmFyL2xpYi9saWJ2aXJ0L3FlbXUvY2FwYWJpbGl0aWVzLnBpZGZpbGUi CiAgICAgICAgaW5jb21pbmcgPSAweDAKICAgICAgICBzaG93X3ZuY19wb3J0ID0gMAogICAg ICAgIGRlZmNvbmZpZyA9IHRydWUKICAgICAgICB1c2VyY29uZmlnID0gZmFsc2UKICAgICAg ICBsb2dfbWFzayA9IDB4MAogICAgICAgIGxvZ19maWxlID0gMHgwCiAgICAgICAgbWVtX3Ry YWNlID0ge21hbGxvYyA9IDB4N2ZhOWNlOTI3NmEyIDxtYWxsb2NfYW5kX3RyYWNlPiwgcmVh bGxvYyA9IDB4N2ZhOWNlOTI3NmQ3IDxyZWFsbG9jX2FuZF90cmFjZT4sIGZyZWUgPSAweDdm YTljZTkyNzcxYiA8ZnJlZV9hbmRfdHJhY2U+LCBjYWxsb2MgPSAweDAsIHRyeV9tYWxsb2Mg PSAweDAsIHRyeV9yZWFsbG9jID0gMHgwfQogICAgICAgIHRyYWNlX2V2ZW50cyA9IDB4MAog ICAgICAgIHRyYWNlX2ZpbGUgPSAweDAKICAgICAgICBtYXhyYW1fc2l6ZSA9IDEzNDIxNzcy OAogICAgICAgIHJhbV9zbG90cyA9IDAKICAgICAgICB2bXN0YXRlX2R1bXBfZmlsZSA9IDB4 MAogICAgICAgIG1haW5fbG9vcF9lcnIgPSAweDAKICAgICAgICBlcnIgPSAweDAKICAgICAg ICBfX2Z1bmNfXyA9ICJtYWluIgo= --------------040400080604060006080309--