From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: Need help with fixing the Xen waitqueue feature Date: Wed, 23 Nov 2011 21:03:39 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="B_3404927025_64467266" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Olaf Hering Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3404927025_64467266 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit On 23/11/2011 19:21, "Keir Fraser" wrote: > On 23/11/2011 18:31, "Olaf Hering" wrote: > >> On Wed, Nov 23, Keir Fraser wrote: >> >>> We have quite a big waitqueue problem actually. The current scheme of >>> per-cpu stacks doesn't work nicely, as the stack pointer will change if a >>> vcpu goes to sleep and then wakes up on a different cpu. This really doesn't >>> work nicely with preempted C code, which may implement frame pointers and/or >>> arbitrarily take the address of on-stack variables. The result will be >>> hideous cross-stack corruptions, as these frame pointers and cached >>> addresses of automatic variables will reference the wrong cpu's stack! >>> Fixing or detecting this in general is not possible afaics. >> >> Yes, I was thinking about that wakeup on different cpu as well. >> As a quick fix/hack, perhaps the scheduler could make sure the vcpu >> wakes up on the same cpu? > > Could save old affinity and then vcpu_set_affinity. That will have to do for > now. Actually it should work okay as long as toolstack doesn't mess with > affinity meanwhile. I'll sort out a patch for this. Attached three patches for you to try. They apply in sequence. 00: A fixed version of "domain_crash on stack overflow" 01: Reorders prepare_to_wait so that the vcpu will always be on the waitqueue on exit (even if it has just been woken). 02: Ensures the vcpu wakes up on the same cpu that it slept on. We need all of these. Just need testing to make sure they aren't horribly broken. You should be able to test multi-processor host again with these. -- Keir > -- Keir > >> Olaf > > --B_3404927025_64467266 Content-type: application/octet-stream; name="00-prep-to-wait-dom-crash" Content-disposition: attachment; filename="00-prep-to-wait-dom-crash" Content-transfer-encoding: base64 IyBIRyBjaGFuZ2VzZXQgcGF0Y2gKIyBQYXJlbnQgODRiM2U0NmFhN2QyNGE0NjA1YzM2OTQw NjA2ZTdkYTk2NzliMGU3ZgoKZGlmZiAtciA4NGIzZTQ2YWE3ZDIgeGVuL2NvbW1vbi93YWl0 LmMKLS0tIGEveGVuL2NvbW1vbi93YWl0LmMJV2VkIE5vdiAyMyAxMjowMzozNyAyMDExICsw MDAwCisrKyBiL3hlbi9jb21tb24vd2FpdC5jCVdlZCBOb3YgMjMgMTk6NDM6MzUgMjAxMSAr MDAwMApAQCAtMTA2LDEzICsxMDYsMTYgQEAgdm9pZCB3YWtlX3VwKHN0cnVjdCB3YWl0cXVl dWVfaGVhZCAqd3EpCiBzdGF0aWMgdm9pZCBfX3ByZXBhcmVfdG9fd2FpdChzdHJ1Y3Qgd2Fp dHF1ZXVlX3ZjcHUgKndxdikKIHsKICAgICBjaGFyICpjcHVfaW5mbyA9IChjaGFyICopZ2V0 X2NwdV9pbmZvKCk7CisKICAgICBhc20gdm9sYXRpbGUgKAogI2lmZGVmIENPTkZJR19YODZf NjQKICAgICAgICAgInB1c2ggJSVyYXg7IHB1c2ggJSVyYng7IHB1c2ggJSVyY3g7IHB1c2gg JSVyZHg7IHB1c2ggJSVyZGk7ICIKICAgICAgICAgInB1c2ggJSVyYnA7IHB1c2ggJSVyODsg cHVzaCAlJXI5OyBwdXNoICUlcjEwOyBwdXNoICUlcjExOyAiCiAgICAgICAgICJwdXNoICUl cjEyOyBwdXNoICUlcjEzOyBwdXNoICUlcjE0OyBwdXNoICUlcjE1OyBjYWxsIDFmOyAiCiAg ICAgICAgICIxOiBtb3YgODAoJSVyc3ApLCUlcmRpOyBtb3YgOTYoJSVyc3ApLCUlcmN4OyBt b3YgJSVyc3AsJSVyc2k7ICIKLSAgICAgICAgInN1YiAlJXJzaSwlJXJjeDsgcmVwIG1vdnNi OyBtb3YgJSVyc3AsJSVyc2k7IHBvcCAlJXJheDsgIgorICAgICAgICAic3ViICUlcnNpLCUl cmN4OyBjbXAgJTMsJSVyY3g7IGpiZSAyZjsgIgorICAgICAgICAieG9yICUlZXNpLCUlZXNp OyBqbXAgM2Y7ICIKKyAgICAgICAgIjI6IHJlcCBtb3ZzYjsgbW92ICUlcnNwLCUlcnNpOyAz OiBwb3AgJSVyYXg7ICIKICAgICAgICAgInBvcCAlJXIxNTsgcG9wICUlcjE0OyBwb3AgJSVy MTM7IHBvcCAlJXIxMjsgIgogICAgICAgICAicG9wICUlcjExOyBwb3AgJSVyMTA7IHBvcCAl JXI5OyBwb3AgJSVyODsgIgogICAgICAgICAicG9wICUlcmJwOyBwb3AgJSVyZGk7IHBvcCAl JXJkeDsgcG9wICUlcmN4OyBwb3AgJSVyYng7IHBvcCAlJXJheCIKQEAgLTEyMCwxMyArMTIz LDIwIEBAIHN0YXRpYyB2b2lkIF9fcHJlcGFyZV90b193YWl0KHN0cnVjdCB3YWkKICAgICAg ICAgInB1c2ggJSVlYXg7IHB1c2ggJSVlYng7IHB1c2ggJSVlY3g7IHB1c2ggJSVlZHg7IHB1 c2ggJSVlZGk7ICIKICAgICAgICAgInB1c2ggJSVlYnA7IGNhbGwgMWY7ICIKICAgICAgICAg IjE6IG1vdiA4KCUlZXNwKSwlJWVkaTsgbW92IDE2KCUlZXNwKSwlJWVjeDsgbW92ICUlZXNw LCUlZXNpOyAiCi0gICAgICAgICJzdWIgJSVlc2ksJSVlY3g7IHJlcCBtb3ZzYjsgbW92ICUl ZXNwLCUlZXNpOyBwb3AgJSVlYXg7ICIKKyAgICAgICAgInN1YiAlJWVzaSwlJWVjeDsgY21w ICUzLCUlZWN4OyBqYmUgMmY7ICIKKyAgICAgICAgInhvciAlJWVzaSwlJWVzaTsgam1wIDNm OyAiCisgICAgICAgICIyOiByZXAgbW92c2I7IG1vdiAlJWVzcCwlJWVzaTsgMzogcG9wICUl ZWF4OyAiCiAgICAgICAgICJwb3AgJSVlYnA7IHBvcCAlJWVkaTsgcG9wICUlZWR4OyBwb3Ag JSVlY3g7IHBvcCAlJWVieDsgcG9wICUlZWF4IgogI2VuZGlmCiAgICAgICAgIDogIj1TIiAo d3F2LT5lc3ApCi0gICAgICAgIDogImMiIChjcHVfaW5mbyksICJEIiAod3F2LT5zdGFjaykK KyAgICAgICAgOiAiYyIgKGNwdV9pbmZvKSwgIkQiICh3cXYtPnN0YWNrKSwgImkiIChQQUdF X1NJWkUpCiAgICAgICAgIDogIm1lbW9yeSIgKTsKLSAgICBCVUdfT04oKGNwdV9pbmZvIC0g KGNoYXIgKil3cXYtPmVzcCkgPiBQQUdFX1NJWkUpOworCisgICAgaWYgKCB1bmxpa2VseSh3 cXYtPmVzcCA9PSAwKSApCisgICAgeworICAgICAgICBnZHByaW50ayhYRU5MT0dfRVJSLCAi U3RhY2sgdG9vIGxhcmdlIGluICVzXG4iLCBfX0ZVTkNUSU9OX18pOworICAgICAgICBkb21h aW5fY3Jhc2hfc3luY2hyb25vdXMoKTsKKyAgICB9CiB9CiAKIHN0YXRpYyB2b2lkIF9fZmlu aXNoX3dhaXQoc3RydWN0IHdhaXRxdWV1ZV92Y3B1ICp3cXYpCkBAIC0xNjIsNiArMTcyLDcg QEAgdm9pZCBwcmVwYXJlX3RvX3dhaXQoc3RydWN0IHdhaXRxdWV1ZV9oZQogICAgIHN0cnVj dCB2Y3B1ICpjdXJyID0gY3VycmVudDsKICAgICBzdHJ1Y3Qgd2FpdHF1ZXVlX3ZjcHUgKndx diA9IGN1cnItPndhaXRxdWV1ZV92Y3B1OwogCisgICAgQVNTRVJUKCFpbl9hdG9taWMoKSk7 CiAgICAgQVNTRVJUKGxpc3RfZW1wdHkoJndxdi0+bGlzdCkpOwogCiAgICAgc3Bpbl9sb2Nr KCZ3cS0+bG9jayk7Cg== --B_3404927025_64467266 Content-type: application/octet-stream; name="01-prep-to-wait-reorder" Content-disposition: attachment; filename="01-prep-to-wait-reorder" Content-transfer-encoding: base64 IyBIRyBjaGFuZ2VzZXQgcGF0Y2gKIyBQYXJlbnQgY2MwNTVkYWIzNjA2NTI5ZmNhYmMyODY4 NWM1YmYwZGViZGZjMjEzYwpkaWZmIC1yIGNjMDU1ZGFiMzYwNiAtciA4NDYyODI5MWU1ODUg eGVuL2NvbW1vbi93YWl0LmMKLS0tIGEveGVuL2NvbW1vbi93YWl0LmMJV2VkIE5vdiAyMyAx ODowMTo0NCAyMDExICswMDAwCisrKyBiL3hlbi9jb21tb24vd2FpdC5jCVdlZCBOb3YgMjMg MTg6MDM6NTUgMjAxMSArMDAwMApAQCAtMTA3LDYgKzEwNyw4IEBAIHN0YXRpYyB2b2lkIF9f cHJlcGFyZV90b193YWl0KHN0cnVjdCB3YWkKIHsKICAgICBjaGFyICpjcHVfaW5mbyA9IChj aGFyICopZ2V0X2NwdV9pbmZvKCk7CiAKKyAgICBBU1NFUlQod3F2LT5lc3AgPT0gMCk7CisK ICAgICBhc20gdm9sYXRpbGUgKAogI2lmZGVmIENPTkZJR19YODZfNjQKICAgICAgICAgInB1 c2ggJSVyYXg7IHB1c2ggJSVyYng7IHB1c2ggJSVyY3g7IHB1c2ggJSVyZHg7IHB1c2ggJSVy ZGk7ICIKQEAgLTE3MywxNCArMTc1LDEzIEBAIHZvaWQgcHJlcGFyZV90b193YWl0KHN0cnVj dCB3YWl0cXVldWVfaGUKICAgICBzdHJ1Y3Qgd2FpdHF1ZXVlX3ZjcHUgKndxdiA9IGN1cnIt PndhaXRxdWV1ZV92Y3B1OwogCiAgICAgQVNTRVJUKCFpbl9hdG9taWMoKSk7CisgICAgX19w cmVwYXJlX3RvX3dhaXQod3F2KTsKKwogICAgIEFTU0VSVChsaXN0X2VtcHR5KCZ3cXYtPmxp c3QpKTsKLQogICAgIHNwaW5fbG9jaygmd3EtPmxvY2spOwogICAgIGxpc3RfYWRkX3RhaWwo Jndxdi0+bGlzdCwgJndxLT5saXN0KTsKICAgICB2Y3B1X3BhdXNlX25vc3luYyhjdXJyKTsK ICAgICBzcGluX3VubG9jaygmd3EtPmxvY2spOwotCi0gICAgX19wcmVwYXJlX3RvX3dhaXQo d3F2KTsKIH0KIAogdm9pZCBmaW5pc2hfd2FpdChzdHJ1Y3Qgd2FpdHF1ZXVlX2hlYWQgKndx KQo= --B_3404927025_64467266 Content-type: application/octet-stream; name="02-waitq-set-vcpu-affinity" Content-disposition: attachment; filename="02-waitq-set-vcpu-affinity" Content-transfer-encoding: base64 IyBIRyBjaGFuZ2VzZXQgcGF0Y2gKIyBQYXJlbnQgMThhMWYzNWFmMWM1NWRkYmQ4N2JkMzlh OTMxN2MzOGZmYTJhMWY3YQpkaWZmIC1yIDE4YTFmMzVhZjFjNSAtciAwZjIyMDY0ZDk4YWUg eGVuL2NvbW1vbi93YWl0LmMKLS0tIGEveGVuL2NvbW1vbi93YWl0LmMJV2VkIE5vdiAyMyAx OTo0Mzo0NiAyMDExICswMDAwCisrKyBiL3hlbi9jb21tb24vd2FpdC5jCVdlZCBOb3YgMjMg MjA6MDI6MDUgMjAxMSArMDAwMApAQCAtMzQsNiArMzQsOCBAQCBzdHJ1Y3Qgd2FpdHF1ZXVl X3ZjcHUgewogICAgICAqLwogICAgIHZvaWQgKmVzcDsKICAgICBjaGFyICpzdGFjazsKKyAg ICBjcHVtYXNrX3Qgc2F2ZWRfYWZmaW5pdHk7CisgICAgdW5zaWduZWQgaW50IHdha2V1cF9j cHU7CiAjZW5kaWYKIH07CiAKQEAgLTEwNiw5ICsxMDgsMTkgQEAgdm9pZCB3YWtlX3VwKHN0 cnVjdCB3YWl0cXVldWVfaGVhZCAqd3EpCiBzdGF0aWMgdm9pZCBfX3ByZXBhcmVfdG9fd2Fp dChzdHJ1Y3Qgd2FpdHF1ZXVlX3ZjcHUgKndxdikKIHsKICAgICBjaGFyICpjcHVfaW5mbyA9 IChjaGFyICopZ2V0X2NwdV9pbmZvKCk7CisgICAgc3RydWN0IHZjcHUgKmN1cnIgPSBjdXJy ZW50OwogCiAgICAgQVNTRVJUKHdxdi0+ZXNwID09IDApOwogCisgICAgLyogU2F2ZSBjdXJy ZW50IFZDUFUgYWZmaW5pdHk7IGZvcmNlIHdha2V1cCBvbiAqdGhpcyogQ1BVIG9ubHkuICov CisgICAgd3F2LT53YWtldXBfY3B1ID0gc21wX3Byb2Nlc3Nvcl9pZCgpOworICAgIGNwdW1h c2tfY29weSgmd3F2LT5zYXZlZF9hZmZpbml0eSwgY3Vyci0+Y3B1X2FmZmluaXR5KTsKKyAg ICBpZiAoIHZjcHVfc2V0X2FmZmluaXR5KGN1cnIsIGNwdW1hc2tfb2Yod3F2LT53YWtldXBf Y3B1KSkgKQorICAgIHsKKyAgICAgICAgZ2RwcmludGsoWEVOTE9HX0VSUiwgIlVuYWJsZSB0 byBzZXQgdmNwdSBhZmZpbml0eVxuIik7CisgICAgICAgIGRvbWFpbl9jcmFzaF9zeW5jaHJv bm91cygpOworICAgIH0KKwogICAgIGFzbSB2b2xhdGlsZSAoCiAjaWZkZWYgQ09ORklHX1g4 Nl82NAogICAgICAgICAicHVzaCAlJXJheDsgcHVzaCAlJXJieDsgcHVzaCAlJXJjeDsgcHVz aCAlJXJkeDsgcHVzaCAlJXJkaTsgIgpAQCAtMTQ0LDYgKzE1Niw3IEBAIHN0YXRpYyB2b2lk IF9fcHJlcGFyZV90b193YWl0KHN0cnVjdCB3YWkKIHN0YXRpYyB2b2lkIF9fZmluaXNoX3dh aXQoc3RydWN0IHdhaXRxdWV1ZV92Y3B1ICp3cXYpCiB7CiAgICAgd3F2LT5lc3AgPSBOVUxM OworICAgICh2b2lkKXZjcHVfc2V0X2FmZmluaXR5KGN1cnJlbnQsICZ3cXYtPnNhdmVkX2Fm ZmluaXR5KTsKIH0KIAogdm9pZCBjaGVja193YWtldXBfZnJvbV93YWl0KHZvaWQpCkBAIC0x NTUsNiArMTY4LDIwIEBAIHZvaWQgY2hlY2tfd2FrZXVwX2Zyb21fd2FpdCh2b2lkKQogICAg IGlmICggbGlrZWx5KHdxdi0+ZXNwID09IE5VTEwpICkKICAgICAgICAgcmV0dXJuOwogCisg ICAgLyogQ2hlY2sgaWYgd2Ugd29rZSB1cCBvbiB0aGUgd3JvbmcgQ1BVLiAqLworICAgIGlm ICggdW5saWtlbHkoc21wX3Byb2Nlc3Nvcl9pZCgpICE9IHdxdi0+d2FrZXVwX2NwdSkgKQor ICAgIHsKKyAgICAgICAgLyogUmUtc2V0IFZDUFUgYWZmaW5pdHkgYW5kIHJlLWVudGVyIHRo ZSBzY2hlZHVsZXIuICovCisgICAgICAgIHN0cnVjdCB2Y3B1ICpjdXJyID0gY3VycmVudDsK KyAgICAgICAgY3B1bWFza19jb3B5KCZ3cXYtPnNhdmVkX2FmZmluaXR5LCBjdXJyLT5jcHVf YWZmaW5pdHkpOworICAgICAgICBpZiAoIHZjcHVfc2V0X2FmZmluaXR5KGN1cnIsIGNwdW1h c2tfb2Yod3F2LT53YWtldXBfY3B1KSkgKQorICAgICAgICB7CisgICAgICAgICAgICBnZHBy aW50ayhYRU5MT0dfRVJSLCAiVW5hYmxlIHRvIHNldCB2Y3B1IGFmZmluaXR5XG4iKTsKKyAg ICAgICAgICAgIGRvbWFpbl9jcmFzaF9zeW5jaHJvbm91cygpOworICAgICAgICB9CisgICAg ICAgIHdhaXQoKTsgLyogdGFrZXMgdXMgYmFjayBpbnRvIHRoZSBzY2hlZHVsZXIgKi8KKyAg ICB9CisKICAgICBhc20gdm9sYXRpbGUgKAogICAgICAgICAibW92ICUxLCUlIl9fT1Aic3A7 IHJlcCBtb3ZzYjsgam1wICooJSUiX19PUCJzcCkiCiAgICAgICAgIDogOiAiUyIgKHdxdi0+ c3RhY2spLCAiRCIgKHdxdi0+ZXNwKSwK --B_3404927025_64467266 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --B_3404927025_64467266--