From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dulloor Subject: Re: RE: [PROPOSAL] Doing work in idle-vcpu context Date: Mon, 19 Apr 2010 02:08:10 -0400 Message-ID: References: <789F9655DD1B8F43B48D77C5D30659731D797DC3@shsmsx501.ccr.corp.intel.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=0016e6d99ead194a6f048490c9f1 Return-path: In-Reply-To: <789F9655DD1B8F43B48D77C5D30659731D797DC3@shsmsx501.ccr.corp.intel.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Jiang, Yunhong" Cc: George Dunlap , "xen-devel@lists.xensource.com" , Keir Fraser List-Id: xen-devel@lists.xenproject.org --0016e6d99ead194a6f048490c9f1 Content-Type: text/plain; charset=ISO-8859-1 On Mon, Apr 19, 2010 at 1:55 AM, Jiang, Yunhong wrote: > > >>-----Original Message----- >>From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] >>Sent: Saturday, April 17, 2010 2:06 AM >>To: Jiang, Yunhong; George Dunlap; xen-devel@lists.xensource.com >>Subject: [PROPOSAL] Doing work in idle-vcpu context >> >>George, Yunhong, and others, >> >>So, it seems that runing stop_machine_run(), and now >>continue_hypercall_on_cpu(), in softirq context is a bit of a problem. >>Because the softirq can stop the currently-running vcpu from being >>descheduled we can end up with subtle deadlocks. For example, with s_m_r() >>we try to rendezvous all cpus in softirq context -- we can have CPU A enter >>the softirq interrupting VCPU X, meanwhile VCPU Y on CPU B is spinning >>trying to pause VCPU X. Hence CPU B doesn't get into softirq, and so CPU A >>never leaves it, and we have deadlock. >> >>There are various possible solutions to this, but one of the architecturally >>neatest would be to run the s_m_r() and c_h_o_c() work in a >>'Linux-workqueue' type of environment -- i.e., in a proper non-guest vcpu >>context. Rather than introducing the whole kthread concept into Xen, one >>possibility would be to schedule this work on the idle vcpus -- effectively >>promoting idle vcpus to a more general kind of 'Xen worker vcpu' whose job >>can include running the idle loop. >> >>One bit of mechanism this would require is the ability to bump the idle vcpu >>priority up - preferably to 'max' priority forcing it to run next until we >>return it to idle/lowest priority. George: how hard would such a mechanism >>be to implement do you think? >> >>More generally: what do people think of this idea? > > The only concern from me is, are there any assumption in other components that idle > vcpu is always for idle, and is always lowest priority? Using the idle_domain as a worker_domain sounds a good idea. And, bumping the credit up doesn't seem to be too difficult. I have attached a quickly whipped working patch (with a test driver) for this. Not many scheduler changes. I have looked at all the other places for idle_vcpu and PRI_IDLE too and they look fine to me. Keir, is this similar to what you are looking for ? > > --jyh > >> >> Thanks, >> Keir >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > --0016e6d99ead194a6f048490c9f1 Content-Type: text/x-patch; charset=US-ASCII; name="workqueue.patch" Content-Disposition: attachment; filename="workqueue.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_g86vqq1b0 ZGlmZiAtciA3ZWU4YmI0MDIwMGEgeGVuL2FyY2gveDg2L2RvbWFpbi5jCi0tLSBhL3hlbi9hcmNo L3g4Ni9kb21haW4uYwlUaHUgQXByIDE1IDE5OjExOjE2IDIwMTAgKzAxMDAKKysrIGIveGVuL2Fy Y2gveDg2L2RvbWFpbi5jCU1vbiBBcHIgMTkgMDI6MDE6MDYgMjAxMCAtMDQwMApAQCAtNTEsNiAr NTEsNyBAQAogI2luY2x1ZGUgPGFzbS9ubWkuaD4KICNpbmNsdWRlIDx4ZW4vbnVtYS5oPgogI2lu Y2x1ZGUgPHhlbi9pb21tdS5oPgorI2luY2x1ZGUgPHhlbi93b3JrcXVldWUuaD4KICNpZmRlZiBD T05GSUdfQ09NUEFUCiAjaW5jbHVkZSA8Y29tcGF0L3ZjcHUuaD4KICNlbmRpZgpAQCAtMTE2LDEy ICsxMTcsNTggQEAKICAgICAoKmRlYWRfaWRsZSkoKTsKIH0KIAorLyogeGVuX3dvcmsgaW1wbGVt ZW50YXRpb24gKi8KK0RFRklORV9QRVJfQ1BVKHN0cnVjdCB4ZW5fd29ya3F1ZXVlX3N0cnVjdCwg eGVuX3dvcmtxdWV1ZSk7CisKK3ZvaWQgaW5pdF94ZW5fd29ya3F1ZXVlcyh2b2lkKQoreworCWlu dCBpOworCWZvcl9lYWNoX3Bvc3NpYmxlX2NwdShpKQorCXsKKwkJc3RydWN0IHhlbl93b3JrcXVl dWVfc3RydWN0ICp3cSA9ICZwZXJfY3B1KHhlbl93b3JrcXVldWUsIGkpOworCQlJTklUX0xJU1Rf SEVBRCgmd3EtPmxpc3QpOworCQlzcGluX2xvY2tfaW5pdCgmd3EtPmxvY2spOworCX0KK30KKwor aW50IHNjaGVkdWxlX3hlbl93b3JrX29uKGludCBjcHUsIHN0cnVjdCB4ZW5fd29ya19zdHJ1Y3Qg KndvcmspCit7CisJc3RydWN0IHhlbl93b3JrcXVldWVfc3RydWN0ICp3cSA9ICZwZXJfY3B1KHhl bl93b3JrcXVldWUsIGNwdSk7CisKKwlzcGluX2xvY2soJndxLT5sb2NrKTsKKwlsaXN0X2FkZF90 YWlsKCZ3b3JrLT5lbnRyeSwgJndxLT5saXN0KTsKKwlzcGluX3VubG9jaygmd3EtPmxvY2spOwor CisJcmV0dXJuIDA7Cit9CisKKy8qIGV4ZWN1dGUgeGVuX3dvcmsgcXVldWVkIG9uIHhlbl93b3Jr cXVldWVbY3B1XSAKKyAqIFhYWDogY2FsbGVkIG9ubHkgZnJvbSBpZGxlX2xvb3AgKi8KK3N0YXRp YyB2b2lkIHByb2Nlc3NfeGVuX3dvcmtxdWV1ZSh2b2lkKQoreworCXN0cnVjdCB4ZW5fd29ya3F1 ZXVlX3N0cnVjdCAqd3EgPSAmdGhpc19jcHUoeGVuX3dvcmtxdWV1ZSk7CisKKwlzcGluX2xvY2so JndxLT5sb2NrKTsKKwl3aGlsZSAoIWxpc3RfZW1wdHkoJndxLT5saXN0KSkKKwl7CisJCXN0cnVj dCB4ZW5fd29ya19zdHJ1Y3QgKndvcmsgPSBsaXN0X2VudHJ5KHdxLT5saXN0Lm5leHQsCisJCQkJ CQkJCQkJc3RydWN0IHhlbl93b3JrX3N0cnVjdCwgZW50cnkpOworCQl4ZW5fd29ya19mdW5jX3Qg ZnVuYyA9IHdvcmstPmZ1bmM7CisJCWxpc3RfZGVsX2luaXQod3EtPmxpc3QubmV4dCk7CisJCXNw aW5fdW5sb2NrKCZ3cS0+bG9jayk7CisJCWZ1bmMod29yayk7CisJCXNwaW5fbG9jaygmd3EtPmxv Y2spOworCX0KKwlzcGluX3VubG9jaygmd3EtPmxvY2spOworfQorCiB2b2lkIGlkbGVfbG9vcCh2 b2lkKQogewogICAgIGZvciAoIDsgOyApCiAgICAgewogICAgICAgICBpZiAoIGNwdV9pc19vZmZs aW5lKHNtcF9wcm9jZXNzb3JfaWQoKSkgKQogICAgICAgICAgICAgcGxheV9kZWFkKCk7CisJCXBy b2Nlc3NfeGVuX3dvcmtxdWV1ZSgpOwogICAgICAgICAoKnBtX2lkbGUpKCk7CiAgICAgICAgIGRv X3NvZnRpcnEoKTsKICAgICB9CmRpZmYgLXIgN2VlOGJiNDAyMDBhIHhlbi9hcmNoL3g4Ni9zZXR1 cC5jCi0tLSBhL3hlbi9hcmNoL3g4Ni9zZXR1cC5jCVRodSBBcHIgMTUgMTk6MTE6MTYgMjAxMCAr MDEwMAorKysgYi94ZW4vYXJjaC94ODYvc2V0dXAuYwlNb24gQXByIDE5IDAyOjAxOjA2IDIwMTAg LTA0MDAKQEAgLTIxLDYgKzIxLDcgQEAKICNpbmNsdWRlIDx4ZW4vdmdhLmg+CiAjaW5jbHVkZSA8 eGVuL2RtaS5oPgogI2luY2x1ZGUgPHhlbi9ub2RlbWFzay5oPgorI2luY2x1ZGUgPHhlbi93b3Jr cXVldWUuaD4KICNpbmNsdWRlIDxwdWJsaWMvdmVyc2lvbi5oPgogI2lmZGVmIENPTkZJR19DT01Q QVQKICNpbmNsdWRlIDxjb21wYXQvcGxhdGZvcm0uaD4KQEAgLTI0Miw2ICsyNDMsNyBAQAogewog ICAgIHN0cnVjdCBkb21haW4gKmlkbGVfZG9tYWluOwogCisJaW5pdF94ZW5fd29ya3F1ZXVlcygp OwogICAgIC8qIERvbWFpbiBjcmVhdGlvbiByZXF1aXJlcyB0aGF0IHNjaGVkdWxlciBzdHJ1Y3R1 cmVzIGFyZSBpbml0aWFsaXNlZC4gKi8KICAgICBzY2hlZHVsZXJfaW5pdCgpOwogCmRpZmYgLXIg N2VlOGJiNDAyMDBhIHhlbi9jb21tb24vc2NoZWRfY3JlZGl0LmMKLS0tIGEveGVuL2NvbW1vbi9z Y2hlZF9jcmVkaXQuYwlUaHUgQXByIDE1IDE5OjExOjE2IDIwMTAgKzAxMDAKKysrIGIveGVuL2Nv bW1vbi9zY2hlZF9jcmVkaXQuYwlNb24gQXByIDE5IDAyOjAxOjA2IDIwMTAgLTA0MDAKQEAgLTIy LDYgKzIyLDcgQEAKICNpbmNsdWRlIDxhc20vYXRvbWljLmg+CiAjaW5jbHVkZSA8eGVuL2Vycm5v Lmg+CiAjaW5jbHVkZSA8eGVuL2tleWhhbmRsZXIuaD4KKyNpbmNsdWRlIDx4ZW4vd29ya3F1ZXVl Lmg+CiAKIC8qCiAgKiBDU0NIRURfU1RBVFMKQEAgLTExNzMsNiArMTE3NCwxMSBAQAogICAgICAg ICBidXJuX2NyZWRpdHMoc2N1cnIsIG5vdyk7CiAgICAgICAgIHNjdXJyLT5zdGFydF90aW1lIC09 IG5vdzsKICAgICB9CisJZWxzZQorCXsgLyogaWRsZSB2Y3B1IC0gcmV2ZXJ0IHRoZSBwcmlvcml0 eSwgaWYgbmVlZGVkICovCisJCWlmIChzY3Vyci0+cHJpICE9IENTQ0hFRF9QUklfSURMRSkKKwkJ CXNjdXJyLT5wcmkgPSBDU0NIRURfUFJJX0lETEU7CisJfQogCiAgICAgLyoKICAgICAgKiBTZWxl Y3QgbmV4dCBydW5uYWJsZSBsb2NhbCBWQ1BVIChpZSB0b3Agb2YgbG9jYWwgcnVucSkKQEAgLTEx ODIsNyArMTE4OCwxNSBAQAogICAgIGVsc2UKICAgICAgICAgQlVHX09OKCBpc19pZGxlX3ZjcHUo Y3VycmVudCkgfHwgbGlzdF9lbXB0eShydW5xKSApOwogCi0gICAgc25leHQgPSBfX3J1bnFfZWxl bShydW5xLT5uZXh0KTsKKwkvKiBUaGVyZSBpcyB3b3JrIHRvIGJlIGRvbmUgKGluIGlkbGVfdmNw dSBjb250ZXh0KS4gVGVtcG9yYXJpbHkgYm9vc3QKKwkgKiB0aGUgcHJpb3JpdHkgb2YgaWRsZV92 Y3B1W2NwdV0gYW5kIHNjaGVkdWxlIGl0ICovCisJaWYgKCF4ZW5fd29ya3F1ZXVlX2VtcHR5KGNw dSkpCisJeworCQlzbmV4dCA9IENTQ0hFRF9WQ1BVKGlkbGVfdmNwdVtjcHVdKTsKKwkJc25leHQt PnByaSA9IENTQ0hFRF9QUklfVFNfQk9PU1Q7CisJfQorCWVsc2UKKyAgICAJc25leHQgPSBfX3J1 bnFfZWxlbShydW5xLT5uZXh0KTsKIAogICAgIC8qCiAgICAgICogU01QIExvYWQgYmFsYW5jZToK QEAgLTEyMTcsNyArMTIzMSw3IEBACiAgICAgLyoKICAgICAgKiBSZXR1cm4gdGFzayB0byBydW4g bmV4dC4uLgogICAgICAqLwotICAgIHJldC50aW1lID0gKGlzX2lkbGVfdmNwdShzbmV4dC0+dmNw dSkgPworICAgIHJldC50aW1lID0gKChpc19pZGxlX3ZjcHUoc25leHQtPnZjcHUpICYmIChzbmV4 dC0+cHJpPT1DU0NIRURfUFJJX0lETEUpKSA/CiAgICAgICAgICAgICAgICAgLTEgOiBNSUxMSVNF Q1MoQ1NDSEVEX01TRUNTX1BFUl9UU0xJQ0UpKTsKICAgICByZXQudGFzayA9IHNuZXh0LT52Y3B1 OwogCmRpZmYgLXIgN2VlOGJiNDAyMDBhIHhlbi9jb21tb24vc2NoZWR1bGUuYwotLS0gYS94ZW4v Y29tbW9uL3NjaGVkdWxlLmMJVGh1IEFwciAxNSAxOToxMToxNiAyMDEwICswMTAwCisrKyBiL3hl bi9jb21tb24vc2NoZWR1bGUuYwlNb24gQXByIDE5IDAyOjAxOjA2IDIwMTAgLTA0MDAKQEAgLTMx LDYgKzMxLDcgQEAKICNpbmNsdWRlIDx4ZW4vZXJybm8uaD4KICNpbmNsdWRlIDx4ZW4vZ3Vlc3Rf YWNjZXNzLmg+CiAjaW5jbHVkZSA8eGVuL211bHRpY2FsbC5oPgorI2luY2x1ZGUgPHhlbi93b3Jr cXVldWUuaD4KICNpbmNsdWRlIDxwdWJsaWMvc2NoZWQuaD4KICNpbmNsdWRlIDx4c20veHNtLmg+ CiAKQEAgLTk1NSwxMiArOTU2LDM1IEBACiAgICAgU0NIRURfT1AoaW5pdCk7CiB9CiAKK3N0YXRp YyBzdHJ1Y3QgeGVuX3dvcmtfc3RydWN0IGR1bW15X3dvcmtbTlJfQ1BVU107CitzdGF0aWMgdm9p ZCBkdW1teV94ZW5fd29ya19mdW5jKHN0cnVjdCB4ZW5fd29ya19zdHJ1Y3QgKndvcmspCit7Cisg ICAgdW5zaWduZWQgaW50IGNwdSA9IHNtcF9wcm9jZXNzb3JfaWQoKTsKKwlzX3RpbWVfdCB0aGVu ID0gKHNfdGltZV90KXdvcmstPmRhdGE7CisJc190aW1lX3Qgbm93ID0gTk9XKCk7CisJcHJpbnRr KCJYRU5fV09SS19DUFVbJTAyZF0gOiAlbHUgLT4gJWx1ICglbHUpXG4iLCBjcHUsIAorCQkJKHVu c2lnbmVkIGxvbmcpdGhlbiwgKHVuc2lnbmVkIGxvbmcpbm93LCAodW5zaWduZWQgbG9uZykobm93 LXRoZW4pKTsKKwlyZXR1cm47Cit9CisKK3N0YXRpYyB2b2lkIHNjaGVkdWxlX3hlbl93b3JrX29u X2FsbCh2b2lkKQoreworCWludCBpOworCWZvcl9lYWNoX29ubGluZV9jcHUoaSkKKwl7CisJCUlO SVRfWEVOX1dPUksoJmR1bW15X3dvcmtbaV0sIGR1bW15X3hlbl93b3JrX2Z1bmMsIE5PVygpKTsK KwkJc2NoZWR1bGVfeGVuX3dvcmtfb24oaSwgJmR1bW15X3dvcmtbaV0pOworCX0KK30KKwogdm9p ZCBkdW1wX3J1bnEodW5zaWduZWQgY2hhciBrZXkpCiB7CiAgICAgc190aW1lX3QgICAgICBub3cg PSBOT1coKTsKICAgICBpbnQgICAgICAgICAgIGk7CiAgICAgdW5zaWduZWQgbG9uZyBmbGFnczsK IAorCXNjaGVkdWxlX3hlbl93b3JrX29uX2FsbCgpOworCiAgICAgbG9jYWxfaXJxX3NhdmUoZmxh Z3MpOwogCiAgICAgcHJpbnRrKCJTY2hlZHVsZXI6ICVzICglcylcbiIsIG9wcy5uYW1lLCBvcHMu b3B0X25hbWUpOwpkaWZmIC1yIDdlZThiYjQwMjAwYSB4ZW4vaW5jbHVkZS94ZW4vd29ya3F1ZXVl LmgKLS0tIC9kZXYvbnVsbAlUaHUgSmFuIDAxIDAwOjAwOjAwIDE5NzAgKzAwMDAKKysrIGIveGVu L2luY2x1ZGUveGVuL3dvcmtxdWV1ZS5oCU1vbiBBcHIgMTkgMDI6MDE6MDYgMjAxMCAtMDQwMApA QCAtMCwwICsxLDU2IEBACisjaWZuZGVmIF9fWEVOX1dPUktRVUVVRV9ICisjZGVmaW5lIF9fWEVO X1dPUktRVUVVRV9ICisKKyNpbmNsdWRlIDx4ZW4vbGlzdC5oPgorI2luY2x1ZGUgPHhlbi9zcGlu bG9jay5oPgorI2luY2x1ZGUgPHhlbi9jb25maWcuaD4KKworLyogeGVuX3dvcmtxdWV1ZSBpcyBh IHBlci1jcHUgd29ya3F1ZXVlIHRocm91Z2ggd2hpY2ggd2UgY2FuIGFzc2lnbiB3b3JrCisgKiB0 byBhIGNwdSB1c2luZyB4ZW5fd29ya19zdHJ1Y3QuIFRoZSB3b3JrIGFjdHVhbGx5IGdldHMgZG9u ZSBpbiB0aGUgY29udGV4dAorICogb2YgaWRsZV9kb21haW4ncyB2Y3B1IChpbiBpZGxlX2xvb3Ap LiAqLworc3RydWN0IHhlbl93b3JrX3N0cnVjdDsKK3R5cGVkZWYgdm9pZCgqeGVuX3dvcmtfZnVu Y190KShzdHJ1Y3QgeGVuX3dvcmtfc3RydWN0ICp3b3JrKTsKK3N0cnVjdCB4ZW5fd29ya19zdHJ1 Y3QKK3sKKwlzdHJ1Y3QgbGlzdF9oZWFkIGVudHJ5OworCXVpbnQ2NF90IGRhdGE7CisJeGVuX3dv cmtfZnVuY190IGZ1bmM7Cit9OworCisKKyNkZWZpbmUgREVDTEFSRV9YRU5fV09SSyhfbikgCQkJ CQkJCQlcCisJc3RydWN0IHhlbl93b3JrX3N0cnVjdCBfbiA9IHsJCQkJCQkJXAorCQkJCQkuZW50 cnkgPSB7JihfbikuZW50cnksICYoX24pLmVudHJ5fSwJXAorCQkJCQkuZGF0YSA9IDAsIAkJCQkJ CQkJXAorCQkJCQkuZnVuYyA9IDAsCQkJCQkJCQlcCisJCQkJCX0KKworI2RlZmluZSBJTklUX1hF Tl9XT1JLKF93LCBfZiwgX2QpCQlcCisJZG97CQkJCQkJCQkJXAorCQlJTklUX0xJU1RfSEVBRCgm KF93KS0+ZW50cnkpOwlcCisJCShfdyktPmRhdGEgPSAoX2QpOwkJCQlcCisJCShfdyktPmZ1bmMg PSAoX2YpOwkJCQlcCisJfXdoaWxlKDApCisKK2V4dGVybiBpbnQgc2NoZWR1bGVfeGVuX3dvcmtf b24oaW50IGNwdSwgc3RydWN0IHhlbl93b3JrX3N0cnVjdCAqd29yayk7CisKK3N0cnVjdCB4ZW5f d29ya3F1ZXVlX3N0cnVjdCB7CisJc3RydWN0IGxpc3RfaGVhZCBsaXN0OworCXNwaW5sb2NrX3Qg bG9jazsKK307CitERUNMQVJFX1BFUl9DUFUoc3RydWN0IHhlbl93b3JrcXVldWVfc3RydWN0LCB4 ZW5fd29ya3F1ZXVlKTsKK2V4dGVybiB2b2lkIGluaXRfeGVuX3dvcmtxdWV1ZXModm9pZCk7CisK Ky8qIHNjaGVkdWxlciBtdXN0IGZpcnN0IGNoZWNrIGZvciBhbnkgcGVuZGluZyB3b3JrIGluIHhl bl93b3JrcXVldWUgKi8KK3N0YXRpYyBpbmxpbmUgaW50IHhlbl93b3JrcXVldWVfZW1wdHkoaW50 IGNwdSkKK3sKKwlpbnQgcmV0OworCS8vdW5zaWduZWQgbG9uZyBmbGFnczsKKwlzdHJ1Y3QgeGVu X3dvcmtxdWV1ZV9zdHJ1Y3QgKndxID0gJnBlcl9jcHUoeGVuX3dvcmtxdWV1ZSwgY3B1KTsKKwkv L3NwaW5fbG9ja19pcnFzYXZlKCZ3cS0+bG9jaywgZmxhZ3MpOworCXJldCA9IGxpc3RfZW1wdHko JndxLT5saXN0KTsKKwkvL3NwaW5fdW5sb2NrX2lycXJlc3RvcmUoJndxLT5sb2NrLCBmbGFncyk7 CisJcmV0dXJuIHJldDsKK30KKworI2VuZGlmCg== --0016e6d99ead194a6f048490c9f1 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --0016e6d99ead194a6f048490c9f1--