From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: [BUG] mistakenly wake in Xen's credit scheduler Date: Wed, 28 Oct 2015 06:41:24 +0100 Message-ID: <1446010884.2937.276.camel@citrix.com> References: <1445942673.2937.89.camel@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7950664338510530669==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: suokun Cc: George Dunlap , xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org --===============7950664338510530669== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-V+J5X8i5Sk12dxleA+yg" --=-V+J5X8i5Sk12dxleA+yg Content-Type: multipart/mixed; boundary="=-pde6VWqPrf4fA3Mjdx3m" --=-pde6VWqPrf4fA3Mjdx3m Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, 2015-10-27 at 14:32 -0600, suokun wrote: > On Tue, Oct 27, 2015 at 4:44 AM, Dario Faggioli > wrote: > Hi, Dario, > Thank you for your reply. > Hi, > Here are my two VMs running on the same physical CPU. > VM-IO: 1-vCPU pinned to a pCPU, running netperf > VM-CPU: 1-vCPU pinned the the same pCPU, running a while(1) loop > Another machine run the netperf client to send the requests to VM-IO. >=20 > My code is very simple: > in VM-IO, as server side: $ netserver -p 12345 > in VM-CPU, just running a while(1) loop: $./loop > in the client, send I/O request to the VM-IO: $ netperf -H > [server_ip] > -l 15 -t TCP_STREAM -p 12345 >=20 Ok, thanks. > The setting that led to the poor IO performance is as follows: > VM-IO: 1-vCPU pinned to a pCPU, running netperf > VM-CPU: 1-vCPU pinned the the same pCPU, running a while(1) loop >=20 > The root cause is that when an IO request comes, VM-IO=E2=80=99s vCPU is > elevated to BOOST and goes through vcpu_wake =E2=80=94> __runq_tickle. In > __runq_tickle, the currently running vCPU (i.e., the vCPU from VM > -CPU) is marked as _VPF_migrating. > Ok. > Then, Xen goes through schedule() to > reschedule the current vCPU (i.e., vCPU from VM-CPU) and schedule the > next vCPU (i.e., the vCPU from VM-IO). Due to the _VPF_migrating=20 > flag, the descheduled vCPU will be migrated in context_saved() and=20 > later woken up in cpu_wake(). > Sure. > Indeed, csched_vcpu_wake() will quit if the > vCPU from VM-CPU is on run queue. But it is actually not. In > csched_schedule(), the vCPU will not be inserted back to run queue > because it is not runnable due to the __VPF_migrating bit in > pause_flags. As such, the vCPU from VM-CPU will boosted and not be > preempted by a later IO request because BOOST can not preempt BOOST. >=20 Aha! Now I see what you mean. From the previous email, I couldn't really tell which one call to schedule you where looking at, during each phase of the analysis... Thanks for clarifying! And, yes, I agree with you that, since the vCPU of VM-CPU fails the vcpu_runnable() test, it's being treated as it is really waking up from sleep, in csched_vcpu_wake(), and hence boosted. > A simple fix would be allowing BOOST to preempt BOOST.=20 > Nah, that would be an hack on top of an hack! :-P > A better fix > would be checking the CPU affinity before setting the __VPF_migrating > flag. >=20 Yeah, I like this better. So, can you try the patch attached to this email? Here at my place, without any patch, I get the following results: idle: throughput =3D 806.64 with noise: throughput =3D 166.50 With the patch, I get this: idle: throughput =3D 807.18 with noise: throughput =3D 731.66 The patch (if you confirm that it works) fixes the bug in this particular situations, where vCPUs are all pinned to the same pCPUs, but does not prevent vCPUs being migrated around the pCPUs to become BOOSTed in Credit2. That is something I think we should avoid, and I've got a (small) patch series ready for that. I'll give some more testing to it before sending it to the list, though, as I want to make sure it's not causing regressions. Thanks and Regards, Dario ---=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-pde6VWqPrf4fA3Mjdx3m Content-Disposition: attachment; filename="xen-sched-credit1-fix-tickle-migrate-cur.patch" Content-Transfer-Encoding: base64 Content-Type: text/x-patch; name="xen-sched-credit1-fix-tickle-migrate-cur.patch"; charset="UTF-8" Y29tbWl0IDE2MzgxOTM2YWQzMjBkMDEwYzc1NjZjOTQ2YTNlNTI4ZjgwM2U3OGEKQXV0aG9yOiBE YXJpbyBGYWdnaW9saSA8ZGFyaW8uZmFnZ2lvbGlAY2l0cml4LmNvbT4KRGF0ZTogICBUdWUgT2N0 IDI3IDIzOjIyOjE2IDIwMTUgKzAxMDAKCiAgICB4ZW46IGNyZWRpdDE6IG9uIHZDUFUgd2FrZXVw LCBraWNrIGF3YXkgY3VycmVudCBvbmx5IGlmIG1ha2VzIHNlbnNlCiAgICAKICAgIEluIGZhY3Qs IHdoZW4gd2FraW5nIHVwIGEgdkNQVSwgX19ydW5xX3RpY2tsZSgpIGlzIGNhbGxlZAogICAgdG8g YWxsb3cgdGhlIG5ldyB2Q1BVIHRvIHJ1biBvbiBhIHBDUFUgKHdoaWNoIG9uZSwgZGVwZW5kcwog ICAgb24gdGhlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIHRoZSBwcmlvcml0eSBvZiB0aGUgbmV3IHZD UFUsCiAgICBhbmQgdGhlIG9uZXMgb2YgdGhlIHZDUFVzIHRoYXQgYXJlIGFscmVhZHkgcnVubmlu ZykuCiAgICAKICAgIElmIHRoZXJlIGlzIG5vIGlkbGUgcHJvY2Vzc29yIG9uIHdoaWNoIHRoZSBu ZXcgdkNQVSBjYW4KICAgIHJ1biAoZS5nLiwgYmVjYXVzZSBvZiBwaW5uaW5nL2FmZmluaXR5KSwg d2UgdHJ5IHRvIG1pZ3JhdGUKICAgIGF3YXkgdGhlIHZDUFUgdGhhdCBpcyBjdXJyZW50bHkgcnVu bmluZyBvbiB0aGUgbmV3IHZDUFUncwogICAgcHJvY2Vzc29yIChpLmUuLCB0aGUgcHJvY2Vzc29y IG9uIHdoaWNoIHRoZSB2Q1BVIGlzIHdha2luZwogICAgdXApLgogICAgCiAgICBOb3csIHRyeWlu ZyB0byBtaWdyYXRlIGEgdkNQVSBoYXMgdGhlIGVmZmVjdCBvZiBwdXNoaW5nIGl0CiAgICB0aHJv dWdoIGEKICAgIAogICAgIHJ1bm5pbmcgLS0+IG9mZmxpbmUgLS0+IHJ1bm5hYmxlCiAgICAKICAg IHRyYW5zaXRpb24sIHdoaWNoLCBpbiB0dXJuLCBoYXMgdGhlIGZvbGxvd2luZyBuZWdhdGl2ZQog ICAgZWZmZWN0czoKICAgIAogICAgIDEpIENyZWRpdDEgY291bnRzIHRoYXQgYXMgYSB3YWtldXAs IGFuZCBpdCBCT09TVHMgdGhlCiAgICAgICAgdkNQVSwgZXZlbiBpZiBpdCBpcyBhIENQVS1ib3Vu ZCBvbmUsIHdoaWNoIHdvdWxkbid0CiAgICAgICAgbm9ybWFsbHkgaGF2ZSBkZXNlcnZlZCBib29z dGluZy4gVGhpcyBjYW4gcHJldmVudAogICAgICAgIGxlZ2l0IElPLWJvdW5kIHZDUFVzIHRvIGdl dCBhaG9sZCBvZiB0aGUgcHJvY2Vzc29yCiAgICAgICAgdW50aWwgc3VjaCBzcHVyaW91cyBib29z dGluZyBleHBpcmVzLCBodXJ0aW5nIHRoZQogICAgICAgIHBlcmZvcm1hbmNlIQogICAgCiAgICAg Mikgc2luY2UgdGhlIHZDUFUgaXMgZmFpbHMgdGhlIHZjcHVfcnVubmFibGUoKSB0ZXN0CiAgICAg ICAgKHdpdGhpbiB0aGUgY2FsbCB0byBjc2NoZWRfc2NoZWR1bGUoKSB0aGF0IGZvbGxvd3MKICAg ICAgICB0aGUgd2FrZXVwLCBhcyBhIGNvbnNlcXVlbmNlIG9mIHRpY2tsaW5nKSB0aGUKICAgICAg ICBzY2hlZHVsaW5nIHJhdGUtbGltaXRpbmcgbWVjaGFuaXNtIGlzIGFsc28gZm9vbGVkLAogICAg ICAgIGkuZS4sIHRoZSBjb250ZXh0IHN3aXRjaCBoYXBwZW5zIGV2ZW4gaWYgbGVzcyB0aGFuCiAg ICAgICAgdGhlIG1pbmltdW0gZXhlY3V0aW9uIGFtb3VudCBvZiB0aW1lIHBhc3NlZC4KICAgIAog ICAgSW4gcGFydGljdWxhciwgMSkgaGFzIGJlZW4gcmVwb3J0ZWQgdG8gY2F1c2UgdGhlIGZvbGxv d2luZwogICAgaXNzdWU6CiAgICAKICAgICAqIFZNLUlPOiAxLXZDUFUgcGlubmVkIHRvIGEgcENQ VSwgcnVubmluZyBuZXRwZXJmCiAgICAgKiBWTS1DUFU6IDEtdkNQVSBwaW5uZWQgdGhlIHRoZSBz YW1lIHBDUFUsIHJ1bm5pbmcgYSBidXN5CiAgICAgICAgICAgICAgIENQVSBsb29wCiAgICAgPT0+ IE9ubHkgVk0tSS9POiB0aHJvdWdocHV0IGlzIDgwNi42NCBNYnBzCiAgICAgPT0+IFZNLUkvTyAr IFZNLUNQVTogdGhyb3VnaHB1dCBpcyAxNjYuNTAgTWJwcwogICAgCiAgICBUaGlzIHBhdGNoIHNv bHZlcyAoZm9yIHRoZSBhYm92ZSBzY2VuYXJpbykgdGhlIHByb2JsZW0KICAgIGJ5IGNoZWNraW5n IHdoZXRoZXIgb3Igbm90IGl0IG1ha2VzIHNlbnNlIHRvIHRyeSB0bwogICAgbWlncmF0ZSBhd2F5 IHRoZSB2Q1BVIGN1cnJlbnRseSBydW5uaW5nIG9uIHRoZSBwcm9jZXNzb3IuCiAgICBJbiBmYWN0 LCB3ZSBzaG91bGRuJ3QgZXZlbiB0cnkgdG8gZG8gaXQsIGlmIHRoZXJlIGFyZW4ndAogICAgaWRs ZSBwcm9jZXNzb3JzIHdoZXJlIHN1Y2ggYSB2Q1BVIGNhbiBleGVjdXRlLiBJbiBzdWNoIGNhc2Us CiAgICBBdHRlbXB0aW5nIHRoZSBtaWdyYXRpb24gaXMganVzdCBmdXRpbGUgKGhhcm1mdWwsIGFj dHVhbGx5ISkuCiAgICAKICAgIFdpdGggdGhpcyBwYXRjaCwgaW4gdGhlIGFib3ZlIGNvbmZpZ3Vy YXRpb24sIHJlc3VsdHMgYXJlOgogICAgCiAgICAgPT0+IE9ubHkgVk0tSS9POiB0aHJvdWdocHV0 IGlzIDgwNy4xOCBNYnBzCiAgICAgPT0+IFZNLUkvTyArIFZNLUNQVTogdGhyb3VnaHB1dCBpcyA3 MzEuNjYgTWJwcwogICAgCiAgICBOb3RlIHRoYXQsIHN0aWxsIGFib3V0IDEpLCBpdCBpcyBfd3Jv bmdfIHRoYXQgQ3JlZGl0MQogICAgdHJlYXRzIHdha2V1cHMgcmVzdWx0aW5nIGZyb20gbWlncmF0 aW9uIG9mIGEgdkNQVSB0bwogICAgYW5vdGhlciBwQ1BVIGFzICJyZWd1bGFyIHdha2V1cHMiLCBo ZW5jZSBncmFudGluZyBCT09TVAogICAgcHJpb3JpdHkgdG8gdGhlIHZDUFVzIGV4cGVyaWVuY2lu ZyB0aGF0LiBIb3dldmVyOgogICAgIC0gZml4aW5nIHRoYXQgaXMgbm9uLXRyaXZpYWwsIGFuZCBy ZXF1aXJlcyBiZWluZyBkb25lCiAgICAgICBpbiBpdHMgb3duIHBhdGNoOwogICAgIC0gdGhhdCBp cyBvcnRob2dvbmFsIHRvIHRoZSBmaXggYmVpbmcgaW50cm9kdWNlZCBoZXJlLgogICAgICAgVGhh dCBpcyB0byBzYXksIGV2ZW4gd2hlbiBDcmVkaXQxIHdpbGwgYmUgZml4ZWQgbm90CiAgICAgICB0 byBib29zdCBtaWdyYXRpbmcgdkNQVXMsIHRoaXMgcGF0Y2ggd2lsbCBiZSBzdGlsbAogICAgICAg Y29yZWN0IGFuZCBuZWNlc3NhcnkuCiAgICAKICAgIFJlcG9ydGVkLWJ5OiBzdW9rdW4gPHN1b2t1 bnN0YXJAZ21haWwuY29tPgogICAgU2lnbmVkLW9mZi1ieTogRGFyaW8gRmFnZ2lvbGkgPGRhcmlv LmZhZ2dpb2xpQGNpdHJpeC5jb20+CiAgICAtLS0KICAgIENjOiBHZW9yZ2UgRHVubGFwIDxnZW9y Z2UuZHVubGFwQGNpdHJpeC5jb20+CiAgICBDYzogc3Vva3VuIDxzdW9rdW5zdGFyQGdtYWlsLmNv bT4KCmRpZmYgLS1naXQgYS94ZW4vY29tbW9uL3NjaGVkX2NyZWRpdC5jIGIveGVuL2NvbW1vbi9z Y2hlZF9jcmVkaXQuYwppbmRleCBiOGYyOGZlLi4xYjMwZTY3IDEwMDY0NAotLS0gYS94ZW4vY29t bW9uL3NjaGVkX2NyZWRpdC5jCisrKyBiL3hlbi9jb21tb24vc2NoZWRfY3JlZGl0LmMKQEAgLTQy Niw5ICs0MjYsMTAgQEAgX19ydW5xX3RpY2tsZSh1bnNpZ25lZCBpbnQgY3B1LCBzdHJ1Y3QgY3Nj aGVkX3ZjcHUgKm5ldykKIAogICAgICAgICAgICAgLyoKICAgICAgICAgICAgICAqIElmIHRoZXJl IGFyZSBubyBzdWl0YWJsZSBpZGxlcnMgZm9yIG5ldywgYW5kIGl0J3MgaGlnaGVyCi0gICAgICAg ICAgICAgKiBwcmlvcml0eSB0aGFuIGN1ciwgYXNrIHRoZSBzY2hlZHVsZXIgdG8gbWlncmF0ZSBj dXIgYXdheS4KLSAgICAgICAgICAgICAqIFdlIGhhdmUgdG8gYWN0IGxpa2UgdGhpcyAoaW5zdGVh ZCBvZiBqdXN0IHdha2luZyBzb21lIG9mCi0gICAgICAgICAgICAgKiB0aGUgaWRsZXJzIHN1aXRh YmxlIGZvciBjdXIpIGJlY2F1c2UgY3VyIGlzIHJ1bm5pbmcuCisgICAgICAgICAgICAgKiBwcmlv cml0eSB0aGFuIGN1ciwgY2hlY2sgd2hldGhlciB3ZSBjYW4gbWlncmF0ZSBjdXIgYXdheS4KKyAg ICAgICAgICAgICAqIChXZSBoYXZlIHRvIGRvIGl0IGluZGlyZWN0bHksIHZpYSBfVlBGX21pZ3Jh dGluZywgaW5zdGVhZAorICAgICAgICAgICAgICogb2YganVzdCB0aWNrbGluZyBhbnkgaWRsZXIg c3VpdGFibGUgZm9yIGN1cikgYmVjYXVzZSBjdXIKKyAgICAgICAgICAgICAqIGlzIHJ1bm5pbmcu KQogICAgICAgICAgICAgICoKICAgICAgICAgICAgICAqIElmIHRoZXJlIGFyZSBzdWl0YWJsZSBp ZGxlcnMgZm9yIG5ldywgbm8gbWF0dGVyIHByaW9yaXRpZXMsCiAgICAgICAgICAgICAgKiBsZWF2 ZSBjdXIgYWxvbmUgKGFzIGl0IGlzIHJ1bm5pbmcgYW5kIGlzLCBsaWtlbHksIGNhY2hlLWhvdCkK QEAgLTQzNywxMSArNDM4LDE4IEBAIF9fcnVucV90aWNrbGUodW5zaWduZWQgaW50IGNwdSwgc3Ry dWN0IGNzY2hlZF92Y3B1ICpuZXcpCiAgICAgICAgICAgICAgKi8KICAgICAgICAgICAgIGlmICgg bmV3X2lkbGVyc19lbXB0eSAmJiBuZXctPnByaSA+IGN1ci0+cHJpICkKICAgICAgICAgICAgIHsK KyAgICAgICAgICAgICAgICBjc2NoZWRfYmFsYW5jZV9jcHVtYXNrKGN1ci0+dmNwdSwgYmFsYW5j ZV9zdGVwLAorICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY3NjaGVkX2Jh bGFuY2VfbWFzayhjcHUpKTsKKyAgICAgICAgICAgICAgICBpZiAoIGNwdW1hc2tfaW50ZXJzZWN0 cyhjc2NoZWRfYmFsYW5jZV9tYXNrKGNwdSksCisgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgJmlkbGVfbWFzaykgKQorICAgICAgICAgICAgICAgIHsKKyAgICAgICAgICAg ICAgICAgICAgU0NIRURfVkNQVV9TVEFUX0NSQU5LKGN1ciwga2lja2VkX2F3YXkpOworICAgICAg ICAgICAgICAgICAgICBTQ0hFRF9WQ1BVX1NUQVRfQ1JBTksoY3VyLCBtaWdyYXRlX3IpOworICAg ICAgICAgICAgICAgICAgICBTQ0hFRF9TVEFUX0NSQU5LKG1pZ3JhdGVfa2lja2VkX2F3YXkpOwor ICAgICAgICAgICAgICAgICAgICBzZXRfYml0KF9WUEZfbWlncmF0aW5nLCAmY3VyLT52Y3B1LT5w YXVzZV9mbGFncyk7CisgICAgICAgICAgICAgICAgfQorICAgICAgICAgICAgICAgIC8qIFRpY2ts ZSBjcHUgYW55d2F5LCB0byBsZXQgbmV3IHByZWVtcHQgY3VyLiAqLwogICAgICAgICAgICAgICAg IFNDSEVEX1NUQVRfQ1JBTksodGlja2xlX2lkbGVyc19ub25lKTsKLSAgICAgICAgICAgICAgICBT Q0hFRF9WQ1BVX1NUQVRfQ1JBTksoY3VyLCBraWNrZWRfYXdheSk7Ci0gICAgICAgICAgICAgICAg U0NIRURfVkNQVV9TVEFUX0NSQU5LKGN1ciwgbWlncmF0ZV9yKTsKLSAgICAgICAgICAgICAgICBT Q0hFRF9TVEFUX0NSQU5LKG1pZ3JhdGVfa2lja2VkX2F3YXkpOwotICAgICAgICAgICAgICAgIHNl dF9iaXQoX1ZQRl9taWdyYXRpbmcsICZjdXItPnZjcHUtPnBhdXNlX2ZsYWdzKTsKICAgICAgICAg ICAgICAgICBfX2NwdW1hc2tfc2V0X2NwdShjcHUsICZtYXNrKTsKICAgICAgICAgICAgIH0KICAg ICAgICAgICAgIGVsc2UgaWYgKCAhbmV3X2lkbGVyc19lbXB0eSApCg== --=-pde6VWqPrf4fA3Mjdx3m-- --=-V+J5X8i5Sk12dxleA+yg Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEABECAAYFAlYwYAQACgkQk4XaBE3IOsTVMACdHwcP6kInaPvKLkW/vNZHHLW6 TwIAn3F46dLsFVTVJegMfN9djDdWBo5b =FtNn -----END PGP SIGNATURE----- --=-V+J5X8i5Sk12dxleA+yg-- --===============7950664338510530669== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============7950664338510530669==--