From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3rYnJT0Mk6zDr9D for ; Tue, 21 Jun 2016 22:35:52 +1000 (AEST) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.11/8.16.0.11) with SMTP id u5LCY0GO118358 for ; Tue, 21 Jun 2016 08:35:51 -0400 Received: from e28smtp09.in.ibm.com (e28smtp09.in.ibm.com [125.16.236.9]) by mx0a-001b2d01.pphosted.com with ESMTP id 23n2u7bhrx-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Tue, 21 Jun 2016 08:35:51 -0400 Received: from localhost by e28smtp09.in.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 21 Jun 2016 18:05:48 +0530 Received: from d28relay06.in.ibm.com (d28relay06.in.ibm.com [9.184.220.150]) by d28dlp01.in.ibm.com (Postfix) with ESMTP id 80A2CE005B for ; Tue, 21 Jun 2016 18:09:27 +0530 (IST) Received: from d28av02.in.ibm.com (d28av02.in.ibm.com [9.184.220.64]) by d28relay06.in.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u5LCZiCv45940794 for ; Tue, 21 Jun 2016 18:05:44 +0530 Received: from d28av02.in.ibm.com (localhost [127.0.0.1]) by d28av02.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u5LCZe4O016168 for ; Tue, 21 Jun 2016 18:05:44 +0530 Date: Tue, 21 Jun 2016 20:35:22 +0800 From: xinhui MIME-Version: 1.0 To: Benjamin Herrenschmidt , Peter Zijlstra CC: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, virtualization@lists.linux-foundation.org, paulus@samba.org, mpe@ellerman.id.au, mingo@redhat.com, paulmck@linux.vnet.ibm.com, waiman.long@hpe.com Subject: Re: [PATCH v5 1/6] qspinlock: powerpc support qspinlock References: <1464859370-5162-1-git-send-email-xinhui.pan@linux.vnet.ibm.com> <1464859370-5162-3-git-send-email-xinhui.pan@linux.vnet.ibm.com> <1464917520.26773.11.camel@kernel.crashing.org> <1464917548.26773.12.camel@au1.ibm.com> <57510353.1020209@linux.vnet.ibm.com> <1464928427.26773.26.camel@kernel.crashing.org> <20160606155907.GH30909@twins.programming.kicks-ass.net> <1465249297.4274.72.camel@kernel.crashing.org> In-Reply-To: <1465249297.4274.72.camel@kernel.crashing.org> Content-Type: text/plain; charset=utf-8; format=flowed Message-Id: <5769348A.1060505@linux.vnet.ibm.com> List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 2016年06月07日 05:41, Benjamin Herrenschmidt wrote: > On Mon, 2016-06-06 at 17:59 +0200, Peter Zijlstra wrote: >> On Fri, Jun 03, 2016 at 02:33:47PM +1000, Benjamin Herrenschmidt wrote: >>> >>> - For the above, can you show (or describe) where the qspinlock >>> improves things compared to our current locks. >> So currently PPC has a fairly straight forward test-and-set spinlock >> IIRC. You have this because LPAR/virt muck and lock holder preemption >> issues etc.. >> qspinlock is 1) a fair lock (like ticket locks) and 2) provides >> out-of-word spinning, reducing cacheline pressure. > > Thanks Peter. I think I understand the theory, but I'd like see it > translate into real numbers. > >> Esp. on multi-socket x86 we saw the out-of-word spinning being a big win >> over our ticket locks. >> >> And fairness, brought to us by the ticket locks a long time ago, >> eliminated starvation issues we had, where a spinner local to the holder >> would 'always' win from a spinner further away. So under heavy enough >> local contention, the spinners on 'remote' CPUs would 'never' get to own >> the lock. > > I think our HW has tweaks to avoid that from happening with the simple > locks in the underlying ll/sc implementation. In any case, what I'm > asking is actual tests to verify it works as expected for us. > IF HW has such tweaks then there mush be performance drop when total cpu's number grows up. And I got such clues one simple benchmark test: it tests how many spin_lock/spin_unlock pairs can be done within 15 seconds on all cpus. say, while(!done) { spin_lock() this_cpu_inc(loops) spin_unlock() } I do the test on two machines, one is using powerKVM, and the other is using pHyp. the result below shows what the sum of loops is in the end, with xxxxK form. cpu count | pv-qspinlock | test-set spinlock| ---------------------------------------------------- 8 (powerKVM) | 62830K | 67340K | ------------------------------------------------ 8 (pHyp) | 49800K | 59330K | ------------------------------------------------ 32 (pHyp) | 87580K | 20990K | ------------------------------------------------- while cpu count grows up, the lock/unlock pairs ops of test-set spinlock drops very much. this is because the cache bouncing in different physical cpus. So to verify how both spinlock impact the data-cache, another simple benchmark test. code looks like: struct _x { spinlock_t lk; unsigned long x; } x; while(!this_cpu_read(stop)) { int i = 0xff spin_lock(x.lk) this_cpu_inc(loops) while(i--) READ_ONCE(x.x); spin_unlock(x.lk) } the result below shows what the sum of loops is in the end, with xxxxK form. cpu count | pv-qspinlock | test-set spinlock| ------------------------------------------------ 8 (pHyp) | 13240K | 9780K | ------------------------------------------------ 32 (pHyp) | 25790K | 9700K | ------------------------------------------------ obviously pv-qspinlock is more cache-friendly, and has better performance than test-set spinlock. More test is going on, I will send out new patch set with the result. HOPE *within* this week. unixbench really takes a long time. thanks xinhui >> pv-qspinlock tries to preserve the fairness while allowing limited lock >> stealing and explicitly managing which vcpus to wake. > > Right. >>> >>> While there's >>> theory and to some extent practice on x86, it would be nice to >>> validate the effects on POWER. >> Right; so that will have to be from benchmarks which I cannot help you >> with ;-) > > Precisely :-) This is what I was asking for ;-) > > Cheers, > Ben. > From mboxrd@z Thu Jan 1 00:00:00 1970 From: xinhui Subject: Re: [PATCH v5 1/6] qspinlock: powerpc support qspinlock Date: Tue, 21 Jun 2016 20:35:22 +0800 Message-ID: <5769348A.1060505@linux.vnet.ibm.com> References: <1464859370-5162-1-git-send-email-xinhui.pan@linux.vnet.ibm.com> <1464859370-5162-3-git-send-email-xinhui.pan@linux.vnet.ibm.com> <1464917520.26773.11.camel@kernel.crashing.org> <1464917548.26773.12.camel@au1.ibm.com> <57510353.1020209@linux.vnet.ibm.com> <1464928427.26773.26.camel@kernel.crashing.org> <20160606155907.GH30909@twins.programming.kicks-ass.net> <1465249297.4274.72.camel@kernel.crashing.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <1465249297.4274.72.camel@kernel.crashing.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Benjamin Herrenschmidt , Peter Zijlstra Cc: mpe@ellerman.id.au, linux-kernel@vger.kernel.org, waiman.long@hpe.com, virtualization@lists.linux-foundation.org, mingo@redhat.com, paulus@samba.org, paulmck@linux.vnet.ibm.com, linuxppc-dev@lists.ozlabs.org List-Id: virtualization@lists.linuxfoundation.org CgpPbiAyMDE25bm0MDbmnIgwN+aXpSAwNTo0MSwgQmVuamFtaW4gSGVycmVuc2NobWlkdCB3cm90 ZToKPiBPbiBNb24sIDIwMTYtMDYtMDYgYXQgMTc6NTkgKzAyMDAsIFBldGVyIFppamxzdHJhIHdy b3RlOgo+PiBPbiBGcmksIEp1biAwMywgMjAxNiBhdCAwMjozMzo0N1BNICsxMDAwLCBCZW5qYW1p biBIZXJyZW5zY2htaWR0IHdyb3RlOgo+Pj4KPj4+ICAgLSBGb3IgdGhlIGFib3ZlLCBjYW4geW91 IHNob3cgKG9yIGRlc2NyaWJlKSB3aGVyZSB0aGUgcXNwaW5sb2NrCj4+PiAgICAgaW1wcm92ZXMg dGhpbmdzIGNvbXBhcmVkIHRvIG91ciBjdXJyZW50IGxvY2tzLgo+PiBTbyBjdXJyZW50bHkgUFBD IGhhcyBhIGZhaXJseSBzdHJhaWdodCBmb3J3YXJkIHRlc3QtYW5kLXNldCBzcGlubG9jawo+PiBJ SVJDLiBZb3UgaGF2ZSB0aGlzIGJlY2F1c2UgTFBBUi92aXJ0IG11Y2sgYW5kIGxvY2sgaG9sZGVy IHByZWVtcHRpb24KPj4gaXNzdWVzIGV0Yy4uCj4+IHFzcGlubG9jayBpcyAxKSBhIGZhaXIgbG9j ayAobGlrZSB0aWNrZXQgbG9ja3MpIGFuZCAyKSBwcm92aWRlcwo+PiBvdXQtb2Ytd29yZCBzcGlu bmluZywgcmVkdWNpbmcgY2FjaGVsaW5lIHByZXNzdXJlLgo+Cj4gVGhhbmtzIFBldGVyLiBJIHRo aW5rIEkgdW5kZXJzdGFuZCB0aGUgdGhlb3J5LCBidXQgSSdkIGxpa2Ugc2VlIGl0Cj4gdHJhbnNs YXRlIGludG8gcmVhbCBudW1iZXJzLgo+Cj4+IEVzcC4gb24gbXVsdGktc29ja2V0IHg4NiB3ZSBz YXcgdGhlIG91dC1vZi13b3JkIHNwaW5uaW5nIGJlaW5nIGEgYmlnIHdpbgo+PiBvdmVyIG91ciB0 aWNrZXQgbG9ja3MuCj4+Cj4+IEFuZCBmYWlybmVzcywgYnJvdWdodCB0byB1cyBieSB0aGUgdGlj a2V0IGxvY2tzIGEgbG9uZyB0aW1lIGFnbywKPj4gZWxpbWluYXRlZCBzdGFydmF0aW9uIGlzc3Vl cyB3ZSBoYWQsIHdoZXJlIGEgc3Bpbm5lciBsb2NhbCB0byB0aGUgaG9sZGVyCj4+IHdvdWxkICdh bHdheXMnIHdpbiBmcm9tIGEgc3Bpbm5lciBmdXJ0aGVyIGF3YXkuIFNvIHVuZGVyIGhlYXZ5IGVu b3VnaAo+PiBsb2NhbCBjb250ZW50aW9uLCB0aGUgc3Bpbm5lcnMgb24gJ3JlbW90ZScgQ1BVcyB3 b3VsZCAnbmV2ZXInIGdldCB0byBvd24KPj4gdGhlIGxvY2suCj4KPiBJIHRoaW5rIG91ciBIVyBo YXMgdHdlYWtzIHRvIGF2b2lkIHRoYXQgZnJvbSBoYXBwZW5pbmcgd2l0aCB0aGUgc2ltcGxlCj4g bG9ja3MgaW4gdGhlIHVuZGVybHlpbmcgbGwvc2MgaW1wbGVtZW50YXRpb24uIEluIGFueSBjYXNl LCB3aGF0IEknbQo+IGFza2luZyBpcyBhY3R1YWwgdGVzdHMgdG8gdmVyaWZ5IGl0IHdvcmtzIGFz IGV4cGVjdGVkIGZvciB1cy4KPgpJRiBIVyBoYXMgc3VjaCB0d2Vha3MgdGhlbiB0aGVyZSBtdXNo IGJlIHBlcmZvcm1hbmNlIGRyb3Agd2hlbiB0b3RhbCBjcHUncyBudW1iZXIgZ3Jvd3MgdXAuCkFu ZCBJIGdvdCBzdWNoIGNsdWVzCgpvbmUgc2ltcGxlIGJlbmNobWFyayB0ZXN0OgppdCB0ZXN0cyBo b3cgbWFueSBzcGluX2xvY2svc3Bpbl91bmxvY2sgcGFpcnMgY2FuIGJlIGRvbmUgd2l0aGluIDE1 IHNlY29uZHMgb24gYWxsIGNwdXMuCnNheSwKd2hpbGUoIWRvbmUpIHsKCXNwaW5fbG9jaygpCgl0 aGlzX2NwdV9pbmMobG9vcHMpCglzcGluX3VubG9jaygpCn0KCkkgZG8gdGhlIHRlc3Qgb24gdHdv IG1hY2hpbmVzLCBvbmUgaXMgdXNpbmcgcG93ZXJLVk0sIGFuZCB0aGUgb3RoZXIgaXMgdXNpbmcg cEh5cC4KdGhlIHJlc3VsdCBiZWxvdyBzaG93cyB3aGF0IHRoZSBzdW0gb2YgbG9vcHMgaXMgaW4g dGhlIGVuZCwgd2l0aCB4eHh4SyBmb3JtLgoKY3B1IGNvdW50CXwgcHYtcXNwaW5sb2NrCXwgdGVz dC1zZXQgc3BpbmxvY2t8Ci0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0KOCAocG93ZXJLVk0pCXwJNjI4MzBLCXwJNjczNDBLCXwKLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCjggKHBIeXApCXwJNDk4MDBLCXwJ NTkzMzBLCXwKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t CjMyIChwSHlwKQl8CTg3NTgwSwl8CTIwOTkwSwl8Ci0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KCndoaWxlIGNwdSBjb3VudCBncm93cyB1cCwgdGhlIGxv Y2svdW5sb2NrIHBhaXJzIG9wcyBvZiB0ZXN0LXNldCBzcGlubG9jayBkcm9wcyB2ZXJ5IG11Y2gu CnRoaXMgaXMgYmVjYXVzZSB0aGUgY2FjaGUgYm91bmNpbmcgaW4gZGlmZmVyZW50IHBoeXNpY2Fs IGNwdXMuCgpTbyB0byB2ZXJpZnkgaG93IGJvdGggc3BpbmxvY2sgaW1wYWN0IHRoZSBkYXRhLWNh Y2hlLAphbm90aGVyIHNpbXBsZSBiZW5jaG1hcmsgdGVzdC4KY29kZSBsb29rcyBsaWtlOgoKc3Ry dWN0IF94IHsKCXNwaW5sb2NrX3QgbGs7Cgl1bnNpZ25lZCBsb25nIHg7Cn0geDsKCndoaWxlKCF0 aGlzX2NwdV9yZWFkKHN0b3ApKSB7CglpbnQgaSA9IDB4ZmYKCXNwaW5fbG9jayh4LmxrKQoJdGhp c19jcHVfaW5jKGxvb3BzKQoJd2hpbGUoaS0tKQoJCVJFQURfT05DRSh4LngpOwoJc3Bpbl91bmxv Y2soeC5saykKfQoKdGhlIHJlc3VsdCBiZWxvdyBzaG93cyB3aGF0IHRoZSBzdW0gb2YgbG9vcHMg aXMgaW4gdGhlIGVuZCwgd2l0aCB4eHh4SyBmb3JtLgoKY3B1IGNvdW50CXwgcHYtcXNwaW5sb2Nr CXwgdGVzdC1zZXQgc3BpbmxvY2t8Ci0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLQo4IChwSHlwKQl8CTEzMjQwSwl8CTk3ODBLCXwKLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCjMyIChwSHlwKQl8CTI1NzkwSwl8CTk3 MDBLCXwKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCgpv YnZpb3VzbHkgcHYtcXNwaW5sb2NrIGlzIG1vcmUgY2FjaGUtZnJpZW5kbHksIGFuZCBoYXMgYmV0 dGVyIHBlcmZvcm1hbmNlIHRoYW4gdGVzdC1zZXQgc3BpbmxvY2suCgpNb3JlIHRlc3QgaXMgZ29p bmcgb24sIEkgd2lsbCBzZW5kIG91dCBuZXcgcGF0Y2ggc2V0IHdpdGggdGhlIHJlc3VsdC4KSE9Q RSAqd2l0aGluKiB0aGlzIHdlZWsuIHVuaXhiZW5jaCByZWFsbHkgdGFrZXMgYSBsb25nIHRpbWUu Cgp0aGFua3MKeGluaHVpCj4+IHB2LXFzcGlubG9jayB0cmllcyB0byBwcmVzZXJ2ZSB0aGUgZmFp cm5lc3Mgd2hpbGUgYWxsb3dpbmcgbGltaXRlZCBsb2NrCj4+IHN0ZWFsaW5nIGFuZCBleHBsaWNp dGx5IG1hbmFnaW5nIHdoaWNoIHZjcHVzIHRvIHdha2UuCj4KPiBSaWdodC4KPj4+Cj4+PiAJV2hp bGUgdGhlcmUncwo+Pj4gICAgIHRoZW9yeSBhbmQgdG8gc29tZSBleHRlbnQgcHJhY3RpY2Ugb24g eDg2LCBpdCB3b3VsZCBiZSBuaWNlIHRvCj4+PiAgICAgdmFsaWRhdGUgdGhlIGVmZmVjdHMgb24g UE9XRVIuCj4+IFJpZ2h0OyBzbyB0aGF0IHdpbGwgaGF2ZSB0byBiZSBmcm9tIGJlbmNobWFya3Mg d2hpY2ggSSBjYW5ub3QgaGVscCB5b3UKPj4gd2l0aCA7LSkKPgo+IFByZWNpc2VseSA6LSkgVGhp cyBpcyB3aGF0IEkgd2FzIGFza2luZyBmb3IgOy0pCj4KPiBDaGVlcnMsCj4gQmVuLgo+CgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpWaXJ0dWFsaXphdGlv biBtYWlsaW5nIGxpc3QKVmlydHVhbGl6YXRpb25AbGlzdHMubGludXgtZm91bmRhdGlvbi5vcmcK aHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGluZm8vdmlydHVh bGl6YXRpb24=