From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ingo Molnar Subject: Re: [PATCH V3 2/2] powerpc/pseries: init fault_around_order for pseries Date: Tue, 6 May 2014 13:29:17 +0200 Message-ID: <20140506112917.GA29525@gmail.com> References: <1398675690-16186-1-git-send-email-maddy@linux.vnet.ibm.com> <1398675690-16186-3-git-send-email-maddy@linux.vnet.ibm.com> <20140429070632.GB27951@gmail.com> <87d2fz47tg.fsf@rustcorp.com.au> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <87d2fz47tg.fsf@rustcorp.com.au> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+glppe-linuxppc-embedded-2=m.gmane.org@lists.ozlabs.org Sender: "Linuxppc-dev" To: Rusty Russell Cc: linux-arch@vger.kernel.org, riel@redhat.com, Madhavan Srinivasan , dave.hansen@intel.com, peterz@infradead.org, x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, ak@linux.intel.com, paulus@samba.org, mgorman@suse.de, Linus Torvalds , akpm@linux-foundation.org, linuxppc-dev@lists.ozlabs.org, kirill.shutemov@linux.intel.com List-Id: linux-arch.vger.kernel.org CiogUnVzdHkgUnVzc2VsbCA8cnVzdHlAcnVzdGNvcnAuY29tLmF1PiB3cm90ZToKCj4gSW5nbyBN b2xuYXIgPG1pbmdvQGtlcm5lbC5vcmc+IHdyaXRlczoKPiA+ICogTWFkaGF2YW4gU3Jpbml2YXNh biA8bWFkZHlAbGludXgudm5ldC5pYm0uY29tPiB3cm90ZToKPiA+Cj4gPj4gUGVyZm9ybWFuY2Ug ZGF0YSBmb3IgZGlmZmVyZW50IEZBVUxUX0FST1VORF9PUkRFUiB2YWx1ZXMgZnJvbSA0IHNvY2tl dAo+ID4+IFBvd2VyNyBzeXN0ZW0gKDEyOCBUaHJlYWRzIGFuZCAxMjhHQiBtZW1vcnkpLiBwZXJm IHN0YXQgd2l0aCByZXBlYXQgb2YgNQo+ID4+IGlzIHVzZWQgdG8gZ2V0IHRoZSBzdGRkZXYgdmFs dWVzLiBUZXN0IHJhbiBpbiB2My4xNCBrZXJuZWwgKEJhc2VsaW5lKSBhbmQKPiA+PiB2My4xNS1y YzEgZm9yIGRpZmZlcmVudCBmYXVsdCBhcm91bmQgb3JkZXIgdmFsdWVzLgo+ID4+IAo+ID4+IEZB VUxUX0FST1VORF9PUkRFUiAgICAgIEJhc2VsaW5lICAgICAgICAxICAgICAgICAgICAgICAgMyAg ICAgICAgICAgICAgIDQgICAgICAgICAgICAgICA1ICAgICAgICAgICAgICAgOAo+ID4+IAo+ID4+ IExpbnV4IGJ1aWxkIChtYWtlIC1qNjQpCj4gPj4gbWlub3ItZmF1bHRzICAgICAgICAgICAgNDcs NDM3LDM1OSAgICAgIDM1LDI3OSwyODYgICAgICAyNSw0MjUsMzQ3ICAgICAgMjMsNDYxLDI3NSAg ICAgIDIyLDAwMiwxODkgICAgICAyMSw0MzUsODM2Cj4gPj4gdGltZXMgaW4gc2Vjb25kcyAgICAg ICAgMzQ3LjMwMjUyODQyMCAgIDM0NC4wNjE1ODg0NjAgICAzNDAuOTc0MDIyMzkxICAgMzQ4LjE5 MzUwODExNiAgIDM0OC42NzM5MDAxNTggICAzNTAuOTg2NTQzNjE4Cj4gPj4gIHN0ZGRldiBmb3Ig dGltZSAgICAgICAgKCArLSAgMS41MCUgKSAgICggKy0gIDAuNzMlICkgICAoICstICAxLjEzJSAp ICAgKCArLSAgMS4wMSUgKSAgICggKy0gIDEuODklICkgICAoICstICAxLjU1JSApCj4gPj4gICVj aGcgdGltZSB0byBiYXNlbGluZSAgICAgICAgICAgICAgICAgIC0wLjklICAgICAgICAgICAtMS44 JSAgICAgICAgICAgMC4yJSAgICAgICAgICAgIDAuMzklICAgICAgICAgICAxLjA2JQo+ID4KPiA+ IFByb2JhYmx5IHRvbyBub2lzeS4KPiAKPiBBIGxpdHRsZSwgYnV0IDMgc3RpbGwgbG9va3MgbGlr ZSB0aGUgd2lubmVyLgo+IAo+ID4+IExpbnV4IHJlYnVpbGQgKG1ha2UgLWo2NCkKPiA+PiBtaW5v ci1mYXVsdHMgICAgICAgICAgICA5NDEsNTUyICAgICAgICAgNzE4LDMxOSAgICAgICAgIDQ4Niw2 MjUgICAgICAgICA0NDAsMTI0ICAgICAgICAgNDEwLDUxMCAgICAgICAgIDM5Nyw0MTYKPiA+PiB0 aW1lcyBpbiBzZWNvbmRzICAgICAgICAzMC41Njk4MzQ3MTggICAgMzEuMjE5NjM3NTM5ICAgIDMx LjMxOTM3MDY0OSAgICAzMS40MzQyODU0NzIgICAgMzEuOTcyMzY3MTc0ICAgIDMxLjQ0MzA0MzU4 MAo+ID4+ICBzdGRkZXYgZm9yIHRpbWUgICAgICAgICggKy0gIDEuMDclICkgICAoICstICAwLjEz JSApICAgKCArLSAgMC40MyUgKSAgICggKy0gIDAuMTglICkgICAoICstICAwLjk1JSApICAgKCAr LSAgMC41OCUgKQo+ID4+ICAlY2hnIHRpbWUgdG8gYmFzZWxpbmUgICAgICAgICAgICAgICAgICAy LjElICAgICAgICAgICAgMi40JSAgICAgICAgICAgIDIuOCUgICAgICAgICAgICA0LjU4JSAgICAg ICAgICAgMi44NSUKPiA+Cj4gPiBIZXJlIGl0IGxvb2tzIGxpa2UgYSBzcGVlZHVwLiBPcHRpbWFs IHZhbHVlOiA1Ky4KPiAKPiBObywgbG93ZXIgdGltZSBpcyBiZXR0ZXIuICBCYXNlbGluZSAobm8g ZmF1bHRhcm91bmQpIHdpbnMuCj4gCj4gCj4gZXRjLgoKYWgsIHllYWgsIHlvdSBhcmUgcmlnaHQu IEJyYWluZmFydCBvZiB0aGUgd2Vlay4uLgoKPiBJdCdzIG5vdCBhIGh1Z2Ugc3VycHJpc2UgdGhh dCBhIDY0ayBwYWdlIGFyY2ggd2FudHMgYSBzbWFsbGVyIHZhbHVlIAo+IHRoYW4gYSA0ayBzeXN0 ZW0uICBCdXQgSSBhZ3JlZTogSSBkb24ndCBzZWUgbXVjaCB1cHNpZGUgZm9yIEZBTyA+IDAsIAo+ IGJ1dCBJIGRvIHNlZSBkb3duc2lkZS4KPiAKPiBNb3N0IGV4dHJlbWUgcmVzdWx0czoKPiBPcmRl ciAxOiAyJSBsb3NzIG9uIHJlY29tcGlsZS4gIDEwJSB3aW4gNCUgbG9zcyBvbiBzZXEuICA5JSBs b3NzIHJhbmRvbS4KPiBPcmRlciAzOiAyJSBsb3NzIG9uIHJlY29tcGlsZS4gIDYlIHdpbiA1JSBs b3NzIG9uIHNlcS4gIDE0JSBsb3NzIG9uIHJhbmRvbS4KPiBPcmRlciA0OiAyLjglIGxvc3Mgb24g cmVjb21waWxlLiAxMCUgd2luIDclIGxvc3Mgb24gc2VxLiAgOSUgbG9zcyBvbiByYW5kb20uCj4g Cj4gPiBJJ20gc3RhcnRpbmcgdG8gc3VzcGVjdCB0aGF0IG1heWJlIHdvcmtsb2FkcyBvdWdodCB0 byBiZSBnaXZlbiBhIAo+ID4gY2hvaWNlIGluIHRoaXMgbWF0dGVyLCB2aWEgbWFkdmlzZSgpIG9y IHN1Y2guCj4gCj4gSSByZWFsbHkgZG9uJ3QgdGhpbmsgdGhleSdsbCBiZSBhYmxlIHRvIHVzZSBp dDsgaXQnbGwgY2hhbmdlIGZhciB0b28gCj4gbXVjaCB3aXRoIG1hY2hpbmUgYW5kIGtlcm5lbCB1 cGRhdGVzLiBbLi4uXQoKRG8gd2Uga25vdyB0aGF0PwoKPiBbLi4uXSBJIHRoaW5rIHdlIHNob3Vs ZCBhcHBseSBwYXRjaAo+ICMxICh3aXRoIGZpeGVzKSB0byBtYWtlIGl0IGEgdmFyaWFibGUsIHRo ZW4gc2V0IGl0IHRvIDAgZm9yIFBQQy4KCk9rLCBhZ3JlZWQgLSBhdCBsZWFzdCB1bnRpbCBjb250 cmFyeSBkYXRhIGNvbWVzIGFyb3VuZC4KClRoYW5rcywKCglJbmdvCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCkxpbnV4cHBjLWRldiBtYWlsaW5nIGxpc3QK TGludXhwcGMtZGV2QGxpc3RzLm96bGFicy5vcmcKaHR0cHM6Ly9saXN0cy5vemxhYnMub3JnL2xp c3RpbmZvL2xpbnV4cHBjLWRldg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ee0-f54.google.com ([74.125.83.54]:46595 "EHLO mail-ee0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934600AbaEFL3X (ORCPT ); Tue, 6 May 2014 07:29:23 -0400 Date: Tue, 6 May 2014 13:29:17 +0200 From: Ingo Molnar Subject: Re: [PATCH V3 2/2] powerpc/pseries: init fault_around_order for pseries Message-ID: <20140506112917.GA29525@gmail.com> References: <1398675690-16186-1-git-send-email-maddy@linux.vnet.ibm.com> <1398675690-16186-3-git-send-email-maddy@linux.vnet.ibm.com> <20140429070632.GB27951@gmail.com> <87d2fz47tg.fsf@rustcorp.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87d2fz47tg.fsf@rustcorp.com.au> Sender: linux-arch-owner@vger.kernel.org List-ID: To: Rusty Russell Cc: Madhavan Srinivasan , linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, x86@kernel.org, benh@kernel.crashing.org, paulus@samba.org, kirill.shutemov@linux.intel.com, akpm@linux-foundation.org, riel@redhat.com, mgorman@suse.de, ak@linux.intel.com, peterz@infradead.org, dave.hansen@intel.com, Linus Torvalds Message-ID: <20140506112917.hcij4dnciMyJeZZvfjRM7KqsB_YWwhKBNxoQvMcomlc@z> * Rusty Russell wrote: > Ingo Molnar writes: > > * Madhavan Srinivasan wrote: > > > >> Performance data for different FAULT_AROUND_ORDER values from 4 socket > >> Power7 system (128 Threads and 128GB memory). perf stat with repeat of 5 > >> is used to get the stddev values. Test ran in v3.14 kernel (Baseline) and > >> v3.15-rc1 for different fault around order values. > >> > >> FAULT_AROUND_ORDER Baseline 1 3 4 5 8 > >> > >> Linux build (make -j64) > >> minor-faults 47,437,359 35,279,286 25,425,347 23,461,275 22,002,189 21,435,836 > >> times in seconds 347.302528420 344.061588460 340.974022391 348.193508116 348.673900158 350.986543618 > >> stddev for time ( +- 1.50% ) ( +- 0.73% ) ( +- 1.13% ) ( +- 1.01% ) ( +- 1.89% ) ( +- 1.55% ) > >> %chg time to baseline -0.9% -1.8% 0.2% 0.39% 1.06% > > > > Probably too noisy. > > A little, but 3 still looks like the winner. > > >> Linux rebuild (make -j64) > >> minor-faults 941,552 718,319 486,625 440,124 410,510 397,416 > >> times in seconds 30.569834718 31.219637539 31.319370649 31.434285472 31.972367174 31.443043580 > >> stddev for time ( +- 1.07% ) ( +- 0.13% ) ( +- 0.43% ) ( +- 0.18% ) ( +- 0.95% ) ( +- 0.58% ) > >> %chg time to baseline 2.1% 2.4% 2.8% 4.58% 2.85% > > > > Here it looks like a speedup. Optimal value: 5+. > > No, lower time is better. Baseline (no faultaround) wins. > > > etc. ah, yeah, you are right. Brainfart of the week... > It's not a huge surprise that a 64k page arch wants a smaller value > than a 4k system. But I agree: I don't see much upside for FAO > 0, > but I do see downside. > > Most extreme results: > Order 1: 2% loss on recompile. 10% win 4% loss on seq. 9% loss random. > Order 3: 2% loss on recompile. 6% win 5% loss on seq. 14% loss on random. > Order 4: 2.8% loss on recompile. 10% win 7% loss on seq. 9% loss on random. > > > I'm starting to suspect that maybe workloads ought to be given a > > choice in this matter, via madvise() or such. > > I really don't think they'll be able to use it; it'll change far too > much with machine and kernel updates. [...] Do we know that? > [...] I think we should apply patch > #1 (with fixes) to make it a variable, then set it to 0 for PPC. Ok, agreed - at least until contrary data comes around. Thanks, Ingo