From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: "Verma, Vishal L" Subject: Re: [Linux-nvdimm] [PATCH v2 19/20] nd_btt: atomic sector updates Date: Mon, 18 May 2015 22:38:11 +0000 Message-ID: <1431988690.15857.31.camel@intel.com> References: <20150428181203.35812.60474.stgit@dwillia2-desk3.amr.corp.intel.com> <20150428182557.35812.38292.stgit@dwillia2-desk3.amr.corp.intel.com> <94D0CD8314A33A4D9D801C0FE68B40295A9097E5@G9W0745.americas.hpqcorp.net> In-Reply-To: <94D0CD8314A33A4D9D801C0FE68B40295A9097E5@G9W0745.americas.hpqcorp.net> Content-Language: en-US Content-Type: text/plain; charset="utf-8" Content-ID: Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org To: "Elliott@hp.com" Cc: "linux-kernel@vger.kernel.org" , "mingo@kernel.org" , "neilb@suse.de" , "hch@lst.de" , "Williams, Dan J" , "axboe@fb.com" , "hpa@zytor.com" , "linux-nvdimm@lists.01.org" , "gregkh@linuxfoundation.org" , "luto@amacapital.net" , "david@fromorbit.com" List-ID: T24gU3VuLCAyMDE1LTA1LTE3IGF0IDAxOjE5ICswMDAwLCBFbGxpb3R0LCBSb2JlcnQgKFNlcnZl ciBTdG9yYWdlKQ0Kd3JvdGU6DQo+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gPiBG cm9tOiBMaW51eC1udmRpbW0gW21haWx0bzpsaW51eC1udmRpbW0tYm91bmNlc0BsaXN0cy4wMS5v cmddIE9uIEJlaGFsZiBPZg0KPiA+IERhbiBXaWxsaWFtcw0KPiA+IFNlbnQ6IFR1ZXNkYXksIEFw cmlsIDI4LCAyMDE1IDE6MjYgUE0NCj4gPiBUbzogbGludXgtbnZkaW1tQGxpc3RzLjAxLm9yZw0K PiA+IENjOiBJbmdvIE1vbG5hcjsgTmVpbCBCcm93bjsgR3JlZyBLSDsgRGF2ZSBDaGlubmVyOyBs aW51eC0NCj4gPiBrZXJuZWxAdmdlci5rZXJuZWwub3JnOyBBbmR5IEx1dG9taXJza2k7IEplbnMg QXhib2U7IEguIFBldGVyIEFudmluOw0KPiA+IENocmlzdG9waCBIZWxsd2lnDQo+ID4gU3ViamVj dDogW0xpbnV4LW52ZGltbV0gW1BBVENIIHYyIDE5LzIwXSBuZF9idHQ6IGF0b21pYyBzZWN0b3Ig dXBkYXRlcw0KPiA+IA0KPiA+IEZyb206IFZpc2hhbCBWZXJtYSA8dmlzaGFsLmwudmVybWFAbGlu dXguaW50ZWwuY29tPg0KPiA+IA0KPiA+IEJUVCBzdGFuZHMgZm9yIEJsb2NrIFRyYW5zbGF0aW9u IFRhYmxlLCBhbmQgaXMgYSB3YXkgdG8gcHJvdmlkZSBwb3dlcg0KPiA+IGZhaWwgc2VjdG9yIGF0 b21pY2l0eSBzZW1hbnRpY3MgZm9yIGJsb2NrIGRldmljZXMgdGhhdCBoYXZlIHRoZSBhYmlsaXR5 DQo+ID4gdG8gcGVyZm9ybSBieXRlIGdyYW51bGFyaXR5IElPLiBJdCByZWxpZXMgb24gdGhlIC0+ cndfYnl0ZXMoKSBjYXBhYmlsaXR5DQo+ID4gb2YgcHJvdmlkZWQgbmQgbmFtZXNwYWNlIGRldmlj ZXMuDQo+ID4gDQo+ID4gVGhlIEJUVCB3b3JrcyBhcyBhIHN0YWNrZWQgYmxvY2tlZCBkZXZpY2Us IGFuZCByZXNlcnZlcyBhIGNodW5rIG9mIHNwYWNlDQo+ID4gZnJvbSB0aGUgYmFja2luZyBkZXZp Y2UgZm9yIGl0cyBhY2NvdW50aW5nIG1ldGFkYXRhLiAgQkxLIG5hbWVzcGFjZXMgbWF5DQo+ID4g bWFuZGF0ZSB1c2Ugb2YgYSBCVFQgYW5kIGV4cGVjdCB0aGUgYnVzIHRvIGluaXRpYWxpemUgYSBC VFQgaWYgbm90DQo+ID4gYWxyZWFkeSBwcmVzZW50LiAgT3RoZXJ3aXNlIGlmIGEgQlRUIGlzIGRl c2lyZWQgZm9yIG90aGVyIG5hbWVzcGFjZXMgKG9yDQo+ID4gcGFydGl0aW9ucyBvZiBhIG5hbWVz cGFjZSkgYSBCVFQgbWF5IGJlIG1hbnVhbGx5IGNvbmZpZ3VyZWQuDQo+IC4uLg0KPiANCj4gUnVu bmluZyBidHQgYWJvdmUgcG1lbSB3aXRoIGEgdmFyaWV0eSBvZiB3b3JrbG9hZHMsIEkgc2VlIGFu IGF3ZnVsIGxvdCANCj4gb2YgdGltZSBzcGVudCBpbiB0d28gcGxhY2VzOg0KPiAqIF9yYXdfc3Bp bl9sb2NrIA0KPiAqIGJ0dF9tYWtlX3JlcXVlc3QNCg0KSSdtIGN1cmlvdXMgYWJvdXQgd2h5IF9y YXdfc3Bpbl9sb2NrIG1pZ2h0IGJlIHRha2luZyB1cCBzbyBtdWNoIHRpbWUgLQ0KZG9lcyB0aGUg dGVzdCBzeXN0ZW0gaGF2ZSBtb3JlIHRoYW4gMjU2IENQVXM/IEV2ZW4gaWdub3JpbmcgRGFuJ3Mg Zml4DQpmb3IgbnJfY3B1cyB3aGljaCB3aWxsIGFsbG93IHRoZSBsb2NrbGVzcyBwYXRoIG9wdGlt aXphdGlvbiB0byBraWNrIGluDQooY29taW5nIGluIHYzKSwgdGhlIGxhbmUgc3BpbmxvY2tzIHNo b3VsZCBiZSBsYXJnZWx5IHVuY29udGVzdGVkIHVubGVzcw0KeW91IGhhdmUgbW9yZSB0aGFuIDI1 NiBDUFVzLg0KDQpPbmUgZXhwbGFuYXRpb24gY291bGQgYmUgdGhhdCB0aGlzIGlzIGJlaW5nIGRv bWluYXRlZCBieSAnbWFwX2xvY2tzJyAtDQpJJ2QgbmVlZCB0byBnZXQgbXkgcGVyZiBzZXR1cCBn b2luZyBhZ2FpbiB0byBmaW5kIG91dC4gKFRoZSBtb3JlIEkgdGhpbmsNCmFib3V0IGl0LCB0aGUg bW9yZSBJJ20gY29udmluY2VkIHRoaXMgaXMgdGhlIGNhc2UuIEZvcnR1bmF0ZWx5IGl0IGlzDQpz aW1wbGUgdG8gcmVkdWNlIG1hcF9sb2NrIGNvbnRlbnRpb24gYnkgaW5jcmVhc2luZyB0aGUgbnVt YmVyIG9mIGxvY2tzKS4NCg0KPiANCj4gVGhpcyBvY2N1cnMgZm9yIGZpbyB0byByYXcgL2Rldi9u ZE4gZGV2aWNlcywgZGRwdCBvdmVyIGV4dDQgb3IgeGZzLA0KPiBjcCAtUiBvZiBsYXJnZSBkaXJl Y3RvcmllcywgYW5kIHJ1bm5pbmcgbWFrZSBvbiB0aGUgbGludXgga2VybmVsLg0KPiANCj4gU29t ZSBzcGVjaWZpYyByZXN1bHRzOg0KPiANCj4gZmlvIDQgS2lCIHJhbmRvbSByZWFkcywgV0MgY2Fj aGUgdHlwZSwgbWVtY3B5Og0KPiAqIDQzMTc1IE1CL3MsICAgOCBNIElPUFMgIHBtZW0wIGFuZCBw bWVtMQ0KPiAqIDE4NTAwIE1CL3MsIDEuNSBNIElPUFMgIG5kMCBhbmQgbmQxDQo+IA0KPiBmaW8g NCBLaUIgcmFuZG9tIHJlYWRzLCBXQyBjYWNoZSB0eXBlLCBtZW1jcHkgd2l0aCBub24tdGVtcG9y YWwNCj4gbG9hZHMgKHdoZW4gZXZlcnl0aGluZyBpcyA2NC1ieXRlIGFsaWduZWQpOg0KPiAqIDMz ODE0IE1CL3MsIDQuMyBNIElPUFMgIG5kMCBhbmQgbmQxDQo+IA0KPiBaZXJvaW5nIG91dCAzMiBN aUIgd2l0aCBkZHB0Og0KPiAqIDE5IHMsIDE4MDAgTWlCL3MJcG1lbQ0KPiAqIDU1IHMsICA2MjUg TWlCL3MJYnR0DQo+IA0KPiBJZiBidHRfbWFrZV9yZXF1ZXN0IG5lZWRzIHRvIHN0YWxsIHRoaXMg bXVjaCwgbWF5YmUgaXQnZCBiZSBiZXR0ZXINCj4gdG8gdXRpbGl6ZSB0aGUgYmxrLW1xIHJlcXVl c3QgcXVldWVzLCBrZWVwaW5nIHJlcXVlc3RzIGluIHBlci1DUFUNCj4gcXVldWVzIHdoaWxlIHRo ZXkncmUgd2FpdGluZywgYW5kIHVzaW5nIElQSXMgZm9yIGNvbXBsZXRpb24gDQo+IGludGVycnVw dHMgd2hlbiB0aGV5J3JlIGZpbmFsbHkgZG9uZS4NCj4gDQo+IA0KPiBmaW8gNCBLaUIgcmFuZG9t IHJlYWRzIHdpdGhvdXQgbm9uLXRlbXBvcmFsIG1lbWNweQ0KPiA9PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KPiBwZXJmIHRvcCBzaG93cyBtZW1jcHlf ZXJtcyB0YWtpbmcgYWxsIHRoZSB0aW1lLCBhIGZ1bmN0aW9uIHRoYXQNCj4gdXNlcyA4LWJ5dGUg UkVQOyBNT1ZTQiBpbnN0cnVjdGlvbnM6DQo+ICA4NS43OCUgIFtrZXJuZWxdICAgICAgICAgICAg IFtrXSBtZW1jcHlfZXJtcw0KPiAgIDEuMjElICBba2VybmVsXSAgICAgICAgICAgICBba10gX3Jh d19zcGluX2xvY2sNCj4gICAwLjcyJSAgW25kX2J0dF0gICAgICAgICAgICAgW2tdIGJ0dF9tYWtl X3JlcXVlc3QNCj4gICAwLjY3JSAgW2tlcm5lbF0gICAgICAgICAgICAgW2tdIGRvX2Jsb2NrZGV2 X2RpcmVjdF9JTw0KPiAgIDAuNDclICBmaW8gICAgICAgICAgICAgICAgICBbLl0gZ2V0X2lvX3UN Cj4gDQo+IGZpbyA0IEtpQiByYW5kb20gcmVhZHMgd2l0aCBub24tdGVtcG9yYWwgbWVtY3B5DQo+ ID09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQo+IHBlcmYg dG9wIHNob3dzIHRoZXJlIGFyZSBzdGlsbCBxdWl0ZSBhIGZldyB1bmFsaWduZWQgYWNjZXNzZXMN Cj4gcmVzdWx0aW5nIGluIGxlZ2FjeSBtZW1jcHksIGJ1dCBhYm91dCBlcXVhbCB0aW1lIGlzIG5v dyBzcGVudA0KPiBpbiBsZWdhY3kgdnMgTlQgbWVtY3B5Og0KDQpJIHRha2UgaXQgbWVtY3B5X2xu dF9zdF82NCBleHBlY3RzIDY0QiBhbGlnbmVkIGFjY2Vzc2VzPyBUaGUgQlRUIHdpbGwNCmFsd2F5 cyBoYXZlIHNvbWUgdW5hbGlnbmVkIGFjY2Vzc2VzIC0gdGhlIG1hcCB1cGRhdGVzIGFyZSA0QiBs b25nLiBGbG9nDQp1cGRhdGVzIGFyZSAxNkIgbG9uZywgYnV0IGFib3V0IGhhbGYgb2YgdGhlbSB3 aWxsIG5vdCBzdGFydCBhdCBhIDY0Qg0KYWxpZ25tZW50IGR1ZSB0byBob3cgaXQgaXMgbGFpZCBv dXQuIEknbSBvcGVuIHRvIGhlYXJpbmcgaWRlYXMgYWJvdXQgaG93DQp0aGlzIGNvdWxkIGJlIGlt cHJvdmVkLiBPbmUgd2F5LCBhcyBEYW4gYWxyZWFkeSBzYWlkIGlzIHJlZHVjaW5nIHRoZQ0KbnVt YmVyIG9mIG1ldGFkYXRhIHVwZGF0ZXMgYnkgY29uZmlndXJpbmcgYSBsYXJnZXIgKDRLKSBzZWN0 b3Igc2l6ZS4NCg0KPiAgMzAuNDclICBba2VybmVsXSAgICAgICAgICAgIFtrXSBtZW1jcHlfZXJt cw0KPiAgMjYuMjclICBba2VybmVsXSAgICAgICAgICAgIFtrXSBtZW1jcHlfbG50X3N0XzY0DQo+ ICAgNS4zNyUgIFtrZXJuZWxdICAgICAgICAgICAgW2tdIF9yYXdfc3Bpbl9sb2NrDQo+ICAgMi4y MCUgIFtrZXJuZWxdICAgICAgICAgICAgW2tdIGJ0dF9tYWtlX3JlcXVlc3QNCj4gICAyLjAzJSAg W2tlcm5lbF0gICAgICAgICAgICBba10gZG9fYmxvY2tkZXZfZGlyZWN0X0lPDQo+ICAgMS40MSUg IGZpbyAgICAgICAgICAgICAgICAgWy5dIGdldF9pb191DQo+ICAgMS4yMiUgIFtrZXJuZWxdICAg ICAgICAgICAgW2tdIGJ0dF9tYXBfcmVhZA0KPiAgIDEuMTUlICBba2VybmVsXSAgICAgICAgICAg IFtrXSBwbWVtX3J3X2J5dGVzDQo+ICAgMS4wMSUgIFtrZXJuZWxdICAgICAgICAgICAgW2tdIG5k X2J0dF9yd19ieXRlcw0KPiAgIDAuOTglICBba2VybmVsXSAgICAgICAgICAgIFtrXSBuZF9yZWdp b25fYWNxdWlyZV9sYW5lDQo+ICAgMC44OSUgIGZpbyAgICAgICAgICAgICAgICAgWy5dIGdldF9u ZXh0X3JhbmRfYmxvY2sNCj4gICAwLjg4JSAgZmlvICAgICAgICAgICAgICAgICBbLl0gdGhyZWFk X21haW4NCj4gICAwLjc5JSAgZmlvICAgICAgICAgICAgICAgICBbLl0gaW9zX2NvbXBsZXRlZA0K PiAgIDAuNzYlICBmaW8gICAgICAgICAgICAgICAgIFsuXSB0ZF9pb19xdWV1ZQ0KPiAgIDAuNzUl ICBba2VybmVsXSAgICAgICAgICAgIFtrXSBfcmF3X3NwaW5fbG9ja19pcnFzYXZlDQo+ICAgMC42 OCUgIFtrZXJuZWxdICAgICAgICAgICAgW2tdIGttZW1fY2FjaGVfZnJlZQ0KPiAgIDAuNjYlICBb a2VybmVsXSAgICAgICAgICAgIFtrXSBrbWVtX2NhY2hlX2FsbG9jDQo+ICAgMC41OSUgIFtrZXJu ZWxdICAgICAgICAgICAgW2tdIF9fYXVkaXRfc3lzY2FsbF9leGl0DQo+ICAgMC41NyUgIFtrZXJu ZWxdICAgICAgICAgICAgW2tdIGFpb19jb21wbGV0ZQ0KPiAgIDAuNTQlICBba2VybmVsXSAgICAg ICAgICAgIFtrXSBkb19pb19zdWJtaXQNCj4gICAwLjUyJSAgW2tlcm5lbF0gICAgICAgICAgICBb a10gX3Jhd19zcGluX3VubG9ja19pcnFyZXN0b3JlDQo+IA0KPiBmaW8gcmFuZHJ3IHdvcmtsb2Fk DQo+ID09PT09PT09PT09PT09PT09PT0NCj4gcGVyZiB0b3Agc2hvd3MgdGhhdCBhZGRpbmcgd3Jp dGVzIHRvIHRoZSBtaXggYnJpbmdzIGJ0dF9tYWtlX3JlcXVlc3QNCj4gaXRzIGNwdV9yZWxheCgp IGxvb3AgdG8gdGhlIGZvcmVmcm9udDoNCg0KVGhpcyBpcyBnb29kIGRhdGEgLSB3ZSd2ZSB0aG91 Z2h0IGFib3V0IGJlbmNobWFya2luZyBhbiBhbHRlcm5hdGUNCnN0cmF0ZWd5IGZvciB0aGUgJ1Jl YWQgVHJhY2tpbmcgVGFibGUnIC0gaW5zdGVhZCBvZiBhIGxpbmVhciBhcnJheSwgdXNlDQphIGhh c2ggdGFibGUgKGhhc2ggbmV3X3Bvc3RtYXAgaW50byBhbiBSVFQgc2xvdCkgLiBJdCB3aWxsIGFk ZCBmYWxzZQ0Kd2FpdGluZyBkdWUgdG8gaGFzaCBjb2xsaXNpb25zLCBidXQgdGhlIHdyaXRlcnMg d29uJ3QgaGF2ZSB0byB3YWxrIHRoZQ0KYXJyYXkgLSB0aGV5IGNhbiBqdXN0IGhhc2ggaW50byB0 aGUgcmlnaHQgc3BvdC4NCg0KQW5vdGhlciBvcHRpb24gdGhhdCBEYW4gc3VnZ2VzdGVkIGNvdWxk IGJlIGluc3RlYWQgb2Ygd2FpdGluZyBpbiB0aGUNCmNwdV9yZWxheCgpIGxvb3AsIHdlIGRyb3Ag dGhlIGxhbmUsIGFuZCBzdGFydCBvdmVyIHdpdGggX2FjcXVpcmVfbGFuZS4NClRoaXMgd2F5LCB3 ZSBhcmUgd2FpdGluZyBpbiB0aGUgc3Bpbl9sb2NrIHN0YWdlLCB0aHVzIGdpdmluZyBhIGNoYW5j ZSB0bw0Kb3RoZXIgdGhyZWFkcy9DUFVzIHRoYXQgcmVzb2x2ZSB0byB0aGUgc2FtZSBzcGlubG9j ayB0byBydW4uDQoNCj4gICAyMS4wOSUgIFtuZF9idHRdICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgW2tdIGJ0dF9tYWtlX3JlcXVlc3QgDQo+ICAgMTkuMDYlICBba2VybmVsXSAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIFtrXSBtZW1jcHlfZXJtcyAgDQo+ICAgMTQuMzUlICBba2Vy bmVsXSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtrXSBfcmF3X3NwaW5fbG9jayAgIA0K PiAgIDEwLjM4JSAgW25kX3BtZW1dICAgICAgICAgICAgICAgICAgICAgICAgICAgICBba10gbWVt Y3B5X2xudF9zdF82NCAgICANCj4gICAgMS41NyUgIFtrZXJuZWxdICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgW2tdIGRvX2Jsb2NrZGV2X2RpcmVjdF9JTyAgIA0KPiAgICAxLjUxJSAgW25k X3BtZW1dICAgICAgICAgICAgICAgICAgICAgICAgICAgICBba10gbWVtY3B5X2x0X3NudF82NCAg ICAgIA0KPiAgICAxLjQzJSAgW25kX2J0dF0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICBb a10gbmRfYnR0X3J3X2J5dGVzICAgICAgIA0KPiAgICAxLjM5JSAgW2tlcm5lbF0gICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBba10gcmFkaXhfdHJlZV9uZXh0X2NodW5rICANCj4gICAgMS4z MyUgIFtrZXJuZWxdICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW2tdIHB1dF9wYWdlICAg ICAgICAgICAgIA0KPiAgICAxLjIxJSAgW25kX3BtZW1dICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBba10gcG1lbV9yd19ieXRlcyAgICAgIA0KPiAgICAxLjExJSAgZmlvICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBbLl0gZ2V0X2lvX3UgICAgICAgICAgDQo+ICAgIDAuOTAl ICBmaW8gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFsuXSBpb191X3F1ZXVlZF9j b21wbGV0ZSAgDQo+ICAgIDAuNzQlICBba2VybmVsXSAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIFtrXSBzeXN0ZW1fY2FsbCAgICAgICAgIA0KPiAgICAwLjcyJSAgW2xpYm5kXSAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBba10gbmRfcmVnaW9uX2FjcXVpcmVfbGFuZSAgIA0KPiAg ICAwLjcxJSAgW25kX2J0dF0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICBba10gYnR0X21h cF9yZWFkICAgICAgICAgICAgDQo+ICAgIDAuNjIlICBmaW8gICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIFsuXSB0aHJlYWRfbWFpbiAgICAgICAgICAgDQo+IA0KPiBpbnNpZGUgYnR0 X21ha2VfcmVxdWVzdDoNCj4gDQo+ICAgICAgICDCpiAgICAgICAgICAgICAgICAgICAgIC8qIFdh aXQgaWYgdGhlIG5ldyBibG9jayBpcyBiZWluZyByZWFkIGZyb20gKi8NCj4gICAgICAgIMKmICAg ICAgICAgICAgICAgICAgICAgZm9yIChpID0gMDsgaSA8IGFyZW5hLT5uZnJlZTsgaSsrKQ0KPiAg IDIuOTggwqYgICAgID8gamUgICAgIDJiNA0KPiAgIDAuMDUgwqYgICAgICAgbW92ICAgIDB4NjAo JXIxNCksJXJheA0KPiAgIDAuMDAgwqYgICAgICAgbW92ICAgICVlYngsJWVkeA0KPiAgICAgICAg wqYgICAgICAgeG9yICAgICVlc2ksJWVzaQ0KPiAgIDAuMDMgwqYgICAgICAgb3IgICAgICQweDgw MDAwMDAwLCVlZHgNCj4gICAwLjA1IMKmICAgICAgIG5vcA0KPiAgICAgICAgwqYgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHdoaWxlIChhcmVuYS0+cnR0W2ldID09IChSVFRfVkFMSUQgfCBu ZXdfcG9zdG1hcCkpDQo+ICAyMi45OCDCpjI5MDogICBtb3YgICAgJWVzaSwlZWRpDQo+ICAgMC4w MSDCpiAgICAgICBjbXAgICAgJWVkeCwoJXJheCwlcmRpLDQpDQo+ICAzMC45NyDCpiAgICAgICBs ZWEgICAgMHgwKCwlcmRpLDQpLCVyY3gNCj4gIDIxLjA1IMKmICAgICA/IGpuZSAgICAyYWINCj4g ICAgICAgIMKmICAgICAgIG5vcA0KPiAgICAgICAgwqYgICAgIH0NCj4gICAgICAgIMKmDQo+ICAg ICAgICDCpiAgICAgLyogUkVQIE5PUCAoUEFVU0UpIGlzIGEgZ29vZCB0aGluZyB0byBpbnNlcnQg aW50byBidXN5LXdhaXQgbG9vcHMuICovDQo+ICAgICAgICDCpiAgICAgc3RhdGljIGlubGluZSB2 b2lkIHJlcF9ub3Aodm9pZCkNCj4gICAgICAgIMKmICAgICB7DQo+ICAgICAgICDCpiAgICAgICAg ICAgICBhc20gdm9sYXRpbGUoInJlcDsgbm9wIiA6OjogIm1lbW9yeSIpOw0KPiAgICAgICAgwqYy YTA6ICAgcGF1c2UNCj4gICAgICAgIMKmICAgICAgIG1vdiAgICAweDYwKCVyMTQpLCVyYXgNCj4g ICAgICAgIMKmICAgICAgIGNtcCAgICAoJXJheCwlcmN4LDEpLCVlZHgNCj4gICAgICAgIMKmICAg ICA/IGplICAgICAyYTANCj4gICAgICAgIMKmICAgICAgICAgICAgICAgICAgICAgfQ0KPiANCj4g DQo+IGRkcHQgemVyb2luZyBvdXQNCj4gPT09PT09PT09PT09PT09PQ0KPiBwZXJmIHRvcCBzaG93 cyAyNyUgaW4gc3BpbmxvY2tzLCBhbmQgMTQlIGluIGJ0dF9tYWtlX3JlcXVlc3QgKGFsbCBpbiAN Cj4gdGhlICJ3YWl0IGlmIHRoZSBuZXcgYmxvY2sgaXMgYmVpbmcgcmVhZCBmcm9tIiBsb29wKS4N Cj4gDQo+ICAgMjYuNDglICBba2VybmVsXSAgICAgICAgICAgICAgICAgICAgICBba10gX3Jhd19z cGluX2xvY2sgICANCj4gICAxNC40NiUgIFtuZF9idHRdICAgICAgICAgICAgICAgICAgICAgIFtr XSBidHRfbWFrZV9yZXF1ZXN0ICANCj4gICAxMy4xNCUgIFtrZXJuZWxdICAgICAgICAgICAgICAg ICAgICAgIFtrXSBtZW1jcHlfZXJtcyAgICANCj4gICAxMC4zNCUgIFtrZXJuZWxdICAgICAgICAg ICAgICAgICAgICAgIFtrXSBjb3B5X3VzZXJfZW5oYW5jZWRfZmFzdF9zdHJpbmcgDQo+ICAgIDMu MTIlICBbbmRfcG1lbV0gICAgICAgICAgICAgICAgICAgICBba10gbWVtY3B5X2x0X3NudF82NCAg DQo+ICAgIDEuMTUlICBba2VybmVsXSAgICAgICAgICAgICAgICAgICAgICBba10gX19ibG9ja19j b21taXRfd3JpdGUuaXNyYS4yMSANCj4gICAgMC45NiUgIFtuZF9wbWVtXSAgICAgICAgICAgICAg ICAgICAgIFtrXSBwbWVtX3J3X2J5dGVzIA0KPiAgICAwLjk2JSAgW25kX2J0dF0gICAgICAgICAg ICAgICAgICAgICAgW2tdIG5kX2J0dF9yd19ieXRlcyANCj4gICAgMC44NiUgIFtrZXJuZWxdICAg ICAgICAgICAgICAgICAgICAgIFtrXSB1bmxvY2tfcGFnZSAgICAgDQo+ICAgIDAuNjUlICBba2Vy bmVsXSAgICAgICAgICAgICAgICAgICAgICBba10gX3Jhd19zcGluX2xvY2tfaXJxc2F2ZSANCj4g ICAgMC41OCUgIFtrZXJuZWxdICAgICAgICAgICAgICAgICAgICAgIFtrXSBiZGV2X3JlYWRfb25s eSANCj4gICAgMC41NiUgIFtrZXJuZWxdICAgICAgICAgICAgICAgICAgICAgIFtrXSByZWxlYXNl X3BhZ2VzICANCj4gICAgMC41NCUgIFtuZF9wbWVtXSAgICAgICAgICAgICAgICAgICAgIFtrXSBt ZW1jcHlfbG50X3N0XzY0ICANCj4gICAgMC41MyUgIFtleHQ0XSAgICAgICAgICAgICAgICAgICAg ICAgIFtrXSBleHQ0X21hcmtfaWxvY19kaXJ0eSAgIA0KPiAgICAwLjUyJSAgW2tlcm5lbF0gICAg ICAgICAgICAgICAgICAgICAgW2tdIF9fd2FrZV91cF9iaXQgICANCj4gICAgMC41MiUgIFtrZXJu ZWxdICAgICAgICAgICAgICAgICAgICAgIFtrXSBfX2NsZWFyX3VzZXIgICANCj4gDQo+IC0tLQ0K PiBSb2JlcnQgRWxsaW90dCwgSFAgU2VydmVyIFN0b3JhZ2UNCg0KVGhhbmtzLCB0aGlzIGlzIGdy ZWF0IGFuYWx5c2lzIC0gSSdkIGJlIHF1aXRlIGludGVyZXN0ZWQgaW4gaG93IHRoZXNlDQpudW1i ZXJzIGxvb2sgbGlrZSB3aXRoIHRoZSB0d28gY2hhbmdlcyBEYW4gbWVudGlvbmVkLg0KDQoJLVZp c2hhbA0K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754797AbbERWiQ (ORCPT ); Mon, 18 May 2015 18:38:16 -0400 Received: from mga02.intel.com ([134.134.136.20]:48823 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754383AbbERWiN (ORCPT ); Mon, 18 May 2015 18:38:13 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.13,455,1427785200"; d="scan'208";a="731403430" From: "Verma, Vishal L" To: "Elliott@hp.com" CC: "linux-kernel@vger.kernel.org" , "mingo@kernel.org" , "neilb@suse.de" , "hch@lst.de" , "Williams, Dan J" , "axboe@fb.com" , "hpa@zytor.com" , "linux-nvdimm@lists.01.org" , "gregkh@linuxfoundation.org" , "luto@amacapital.net" , "david@fromorbit.com" Subject: Re: [Linux-nvdimm] [PATCH v2 19/20] nd_btt: atomic sector updates Thread-Topic: [Linux-nvdimm] [PATCH v2 19/20] nd_btt: atomic sector updates Thread-Index: AQHQkbtRxwNKdplhG0iltM6wnt3fbA== Date: Mon, 18 May 2015 22:38:11 +0000 Message-ID: <1431988690.15857.31.camel@intel.com> References: <20150428181203.35812.60474.stgit@dwillia2-desk3.amr.corp.intel.com> <20150428182557.35812.38292.stgit@dwillia2-desk3.amr.corp.intel.com> <94D0CD8314A33A4D9D801C0FE68B40295A9097E5@G9W0745.americas.hpqcorp.net> In-Reply-To: <94D0CD8314A33A4D9D801C0FE68B40295A9097E5@G9W0745.americas.hpqcorp.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.232.112.38] Content-Type: text/plain; charset="utf-8" Content-ID: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id t4IMcMpb029386 On Sun, 2015-05-17 at 01:19 +0000, Elliott, Robert (Server Storage) wrote: > > -----Original Message----- > > From: Linux-nvdimm [mailto:linux-nvdimm-bounces@lists.01.org] On Behalf Of > > Dan Williams > > Sent: Tuesday, April 28, 2015 1:26 PM > > To: linux-nvdimm@lists.01.org > > Cc: Ingo Molnar; Neil Brown; Greg KH; Dave Chinner; linux- > > kernel@vger.kernel.org; Andy Lutomirski; Jens Axboe; H. Peter Anvin; > > Christoph Hellwig > > Subject: [Linux-nvdimm] [PATCH v2 19/20] nd_btt: atomic sector updates > > > > From: Vishal Verma > > > > BTT stands for Block Translation Table, and is a way to provide power > > fail sector atomicity semantics for block devices that have the ability > > to perform byte granularity IO. It relies on the ->rw_bytes() capability > > of provided nd namespace devices. > > > > The BTT works as a stacked blocked device, and reserves a chunk of space > > from the backing device for its accounting metadata. BLK namespaces may > > mandate use of a BTT and expect the bus to initialize a BTT if not > > already present. Otherwise if a BTT is desired for other namespaces (or > > partitions of a namespace) a BTT may be manually configured. > ... > > Running btt above pmem with a variety of workloads, I see an awful lot > of time spent in two places: > * _raw_spin_lock > * btt_make_request I'm curious about why _raw_spin_lock might be taking up so much time - does the test system have more than 256 CPUs? Even ignoring Dan's fix for nr_cpus which will allow the lockless path optimization to kick in (coming in v3), the lane spinlocks should be largely uncontested unless you have more than 256 CPUs. One explanation could be that this is being dominated by 'map_locks' - I'd need to get my perf setup going again to find out. (The more I think about it, the more I'm convinced this is the case. Fortunately it is simple to reduce map_lock contention by increasing the number of locks). > > This occurs for fio to raw /dev/ndN devices, ddpt over ext4 or xfs, > cp -R of large directories, and running make on the linux kernel. > > Some specific results: > > fio 4 KiB random reads, WC cache type, memcpy: > * 43175 MB/s, 8 M IOPS pmem0 and pmem1 > * 18500 MB/s, 1.5 M IOPS nd0 and nd1 > > fio 4 KiB random reads, WC cache type, memcpy with non-temporal > loads (when everything is 64-byte aligned): > * 33814 MB/s, 4.3 M IOPS nd0 and nd1 > > Zeroing out 32 MiB with ddpt: > * 19 s, 1800 MiB/s pmem > * 55 s, 625 MiB/s btt > > If btt_make_request needs to stall this much, maybe it'd be better > to utilize the blk-mq request queues, keeping requests in per-CPU > queues while they're waiting, and using IPIs for completion > interrupts when they're finally done. > > > fio 4 KiB random reads without non-temporal memcpy > ================================================== > perf top shows memcpy_erms taking all the time, a function that > uses 8-byte REP; MOVSB instructions: > 85.78% [kernel] [k] memcpy_erms > 1.21% [kernel] [k] _raw_spin_lock > 0.72% [nd_btt] [k] btt_make_request > 0.67% [kernel] [k] do_blockdev_direct_IO > 0.47% fio [.] get_io_u > > fio 4 KiB random reads with non-temporal memcpy > =============================================== > perf top shows there are still quite a few unaligned accesses > resulting in legacy memcpy, but about equal time is now spent > in legacy vs NT memcpy: I take it memcpy_lnt_st_64 expects 64B aligned accesses? The BTT will always have some unaligned accesses - the map updates are 4B long. Flog updates are 16B long, but about half of them will not start at a 64B alignment due to how it is laid out. I'm open to hearing ideas about how this could be improved. One way, as Dan already said is reducing the number of metadata updates by configuring a larger (4K) sector size. > 30.47% [kernel] [k] memcpy_erms > 26.27% [kernel] [k] memcpy_lnt_st_64 > 5.37% [kernel] [k] _raw_spin_lock > 2.20% [kernel] [k] btt_make_request > 2.03% [kernel] [k] do_blockdev_direct_IO > 1.41% fio [.] get_io_u > 1.22% [kernel] [k] btt_map_read > 1.15% [kernel] [k] pmem_rw_bytes > 1.01% [kernel] [k] nd_btt_rw_bytes > 0.98% [kernel] [k] nd_region_acquire_lane > 0.89% fio [.] get_next_rand_block > 0.88% fio [.] thread_main > 0.79% fio [.] ios_completed > 0.76% fio [.] td_io_queue > 0.75% [kernel] [k] _raw_spin_lock_irqsave > 0.68% [kernel] [k] kmem_cache_free > 0.66% [kernel] [k] kmem_cache_alloc > 0.59% [kernel] [k] __audit_syscall_exit > 0.57% [kernel] [k] aio_complete > 0.54% [kernel] [k] do_io_submit > 0.52% [kernel] [k] _raw_spin_unlock_irqrestore > > fio randrw workload > =================== > perf top shows that adding writes to the mix brings btt_make_request > its cpu_relax() loop to the forefront: This is good data - we've thought about benchmarking an alternate strategy for the 'Read Tracking Table' - instead of a linear array, use a hash table (hash new_postmap into an RTT slot) . It will add false waiting due to hash collisions, but the writers won't have to walk the array - they can just hash into the right spot. Another option that Dan suggested could be instead of waiting in the cpu_relax() loop, we drop the lane, and start over with _acquire_lane. This way, we are waiting in the spin_lock stage, thus giving a chance to other threads/CPUs that resolve to the same spinlock to run. > 21.09% [nd_btt] [k] btt_make_request > 19.06% [kernel] [k] memcpy_erms > 14.35% [kernel] [k] _raw_spin_lock > 10.38% [nd_pmem] [k] memcpy_lnt_st_64 > 1.57% [kernel] [k] do_blockdev_direct_IO > 1.51% [nd_pmem] [k] memcpy_lt_snt_64 > 1.43% [nd_btt] [k] nd_btt_rw_bytes > 1.39% [kernel] [k] radix_tree_next_chunk > 1.33% [kernel] [k] put_page > 1.21% [nd_pmem] [k] pmem_rw_bytes > 1.11% fio [.] get_io_u > 0.90% fio [.] io_u_queued_complete > 0.74% [kernel] [k] system_call > 0.72% [libnd] [k] nd_region_acquire_lane > 0.71% [nd_btt] [k] btt_map_read > 0.62% fio [.] thread_main > > inside btt_make_request: > > ¦ /* Wait if the new block is being read from */ > ¦ for (i = 0; i < arena->nfree; i++) > 2.98 ¦ ? je 2b4 > 0.05 ¦ mov 0x60(%r14),%rax > 0.00 ¦ mov %ebx,%edx > ¦ xor %esi,%esi > 0.03 ¦ or $0x80000000,%edx > 0.05 ¦ nop > ¦ while (arena->rtt[i] == (RTT_VALID | new_postmap)) > 22.98 ¦290: mov %esi,%edi > 0.01 ¦ cmp %edx,(%rax,%rdi,4) > 30.97 ¦ lea 0x0(,%rdi,4),%rcx > 21.05 ¦ ? jne 2ab > ¦ nop > ¦ } > ¦ > ¦ /* REP NOP (PAUSE) is a good thing to insert into busy-wait loops. */ > ¦ static inline void rep_nop(void) > ¦ { > ¦ asm volatile("rep; nop" ::: "memory"); > ¦2a0: pause > ¦ mov 0x60(%r14),%rax > ¦ cmp (%rax,%rcx,1),%edx > ¦ ? je 2a0 > ¦ } > > > ddpt zeroing out > ================ > perf top shows 27% in spinlocks, and 14% in btt_make_request (all in > the "wait if the new block is being read from" loop). > > 26.48% [kernel] [k] _raw_spin_lock > 14.46% [nd_btt] [k] btt_make_request > 13.14% [kernel] [k] memcpy_erms > 10.34% [kernel] [k] copy_user_enhanced_fast_string > 3.12% [nd_pmem] [k] memcpy_lt_snt_64 > 1.15% [kernel] [k] __block_commit_write.isra.21 > 0.96% [nd_pmem] [k] pmem_rw_bytes > 0.96% [nd_btt] [k] nd_btt_rw_bytes > 0.86% [kernel] [k] unlock_page > 0.65% [kernel] [k] _raw_spin_lock_irqsave > 0.58% [kernel] [k] bdev_read_only > 0.56% [kernel] [k] release_pages > 0.54% [nd_pmem] [k] memcpy_lnt_st_64 > 0.53% [ext4] [k] ext4_mark_iloc_dirty > 0.52% [kernel] [k] __wake_up_bit > 0.52% [kernel] [k] __clear_user > > --- > Robert Elliott, HP Server Storage Thanks, this is great analysis - I'd be quite interested in how these numbers look like with the two changes Dan mentioned. -Vishal {.n++%ݶw{.n+{G{ayʇڙ,jfhz_(階ݢj"mG?&~iOzv^m ?I