From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: "Kani, Toshi" To: "torvalds@linux-foundation.org" CC: "linux-kernel@vger.kernel.org" , "alex.williamson@redhat.com" , "linux-block@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "bhelgaas@google.com" , "hch@lst.de" , "axboe@kernel.dk" , "linux-nvdimm@lists.01.org" , "jglisse@redhat.com" , "linux-nvme@lists.infradead.org" , "maxg@mellanox.com" , "linux-pci@vger.kernel.org" , "keith.busch@intel.com" , "benh@kernel.crashing.org" , "jgg@ziepe.ca" , "oliveroh@au1.ibm.com" Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory Date: Fri, 2 Mar 2018 17:38:58 +0000 Message-ID: <1520015040.2693.38.camel@hpe.com> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> <1519936815.4592.25.camel@au1.ibm.com> <20180301205315.GJ19007@ziepe.ca> <1519942012.4592.31.camel@au1.ibm.com> <1519943658.4592.34.camel@kernel.crashing.org> <1520010446.2693.19.camel@hpe.com> In-Reply-To: Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 List-ID: T24gRnJpLCAyMDE4LTAzLTAyIGF0IDA4OjU3IC0wODAwLCBMaW51cyBUb3J2YWxkcyB3cm90ZToN Cj4gT24gRnJpLCBNYXIgMiwgMjAxOCBhdCA4OjIyIEFNLCBLYW5pLCBUb3NoaSA8dG9zaGkua2Fu aUBocGUuY29tPiB3cm90ZToNCj4gPiANCj4gPiBGV0lXLCB0aGlzIHRoaW5nIGlzIGNhbGxlZCBN VFJScyBvbiB4ODYsIHdoaWNoIGFyZSBpbml0aWFsaXplZCBieSBCSU9TLg0KPiANCj4gTm8uDQo+ IA0KPiBPciByYXRoZXIsIHRoYXQncyBzaW1wbHkganVzdCBhbm90aGVyIChzbWFsbCkgcGFydCBv ZiBpdCBhbGwgLSBhbmQgYW4NCj4gYXJjaGl0ZWN0ZWQgYW5kIGRvY3VtZW50ZWQgb25lIGF0IHRo YXQuDQo+IA0KPiBMaWtlIHRoZSBwYWdlIHRhYmxlIGNhY2hpbmcgZW50cmllcywgdGhlIG1lbW9y eSB0eXBlIHJhbmdlIHJlZ2lzdGVycw0KPiBhcmUgcmVhbGx5IGp1c3QgInNlY29uZGFyeSBpbmZv cm1hdGlvbiIuIFRoZXkgZG9uJ3QgYWN0dWFsbHkgc2VsZWN0DQo+IGJldHdlZW4gUENJZSBhbmQg UkFNLCB0aGV5IGp1c3QgYWZmZWN0IHRoZSBiZWhhdmlvciBvbiB0b3Agb2YgdGhhdC4NCj4gDQo+ IFRoZSByZWFsbHkgbml0dHktZ3JpdHR5IHN0dWZmIGlzIG5vdCBhcmNoaXRlY3RlZCwgYW5kIGdl bmVyYWxseSBub3QNCj4gZG9jdW1lbnRlZCBvdXRzaWRlIChwb3NzaWJseSkgdGhlIEJJT1Mgd3Jp dGVyJ3MgZ3VpZGUgdGhhdCBpcyBub3QgbWFkZQ0KPiBwdWJsaWMuDQo+IA0KPiBUaG9zZSBtYWdp Y2FsIHJlZ2lzdGVycyBjb250YWluIGRldGFpbHMgbGlrZSBob3cgdGhlIERSQU0gaXMNCj4gaW50 ZXJsZWF2ZWQgKGlmIGl0IGlzKSwgd2hhdCB0aGUgdGltaW5ncyBhcmUsIHdoZXJlIHdoaWNoIG1l bW9yeQ0KPiBjb250cm9sbGVyIGhhbmRsZXMgd2hpY2ggbWVtb3J5IHJhbmdlLCBhbmQgd2hhdCBh cmUgZ29lcyB0byBQQ0llIGV0Yy4NCj4gDQo+IEJhc2ljYWxseSBhbGwgdGhlIGFjdHVhbCAqc3Rl ZXJpbmcqIGluZm9ybWF0aW9uIGlzIHZlcnkgbXVjaCBoaWRkZW4NCj4gYXdheSBmcm9tIHRoZSBr ZXJuZWwgKGFuZCBvZnRlbiBmcm9tIHRoZSBCSU9TIHRvbykuIFRoZSBwYXJ0cyB3ZSBzZWUNCj4g YXQgYSBoaWdoZXIgbGV2ZWwgYXJlIGp1c3QgdHVuaW5nIGFuZCB0d2Vha3MuDQo+IA0KPiBOb3Rl OiB0aGUgZGV0YWlscyBkaWZmZXIgX2Vub3Jtb3VzbHlfIGJldHdlZW4gZGlmZmVyZW50IGNoaXBz LiBUaGUNCj4gc2V0dXAgY2FuIGJlIHZlcnkgZGlmZmVyZW50LCB3aXRoIHRoaW5ncyBsaWtlIEtu aWdodHMgTGFuZGluZyBoYXZpbmcNCj4gdGhlIGV4dGVybmFsIGNhY2hlIHRoYXQgY2FuIGFsc28g YWN0IGFzIGxvY2FsIG1lbW9yeSB0aGF0IGlzbid0IGENCj4gY2FjaGUgYnV0IG1hcHMgYXQgYSBk aWZmZXJlbnQgcGh5c2ljYWwgYWRkcmVzcyBpbnN0ZWFkIGV0Yy4gVGhhdCdzIHRoZQ0KPiBraW5k IG9mIHN0ZWVyaW5nIEknbSB0YWxraW5nIGFib3V0IC0gYXQgYSBsb3cgbGV2ZWwgaG93IHBoeXNp Y2FsDQo+IGFkZHJlc3NlcyBnZXQgbWFwcGVkIHRvIGRpZmZlcmVudCBjYWNoZSBwYXJ0aXRpb25z LCBtZW1vcnkNCj4gY29udHJvbGxlcnMsIG9yIHRvIHRoZSBJTyBzeXN0ZW0gZXRjLg0KDQpSaWdo dCwgTVJDIGNvZGUgaXMgbm90IGRvY3VtZW50ZWQgcHVibGljbHksIGFuZCBpdCBpcyB2ZXJ5IG11 Y2ggQ1BVDQpkZXBlbmRlbnQuICBJdCBwcm9ncmFtcyBhZGRyZXNzIGRlY29kZXJzIGFuZCBtYXBz IERSQU1zIHRvIHBoeXNpY2FsDQphZGRyZXNzIGFzIHlvdSBkZXNjcmliZWQuICBNVFJScyBoYXZl IG5vdGhpbmcgdG8gZG8gd2l0aCB0aGlzIG1lbW9yeQ0KY29udHJvbGxlciBzZXR0aW5nLiAgVGhh dCBzYWlkLCBNVFJScyBzcGVjaWZ5IENQVSdzIG1lbW9yeSBhY2Nlc3MgdHlwZSwNCnN1Y2ggYXMg VUMgYW5kIFdCLg0KDQpUaGFua3MsDQotVG9zaGkNCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from g2t2354.austin.hpe.com (g2t2354.austin.hpe.com [15.233.44.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 9FE652249273B for ; Fri, 2 Mar 2018 09:32:52 -0800 (PST) From: "Kani, Toshi" Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory Date: Fri, 2 Mar 2018 17:38:58 +0000 Message-ID: <1520015040.2693.38.camel@hpe.com> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> <1519936815.4592.25.camel@au1.ibm.com> <20180301205315.GJ19007@ziepe.ca> <1519942012.4592.31.camel@au1.ibm.com> <1519943658.4592.34.camel@kernel.crashing.org> <1520010446.2693.19.camel@hpe.com> In-Reply-To: Content-Language: en-US Content-ID: MIME-Version: 1.0 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: "torvalds@linux-foundation.org" Cc: "axboe@kernel.dk" , "keith.busch@intel.com" , "oliveroh@au1.ibm.com" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-block@vger.kernel.org" , "jgg@ziepe.ca" , "alex.williamson@redhat.com" , "jglisse@redhat.com" , "benh@kernel.crashing.org" , "bhelgaas@google.com" , "maxg@mellanox.com" , "hch@lst.de" List-ID: On Fri, 2018-03-02 at 08:57 -0800, Linus Torvalds wrote: > On Fri, Mar 2, 2018 at 8:22 AM, Kani, Toshi wrote: > > > > FWIW, this thing is called MTRRs on x86, which are initialized by BIOS. > > No. > > Or rather, that's simply just another (small) part of it all - and an > architected and documented one at that. > > Like the page table caching entries, the memory type range registers > are really just "secondary information". They don't actually select > between PCIe and RAM, they just affect the behavior on top of that. > > The really nitty-gritty stuff is not architected, and generally not > documented outside (possibly) the BIOS writer's guide that is not made > public. > > Those magical registers contain details like how the DRAM is > interleaved (if it is), what the timings are, where which memory > controller handles which memory range, and what are goes to PCIe etc. > > Basically all the actual *steering* information is very much hidden > away from the kernel (and often from the BIOS too). The parts we see > at a higher level are just tuning and tweaks. > > Note: the details differ _enormously_ between different chips. The > setup can be very different, with things like Knights Landing having > the external cache that can also act as local memory that isn't a > cache but maps at a different physical address instead etc. That's the > kind of steering I'm talking about - at a low level how physical > addresses get mapped to different cache partitions, memory > controllers, or to the IO system etc. Right, MRC code is not documented publicly, and it is very much CPU dependent. It programs address decoders and maps DRAMs to physical address as you described. MTRRs have nothing to do with this memory controller setting. That said, MTRRs specify CPU's memory access type, such as UC and WB. Thanks, -Toshi _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 From: toshi.kani@hpe.com (Kani, Toshi) Date: Fri, 2 Mar 2018 17:38:58 +0000 Subject: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory In-Reply-To: References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> <1519936815.4592.25.camel@au1.ibm.com> <20180301205315.GJ19007@ziepe.ca> <1519942012.4592.31.camel@au1.ibm.com> <1519943658.4592.34.camel@kernel.crashing.org> <1520010446.2693.19.camel@hpe.com> Message-ID: <1520015040.2693.38.camel@hpe.com> On Fri, 2018-03-02@08:57 -0800, Linus Torvalds wrote: > On Fri, Mar 2, 2018@8:22 AM, Kani, Toshi wrote: > > > > FWIW, this thing is called MTRRs on x86, which are initialized by BIOS. > > No. > > Or rather, that's simply just another (small) part of it all - and an > architected and documented one at that. > > Like the page table caching entries, the memory type range registers > are really just "secondary information". They don't actually select > between PCIe and RAM, they just affect the behavior on top of that. > > The really nitty-gritty stuff is not architected, and generally not > documented outside (possibly) the BIOS writer's guide that is not made > public. > > Those magical registers contain details like how the DRAM is > interleaved (if it is), what the timings are, where which memory > controller handles which memory range, and what are goes to PCIe etc. > > Basically all the actual *steering* information is very much hidden > away from the kernel (and often from the BIOS too). The parts we see > at a higher level are just tuning and tweaks. > > Note: the details differ _enormously_ between different chips. The > setup can be very different, with things like Knights Landing having > the external cache that can also act as local memory that isn't a > cache but maps at a different physical address instead etc. That's the > kind of steering I'm talking about - at a low level how physical > addresses get mapped to different cache partitions, memory > controllers, or to the IO system etc. Right, MRC code is not documented publicly, and it is very much CPU dependent. It programs address decoders and maps DRAMs to physical address as you described. MTRRs have nothing to do with this memory controller setting. That said, MTRRs specify CPU's memory access type, such as UC and WB. Thanks, -Toshi From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Kani, Toshi" Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory Date: Fri, 2 Mar 2018 17:38:58 +0000 Message-ID: <1520015040.2693.38.camel@hpe.com> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> <1519936815.4592.25.camel@au1.ibm.com> <20180301205315.GJ19007@ziepe.ca> <1519942012.4592.31.camel@au1.ibm.com> <1519943658.4592.34.camel@kernel.crashing.org> <1520010446.2693.19.camel@hpe.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Content-Language: en-US Content-ID: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: "torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org" Cc: "axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org" , "keith.busch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org" , "oliveroh-8fk3Idey6ehBDgjK7y7TUQ@public.gmane.org" , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" , "linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "jgg-uk2M96/98Pc@public.gmane.org" , "alex.williamson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org" , "jglisse-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org" , "benh-XVmvHMARGAS8U2dJNN8I7kB+6BGkLq7r@public.gmane.org" , "bhelgaas-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org" , "maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org" , "hch-jcswGhMUV9g@public.gmane.org" List-Id: linux-rdma@vger.kernel.org On Fri, 2018-03-02 at 08:57 -0800, Linus Torvalds wrote: > On Fri, Mar 2, 2018 at 8:22 AM, Kani, Toshi wrote: > > > > FWIW, this thing is called MTRRs on x86, which are initialized by BIOS. > > No. > > Or rather, that's simply just another (small) part of it all - and an > architected and documented one at that. > > Like the page table caching entries, the memory type range registers > are really just "secondary information". They don't actually select > between PCIe and RAM, they just affect the behavior on top of that. > > The really nitty-gritty stuff is not architected, and generally not > documented outside (possibly) the BIOS writer's guide that is not made > public. > > Those magical registers contain details like how the DRAM is > interleaved (if it is), what the timings are, where which memory > controller handles which memory range, and what are goes to PCIe etc. > > Basically all the actual *steering* information is very much hidden > away from the kernel (and often from the BIOS too). The parts we see > at a higher level are just tuning and tweaks. > > Note: the details differ _enormously_ between different chips. The > setup can be very different, with things like Knights Landing having > the external cache that can also act as local memory that isn't a > cache but maps at a different physical address instead etc. That's the > kind of steering I'm talking about - at a low level how physical > addresses get mapped to different cache partitions, memory > controllers, or to the IO system etc. Right, MRC code is not documented publicly, and it is very much CPU dependent. It programs address decoders and maps DRAMs to physical address as you described. MTRRs have nothing to do with this memory controller setting. That said, MTRRs specify CPU's memory access type, such as UC and WB. Thanks, -Toshi From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1164909AbeCBRjE (ORCPT ); Fri, 2 Mar 2018 12:39:04 -0500 Received: from g2t2354.austin.hpe.com ([15.233.44.27]:45506 "EHLO g2t2354.austin.hpe.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1946772AbeCBRjC (ORCPT ); Fri, 2 Mar 2018 12:39:02 -0500 From: "Kani, Toshi" To: "torvalds@linux-foundation.org" CC: "linux-kernel@vger.kernel.org" , "alex.williamson@redhat.com" , "linux-block@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "bhelgaas@google.com" , "hch@lst.de" , "axboe@kernel.dk" , "linux-nvdimm@lists.01.org" , "jglisse@redhat.com" , "linux-nvme@lists.infradead.org" , "maxg@mellanox.com" , "linux-pci@vger.kernel.org" , "keith.busch@intel.com" , "benh@kernel.crashing.org" , "jgg@ziepe.ca" , "oliveroh@au1.ibm.com" Subject: Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory Thread-Topic: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory Thread-Index: AQHTsO2SauImX1OM+UWgCf/w5DnppaO6wDuAgAAAYICAAQJ2gIAAFIKAgAABk4CAAAOhgIAAFJIAgAAG2oCAAADQAIABNwIA///9QACAABgkAA== Date: Fri, 2 Mar 2018 17:38:58 +0000 Message-ID: <1520015040.2693.38.camel@hpe.com> References: <20180228234006.21093-1-logang@deltatee.com> <1519876489.4592.3.camel@kernel.crashing.org> <1519876569.4592.4.camel@au1.ibm.com> <1519936477.4592.23.camel@au1.ibm.com> <1519936815.4592.25.camel@au1.ibm.com> <20180301205315.GJ19007@ziepe.ca> <1519942012.4592.31.camel@au1.ibm.com> <1519943658.4592.34.camel@kernel.crashing.org> <1520010446.2693.19.camel@hpe.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=toshi.kani@hpe.com; x-originating-ip: [15.219.147.8] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;AT5PR8401MB0353;7:4j7SDDbuN5iH8dyYhrB6n1zwQd/OXQEDNp1F7qXYhs+eFLYHIALd0H9sAYWl7KU9leUwsWMD2RbqBqoExD3M1vQHfMDNkyqDmjpOH2/G/BnqizlokZbWbbAfM3jXloqx5nhQpnJO9lGBJnJjCNWx55cS4cAUaIpFrc+OgMiXv6dNLeTBNnzY6Cxxizy/DWJtxyE8IDYwiPut1Nkdi4B7lF5c6kIdTWAZJJFNqqSExtCsGT0Zuv/BpJaqAsLDTTvX x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: e0d037fb-75f0-4bb0-d809-08d580647a80 x-microsoft-antispam: UriScan:(222181515654134);BCL:0;PCL:0;RULEID:(7020095)(4652020)(8989060)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(48565401081)(5600026)(4604075)(3008032)(2017052603307)(7153060)(7193020);SRVR:AT5PR8401MB0353; x-ms-traffictypediagnostic: AT5PR8401MB0353: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(227479698468861)(222181515654134); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(8211001083)(6040501)(2401047)(5005006)(8121501046)(3002001)(3231220)(944501242)(52105095)(93006095)(93001095)(10201501046)(6055026)(6041288)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123560045)(20161123564045)(6072148)(201708071742011);SRVR:AT5PR8401MB0353;BCL:0;PCL:0;RULEID:;SRVR:AT5PR8401MB0353; x-forefront-prvs: 05991796DF x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(366004)(39860400002)(39380400002)(376002)(346002)(396003)(189003)(377424004)(199004)(36756003)(99286004)(76176011)(2900100001)(97736004)(3660700001)(316002)(478600001)(54906003)(68736007)(81156014)(81166006)(1730700003)(26005)(8676002)(6116002)(59450400001)(186003)(6506007)(102836004)(53546011)(3846002)(93886005)(6246003)(6486002)(7736002)(2501003)(4326008)(229853002)(86362001)(6436002)(3280700002)(305945005)(105586002)(5250100002)(8936002)(5660300001)(2906002)(5640700003)(7416002)(6512007)(14454004)(103116003)(66066001)(53936002)(6916009)(2950100002)(25786009)(2351001)(106356001);DIR:OUT;SFP:1102;SCL:1;SRVR:AT5PR8401MB0353;H:AT5PR8401MB1297.NAMPRD84.PROD.OUTLOOK.COM;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; x-microsoft-antispam-message-info: xxRSHtSoDM+ibcBnfF7IVC3k6bhJo2ioPUoLZrUoCuPsswoN79CtF368Lg4J0eEpdwROiyMNN4PPML8XIB7QZw/OVEXy98ftxod4qUzsvfb0G8MiF+rth+hhhRz/8E5A2F0F5QYFXGYQM0vY6ub46NZ0UfP/UNZSZNL4qZtC3NA= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: e0d037fb-75f0-4bb0-d809-08d580647a80 X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Mar 2018 17:38:58.2204 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 105b2061-b669-4b31-92ac-24d304d195dc X-MS-Exchange-Transport-CrossTenantHeadersStamped: AT5PR8401MB0353 X-OriginatorOrg: hpe.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id w22HdDCn029761 On Fri, 2018-03-02 at 08:57 -0800, Linus Torvalds wrote: > On Fri, Mar 2, 2018 at 8:22 AM, Kani, Toshi wrote: > > > > FWIW, this thing is called MTRRs on x86, which are initialized by BIOS. > > No. > > Or rather, that's simply just another (small) part of it all - and an > architected and documented one at that. > > Like the page table caching entries, the memory type range registers > are really just "secondary information". They don't actually select > between PCIe and RAM, they just affect the behavior on top of that. > > The really nitty-gritty stuff is not architected, and generally not > documented outside (possibly) the BIOS writer's guide that is not made > public. > > Those magical registers contain details like how the DRAM is > interleaved (if it is), what the timings are, where which memory > controller handles which memory range, and what are goes to PCIe etc. > > Basically all the actual *steering* information is very much hidden > away from the kernel (and often from the BIOS too). The parts we see > at a higher level are just tuning and tweaks. > > Note: the details differ _enormously_ between different chips. The > setup can be very different, with things like Knights Landing having > the external cache that can also act as local memory that isn't a > cache but maps at a different physical address instead etc. That's the > kind of steering I'm talking about - at a low level how physical > addresses get mapped to different cache partitions, memory > controllers, or to the IO system etc. Right, MRC code is not documented publicly, and it is very much CPU dependent. It programs address decoders and maps DRAMs to physical address as you described. MTRRs have nothing to do with this memory controller setting. That said, MTRRs specify CPU's memory access type, such as UC and WB. Thanks, -Toshi