From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Message-ID: <583BC14B.2040809@amd.com> Date: Mon, 28 Nov 2016 13:31:55 +0800 From: zhoucm1 MIME-Version: 1.0 Subject: Re: Enabling peer to peer device transactions for PCIe devices References: <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <3f2d2db3-fb75-2422-2a18-a8497fd5d70e@amd.com> <20161125193252.GC16504@obsidianresearch.com> <0087fba9-7bcb-8bb3-c26e-4ef3e4970c34@amd.com> In-Reply-To: <0087fba9-7bcb-8bb3-c26e-4ef3e4970c34@amd.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: =?UTF-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Haggai Eran , Jason Gunthorpe" , "Yu, Qiang" Cc: "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "Kuehling, Felix , Serguei Sagalovitch" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , Sander, , "Suravee , linux-pci@vger.kernel.org" , "Deucher, Alexander , Max Gurtovoy , Blinzer, Paul" , "Linux-media@vger.kernel.org" List-ID: K1FpYW5nLCB3aG8gaXMgd29ya2luZyBvbiBpdC4KCk9uIDIwMTblubQxMeaciDI35pelIDIyOjA3 LCBDaHJpc3RpYW4gS8O2bmlnIHdyb3RlOgo+IEFtIDI3LjExLjIwMTYgdW0gMTU6MDIgc2Nocmll YiBIYWdnYWkgRXJhbjoKPj4gT24gMTEvMjUvMjAxNiA5OjMyIFBNLCBKYXNvbiBHdW50aG9ycGUg d3JvdGU6Cj4+PiBPbiBGcmksIE5vdiAyNSwgMjAxNiBhdCAwMjoyMjoxN1BNICswMTAwLCBDaHJp c3RpYW4gS8O2bmlnIHdyb3RlOgo+Pj4KPj4+Pj4gTGlrZSB5b3Ugc2F5IGJlbG93IHdlIGhhdmUg dG8gaGFuZGxlIHNob3J0IGxpdmVkIGluIHRoZSB1c3VhbCB3YXksIAo+Pj4+PiBhbmQKPj4+Pj4g dGhhdCBjb3ZlcnMgYmFzaWNhbGx5IGV2ZXJ5IGRldmljZSBleGNlcHQgSUIgTVJzLCBpbmNsdWRp bmcgdGhlCj4+Pj4+IGNvbW1hbmQgcXVldWUgb24gYSBOVk1lIGRyaXZlLgo+Pj4+IFdlbGwgYSBw cm9ibGVtIHdoaWNoIHdhc24ndCBtZW50aW9uZWQgc28gZmFyIGlzIHRoYXQgd2hpbGUgR1BVcyBk byAKPj4+PiBoYXZlIGEKPj4+PiBwYWdlIHRhYmxlIHRvIG1pcnJvciB0aGUgQ1BVIHBhZ2UgdGFi bGUsIHRoZXkgdXN1YWxseSBjYW4ndCByZWNvdmVyIAo+Pj4+IGZyb20KPj4+PiBwYWdlIGZhdWx0 cy4KPj4+PiBTbyB3aGF0IHdlIGRvIGlzIG1ha2luZyBzdXJlIHRoYXQgYWxsIG1lbW9yeSBhY2Nl c3NlZCBieSB0aGUgR1BVIAo+Pj4+IEpvYnMgc3RheXMKPj4+PiBpbiBwbGFjZSB3aGlsZSB0aG9z ZSBqb2JzIHJ1biAocHJldHR5IG11Y2ggdGhlIHNhbWUgcGlubmluZyB5b3UgZG8gCj4+Pj4gZm9y IHRoZQo+Pj4+IERNQSkuCj4+PiBZZXMsIGl0IGlzIERNQSwgc28gdGhpcyBpcyBhIHZhbGlkIGFw cHJvYWNoLgo+Pj4KPj4+IEJ1dCwgeW91IGRvbid0IG5lZWQgcGFnZSBmYXVsdHMgZnJvbSB0aGUg R1BVIHRvIGRvIHByb3BlciBjb2hlcmVudAo+Pj4gcGFnZSB0YWJsZSBtaXJyb3JpbmcuIEJhc2lj YWxseSB3aGVuIHRoZSBkcml2ZXIgc3VibWl0cyB0aGUgd29yayB0bwo+Pj4gdGhlIEdQVSBpdCAn ZmF1bHRzJyB0aGUgcGFnZXMgaW50byB0aGUgQ1BVIGFuZCBtaXJyb3IgdHJhbnNsYXRpb24KPj4+ IHRhYmxlIChpbnN0ZWFkIG9mIHBpbm5pbmcpLgo+Pj4KPj4+IExpa2UgaW4gT0RQLCBNTVUgbm90 aWZpZXJzL0hNTSBhcmUgdXNlZCB0byBtb25pdG9yIGZvciB0cmFuc2xhdGlvbgo+Pj4gY2hhbmdl cy4gSWYgYSBjaGFuZ2UgY29tZXMgaW4gdGhlIEdQVSBkcml2ZXIgY2hlY2tzIGlmIGFuIGV4ZWN1 dGluZwo+Pj4gY29tbWFuZCBpcyB0b3VjaGluZyB0aG9zZSBwYWdlcyBhbmQgYmxvY2tzIHRoZSBN TVUgbm90aWZpZXIgdW50aWwgdGhlCj4+PiBjb21tYW5kIGZsdXNoZXMsIHRoZW4gdW5mYXVsdHMg dGhlIHBhZ2UgKGJsb2NraW5nIGZ1dHVyZSBjb21tYW5kcykgYW5kCj4+PiB1bmJsb2NrcyB0aGUg bW11IG5vdGlmaWVyLgo+PiBJIHRoaW5rIGJsb2NraW5nIG1tdSBub3RpZmllcnMgYWdhaW5zdCBz b21ldGhpbmcgdGhhdCBpcyBiYXNpY2FsbHkKPj4gY29udHJvbGxlZCBieSB1c2VyLXNwYWNlIGNh biBiZSBwcm9ibGVtYXRpYy4gVGhpcyBjYW4gYmxvY2sgdGhpbmdzIGxpa2UKPj4gbWVtb3J5IHJl Y2xhaW0uIElmIHlvdSBoYXZlIHVzZXItc3BhY2UgYWNjZXNzIHRvIHRoZSBkZXZpY2UncyBxdWV1 ZXMsCj4+IHVzZXItc3BhY2UgY2FuIGJsb2NrIHRoZSBtbXUgbm90aWZpZXIgZm9yZXZlci4KPiBS ZWFsbHkgZ29vZCBwb2ludC4KPgo+IEkgdGhpbmsgdGhpcyBtZWFucyB0aGUgYmFyZSBtaW5pbXVt IGlmIHdlIGRvbid0IGhhdmUgcmVjb3ZlcmFibGUgcGFnZSAKPiBmYXVsdHMgaXMgdG8gaGF2ZSBw cmVlbXB0aW9uIHN1cHBvcnQgbGlrZSBGZWxpeCBkZXNjcmliZWQgaW4gaGlzIAo+IGFuc3dlciBh cyB3ZWxsLgo+Cj4gR29pbmcgdG8ga2VlcCB0aGF0IGluIG1pbmQsCj4gQ2hyaXN0aWFuLgo+Cj4g X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPiBkcmktZGV2 ZWwgbWFpbGluZyBsaXN0Cj4gZHJpLWRldmVsQGxpc3RzLmZyZWVkZXNrdG9wLm9yZwo+IGh0dHBz Oi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJpLWRldmVsCgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpMaW51eC1udmRpbW0g bWFpbGluZyBsaXN0CkxpbnV4LW52ZGltbUBsaXN0cy4wMS5vcmcKaHR0cHM6Ly9saXN0cy4wMS5v cmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1udmRpbW0K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Message-ID: <583BC14B.2040809@amd.com> Date: Mon, 28 Nov 2016 13:31:55 +0800 From: zhoucm1 MIME-Version: 1.0 To: =?UTF-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Haggai Eran" , Jason Gunthorpe , "Yu, Qiang" CC: "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "Kuehling, Felix" , Serguei Sagalovitch , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Blinzer, Paul" , "Suthikulpanit, Suravee" , "linux-pci@vger.kernel.org" , "Deucher, Alexander" , Max Gurtovoy , Dan Williams , Logan Gunthorpe , "Sander, Ben" , "Linux-media@vger.kernel.org" Subject: Re: Enabling peer to peer device transactions for PCIe devices References: <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <3f2d2db3-fb75-2422-2a18-a8497fd5d70e@amd.com> <20161125193252.GC16504@obsidianresearch.com> <0087fba9-7bcb-8bb3-c26e-4ef3e4970c34@amd.com> In-Reply-To: <0087fba9-7bcb-8bb3-c26e-4ef3e4970c34@amd.com> Content-Type: text/plain; charset="utf-8"; format=flowed Sender: linux-kernel-owner@vger.kernel.org List-ID: +Qiang, who is working on it. On 2016年11月27日 22:07, Christian König wrote: > Am 27.11.2016 um 15:02 schrieb Haggai Eran: >> On 11/25/2016 9:32 PM, Jason Gunthorpe wrote: >>> On Fri, Nov 25, 2016 at 02:22:17PM +0100, Christian König wrote: >>> >>>>> Like you say below we have to handle short lived in the usual way, >>>>> and >>>>> that covers basically every device except IB MRs, including the >>>>> command queue on a NVMe drive. >>>> Well a problem which wasn't mentioned so far is that while GPUs do >>>> have a >>>> page table to mirror the CPU page table, they usually can't recover >>>> from >>>> page faults. >>>> So what we do is making sure that all memory accessed by the GPU >>>> Jobs stays >>>> in place while those jobs run (pretty much the same pinning you do >>>> for the >>>> DMA). >>> Yes, it is DMA, so this is a valid approach. >>> >>> But, you don't need page faults from the GPU to do proper coherent >>> page table mirroring. Basically when the driver submits the work to >>> the GPU it 'faults' the pages into the CPU and mirror translation >>> table (instead of pinning). >>> >>> Like in ODP, MMU notifiers/HMM are used to monitor for translation >>> changes. If a change comes in the GPU driver checks if an executing >>> command is touching those pages and blocks the MMU notifier until the >>> command flushes, then unfaults the page (blocking future commands) and >>> unblocks the mmu notifier. >> I think blocking mmu notifiers against something that is basically >> controlled by user-space can be problematic. This can block things like >> memory reclaim. If you have user-space access to the device's queues, >> user-space can block the mmu notifier forever. > Really good point. > > I think this means the bare minimum if we don't have recoverable page > faults is to have preemption support like Felix described in his > answer as well. > > Going to keep that in mind, > Christian. > > _______________________________________________ > dri-devel mailing list > dri-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/dri-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: zhoucm1 Subject: Re: Enabling peer to peer device transactions for PCIe devices Date: Mon, 28 Nov 2016 13:31:55 +0800 Message-ID: <583BC14B.2040809@amd.com> References: <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <3f2d2db3-fb75-2422-2a18-a8497fd5d70e@amd.com> <20161125193252.GC16504@obsidianresearch.com> <0087fba9-7bcb-8bb3-c26e-4ef3e4970c34@amd.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <0087fba9-7bcb-8bb3-c26e-4ef3e4970c34-5C7GfCeVMHo@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: =?UTF-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , Haggai Eran , Jason Gunthorpe , "Yu, Qiang" Cc: "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , "Kuehling, Felix" , Serguei Sagalovitch , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org" , "Sander, Ben" , "Suthikulpanit, Suravee" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "Deucher, Alexander" , Max Gurtovoy , "Blinzer, Paul" , "Linux-media-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" List-Id: linux-rdma@vger.kernel.org K1FpYW5nLCB3aG8gaXMgd29ya2luZyBvbiBpdC4KCk9uIDIwMTblubQxMeaciDI35pelIDIyOjA3 LCBDaHJpc3RpYW4gS8O2bmlnIHdyb3RlOgo+IEFtIDI3LjExLjIwMTYgdW0gMTU6MDIgc2Nocmll YiBIYWdnYWkgRXJhbjoKPj4gT24gMTEvMjUvMjAxNiA5OjMyIFBNLCBKYXNvbiBHdW50aG9ycGUg d3JvdGU6Cj4+PiBPbiBGcmksIE5vdiAyNSwgMjAxNiBhdCAwMjoyMjoxN1BNICswMTAwLCBDaHJp c3RpYW4gS8O2bmlnIHdyb3RlOgo+Pj4KPj4+Pj4gTGlrZSB5b3Ugc2F5IGJlbG93IHdlIGhhdmUg dG8gaGFuZGxlIHNob3J0IGxpdmVkIGluIHRoZSB1c3VhbCB3YXksIAo+Pj4+PiBhbmQKPj4+Pj4g dGhhdCBjb3ZlcnMgYmFzaWNhbGx5IGV2ZXJ5IGRldmljZSBleGNlcHQgSUIgTVJzLCBpbmNsdWRp bmcgdGhlCj4+Pj4+IGNvbW1hbmQgcXVldWUgb24gYSBOVk1lIGRyaXZlLgo+Pj4+IFdlbGwgYSBw cm9ibGVtIHdoaWNoIHdhc24ndCBtZW50aW9uZWQgc28gZmFyIGlzIHRoYXQgd2hpbGUgR1BVcyBk byAKPj4+PiBoYXZlIGEKPj4+PiBwYWdlIHRhYmxlIHRvIG1pcnJvciB0aGUgQ1BVIHBhZ2UgdGFi bGUsIHRoZXkgdXN1YWxseSBjYW4ndCByZWNvdmVyIAo+Pj4+IGZyb20KPj4+PiBwYWdlIGZhdWx0 cy4KPj4+PiBTbyB3aGF0IHdlIGRvIGlzIG1ha2luZyBzdXJlIHRoYXQgYWxsIG1lbW9yeSBhY2Nl c3NlZCBieSB0aGUgR1BVIAo+Pj4+IEpvYnMgc3RheXMKPj4+PiBpbiBwbGFjZSB3aGlsZSB0aG9z ZSBqb2JzIHJ1biAocHJldHR5IG11Y2ggdGhlIHNhbWUgcGlubmluZyB5b3UgZG8gCj4+Pj4gZm9y IHRoZQo+Pj4+IERNQSkuCj4+PiBZZXMsIGl0IGlzIERNQSwgc28gdGhpcyBpcyBhIHZhbGlkIGFw cHJvYWNoLgo+Pj4KPj4+IEJ1dCwgeW91IGRvbid0IG5lZWQgcGFnZSBmYXVsdHMgZnJvbSB0aGUg R1BVIHRvIGRvIHByb3BlciBjb2hlcmVudAo+Pj4gcGFnZSB0YWJsZSBtaXJyb3JpbmcuIEJhc2lj YWxseSB3aGVuIHRoZSBkcml2ZXIgc3VibWl0cyB0aGUgd29yayB0bwo+Pj4gdGhlIEdQVSBpdCAn ZmF1bHRzJyB0aGUgcGFnZXMgaW50byB0aGUgQ1BVIGFuZCBtaXJyb3IgdHJhbnNsYXRpb24KPj4+ IHRhYmxlIChpbnN0ZWFkIG9mIHBpbm5pbmcpLgo+Pj4KPj4+IExpa2UgaW4gT0RQLCBNTVUgbm90 aWZpZXJzL0hNTSBhcmUgdXNlZCB0byBtb25pdG9yIGZvciB0cmFuc2xhdGlvbgo+Pj4gY2hhbmdl cy4gSWYgYSBjaGFuZ2UgY29tZXMgaW4gdGhlIEdQVSBkcml2ZXIgY2hlY2tzIGlmIGFuIGV4ZWN1 dGluZwo+Pj4gY29tbWFuZCBpcyB0b3VjaGluZyB0aG9zZSBwYWdlcyBhbmQgYmxvY2tzIHRoZSBN TVUgbm90aWZpZXIgdW50aWwgdGhlCj4+PiBjb21tYW5kIGZsdXNoZXMsIHRoZW4gdW5mYXVsdHMg dGhlIHBhZ2UgKGJsb2NraW5nIGZ1dHVyZSBjb21tYW5kcykgYW5kCj4+PiB1bmJsb2NrcyB0aGUg bW11IG5vdGlmaWVyLgo+PiBJIHRoaW5rIGJsb2NraW5nIG1tdSBub3RpZmllcnMgYWdhaW5zdCBz b21ldGhpbmcgdGhhdCBpcyBiYXNpY2FsbHkKPj4gY29udHJvbGxlZCBieSB1c2VyLXNwYWNlIGNh biBiZSBwcm9ibGVtYXRpYy4gVGhpcyBjYW4gYmxvY2sgdGhpbmdzIGxpa2UKPj4gbWVtb3J5IHJl Y2xhaW0uIElmIHlvdSBoYXZlIHVzZXItc3BhY2UgYWNjZXNzIHRvIHRoZSBkZXZpY2UncyBxdWV1 ZXMsCj4+IHVzZXItc3BhY2UgY2FuIGJsb2NrIHRoZSBtbXUgbm90aWZpZXIgZm9yZXZlci4KPiBS ZWFsbHkgZ29vZCBwb2ludC4KPgo+IEkgdGhpbmsgdGhpcyBtZWFucyB0aGUgYmFyZSBtaW5pbXVt IGlmIHdlIGRvbid0IGhhdmUgcmVjb3ZlcmFibGUgcGFnZSAKPiBmYXVsdHMgaXMgdG8gaGF2ZSBw cmVlbXB0aW9uIHN1cHBvcnQgbGlrZSBGZWxpeCBkZXNjcmliZWQgaW4gaGlzIAo+IGFuc3dlciBh cyB3ZWxsLgo+Cj4gR29pbmcgdG8ga2VlcCB0aGF0IGluIG1pbmQsCj4gQ2hyaXN0aWFuLgo+Cj4g X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPiBkcmktZGV2 ZWwgbWFpbGluZyBsaXN0Cj4gZHJpLWRldmVsQGxpc3RzLmZyZWVkZXNrdG9wLm9yZwo+IGh0dHBz Oi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJpLWRldmVsCgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpMaW51eC1udmRpbW0g bWFpbGluZyBsaXN0CkxpbnV4LW52ZGltbUBsaXN0cy4wMS5vcmcKaHR0cHM6Ly9saXN0cy4wMS5v cmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1udmRpbW0K