From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [RFC PATCH 3/5] mm/vma: add support for peer to peer to device vma Date: Wed, 30 Jan 2019 17:30:27 -0500 Message-ID: <20190130223027.GH5061@redhat.com> References: <20190130185652.GB17080@mellanox.com> <20190130192234.GD5061@redhat.com> <20190130193759.GE17080@mellanox.com> <20190130201114.GB17915@mellanox.com> <20190130204332.GF5061@redhat.com> <20190130204954.GI17080@mellanox.com> <20190130214525.GG5061@redhat.com> <20190130215600.GM17080@mellanox.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <20190130215600.GM17080@mellanox.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Jason Gunthorpe Cc: Joerg Roedel , "Rafael J . Wysocki" , Greg Kroah-Hartman , Felix Kuehling , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , Christoph Hellwig , "linux-mm@kvack.org" , "iommu@lists.linux-foundation.org" , "linux-pci@vger.kernel.org" , Bjorn Helgaas , Robin Murphy , Logan Gunthorpe , Christian Koenig , Marek Szyprowski List-Id: iommu@lists.linux-foundation.org T24gV2VkLCBKYW4gMzAsIDIwMTkgYXQgMDk6NTY6MDdQTSArMDAwMCwgSmFzb24gR3VudGhvcnBl IHdyb3RlOgo+IE9uIFdlZCwgSmFuIDMwLCAyMDE5IGF0IDA0OjQ1OjI1UE0gLTA1MDAsIEplcm9t ZSBHbGlzc2Ugd3JvdGU6Cj4gPiBPbiBXZWQsIEphbiAzMCwgMjAxOSBhdCAwODo1MDowMFBNICsw MDAwLCBKYXNvbiBHdW50aG9ycGUgd3JvdGU6Cj4gPiA+IE9uIFdlZCwgSmFuIDMwLCAyMDE5IGF0 IDAzOjQzOjMyUE0gLTA1MDAsIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4gPiA+ID4gT24gV2VkLCBK YW4gMzAsIDIwMTkgYXQgMDg6MTE6MTlQTSArMDAwMCwgSmFzb24gR3VudGhvcnBlIHdyb3RlOgo+ ID4gPiA+ID4gT24gV2VkLCBKYW4gMzAsIDIwMTkgYXQgMDE6MDA6MDJQTSAtMDcwMCwgTG9nYW4g R3VudGhvcnBlIHdyb3RlOgo+ID4gPiA+ID4gCj4gPiA+ID4gPiA+IFdlIG5ldmVyIGNoYW5nZWQg U0dMcy4gV2Ugc3RpbGwgdXNlIHRoZW0gdG8gcGFzcyBwMnBkbWEgcGFnZXMsIG9ubHkgd2UKPiA+ ID4gPiA+ID4gbmVlZCB0byBiZSBhIGJpdCBjYXJlZnVsIHdoZXJlIHdlIHNlbmQgdGhlIGVudGly ZSBTR0wuIEkgc2VlIG5vIHJlYXNvbgo+ID4gPiA+ID4gPiB3aHkgd2UgY2FuJ3QgY29udGludWUg dG8gYmUgY2FyZWZ1bCBvbmNlIHRoZWlyIGluIHVzZXJzcGFjZSBpZiB0aGVyZSdzCj4gPiA+ID4g PiA+IHNvbWV0aGluZyBpbiBHVVAgdG8gZGVueSB0aGVtLgo+ID4gPiA+ID4gPiAKPiA+ID4gPiA+ ID4gSXQgd291bGQgYmUgbmljZSB0byBoYXZlIGhldGVyb2dlbmVvdXMgU0dMcyBhbmQgaXQgaXMg c29tZXRoaW5nIHdlCj4gPiA+ID4gPiA+IHNob3VsZCB3b3JrIHRvd2FyZCBidXQgaW4gcHJhY3Rp Y2UgdGhleSBhcmVuJ3QgcmVhbGx5IG5lY2Vzc2FyeSBhdCB0aGUKPiA+ID4gPiA+ID4gbW9tZW50 Lgo+ID4gPiA+ID4gCj4gPiA+ID4gPiBSRE1BIGdlbmVyYWxseSBjYW5ub3QgY29wZSB3ZWxsIHdp dGggYW4gQVBJIHRoYXQgcmVxdWlyZXMgaG9tb2dlbmVvdXMKPiA+ID4gPiA+IFNHTHMuLiBVc2Vy IHNwYWNlIGNhbiBjb25zdHJ1Y3QgY29tcGxleCBNUnMgKHBhcnRpY3VsYXJseSB3aXRoIHRoZQo+ ID4gPiA+ID4gcHJvcG9zZWQgU0dMIE1SIGZsb3cpIGFuZCB3ZSBtdXN0IG1hcnNoYWwgdGhhdCBp bnRvIGEgc2luZ2xlIFNHTCBvcgo+ID4gPiA+ID4gdGhlIGRyaXZlcnMgZmFsbCBhcGFydC4KPiA+ ID4gPiA+IAo+ID4gPiA+ID4gSmVyb21lIGV4cGxhaW5lZCB0aGF0IEdQVSBpcyB3b3JzZSwgYSBz aW5nbGUgVk1BIG1heSBoYXZlIGEgcmFuZG9tIG1peAo+ID4gPiA+ID4gb2YgQ1BVIG9yIGRldmlj ZSBwYWdlcy4uCj4gPiA+ID4gPiAKPiA+ID4gPiA+IFRoaXMgaXMgYSBwcmV0dHkgYmlnIGJsb2Nr ZXIgdGhhdCB3b3VsZCBoYXZlIHRvIHNvbWVob3cgYmUgZml4ZWQuCj4gPiA+ID4gCj4gPiA+ID4g Tm90ZSB0aGF0IEhNTSB0YWtlcyBjYXJlIG9mIHRoYXQgUkRNQSBPRFAgd2l0aCBteSBPRFAgdG8g SE1NIHBhdGNoLAo+ID4gPiA+IHNvIHdoYXQgeW91IGdldCBmb3IgYW4gT0RQIHVtZW0gaXMganVz dCBhIGxpc3Qgb2YgZG1hIGFkZHJlc3MgeW91Cj4gPiA+ID4gY2FuIHByb2dyYW0geW91ciBkZXZp Y2UgdG8uIFRoZSBhaW0gaXMgdG8gYXZvaWQgdGhlIGRyaXZlciB0byBjYXJlCj4gPiA+ID4gYWJv dXQgdGhhdC4gVGhlIGFjY2VzcyBwb2xpY3kgd2hlbiB0aGUgVU1FTSBvYmplY3QgaXMgY3JlYXRl ZCBieQo+ID4gPiA+IHVzZXJzcGFjZSB0aHJvdWdoIHZlcmJzIEFQSSBzaG91bGQgaG93ZXZlciBh c2NlcnRhaW4gdGhhdCBmb3IgbW1hcAo+ID4gPiA+IG9mIGRldmljZSBmaWxlIGl0IGlzIG9ubHkg Y3JlYXRpbmcgYSBVTUVNIHRoYXQgaXMgZnVsbHkgY292ZXJlZCBieQo+ID4gPiA+IG9uZSBhbmQg b25seSBvbmUgdm1hLiBHUFUgZGV2aWNlIGRyaXZlciB3aWxsIGhhdmUgb25lIHZtYSBwZXIgbG9n aWNhbAo+ID4gPiA+IEdQVSBvYmplY3QuIEkgZXhwZWN0IG90aGVyIGtpbmQgb2YgZGV2aWNlIGRv IHRoYXQgc2FtZSBzbyB0aGF0IHRoZXkKPiA+ID4gPiBjYW4gbWF0Y2ggYSB2bWEgdG8gYSB1bmlx dWUgb2JqZWN0IGluIHRoZWlyIGRyaXZlci4KPiA+ID4gCj4gPiA+IEEgb25lIFZNQSBydWxlIGlz IG5vdCByZWFsbHkgd29ya2FibGUuCj4gPiA+IAo+ID4gPiBXaXRoIE9EUCBWTUEgYm91bmRhcmll cyBjYW4gbW92ZSBhcm91bmQgYWNyb3NzIHRoZSBsaWZldGltZSBvZiB0aGUgTVIKPiA+ID4gYW5k IHdlIGhhdmUgbm8gb2J2aW91cyB3YXkgdG8gZmFpbCBhbnl0aGluZyBpZiB1c2VycGFjZSBwdXRz IGEgVk1BCj4gPiA+IGJvdW5kYXJ5IGluIHRoZSBtaWRkbGUgb2YgYW4gZXhpc3RpbmcgT0RQIE1S IGFkZHJlc3MgcmFuZ2UuCj4gPiAKPiA+IFRoaXMgaXMgdHJ1ZSBvbmx5IGZvciB2bWEgdGhhdCBh cmUgbm90IG1tYXAgb2YgYSBkZXZpY2UgZmlsZS4gVGhpcyBpcwo+ID4gd2hhdCBpIHdhcyB0cnlp bmcgdG8gZ2V0IGFjY3Jvc3MuIEFuIG1tYXAgb2YgYSBmaWxlIGlzIG5ldmVyIG1lcmdlCj4gPiBz byBpdCBjYW4gb25seSBnZXQgc3BsaXQvYnV0Y2hlciBieSBtdW5tYXAvbXJlbWFwIGJ1dCB3aGVu IHRoYXQgaGFwcGVuCj4gPiB5b3UgYWxzbyBuZWVkIHRvIHJlZmxlY3QgdGhlIHZpcnR1YWwgYWRk cmVzcyBzcGFjZSBjaGFuZ2UgdG8gdGhlCj4gPiBkZXZpY2UgaWUgYW55IGFjY2VzcyB0byBhIG5v dyBpbnZhbGlkIHJhbmdlIG11c3QgdHJpZ2dlciBlcnJvci4KPiAKPiBXaHkgaXMgaXQgaW52YWxp ZD8gVGhlIGFkZHJlc3MgcmFuZ2Ugc3RpbGwgaGFzIHZhbGlkIHByb2Nlc3MgbWVtb3J5PwoKSWYg eW91IGRvIG11bm1hcChBLCBzaXplKSB0aGVuIGFsbCBhZGRyZXNzIGluIHRoZSByYW5nZSBbQSwg QStzaXplXQphcmUgaW52YWxpZC4gVGhpcyBpcyB3aGF0IGkgYW0gcmVmZXJpbmcgdG9vIGhlcmUu IFNhbWUgZm9yIG1yZW1hcC4KCj4gCj4gV2hhdCBpcyB0aGUgcHJvYmxlbSBpbiB0aGUgSE1NIG1p cnJvciB0aGF0IGl0IG5lZWRzIHRoaXMgcmVzdHJpY3Rpb24/CgpObyByZXN0cmljdGlvbiBhdCBh bGwgaGVyZS4gSSB0aGluayBpIGp1c3Qgd2Fzbid0IHVuZGVyc3Rvb2QuCgo+IFRoZXJlIGlzIGFs c28gdGhlIHNpdHVhdGlvbiB3aGVyZSB3ZSBjcmVhdGUgYW4gT0RQIE1SIHRoYXQgc3BhbnMgMCAt Pgo+IFU2NF9NQVggaW4gdGhlIHByb2Nlc3MgYWRkcmVzcyBzcGFjZS4gSW4gdGhpcyBjYXNlIHRo ZXJlIGFyZSBsb3RzIG9mCj4gZGlmZmVyZW50IFZNQXMgaXQgY292ZXJzIGFuZCB3ZSBleHBlY3Qg aXQgdG8gZnVsbHkgdHJhY2sgYWxsIGNoYW5nZXMKPiB0byBhbGwgVk1Bcy4KClllcyBhbmQgdGhh dCB3b3JrcyBob3dldmVyIGFueSBtZW1vcnkgYWNjZXNzIGFib3ZlIFRBU0tfU0laRSB3aWxsCnJl dHVybiAtRUZBVUxUIGFzIHRoaXMgaXMga2VybmVsIGFkZHJlc3Mgc3BhY2Ugc28geW91IGNhbiBv bmx5IGFjY2Vzcwphbnl0aGluZyB0aGF0IGlzIGEgdmFsaWQgcHJvY2VzcyB2aXJ0dWFsIGFkZHJl c3MuCgo+IAo+IFNvIHdlIGhhdmUgdG8gc3BpbiB1cCBkZWRpY2F0ZWQgdW1lbV9vZHBzIHRoYXQg Y2FyZWZ1bGx5IHNwYW4gc2luZ2xlCj4gVk1BcywgYW5kIHNvbWVob3cgdHJhY2sgY2hhbmdlcyB0 byBWTUEgPwoKTm8geW91IGRvIG5vdC4KCj4gCj4gbWx4NSBvZHAgZG9lcyBzb21lIG9mIHRoaXMg YWxyZWFkeS4uIEJ1dCB5aWtlcywgdGhpcyBuZWVkcyBzb21lIHByZXR0eQo+IGNhcmVmdWwgdGVz dGluZyBpbiBhbGwgdGhlc2Ugc2l0dWF0aW9ucy4KClNvcnJ5IGlmIGkgY29uZnVzZWQgeW91IGV2 ZW4gbW9yZSB0aGFuIHRoZSBmaXJzdCB0aW1lLiBFdmVyeXRoaW5nIHdvcmtzCnlvdSBoYXZlIG5v dGhpbmcgdG8gd29ycnkgYWJvdXQgOikKCj4gCj4gPiA+IEkgdGhpbmsgdGhlIEhNTSBtaXJyb3Ig QVBJIHJlYWxseSBuZWVkcyB0byBkZWFsIHdpdGggdGhpcyBmb3IgdGhlCj4gPiA+IGRyaXZlciBz b21laG93Lgo+ID4gCj4gPiBZZXMgdGhlIEhNTSBkb2VzIGRlYWwgd2l0aCB0aGlzIGZvciB5b3Us IHlvdSBkbyBub3QgaGF2ZSB0byB3b3JyeSBhYm91dAo+ID4gaXQuIFNvcnJ5IGlmIHRoYXQgd2Fz IG5vdCBjbGVhci4gSSBqdXN0IHdhbnRlZCB0byBzdHJlc3MgdGhhdCB2bWEgdGhhdAo+ID4gYXJl IG1tYXAgb2YgYSBmaWxlIGRvIG5vdCBiZWhhdmUgbGlrZSBvdGhlciB2bWEgaGVuY2Ugd2hlbiB5 b3UgY3JlYXRlCj4gPiB0aGUgVU1FTSB5b3UgY2FuIGNoZWNrIGZvciB0aG9zZSBpZiB5b3UgZmVl bCB0aGUgbmVlZC4KPiAKPiBXaGF0IHByb3BlcnRpZXMgZG8gd2UgZ2V0IGZyb20gSE1NIG1pcnJv cj8gV2lsbCBpdCB0ZWxsIHVzIHdoZW4gdG8KPiBjcmVhdGUgbW9yZSB1bWVtcyB0byBjb3ZlciBW TUEgc2VhbXMgb3Igd2lsbCBpdCBqdXN0IGNhdXNlIHVuZGVzaXJlZAo+IG5vLW1hcHBlZCBmYWls dXJlcyBpbiBzb21lIGNhc2VzPwoKWW91IGRvIG5vdCBnZXQgYW55dGhpbmcgZnJvbSBITU0gbWly cm9yLCBpIG1pZ2h0IGFkZCBhIGZsYWcgc28gdGhhdApITU0gY2FuIHJlcG9ydCB0aGlzIHNwZWNp YWwgY29uZGl0aW9uIGlmIGRyaXZlciB3YW50cyB0byBrbm93LiBJZgp5b3Ugd2FudCB0byBrbm93 IHlvdSBoYXZlIHRvIGxvb2sgYXQgdGhlIHZtYSBieSB5b3Vyc2VsZi4gR1BVIGRyaXZlcgp3aWxs IGRlZmluaXRseSB3YW50IHRvIGtub3cgd2hlbSBpbXBvcnRpbmcgc28gaSBtaWdodCBhZGQgYSBm bGFnIHNvCnRoYXQgdGhleSBkbyBub3QgaGF2ZSB0byBsb29rdXAgdGhlIHZtYSB0aGVtc2VsZiB0 byBrbm93LgoKQWdhaW4gaWYgeW91IGRvIG5vdCBjYXJlIHRoZW4ganVzdCBpZ25vcmUgZXZlcnl0 aGluZyBoZXJlLCBpdCBpcwpoYW5kbGVkIGJ5IEhNTSBhbmQgeW91IGRvIG5vdCBoYXZlIHRvIHdv cnJ5IG9uZSBiaXQuIElmIGl0IHdvcmtlZAp3aXRoIEdVUCBpdCB3aWxsIHdvcmsgd2l0aCBITU0g YW5kIHdpdGggdGhvc2UgcDJwIHBhdGNoZXMgaWYgaXQKd2lsbCBldmVuIHdvcmtzIGFnYWluc3Qg dm1hIHRoYXQgYXJlIG1tYXAgb2YgYSBmaWxlIGFuZCB0aGF0IHNldAp0aGUgcDJwX21hcCBmdW5j dGlvbi4KCkNoZWVycywKSsOpcsO0bWUKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX18KZHJpLWRldmVsIG1haWxpbmcgbGlzdApkcmktZGV2ZWxAbGlzdHMuZnJl ZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGlu Zm8vZHJpLWRldmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6119EC282D7 for ; Wed, 30 Jan 2019 23:09:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 16FAC20881 for ; Wed, 30 Jan 2019 23:09:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726865AbfA3XJl (ORCPT ); Wed, 30 Jan 2019 18:09:41 -0500 Received: from mx1.redhat.com ([209.132.183.28]:43860 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726198AbfA3XJl (ORCPT ); Wed, 30 Jan 2019 18:09:41 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BACE8C0C6C2B; Wed, 30 Jan 2019 22:30:31 +0000 (UTC) Received: from redhat.com (ovpn-126-0.rdu2.redhat.com [10.10.126.0]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 98B08608C6; Wed, 30 Jan 2019 22:30:29 +0000 (UTC) Date: Wed, 30 Jan 2019 17:30:27 -0500 From: Jerome Glisse To: Jason Gunthorpe Cc: Logan Gunthorpe , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Greg Kroah-Hartman , "Rafael J . Wysocki" , Bjorn Helgaas , Christian Koenig , Felix Kuehling , "linux-pci@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Joerg Roedel , "iommu@lists.linux-foundation.org" Subject: Re: [RFC PATCH 3/5] mm/vma: add support for peer to peer to device vma Message-ID: <20190130223027.GH5061@redhat.com> References: <20190130185652.GB17080@mellanox.com> <20190130192234.GD5061@redhat.com> <20190130193759.GE17080@mellanox.com> <20190130201114.GB17915@mellanox.com> <20190130204332.GF5061@redhat.com> <20190130204954.GI17080@mellanox.com> <20190130214525.GG5061@redhat.com> <20190130215600.GM17080@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20190130215600.GM17080@mellanox.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Wed, 30 Jan 2019 22:30:32 +0000 (UTC) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On Wed, Jan 30, 2019 at 09:56:07PM +0000, Jason Gunthorpe wrote: > On Wed, Jan 30, 2019 at 04:45:25PM -0500, Jerome Glisse wrote: > > On Wed, Jan 30, 2019 at 08:50:00PM +0000, Jason Gunthorpe wrote: > > > On Wed, Jan 30, 2019 at 03:43:32PM -0500, Jerome Glisse wrote: > > > > On Wed, Jan 30, 2019 at 08:11:19PM +0000, Jason Gunthorpe wrote: > > > > > On Wed, Jan 30, 2019 at 01:00:02PM -0700, Logan Gunthorpe wrote: > > > > > > > > > > > We never changed SGLs. We still use them to pass p2pdma pages, only we > > > > > > need to be a bit careful where we send the entire SGL. I see no reason > > > > > > why we can't continue to be careful once their in userspace if there's > > > > > > something in GUP to deny them. > > > > > > > > > > > > It would be nice to have heterogeneous SGLs and it is something we > > > > > > should work toward but in practice they aren't really necessary at the > > > > > > moment. > > > > > > > > > > RDMA generally cannot cope well with an API that requires homogeneous > > > > > SGLs.. User space can construct complex MRs (particularly with the > > > > > proposed SGL MR flow) and we must marshal that into a single SGL or > > > > > the drivers fall apart. > > > > > > > > > > Jerome explained that GPU is worse, a single VMA may have a random mix > > > > > of CPU or device pages.. > > > > > > > > > > This is a pretty big blocker that would have to somehow be fixed. > > > > > > > > Note that HMM takes care of that RDMA ODP with my ODP to HMM patch, > > > > so what you get for an ODP umem is just a list of dma address you > > > > can program your device to. The aim is to avoid the driver to care > > > > about that. The access policy when the UMEM object is created by > > > > userspace through verbs API should however ascertain that for mmap > > > > of device file it is only creating a UMEM that is fully covered by > > > > one and only one vma. GPU device driver will have one vma per logical > > > > GPU object. I expect other kind of device do that same so that they > > > > can match a vma to a unique object in their driver. > > > > > > A one VMA rule is not really workable. > > > > > > With ODP VMA boundaries can move around across the lifetime of the MR > > > and we have no obvious way to fail anything if userpace puts a VMA > > > boundary in the middle of an existing ODP MR address range. > > > > This is true only for vma that are not mmap of a device file. This is > > what i was trying to get accross. An mmap of a file is never merge > > so it can only get split/butcher by munmap/mremap but when that happen > > you also need to reflect the virtual address space change to the > > device ie any access to a now invalid range must trigger error. > > Why is it invalid? The address range still has valid process memory? If you do munmap(A, size) then all address in the range [A, A+size] are invalid. This is what i am refering too here. Same for mremap. > > What is the problem in the HMM mirror that it needs this restriction? No restriction at all here. I think i just wasn't understood. > There is also the situation where we create an ODP MR that spans 0 -> > U64_MAX in the process address space. In this case there are lots of > different VMAs it covers and we expect it to fully track all changes > to all VMAs. Yes and that works however any memory access above TASK_SIZE will return -EFAULT as this is kernel address space so you can only access anything that is a valid process virtual address. > > So we have to spin up dedicated umem_odps that carefully span single > VMAs, and somehow track changes to VMA ? No you do not. > > mlx5 odp does some of this already.. But yikes, this needs some pretty > careful testing in all these situations. Sorry if i confused you even more than the first time. Everything works you have nothing to worry about :) > > > > I think the HMM mirror API really needs to deal with this for the > > > driver somehow. > > > > Yes the HMM does deal with this for you, you do not have to worry about > > it. Sorry if that was not clear. I just wanted to stress that vma that > > are mmap of a file do not behave like other vma hence when you create > > the UMEM you can check for those if you feel the need. > > What properties do we get from HMM mirror? Will it tell us when to > create more umems to cover VMA seams or will it just cause undesired > no-mapped failures in some cases? You do not get anything from HMM mirror, i might add a flag so that HMM can report this special condition if driver wants to know. If you want to know you have to look at the vma by yourself. GPU driver will definitly want to know whem importing so i might add a flag so that they do not have to lookup the vma themself to know. Again if you do not care then just ignore everything here, it is handled by HMM and you do not have to worry one bit. If it worked with GUP it will work with HMM and with those p2p patches if it will even works against vma that are mmap of a file and that set the p2p_map function. Cheers, Jérôme