From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive Date: Wed, 8 Aug 2018 11:18:35 -0400 Message-ID: <20180808151835.GA3429@redhat.com> References: <20180801102221.5308-1-nek.in.cn@gmail.com> <20180801165644.GA3820@redhat.com> <20180802040557.GL160746@Turing-Arch-b> <20180802142243.GA3481@redhat.com> <20180803034721.GC91035@Turing-Arch-b> <20180803143944.GA4079@redhat.com> <20180806031252.GG91035@Turing-Arch-b> <20180806153257.GB6002@redhat.com> <11bace0e-dc14-5d2c-f65c-25b852f4e9ca@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: "Tian, Kevin" , Herbert Xu , "kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Jonathan Corbet , Greg Kroah-Hartman , Zaibo Xu , "linux-doc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "Kumar, Sanjay K" , Hao Fang , "iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linuxarm-hv44wF8Li93QT0dZR+AlfA@public.gmane.org" , Alex Williamson , "linux-crypto-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Philippe Ombredanne , Thomas Gleixner , Kenneth Lee , "David S . Miller" , "linux-accelerators-uLR06cmDAlY/bJ5BZ2RsiQ@public.gmane.org" Return-path: Content-Disposition: inline In-Reply-To: <11bace0e-dc14-5d2c-f65c-25b852f4e9ca-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Errors-To: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org List-Id: linux-crypto.vger.kernel.org T24gV2VkLCBBdWcgMDgsIDIwMTggYXQgMDk6MDg6NDJBTSArMDgwMCwgS2VubmV0aCBMZWUgd3Jv dGU6Cj4gCj4gCj4g5ZyoIDIwMTjlubQwOOaciDA25pelIOaYn+acn+S4gCAxMTozMiDkuIvljYgs IEplcm9tZSBHbGlzc2Ug5YaZ6YGTOgo+ID4gT24gTW9uLCBBdWcgMDYsIDIwMTggYXQgMTE6MTI6 NTJBTSArMDgwMCwgS2VubmV0aCBMZWUgd3JvdGU6Cj4gPiA+IE9uIEZyaSwgQXVnIDAzLCAyMDE4 IGF0IDEwOjM5OjQ0QU0gLTA0MDAsIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4gPiA+ID4gT24gRnJp LCBBdWcgMDMsIDIwMTggYXQgMTE6NDc6MjFBTSArMDgwMCwgS2VubmV0aCBMZWUgd3JvdGU6Cj4g PiA+ID4gPiBPbiBUaHUsIEF1ZyAwMiwgMjAxOCBhdCAxMDoyMjo0M0FNIC0wNDAwLCBKZXJvbWUg R2xpc3NlIHdyb3RlOgo+ID4gPiA+ID4gPiBPbiBUaHUsIEF1ZyAwMiwgMjAxOCBhdCAxMjowNTo1 N1BNICswODAwLCBLZW5uZXRoIExlZSB3cm90ZToKPiA+ID4gPiA+ID4gPiBPbiBUaHUsIEF1ZyAw MiwgMjAxOCBhdCAwMjozMzoxMkFNICswMDAwLCBUaWFuLCBLZXZpbiB3cm90ZToKPiA+ID4gPiA+ ID4gPiA+ID4gT24gV2VkLCBBdWcgMDEsIDIwMTggYXQgMDY6MjI6MTRQTSArMDgwMCwgS2VubmV0 aCBMZWUgd3JvdGU6CgpbLi4uXQoKPiA+ID4gPiA+ID4gPiA+ID4gTXkgbW9yZSBnZW5lcmFsIHF1 ZXN0aW9uIGlzIGRvIHdlIHdhbnQgdG8gZ3JvdyBWRklPIHRvIGJlY29tZQo+ID4gPiA+ID4gPiA+ ID4gPiBhIG1vcmUgZ2VuZXJpYyBkZXZpY2UgZHJpdmVyIEFQSS4gVGhpcyBwYXRjaHNldCBhZGRz IGEgY29tbWFuZAo+ID4gPiA+ID4gPiA+ID4gPiBxdWV1ZSBjb25jZXB0IHRvIGl0IChpIGRvbid0 IHRoaW5rIGl0IGV4aXN0IHRvZGF5IGJ1dCBpIGhhdmUKPiA+ID4gPiA+ID4gPiA+ID4gbm90IGZv bGxvdyBWRklPIGNsb3NlbHkpLgo+ID4gPiA+ID4gPiA+ID4gPiAKPiA+ID4gPiA+ID4gPiBUaGUg dGhpbmcgaXMsIFZGSU8gaXMgdGhlIG9ubHkgcGxhY2UgdG8gc3VwcG9ydCBETUEgZnJvbSB1c2Vy IGxhbmQuIElmIHdlIGRvbid0Cj4gPiA+ID4gPiA+ID4gcHV0IGl0IGhlcmUsIHdlIGhhdmUgdG8g Y3JlYXRlIGFub3RoZXIgc2ltaWxhciBmYWNpbGl0eSB0byBzdXBwb3J0IHRoZSBzYW1lLgo+ID4g PiA+ID4gPiBObyBpdCBpcyBub3QsIG5ldHdvcmsgZGV2aWNlLCBHUFUsIGJsb2NrIGRldmljZSwg Li4uIHRoZXkgYWxsIGRvCj4gPiA+ID4gPiA+IHN1cHBvcnQgRE1BLiBUaGUgcG9pbnQgaSBhbSB0 cnlpbmcgdG8gbWFrZSBoZXJlIGlzIHRoYXQgZXZlbiBpbgo+ID4gPiA+ID4gU29ycnksIHdhaXQg YSBtaW51dGUsIGFyZSB3ZSB0YWxraW5nIHRoZSBzYW1lIHRoaW5nPyBJIG1lYW50ICJETUEgZnJv bSB1c2VyCj4gPiA+ID4gPiBsYW5kIiwgbm90ICJETUEgZnJvbSBrZXJuZWwgZHJpdmVyIi4gVG8g ZG8gdGhhdCB3ZSBoYXZlIHRvIG1hbmlwdWxhdGUgdGhlCj4gPiA+ID4gPiBJT01NVShVbml0KS4g SSB0aGluayBpdCBjYW4gb25seSBiZSBkb25lIGJ5IGRlZmF1bHRfZG9tYWluIG9yIHZmaW8gZG9t YWluLiBPcgo+ID4gPiA+ID4gdGhlIHVzZXIgc3BhY2UgaGF2ZSB0byBkaXJlY3RseSBhY2Nlc3Mg dGhlIElPTU1VLgo+ID4gPiA+IEdQVSBkbyBETUEgaW4gdGhlIHNlbnNlIHRoYXQgeW91IHBhc3Mg dG8gdGhlIGtlcm5lbCBhIHZhbGlkCj4gPiA+ID4gdmlydHVhbCBhZGRyZXNzIChrZXJuZWwgZHJp dmVyIGRvIGFsbCB0aGUgcHJvcGVyIGNoZWNrKSBhbmQKPiA+ID4gPiB0aGVuIHlvdSBjYW4gdXNl IHRoZSBHUFUgdG8gY29weSBmcm9tIG9yIHRvIHRoYXQgcmFuZ2Ugb2YKPiA+ID4gPiB2aXJ0dWFs IGFkZHJlc3MuIEV4YWN0bHkgaG93IHlvdSB3YW50IHRvIHVzZSB0aGlzIGNvbXByZXNzaW9uCj4g PiA+ID4gZW5naW5lLiBJdCBkb2VzIG5vdCByZWx5IG9uIFNWTSBidXQgU1ZNIGdvaW5nIGZvcndh cmQgd291bGQKPiA+ID4gPiBzdGlsbCBiZSB0aGUgcHJlZmVyZWQgb3B0aW9uLgo+ID4gPiA+IAo+ ID4gPiBObywgU1ZNIGlzIG5vdCB0aGUgcmVhc29uIHdoeSB3ZSByZWx5IG9uIEplYW4ncyBTVk0o U1ZBKSBzZXJpZXMuIFdlIHJlbHkgb24KPiA+ID4gSmVhbidzIHNlcmllcyBiZWNhdXNlIG9mIG11 bHRpLXByb2Nlc3MgKFBBU0lEIG9yIHN1YnN0cmVhbSBJRCkgc3VwcG9ydC4KPiA+ID4gCj4gPiA+ IEJ1dCBvZiBjb3VzZSwgV2FycERyaXZlIGNhbiBzdGlsbCBiZW5lZml0IGZyb20gdGhlIFNWTSBm ZWF0dXJlLgo+ID4gV2UgYXJlIGdldHRpbmcgc2lkZSB0cmFja2VkIGhlcmUuIFBBU0lEL0lEIGRv IG5vdCByZXF1aXJlIFZGSU8uCj4gPiAKPiBZZXMsIFBBU0lEIGl0c2VsZiBkbyBub3QgcmVxdWly ZSBWRklPLiBCdXQgd2hhdCBpZjoKPiAKPiAxLiBTdXBwb3J0IERNQSBmcm9tIHVzZXIgc3BhY2Uu Cj4gMi4gVGhlIGhhcmR3YXJlIG1ha2VzIHVzZSBvZiBzdGFuZGFyZCBJT01NVS9TTU1VIGZvciBJ TyBhZGRyZXNzIHRyYW5zbGF0aW9uLgo+IDMuIFRoZSBJT01NVSBmYWNpbGl0eSBpcyBzaGFyZWQg YnkgYm90aCBrZXJuZWwgYW5kIHVzZXIgZHJpdmVycy4KPiA0LiBTdXBwb3J0IFBBU0lEIHdpdGgg dGhlIGN1cnJlbnQgSU9NTVUgZmFjaWxpdHkKCkkgZG8gbm90IHNlZSBob3cgYW55IG9mIHRoaXMg bWVhbnMgaXQgaGFzIHRvIGJlIGluIFZGSU8uCk90aGVyIGRldmljZXMgZG8ganVzdCB0aGF0LiBH UFVzIGRyaXZlciBmb3IgaW5zdGFuY2Ugc2hhcmUKRE1BIGVuZ2luZSAodGhhdCBjb3B5IGRhdGEg YXJvdW5kKSBiZXR3ZWVuIGtlcm5lbCBhbmQgdXNlcgpzcGFjZS4gU29tZXRpbWUga2VybmVsIHVz ZSBpdCB0byBtb3ZlIHRoaW5ncyBhcm91bmQuIEV2aWN0CnNvbWUgbWVtb3J5IHRvIG1ha2Ugcm9v bSBmb3IgYSBuZXcgcHJvY2VzcyBpcyB0aGUgY29tbW9uCmV4YW1wbGUuIFNhbWUgRE1BIGVuZ2lu ZXMgaXMgb2Z0ZW4gdXNlIGJ5IHVzZXJzcGFjZSBpdHNlbGYKZHVyaW5nIHJlbmRlcmluZyBvciBj b21wdXRlIChwcm9ncmFtIG1vdmluZyB0aGluZ3Mgb24gdGhlcmUKb3duKS4gU28gdGhleSBhcmUg YWxyZWFkeSBrZXJuZWwgZHJpdmVyIHRoYXQgZG8gYWxsIDQgb2YKdGhlIGFib3ZlIGFuZCBhcmUg bm90IGluIFZGSU8uCgoKPiA+ID4gPiA+ID4geW91ciBtZWNoYW5pc21zIHRoZSB1c2Vyc3BhY2Ug bXVzdCBoYXZlIGEgc3BlY2lmaWMgdXNlcnNwYWNlCj4gPiA+ID4gPiA+IGRyaXZlcnMgZm9yIGVh Y2ggaGFyZHdhcmUgYW5kIHRodXMgdGhlcmUgYXJlIHZpcnR1YWxseSBubwo+ID4gPiA+ID4gPiBk aWZmZXJlbmNlcyBiZXR3ZWVuIGhhdmluZyB0aGlzIHVzZXJzcGFjZSBkcml2ZXIgb3BlbiBhIGRl dmljZQo+ID4gPiA+ID4gPiBmaWxlIGluIHZmaW8gb3Igc29tZXdoZXJlIGVsc2UgaW4gdGhlIGRl dmljZSBmaWxlc3lzdGVtLiBUaGlzIGlzCj4gPiA+ID4gPiA+IGp1c3QgYSBkaWZmZXJlbnQgcGF0 aC4KPiA+ID4gPiA+ID4gCj4gPiA+ID4gPiBUaGUgYmFzaWMgcHJvYmxlbSBXYXJwRHJpdmUgd2Fu dCB0byBzb2x2ZSBpdCB0byBhdm9pZCBzeXNjYWxsLiBUaGlzIGlzIGltcG9ydGFudAo+ID4gPiA+ ID4gdG8gYWNjZWxlcmF0b3JzLiBXZSBoYXZlIHNvbWUgZGF0YSBoZXJlOgo+ID4gPiA+ID4gaHR0 cHM6Ly93d3cuc2xpZGVzaGFyZS5uZXQvbGluYXJvb3JnL3Byb2dyZXNzLWFuZC1kZW1vbnN0cmF0 aW9uLW9mLXdyYXBkcml2ZS1hLWFjY2VsZXJhdG9yLWZyYW1ld29yay1zZm8xNzMxNwo+ID4gPiA+ ID4gCj4gPiA+ID4gPiAoc2VlIHBhZ2UgMykKPiA+ID4gPiA+IAo+ID4gPiA+ID4gVGhlIHBlcmZv cm1hbmNlIGlzIGRpZmZlcmVudCBvbiB1c2luZyBrZXJuZWwgYW5kIHVzZXIgZHJpdmVycy4KPiA+ ID4gPiBZZXMgYW5kIGV4YW1wbGUgaSBwb2ludCB0byBpcyBleGFjdGx5IHRoYXQuIFlvdSBoYXZl IGEgb25lIHRpbWUgc2V0dXAKPiA+ID4gPiBjb3N0IChjcmVhdGluZyBjb21tYW5kIGJ1ZmZlciBi aW5kaW5nIFBBU0lEIHdpdGggY29tbWFuZCBidWZmZXIgYW5kCj4gPiA+ID4gY291cGxlIG90aGVy IHNldHVwIHN0ZXBzKS4gVGhlbiB1c2Vyc3BhY2Ugbm8gbG9uZ2VyIGhhdmUgdG8gZG8gYW55Cj4g PiA+ID4gaW9jdGwgdG8gc2NoZWR1bGUgd29yayBvbiB0aGUgR1BVLiBJdCBpcyBhbGwgZG93biBm cm9tIHVzZXJzcGFjZSBhbmQKPiA+ID4gPiBpdCB1c2UgYSBkb29yYmVsbCB0byBub3RpZnkgaGFy ZHdhcmUgd2hlbiBpdCBzaG91bGQgZ28gbG9vayBhdCBjb21tYW5kCj4gPiA+ID4gYnVmZmVyIGZv ciBuZXcgdGhpbmcgdG8gZXhlY3V0ZS4KPiA+ID4gPiAKPiA+ID4gPiBNeSBwb2ludCBzdGFuZHMg b24gdGhhdC4gWW91IGhhdmUgZXhpc3RpbmcgZHJpdmVyIGFscmVhZHkgZG9pbmcgc28KPiA+ID4g PiB3aXRoIG5vIG5ldyBmcmFtZXdvcmsgYW5kIGluIHlvdXIgc2NoZW1lIHlvdSBuZWVkIGEgdXNl cnNwYWNlIGRyaXZlci4KPiA+ID4gPiBTbyBpIGRvIG5vdCBzZWUgdGhlIHZhbHVlIGFkZCwgdXNp bmcgb25lIHBhdGggb3IgdGhlIG90aGVyIGluIHRoZQo+ID4gPiA+IHVzZXJzcGFjZSBkcml2ZXIg aXMgbGl0dGVyYWx5IG9uZSBsaW5lIHRvIGNoYW5nZS4KPiA+ID4gPiAKPiA+ID4gU29ycnksIEkn ZCBnb3QgY29uZnVzZSBoZXJlLiBJIHBhcnRpYWxseSBhZ3JlZSB0aGF0IHRoZSB1c2VyIGRyaXZl ciBpcwo+ID4gPiByZWR1bmRhbmNlIG9mIGtlcm5lbCBkcml2ZXIuIChCdXQgZm9yIFdhcnBEcml2 ZSwgdGhlIGtlcm5lbCBkcml2ZXIgaXMgYSBmdWxsCj4gPiA+IGRyaXZlciBpbmNsdWRlIGFsbCBw cmVwYXJhdGlvbiBhbmQgc2V0dXAgc3R1ZmYgZm9yIHRoZSBoYXJkd2FyZSwgdGhlIHVzZXIgZHJp dmVyCj4gPiA+IGlzIHNpbXBseSB0byBzZW5kIHJlcXVlc3QgYW5kIHJlY2VpdmUgYW5zd2VyKS4g WWVzLCBpdCBpcyBqdXN0IGEgY2hvaWNlIG9mIHBhdGguCj4gPiA+IEJ1dCB0aGUgdXNlciBwYXRo IGlzIGZhc3RlciBpZiB0aGUgcmVxdWVzdCBjb21lIGZyb20gdXNlIHNwYWNlLiBBbmQgdG8gZG8g dGhhdCwKPiA+ID4gd2UgbmVlZCB1c2VyIGxhbmQgRE1BIHN1cHBvcnQuIFRoZW4gd2h5IGlzIGl0 IGludmFsdWFibGUgdG8gbGV0IFZGSU8gaW52b2x2ZWQ/Cj4gPiBTb21lIGRyaXZlcnMgaW4gdGhl IGtlcm5lbCBhbHJlYWR5IGRvIGV4YWN0bHkgd2hhdCB5b3Ugc2FpZC4gVGhlIHVzZXIKPiA+IHNw YWNlIGVtaXQgY29tbWFuZHMgd2l0aG91dCBldmVyIGdvaW5nIGludG8ga2VybmVsIGJ5IGRpcmVj dGx5IHNjaGVkdWxpbmcKPiA+IGNvbW1hbmRzIGFuZCByaW5naW5nIGEgZG9vcmJlbGwuIFRoZXkg ZG8gbm90IG5lZWQgVkZJTyBlaXRoZXIgYW5kIHRoZXkKPiA+IGNhbiBtYXAgdXNlcnNwYWNlIGFk ZHJlc3MgaW50byB0aGUgRE1BIGFkZHJlc3Mgc3BhY2Ugb2YgdGhlIGRldmljZSBhbmQKPiA+IGFn YWluIHRoZXkgZG8gbm90IG5lZWQgVkZJTyBmb3IgdGhhdC4KPiBDb3VsZCB5b3UgcGxlYXNlIGRp cmVjdGx5IHBvaW50IG91dCB3aGljaCBkcml2ZXIgeW91IHJlZmVyIHRvIGhlcmU/IFRoYW5rCj4g eW91LgoKZHJpdmVycy9ncHUvZHJtL2FtZC8KClN1Yi1kaXJlY3Rvcnkgb2YgaW50ZXJlc3QgaXMg YW1ka2ZkCgpCZWNhdXNlIGl0IGlzIGEgYmlnIGRyaXZlciBoZXJlIGlzIGEgaGlnaGxldmVsIG92 ZXJ2aWV3IG9mIGhvdyBpdCB3b3JrcwoodGhpcyBpcyBhIHNpbXBsaWZpY2F0aW9uKToKICAtIFBy b2Nlc3MgY2FuIGFsbG9jYXRlIEdQVXMgYnVmZmVyICh0aHJvdWdoIGlvY2x0KSBhbmQgbWFwIHRo ZW0gaW50bwogICAgaXRzIGFkZHJlc3Mgc3BhY2UgKHRocm91Z2ggbW1hcCBvZiBkZXZpY2UgZmls ZSBhdCBidWZmZXIgb2JqZWN0CiAgICBzcGVjaWZpYyBvZmZzZXQpLgogIC0gUHJvY2VzcyBjYW4g bWFwIGFueSB2YWxpZCByYW5nZSBvZiB2aXJ0dWFsIGFkZHJlc3Mgc3BhY2UgaW50byBkZXZpY2UK ICAgIGFkZHJlc3Mgc3BhY2UgKElPTU1VIG1hcHBpbmcpLiBUaGlzIG11c3QgYmUgcmVndWxhciBt ZW1vcnkgaWUgbm90IGFuCiAgICBtbWFwIG9mIGEgZGV2aWNlIGZpbGUgb3IgYW55IHNwZWNpYWwg ZmlsZSAodGhpcyBpcyB0aGUgbm9uIFBBU0lECiAgICBwYXRoKQogIC0gUHJvY2VzcyBjYW4gY3Jl YXRlIGEgY29tbWFuZCBxdWV1ZSBhbmQgYmluZCBpdHMgcHJvY2VzcyB0byBpdCBha2EKICAgIFBB U0lELCB0aGlzIGlzIGRvbmUgdGhyb3VnaCBhbiBpb2N0bC4KICAtIFByb2Nlc3MgY2FuIHNjaGVk dWxlIGNvbW1hbmRzIG9udG8gcXVldWVzIGl0IGNyZWF0ZWQgZnJvbSB1c2Vyc3BhY2UKICAgIHdp dGhvdXQgaW9jdGwuIEZvciB0aGF0IGl0IGp1c3Qgd3JpdGUgY29tbWFuZCBpbnRvIGEgcmluZyBi dWZmZXIKICAgIHRoYXQgaXQgbWFwcGVkIGR1cmluZyB0aGUgY29tbWFuZCBxdWV1ZSBjcmVhdGlv biBwcm9jZXNzIGFuZCBpdAogICAgcmluZ3MgYSBkb29yYmVsbCB3aGVuIGNvbW1hbmRzIGFyZSBy ZWFkeSB0byBiZSBjb25zdW1lIGJ5IHRoZQogICAgaGFyZHdhcmUuCiAgLSBDb21tYW5kcyBjYW4g cmVmZXJlbmNlIChhY2Nlc3MpIGFsbCAzIHR5cGVzIG9mIG9iamVjdCBhYm92ZSBpZQogICAgZWl0 aGVyIGZ1bGwgR1BVcyBidWZmZXIsIHByb2Nlc3MgcmVndWxhciBtZW1vcnkgbWFwZWQgYXMgb2Jq ZWN0CiAgICAobm9uIFBBU0lEKSBhbmQgUEFTSUQgbWVtb3J5IGFsbCBhdCB0aGUgc2FtZSB0aW1l IGllIHlvdSBjYW4KICAgIG1peCBhbGwgb2YgdGhlIGFib3ZlIGluIHNhbWUgY29tbWFuZHMgcXVl dWUuCiAgLSBLZXJuZWwgY2FuIGV2aWN0LCB1bmJpbmQgYW55IHByb2Nlc3MgY29tbWFuZCBxdWV1 ZXMsIHVuYmluZCBjb21tYW5kcwogICAgcXVldWUgYXJlIHN0aWxsIHZhbGlkIGZyb20gcHJvY2Vz cyBwb2ludCBvZiB2aWV3IGJ1dCBjb21tYW5kcwogICAgcHJvY2VzcyBzY2hlZHVsZXMgb24gdGhl bSB3aWxsIG5vdCBiZSBleGVjdXRlZCB1bnRpbCBrZXJuZWwgcmUtYmluZAogICAgdGhlIHF1ZXVl LgogIC0gS2VybmVsIGNhbiBzY2hlZHVsZSBjb21tYW5kcyBpdHNlbGYgb250byBpdHMgZGVkaWNh dGVkIGNvbW1hbmQKICAgIHF1ZXVlcyAoa2VybmVsIGRyaXZlciBjcmVhdGUgaXRzIG93biBjb21t YW5kIHF1ZXVlcykuCiAgLSBLZXJuZWwgY2FuIGNvbnRyb2wgcHJpb3JpdGllcyBiZXR3ZWVuIGFs bCB0aGUgcXVldWVzIGllIGl0IGNhbgogICAgZGVjaWRlcyB3aGljaCBxdWV1ZXMgc2hvdWxkIHRo ZSBoYXJkd2FyZSBleGVjdXRlZCBmaXJzdCBuZXh0LgoKSSBiZWxpZXZlIGFsbCBvZiB0aGUgYWJv dmUgYXJlIHRoZSBhc3BlY3RzIHRoYXQgbWF0dGVycyB0byB5b3UuIFRoZSBtYWluCnJlYXNvbiBp IGRvbid0IGxpa2UgY3JlYXRpbmcgYSBuZXcgZHJpdmVyIGluZnJhc3RydWN0dXJlIGlzIHRoYXQg YSBsb3QKb2YgZXhpc3RpbmcgZHJpdmVycyB3aWxsIHdhbnQgdG8gdXNlIHNvbWUgb2YgdGhlIG5l dyBmZWF0dXJlcyB0aGF0IGFyZQpjb21pbmcgKG1lbW9yeSB0b3BvbG9neSwgd2hlcmUgdG8gcGxh Y2UgcHJvY2VzcyBtZW1vcnksIHBpcGVsaW5lIGRldmljZXMsCi4uLikgYW5kIHRodXMgZXhpc3Rp bmcgZHJpdmVycyBhcmUgYmlnIChHUFUgZHJpdmVycyBhcmUgdGhlIGJpZ2dlc3Qgb2YKYWxsIHRo ZSBrZXJuZWwgZHJpdmVycykuCgpTbyByZXdyaXR0aW5nIHRob3NlIGV4aXN0aW5nIGRyaXZlcnMg aW50byBWRklPIG9yIGludG8gYW55IG5ldyBpbmZyYS0Kc3RydWN0dXJlIHNvIHRoYXQgdGhleSBj YW4gbGV2ZXJhZ2UgbmV3IGZlYXR1cmVzIGlzIGEgbm8gZ28gZnJvbSBteQpwb2ludCBvZiB2aWV3 LgoKSSB3b3VsZCByYXRoZXIgc2VlIGEgc2V0IG9mIGhlbHBlcnMgc28gdGhhdCBlYWNoIGZlYXR1 cmVzIGNhbiBiZSB1c2UKZWl0aGVyIGJ5IG5ldyBkcml2ZXJzIG9yIGV4aXN0aW5nIGRyaXZlcnMu IEZvciBpbnN0YW5jZSBhIG5ldyB3YXkgdG8KZXhwb3NlIG1lbW9yeSB0b3BvbG9neS4gQSBuZXcg d2F5IHRvIGV4cG9zZSBob3cgeW91IGNhbiBwaXBlIGRldmljZXMKZnJvbSBvbmUgdG8gYW5vdGhl ciAuLi4KCgpIZW5jZSBpIGRvIG5vdCBzZWUgYW55IHZhbHVlIGluIGEgd2hvbGUgbmV3IGluZnJh LXN0cnVjdHVyZSBpbiB3aGljaApkcml2ZXJzIG11c3QgYmUgcGFydCBvZiB0byBsZXZlcmFnZSBu ZXcgZmVhdHVyZXMuCgpDaGVlcnMsCkrDqXLDtG1lCl9fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fCmlvbW11IG1haWxpbmcgbGlzdAppb21tdUBsaXN0cy5saW51 eC1mb3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3RzLmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFpbG1h bi9saXN0aW5mby9pb21tdQ== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on archive.lwn.net X-Spam-Level: X-Spam-Status: No, score=-6.0 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham autolearn_force=no version=3.4.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by archive.lwn.net (Postfix) with ESMTP id 3A2DE7E3B8 for ; Wed, 8 Aug 2018 15:18:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727191AbeHHRit (ORCPT ); Wed, 8 Aug 2018 13:38:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:36744 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727081AbeHHRit (ORCPT ); Wed, 8 Aug 2018 13:38:49 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4108E87A77; Wed, 8 Aug 2018 15:18:40 +0000 (UTC) Received: from redhat.com (unknown [10.20.6.215]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EFCB520180F4; Wed, 8 Aug 2018 15:18:36 +0000 (UTC) Date: Wed, 8 Aug 2018 11:18:35 -0400 From: Jerome Glisse To: Kenneth Lee Cc: Kenneth Lee , "Tian, Kevin" , Alex Williamson , Herbert Xu , "kvm@vger.kernel.org" , Jonathan Corbet , Greg Kroah-Hartman , Zaibo Xu , "linux-doc@vger.kernel.org" , "Kumar, Sanjay K" , Hao Fang , "linux-kernel@vger.kernel.org" , "linuxarm@huawei.com" , "iommu@lists.linux-foundation.org" , "linux-crypto@vger.kernel.org" , Philippe Ombredanne , Thomas Gleixner , "David S . Miller" , "linux-accelerators@lists.ozlabs.org" Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive Message-ID: <20180808151835.GA3429@redhat.com> References: <20180801102221.5308-1-nek.in.cn@gmail.com> <20180801165644.GA3820@redhat.com> <20180802040557.GL160746@Turing-Arch-b> <20180802142243.GA3481@redhat.com> <20180803034721.GC91035@Turing-Arch-b> <20180803143944.GA4079@redhat.com> <20180806031252.GG91035@Turing-Arch-b> <20180806153257.GB6002@redhat.com> <11bace0e-dc14-5d2c-f65c-25b852f4e9ca@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <11bace0e-dc14-5d2c-f65c-25b852f4e9ca@gmail.com> User-Agent: Mutt/1.10.0 (2018-05-17) X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 15:18:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 15:18:40 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'jglisse@redhat.com' RCPT:'' Sender: linux-doc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-doc@vger.kernel.org On Wed, Aug 08, 2018 at 09:08:42AM +0800, Kenneth Lee wrote: > > > 在 2018年08月06日 星期一 11:32 下午, Jerome Glisse 写道: > > On Mon, Aug 06, 2018 at 11:12:52AM +0800, Kenneth Lee wrote: > > > On Fri, Aug 03, 2018 at 10:39:44AM -0400, Jerome Glisse wrote: > > > > On Fri, Aug 03, 2018 at 11:47:21AM +0800, Kenneth Lee wrote: > > > > > On Thu, Aug 02, 2018 at 10:22:43AM -0400, Jerome Glisse wrote: > > > > > > On Thu, Aug 02, 2018 at 12:05:57PM +0800, Kenneth Lee wrote: > > > > > > > On Thu, Aug 02, 2018 at 02:33:12AM +0000, Tian, Kevin wrote: > > > > > > > > > On Wed, Aug 01, 2018 at 06:22:14PM +0800, Kenneth Lee wrote: [...] > > > > > > > > > My more general question is do we want to grow VFIO to become > > > > > > > > > a more generic device driver API. This patchset adds a command > > > > > > > > > queue concept to it (i don't think it exist today but i have > > > > > > > > > not follow VFIO closely). > > > > > > > > > > > > > > > > The thing is, VFIO is the only place to support DMA from user land. If we don't > > > > > > > put it here, we have to create another similar facility to support the same. > > > > > > No it is not, network device, GPU, block device, ... they all do > > > > > > support DMA. The point i am trying to make here is that even in > > > > > Sorry, wait a minute, are we talking the same thing? I meant "DMA from user > > > > > land", not "DMA from kernel driver". To do that we have to manipulate the > > > > > IOMMU(Unit). I think it can only be done by default_domain or vfio domain. Or > > > > > the user space have to directly access the IOMMU. > > > > GPU do DMA in the sense that you pass to the kernel a valid > > > > virtual address (kernel driver do all the proper check) and > > > > then you can use the GPU to copy from or to that range of > > > > virtual address. Exactly how you want to use this compression > > > > engine. It does not rely on SVM but SVM going forward would > > > > still be the prefered option. > > > > > > > No, SVM is not the reason why we rely on Jean's SVM(SVA) series. We rely on > > > Jean's series because of multi-process (PASID or substream ID) support. > > > > > > But of couse, WarpDrive can still benefit from the SVM feature. > > We are getting side tracked here. PASID/ID do not require VFIO. > > > Yes, PASID itself do not require VFIO. But what if: > > 1. Support DMA from user space. > 2. The hardware makes use of standard IOMMU/SMMU for IO address translation. > 3. The IOMMU facility is shared by both kernel and user drivers. > 4. Support PASID with the current IOMMU facility I do not see how any of this means it has to be in VFIO. Other devices do just that. GPUs driver for instance share DMA engine (that copy data around) between kernel and user space. Sometime kernel use it to move things around. Evict some memory to make room for a new process is the common example. Same DMA engines is often use by userspace itself during rendering or compute (program moving things on there own). So they are already kernel driver that do all 4 of the above and are not in VFIO. > > > > > > your mechanisms the userspace must have a specific userspace > > > > > > drivers for each hardware and thus there are virtually no > > > > > > differences between having this userspace driver open a device > > > > > > file in vfio or somewhere else in the device filesystem. This is > > > > > > just a different path. > > > > > > > > > > > The basic problem WarpDrive want to solve it to avoid syscall. This is important > > > > > to accelerators. We have some data here: > > > > > https://www.slideshare.net/linaroorg/progress-and-demonstration-of-wrapdrive-a-accelerator-framework-sfo17317 > > > > > > > > > > (see page 3) > > > > > > > > > > The performance is different on using kernel and user drivers. > > > > Yes and example i point to is exactly that. You have a one time setup > > > > cost (creating command buffer binding PASID with command buffer and > > > > couple other setup steps). Then userspace no longer have to do any > > > > ioctl to schedule work on the GPU. It is all down from userspace and > > > > it use a doorbell to notify hardware when it should go look at command > > > > buffer for new thing to execute. > > > > > > > > My point stands on that. You have existing driver already doing so > > > > with no new framework and in your scheme you need a userspace driver. > > > > So i do not see the value add, using one path or the other in the > > > > userspace driver is litteraly one line to change. > > > > > > > Sorry, I'd got confuse here. I partially agree that the user driver is > > > redundance of kernel driver. (But for WarpDrive, the kernel driver is a full > > > driver include all preparation and setup stuff for the hardware, the user driver > > > is simply to send request and receive answer). Yes, it is just a choice of path. > > > But the user path is faster if the request come from use space. And to do that, > > > we need user land DMA support. Then why is it invaluable to let VFIO involved? > > Some drivers in the kernel already do exactly what you said. The user > > space emit commands without ever going into kernel by directly scheduling > > commands and ringing a doorbell. They do not need VFIO either and they > > can map userspace address into the DMA address space of the device and > > again they do not need VFIO for that. > Could you please directly point out which driver you refer to here? Thank > you. drivers/gpu/drm/amd/ Sub-directory of interest is amdkfd Because it is a big driver here is a highlevel overview of how it works (this is a simplification): - Process can allocate GPUs buffer (through ioclt) and map them into its address space (through mmap of device file at buffer object specific offset). - Process can map any valid range of virtual address space into device address space (IOMMU mapping). This must be regular memory ie not an mmap of a device file or any special file (this is the non PASID path) - Process can create a command queue and bind its process to it aka PASID, this is done through an ioctl. - Process can schedule commands onto queues it created from userspace without ioctl. For that it just write command into a ring buffer that it mapped during the command queue creation process and it rings a doorbell when commands are ready to be consume by the hardware. - Commands can reference (access) all 3 types of object above ie either full GPUs buffer, process regular memory maped as object (non PASID) and PASID memory all at the same time ie you can mix all of the above in same commands queue. - Kernel can evict, unbind any process command queues, unbind commands queue are still valid from process point of view but commands process schedules on them will not be executed until kernel re-bind the queue. - Kernel can schedule commands itself onto its dedicated command queues (kernel driver create its own command queues). - Kernel can control priorities between all the queues ie it can decides which queues should the hardware executed first next. I believe all of the above are the aspects that matters to you. The main reason i don't like creating a new driver infrastructure is that a lot of existing drivers will want to use some of the new features that are coming (memory topology, where to place process memory, pipeline devices, ...) and thus existing drivers are big (GPU drivers are the biggest of all the kernel drivers). So rewritting those existing drivers into VFIO or into any new infra- structure so that they can leverage new features is a no go from my point of view. I would rather see a set of helpers so that each features can be use either by new drivers or existing drivers. For instance a new way to expose memory topology. A new way to expose how you can pipe devices from one to another ... Hence i do not see any value in a whole new infra-structure in which drivers must be part of to leverage new features. Cheers, Jérôme -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive Date: Wed, 8 Aug 2018 11:18:35 -0400 Message-ID: <20180808151835.GA3429@redhat.com> References: <20180801102221.5308-1-nek.in.cn@gmail.com> <20180801165644.GA3820@redhat.com> <20180802040557.GL160746@Turing-Arch-b> <20180802142243.GA3481@redhat.com> <20180803034721.GC91035@Turing-Arch-b> <20180803143944.GA4079@redhat.com> <20180806031252.GG91035@Turing-Arch-b> <20180806153257.GB6002@redhat.com> <11bace0e-dc14-5d2c-f65c-25b852f4e9ca@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <11bace0e-dc14-5d2c-f65c-25b852f4e9ca-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Errors-To: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org To: Kenneth Lee Cc: "Tian, Kevin" , Herbert Xu , "kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Jonathan Corbet , Greg Kroah-Hartman , Zaibo Xu , "linux-doc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "Kumar, Sanjay K" , Hao Fang , "iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linuxarm-hv44wF8Li93QT0dZR+AlfA@public.gmane.org" , Alex Williamson , "linux-crypto-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Philippe Ombredanne , Thomas Gleixner , Kenneth Lee , "David S . Miller" , "linux-accelerators-uLR06cmDAlY/bJ5BZ2RsiQ@public.gmane.org" List-Id: iommu@lists.linux-foundation.org T24gV2VkLCBBdWcgMDgsIDIwMTggYXQgMDk6MDg6NDJBTSArMDgwMCwgS2VubmV0aCBMZWUgd3Jv dGU6Cj4gCj4gCj4g5ZyoIDIwMTjlubQwOOaciDA25pelIOaYn+acn+S4gCAxMTozMiDkuIvljYgs IEplcm9tZSBHbGlzc2Ug5YaZ6YGTOgo+ID4gT24gTW9uLCBBdWcgMDYsIDIwMTggYXQgMTE6MTI6 NTJBTSArMDgwMCwgS2VubmV0aCBMZWUgd3JvdGU6Cj4gPiA+IE9uIEZyaSwgQXVnIDAzLCAyMDE4 IGF0IDEwOjM5OjQ0QU0gLTA0MDAsIEplcm9tZSBHbGlzc2Ugd3JvdGU6Cj4gPiA+ID4gT24gRnJp LCBBdWcgMDMsIDIwMTggYXQgMTE6NDc6MjFBTSArMDgwMCwgS2VubmV0aCBMZWUgd3JvdGU6Cj4g PiA+ID4gPiBPbiBUaHUsIEF1ZyAwMiwgMjAxOCBhdCAxMDoyMjo0M0FNIC0wNDAwLCBKZXJvbWUg R2xpc3NlIHdyb3RlOgo+ID4gPiA+ID4gPiBPbiBUaHUsIEF1ZyAwMiwgMjAxOCBhdCAxMjowNTo1 N1BNICswODAwLCBLZW5uZXRoIExlZSB3cm90ZToKPiA+ID4gPiA+ID4gPiBPbiBUaHUsIEF1ZyAw MiwgMjAxOCBhdCAwMjozMzoxMkFNICswMDAwLCBUaWFuLCBLZXZpbiB3cm90ZToKPiA+ID4gPiA+ ID4gPiA+ID4gT24gV2VkLCBBdWcgMDEsIDIwMTggYXQgMDY6MjI6MTRQTSArMDgwMCwgS2VubmV0 aCBMZWUgd3JvdGU6CgpbLi4uXQoKPiA+ID4gPiA+ID4gPiA+ID4gTXkgbW9yZSBnZW5lcmFsIHF1 ZXN0aW9uIGlzIGRvIHdlIHdhbnQgdG8gZ3JvdyBWRklPIHRvIGJlY29tZQo+ID4gPiA+ID4gPiA+ ID4gPiBhIG1vcmUgZ2VuZXJpYyBkZXZpY2UgZHJpdmVyIEFQSS4gVGhpcyBwYXRjaHNldCBhZGRz IGEgY29tbWFuZAo+ID4gPiA+ID4gPiA+ID4gPiBxdWV1ZSBjb25jZXB0IHRvIGl0IChpIGRvbid0 IHRoaW5rIGl0IGV4aXN0IHRvZGF5IGJ1dCBpIGhhdmUKPiA+ID4gPiA+ID4gPiA+ID4gbm90IGZv bGxvdyBWRklPIGNsb3NlbHkpLgo+ID4gPiA+ID4gPiA+ID4gPiAKPiA+ID4gPiA+ID4gPiBUaGUg dGhpbmcgaXMsIFZGSU8gaXMgdGhlIG9ubHkgcGxhY2UgdG8gc3VwcG9ydCBETUEgZnJvbSB1c2Vy IGxhbmQuIElmIHdlIGRvbid0Cj4gPiA+ID4gPiA+ID4gcHV0IGl0IGhlcmUsIHdlIGhhdmUgdG8g Y3JlYXRlIGFub3RoZXIgc2ltaWxhciBmYWNpbGl0eSB0byBzdXBwb3J0IHRoZSBzYW1lLgo+ID4g PiA+ID4gPiBObyBpdCBpcyBub3QsIG5ldHdvcmsgZGV2aWNlLCBHUFUsIGJsb2NrIGRldmljZSwg Li4uIHRoZXkgYWxsIGRvCj4gPiA+ID4gPiA+IHN1cHBvcnQgRE1BLiBUaGUgcG9pbnQgaSBhbSB0 cnlpbmcgdG8gbWFrZSBoZXJlIGlzIHRoYXQgZXZlbiBpbgo+ID4gPiA+ID4gU29ycnksIHdhaXQg YSBtaW51dGUsIGFyZSB3ZSB0YWxraW5nIHRoZSBzYW1lIHRoaW5nPyBJIG1lYW50ICJETUEgZnJv bSB1c2VyCj4gPiA+ID4gPiBsYW5kIiwgbm90ICJETUEgZnJvbSBrZXJuZWwgZHJpdmVyIi4gVG8g ZG8gdGhhdCB3ZSBoYXZlIHRvIG1hbmlwdWxhdGUgdGhlCj4gPiA+ID4gPiBJT01NVShVbml0KS4g SSB0aGluayBpdCBjYW4gb25seSBiZSBkb25lIGJ5IGRlZmF1bHRfZG9tYWluIG9yIHZmaW8gZG9t YWluLiBPcgo+ID4gPiA+ID4gdGhlIHVzZXIgc3BhY2UgaGF2ZSB0byBkaXJlY3RseSBhY2Nlc3Mg dGhlIElPTU1VLgo+ID4gPiA+IEdQVSBkbyBETUEgaW4gdGhlIHNlbnNlIHRoYXQgeW91IHBhc3Mg dG8gdGhlIGtlcm5lbCBhIHZhbGlkCj4gPiA+ID4gdmlydHVhbCBhZGRyZXNzIChrZXJuZWwgZHJp dmVyIGRvIGFsbCB0aGUgcHJvcGVyIGNoZWNrKSBhbmQKPiA+ID4gPiB0aGVuIHlvdSBjYW4gdXNl IHRoZSBHUFUgdG8gY29weSBmcm9tIG9yIHRvIHRoYXQgcmFuZ2Ugb2YKPiA+ID4gPiB2aXJ0dWFs IGFkZHJlc3MuIEV4YWN0bHkgaG93IHlvdSB3YW50IHRvIHVzZSB0aGlzIGNvbXByZXNzaW9uCj4g PiA+ID4gZW5naW5lLiBJdCBkb2VzIG5vdCByZWx5IG9uIFNWTSBidXQgU1ZNIGdvaW5nIGZvcndh cmQgd291bGQKPiA+ID4gPiBzdGlsbCBiZSB0aGUgcHJlZmVyZWQgb3B0aW9uLgo+ID4gPiA+IAo+ ID4gPiBObywgU1ZNIGlzIG5vdCB0aGUgcmVhc29uIHdoeSB3ZSByZWx5IG9uIEplYW4ncyBTVk0o U1ZBKSBzZXJpZXMuIFdlIHJlbHkgb24KPiA+ID4gSmVhbidzIHNlcmllcyBiZWNhdXNlIG9mIG11 bHRpLXByb2Nlc3MgKFBBU0lEIG9yIHN1YnN0cmVhbSBJRCkgc3VwcG9ydC4KPiA+ID4gCj4gPiA+ IEJ1dCBvZiBjb3VzZSwgV2FycERyaXZlIGNhbiBzdGlsbCBiZW5lZml0IGZyb20gdGhlIFNWTSBm ZWF0dXJlLgo+ID4gV2UgYXJlIGdldHRpbmcgc2lkZSB0cmFja2VkIGhlcmUuIFBBU0lEL0lEIGRv IG5vdCByZXF1aXJlIFZGSU8uCj4gPiAKPiBZZXMsIFBBU0lEIGl0c2VsZiBkbyBub3QgcmVxdWly ZSBWRklPLiBCdXQgd2hhdCBpZjoKPiAKPiAxLiBTdXBwb3J0IERNQSBmcm9tIHVzZXIgc3BhY2Uu Cj4gMi4gVGhlIGhhcmR3YXJlIG1ha2VzIHVzZSBvZiBzdGFuZGFyZCBJT01NVS9TTU1VIGZvciBJ TyBhZGRyZXNzIHRyYW5zbGF0aW9uLgo+IDMuIFRoZSBJT01NVSBmYWNpbGl0eSBpcyBzaGFyZWQg YnkgYm90aCBrZXJuZWwgYW5kIHVzZXIgZHJpdmVycy4KPiA0LiBTdXBwb3J0IFBBU0lEIHdpdGgg dGhlIGN1cnJlbnQgSU9NTVUgZmFjaWxpdHkKCkkgZG8gbm90IHNlZSBob3cgYW55IG9mIHRoaXMg bWVhbnMgaXQgaGFzIHRvIGJlIGluIFZGSU8uCk90aGVyIGRldmljZXMgZG8ganVzdCB0aGF0LiBH UFVzIGRyaXZlciBmb3IgaW5zdGFuY2Ugc2hhcmUKRE1BIGVuZ2luZSAodGhhdCBjb3B5IGRhdGEg YXJvdW5kKSBiZXR3ZWVuIGtlcm5lbCBhbmQgdXNlcgpzcGFjZS4gU29tZXRpbWUga2VybmVsIHVz ZSBpdCB0byBtb3ZlIHRoaW5ncyBhcm91bmQuIEV2aWN0CnNvbWUgbWVtb3J5IHRvIG1ha2Ugcm9v bSBmb3IgYSBuZXcgcHJvY2VzcyBpcyB0aGUgY29tbW9uCmV4YW1wbGUuIFNhbWUgRE1BIGVuZ2lu ZXMgaXMgb2Z0ZW4gdXNlIGJ5IHVzZXJzcGFjZSBpdHNlbGYKZHVyaW5nIHJlbmRlcmluZyBvciBj b21wdXRlIChwcm9ncmFtIG1vdmluZyB0aGluZ3Mgb24gdGhlcmUKb3duKS4gU28gdGhleSBhcmUg YWxyZWFkeSBrZXJuZWwgZHJpdmVyIHRoYXQgZG8gYWxsIDQgb2YKdGhlIGFib3ZlIGFuZCBhcmUg bm90IGluIFZGSU8uCgoKPiA+ID4gPiA+ID4geW91ciBtZWNoYW5pc21zIHRoZSB1c2Vyc3BhY2Ug bXVzdCBoYXZlIGEgc3BlY2lmaWMgdXNlcnNwYWNlCj4gPiA+ID4gPiA+IGRyaXZlcnMgZm9yIGVh Y2ggaGFyZHdhcmUgYW5kIHRodXMgdGhlcmUgYXJlIHZpcnR1YWxseSBubwo+ID4gPiA+ID4gPiBk aWZmZXJlbmNlcyBiZXR3ZWVuIGhhdmluZyB0aGlzIHVzZXJzcGFjZSBkcml2ZXIgb3BlbiBhIGRl dmljZQo+ID4gPiA+ID4gPiBmaWxlIGluIHZmaW8gb3Igc29tZXdoZXJlIGVsc2UgaW4gdGhlIGRl dmljZSBmaWxlc3lzdGVtLiBUaGlzIGlzCj4gPiA+ID4gPiA+IGp1c3QgYSBkaWZmZXJlbnQgcGF0 aC4KPiA+ID4gPiA+ID4gCj4gPiA+ID4gPiBUaGUgYmFzaWMgcHJvYmxlbSBXYXJwRHJpdmUgd2Fu dCB0byBzb2x2ZSBpdCB0byBhdm9pZCBzeXNjYWxsLiBUaGlzIGlzIGltcG9ydGFudAo+ID4gPiA+ ID4gdG8gYWNjZWxlcmF0b3JzLiBXZSBoYXZlIHNvbWUgZGF0YSBoZXJlOgo+ID4gPiA+ID4gaHR0 cHM6Ly93d3cuc2xpZGVzaGFyZS5uZXQvbGluYXJvb3JnL3Byb2dyZXNzLWFuZC1kZW1vbnN0cmF0 aW9uLW9mLXdyYXBkcml2ZS1hLWFjY2VsZXJhdG9yLWZyYW1ld29yay1zZm8xNzMxNwo+ID4gPiA+ ID4gCj4gPiA+ID4gPiAoc2VlIHBhZ2UgMykKPiA+ID4gPiA+IAo+ID4gPiA+ID4gVGhlIHBlcmZv cm1hbmNlIGlzIGRpZmZlcmVudCBvbiB1c2luZyBrZXJuZWwgYW5kIHVzZXIgZHJpdmVycy4KPiA+ ID4gPiBZZXMgYW5kIGV4YW1wbGUgaSBwb2ludCB0byBpcyBleGFjdGx5IHRoYXQuIFlvdSBoYXZl IGEgb25lIHRpbWUgc2V0dXAKPiA+ID4gPiBjb3N0IChjcmVhdGluZyBjb21tYW5kIGJ1ZmZlciBi aW5kaW5nIFBBU0lEIHdpdGggY29tbWFuZCBidWZmZXIgYW5kCj4gPiA+ID4gY291cGxlIG90aGVy IHNldHVwIHN0ZXBzKS4gVGhlbiB1c2Vyc3BhY2Ugbm8gbG9uZ2VyIGhhdmUgdG8gZG8gYW55Cj4g PiA+ID4gaW9jdGwgdG8gc2NoZWR1bGUgd29yayBvbiB0aGUgR1BVLiBJdCBpcyBhbGwgZG93biBm cm9tIHVzZXJzcGFjZSBhbmQKPiA+ID4gPiBpdCB1c2UgYSBkb29yYmVsbCB0byBub3RpZnkgaGFy ZHdhcmUgd2hlbiBpdCBzaG91bGQgZ28gbG9vayBhdCBjb21tYW5kCj4gPiA+ID4gYnVmZmVyIGZv ciBuZXcgdGhpbmcgdG8gZXhlY3V0ZS4KPiA+ID4gPiAKPiA+ID4gPiBNeSBwb2ludCBzdGFuZHMg b24gdGhhdC4gWW91IGhhdmUgZXhpc3RpbmcgZHJpdmVyIGFscmVhZHkgZG9pbmcgc28KPiA+ID4g PiB3aXRoIG5vIG5ldyBmcmFtZXdvcmsgYW5kIGluIHlvdXIgc2NoZW1lIHlvdSBuZWVkIGEgdXNl cnNwYWNlIGRyaXZlci4KPiA+ID4gPiBTbyBpIGRvIG5vdCBzZWUgdGhlIHZhbHVlIGFkZCwgdXNp bmcgb25lIHBhdGggb3IgdGhlIG90aGVyIGluIHRoZQo+ID4gPiA+IHVzZXJzcGFjZSBkcml2ZXIg aXMgbGl0dGVyYWx5IG9uZSBsaW5lIHRvIGNoYW5nZS4KPiA+ID4gPiAKPiA+ID4gU29ycnksIEkn ZCBnb3QgY29uZnVzZSBoZXJlLiBJIHBhcnRpYWxseSBhZ3JlZSB0aGF0IHRoZSB1c2VyIGRyaXZl ciBpcwo+ID4gPiByZWR1bmRhbmNlIG9mIGtlcm5lbCBkcml2ZXIuIChCdXQgZm9yIFdhcnBEcml2 ZSwgdGhlIGtlcm5lbCBkcml2ZXIgaXMgYSBmdWxsCj4gPiA+IGRyaXZlciBpbmNsdWRlIGFsbCBw cmVwYXJhdGlvbiBhbmQgc2V0dXAgc3R1ZmYgZm9yIHRoZSBoYXJkd2FyZSwgdGhlIHVzZXIgZHJp dmVyCj4gPiA+IGlzIHNpbXBseSB0byBzZW5kIHJlcXVlc3QgYW5kIHJlY2VpdmUgYW5zd2VyKS4g WWVzLCBpdCBpcyBqdXN0IGEgY2hvaWNlIG9mIHBhdGguCj4gPiA+IEJ1dCB0aGUgdXNlciBwYXRo IGlzIGZhc3RlciBpZiB0aGUgcmVxdWVzdCBjb21lIGZyb20gdXNlIHNwYWNlLiBBbmQgdG8gZG8g dGhhdCwKPiA+ID4gd2UgbmVlZCB1c2VyIGxhbmQgRE1BIHN1cHBvcnQuIFRoZW4gd2h5IGlzIGl0 IGludmFsdWFibGUgdG8gbGV0IFZGSU8gaW52b2x2ZWQ/Cj4gPiBTb21lIGRyaXZlcnMgaW4gdGhl IGtlcm5lbCBhbHJlYWR5IGRvIGV4YWN0bHkgd2hhdCB5b3Ugc2FpZC4gVGhlIHVzZXIKPiA+IHNw YWNlIGVtaXQgY29tbWFuZHMgd2l0aG91dCBldmVyIGdvaW5nIGludG8ga2VybmVsIGJ5IGRpcmVj dGx5IHNjaGVkdWxpbmcKPiA+IGNvbW1hbmRzIGFuZCByaW5naW5nIGEgZG9vcmJlbGwuIFRoZXkg ZG8gbm90IG5lZWQgVkZJTyBlaXRoZXIgYW5kIHRoZXkKPiA+IGNhbiBtYXAgdXNlcnNwYWNlIGFk ZHJlc3MgaW50byB0aGUgRE1BIGFkZHJlc3Mgc3BhY2Ugb2YgdGhlIGRldmljZSBhbmQKPiA+IGFn YWluIHRoZXkgZG8gbm90IG5lZWQgVkZJTyBmb3IgdGhhdC4KPiBDb3VsZCB5b3UgcGxlYXNlIGRp cmVjdGx5IHBvaW50IG91dCB3aGljaCBkcml2ZXIgeW91IHJlZmVyIHRvIGhlcmU/IFRoYW5rCj4g eW91LgoKZHJpdmVycy9ncHUvZHJtL2FtZC8KClN1Yi1kaXJlY3Rvcnkgb2YgaW50ZXJlc3QgaXMg YW1ka2ZkCgpCZWNhdXNlIGl0IGlzIGEgYmlnIGRyaXZlciBoZXJlIGlzIGEgaGlnaGxldmVsIG92 ZXJ2aWV3IG9mIGhvdyBpdCB3b3JrcwoodGhpcyBpcyBhIHNpbXBsaWZpY2F0aW9uKToKICAtIFBy b2Nlc3MgY2FuIGFsbG9jYXRlIEdQVXMgYnVmZmVyICh0aHJvdWdoIGlvY2x0KSBhbmQgbWFwIHRo ZW0gaW50bwogICAgaXRzIGFkZHJlc3Mgc3BhY2UgKHRocm91Z2ggbW1hcCBvZiBkZXZpY2UgZmls ZSBhdCBidWZmZXIgb2JqZWN0CiAgICBzcGVjaWZpYyBvZmZzZXQpLgogIC0gUHJvY2VzcyBjYW4g bWFwIGFueSB2YWxpZCByYW5nZSBvZiB2aXJ0dWFsIGFkZHJlc3Mgc3BhY2UgaW50byBkZXZpY2UK ICAgIGFkZHJlc3Mgc3BhY2UgKElPTU1VIG1hcHBpbmcpLiBUaGlzIG11c3QgYmUgcmVndWxhciBt ZW1vcnkgaWUgbm90IGFuCiAgICBtbWFwIG9mIGEgZGV2aWNlIGZpbGUgb3IgYW55IHNwZWNpYWwg ZmlsZSAodGhpcyBpcyB0aGUgbm9uIFBBU0lECiAgICBwYXRoKQogIC0gUHJvY2VzcyBjYW4gY3Jl YXRlIGEgY29tbWFuZCBxdWV1ZSBhbmQgYmluZCBpdHMgcHJvY2VzcyB0byBpdCBha2EKICAgIFBB U0lELCB0aGlzIGlzIGRvbmUgdGhyb3VnaCBhbiBpb2N0bC4KICAtIFByb2Nlc3MgY2FuIHNjaGVk dWxlIGNvbW1hbmRzIG9udG8gcXVldWVzIGl0IGNyZWF0ZWQgZnJvbSB1c2Vyc3BhY2UKICAgIHdp dGhvdXQgaW9jdGwuIEZvciB0aGF0IGl0IGp1c3Qgd3JpdGUgY29tbWFuZCBpbnRvIGEgcmluZyBi dWZmZXIKICAgIHRoYXQgaXQgbWFwcGVkIGR1cmluZyB0aGUgY29tbWFuZCBxdWV1ZSBjcmVhdGlv biBwcm9jZXNzIGFuZCBpdAogICAgcmluZ3MgYSBkb29yYmVsbCB3aGVuIGNvbW1hbmRzIGFyZSBy ZWFkeSB0byBiZSBjb25zdW1lIGJ5IHRoZQogICAgaGFyZHdhcmUuCiAgLSBDb21tYW5kcyBjYW4g cmVmZXJlbmNlIChhY2Nlc3MpIGFsbCAzIHR5cGVzIG9mIG9iamVjdCBhYm92ZSBpZQogICAgZWl0 aGVyIGZ1bGwgR1BVcyBidWZmZXIsIHByb2Nlc3MgcmVndWxhciBtZW1vcnkgbWFwZWQgYXMgb2Jq ZWN0CiAgICAobm9uIFBBU0lEKSBhbmQgUEFTSUQgbWVtb3J5IGFsbCBhdCB0aGUgc2FtZSB0aW1l IGllIHlvdSBjYW4KICAgIG1peCBhbGwgb2YgdGhlIGFib3ZlIGluIHNhbWUgY29tbWFuZHMgcXVl dWUuCiAgLSBLZXJuZWwgY2FuIGV2aWN0LCB1bmJpbmQgYW55IHByb2Nlc3MgY29tbWFuZCBxdWV1 ZXMsIHVuYmluZCBjb21tYW5kcwogICAgcXVldWUgYXJlIHN0aWxsIHZhbGlkIGZyb20gcHJvY2Vz cyBwb2ludCBvZiB2aWV3IGJ1dCBjb21tYW5kcwogICAgcHJvY2VzcyBzY2hlZHVsZXMgb24gdGhl bSB3aWxsIG5vdCBiZSBleGVjdXRlZCB1bnRpbCBrZXJuZWwgcmUtYmluZAogICAgdGhlIHF1ZXVl LgogIC0gS2VybmVsIGNhbiBzY2hlZHVsZSBjb21tYW5kcyBpdHNlbGYgb250byBpdHMgZGVkaWNh dGVkIGNvbW1hbmQKICAgIHF1ZXVlcyAoa2VybmVsIGRyaXZlciBjcmVhdGUgaXRzIG93biBjb21t YW5kIHF1ZXVlcykuCiAgLSBLZXJuZWwgY2FuIGNvbnRyb2wgcHJpb3JpdGllcyBiZXR3ZWVuIGFs bCB0aGUgcXVldWVzIGllIGl0IGNhbgogICAgZGVjaWRlcyB3aGljaCBxdWV1ZXMgc2hvdWxkIHRo ZSBoYXJkd2FyZSBleGVjdXRlZCBmaXJzdCBuZXh0LgoKSSBiZWxpZXZlIGFsbCBvZiB0aGUgYWJv dmUgYXJlIHRoZSBhc3BlY3RzIHRoYXQgbWF0dGVycyB0byB5b3UuIFRoZSBtYWluCnJlYXNvbiBp IGRvbid0IGxpa2UgY3JlYXRpbmcgYSBuZXcgZHJpdmVyIGluZnJhc3RydWN0dXJlIGlzIHRoYXQg YSBsb3QKb2YgZXhpc3RpbmcgZHJpdmVycyB3aWxsIHdhbnQgdG8gdXNlIHNvbWUgb2YgdGhlIG5l dyBmZWF0dXJlcyB0aGF0IGFyZQpjb21pbmcgKG1lbW9yeSB0b3BvbG9neSwgd2hlcmUgdG8gcGxh Y2UgcHJvY2VzcyBtZW1vcnksIHBpcGVsaW5lIGRldmljZXMsCi4uLikgYW5kIHRodXMgZXhpc3Rp bmcgZHJpdmVycyBhcmUgYmlnIChHUFUgZHJpdmVycyBhcmUgdGhlIGJpZ2dlc3Qgb2YKYWxsIHRo ZSBrZXJuZWwgZHJpdmVycykuCgpTbyByZXdyaXR0aW5nIHRob3NlIGV4aXN0aW5nIGRyaXZlcnMg aW50byBWRklPIG9yIGludG8gYW55IG5ldyBpbmZyYS0Kc3RydWN0dXJlIHNvIHRoYXQgdGhleSBj YW4gbGV2ZXJhZ2UgbmV3IGZlYXR1cmVzIGlzIGEgbm8gZ28gZnJvbSBteQpwb2ludCBvZiB2aWV3 LgoKSSB3b3VsZCByYXRoZXIgc2VlIGEgc2V0IG9mIGhlbHBlcnMgc28gdGhhdCBlYWNoIGZlYXR1 cmVzIGNhbiBiZSB1c2UKZWl0aGVyIGJ5IG5ldyBkcml2ZXJzIG9yIGV4aXN0aW5nIGRyaXZlcnMu IEZvciBpbnN0YW5jZSBhIG5ldyB3YXkgdG8KZXhwb3NlIG1lbW9yeSB0b3BvbG9neS4gQSBuZXcg d2F5IHRvIGV4cG9zZSBob3cgeW91IGNhbiBwaXBlIGRldmljZXMKZnJvbSBvbmUgdG8gYW5vdGhl ciAuLi4KCgpIZW5jZSBpIGRvIG5vdCBzZWUgYW55IHZhbHVlIGluIGEgd2hvbGUgbmV3IGluZnJh LXN0cnVjdHVyZSBpbiB3aGljaApkcml2ZXJzIG11c3QgYmUgcGFydCBvZiB0byBsZXZlcmFnZSBu ZXcgZmVhdHVyZXMuCgpDaGVlcnMsCkrDqXLDtG1lCl9fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fCmlvbW11IG1haWxpbmcgbGlzdAppb21tdUBsaXN0cy5saW51 eC1mb3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3RzLmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFpbG1h bi9saXN0aW5mby9pb21tdQ== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD9A0C46470 for ; Wed, 8 Aug 2018 15:18:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6C25321A13 for ; Wed, 8 Aug 2018 15:18:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6C25321A13 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727521AbeHHRit (ORCPT ); Wed, 8 Aug 2018 13:38:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:36744 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727081AbeHHRit (ORCPT ); Wed, 8 Aug 2018 13:38:49 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4108E87A77; Wed, 8 Aug 2018 15:18:40 +0000 (UTC) Received: from redhat.com (unknown [10.20.6.215]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EFCB520180F4; Wed, 8 Aug 2018 15:18:36 +0000 (UTC) Date: Wed, 8 Aug 2018 11:18:35 -0400 From: Jerome Glisse To: Kenneth Lee Cc: Kenneth Lee , "Tian, Kevin" , Alex Williamson , Herbert Xu , "kvm@vger.kernel.org" , Jonathan Corbet , Greg Kroah-Hartman , Zaibo Xu , "linux-doc@vger.kernel.org" , "Kumar, Sanjay K" , Hao Fang , "linux-kernel@vger.kernel.org" , "linuxarm@huawei.com" , "iommu@lists.linux-foundation.org" , "linux-crypto@vger.kernel.org" , Philippe Ombredanne , Thomas Gleixner , "David S . Miller" , "linux-accelerators@lists.ozlabs.org" Subject: Re: [RFC PATCH 0/7] A General Accelerator Framework, WarpDrive Message-ID: <20180808151835.GA3429@redhat.com> References: <20180801102221.5308-1-nek.in.cn@gmail.com> <20180801165644.GA3820@redhat.com> <20180802040557.GL160746@Turing-Arch-b> <20180802142243.GA3481@redhat.com> <20180803034721.GC91035@Turing-Arch-b> <20180803143944.GA4079@redhat.com> <20180806031252.GG91035@Turing-Arch-b> <20180806153257.GB6002@redhat.com> <11bace0e-dc14-5d2c-f65c-25b852f4e9ca@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <11bace0e-dc14-5d2c-f65c-25b852f4e9ca@gmail.com> User-Agent: Mutt/1.10.0 (2018-05-17) X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 15:18:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 15:18:40 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'jglisse@redhat.com' RCPT:'' Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 08, 2018 at 09:08:42AM +0800, Kenneth Lee wrote: > > > 在 2018年08月06日 星期一 11:32 下午, Jerome Glisse 写道: > > On Mon, Aug 06, 2018 at 11:12:52AM +0800, Kenneth Lee wrote: > > > On Fri, Aug 03, 2018 at 10:39:44AM -0400, Jerome Glisse wrote: > > > > On Fri, Aug 03, 2018 at 11:47:21AM +0800, Kenneth Lee wrote: > > > > > On Thu, Aug 02, 2018 at 10:22:43AM -0400, Jerome Glisse wrote: > > > > > > On Thu, Aug 02, 2018 at 12:05:57PM +0800, Kenneth Lee wrote: > > > > > > > On Thu, Aug 02, 2018 at 02:33:12AM +0000, Tian, Kevin wrote: > > > > > > > > > On Wed, Aug 01, 2018 at 06:22:14PM +0800, Kenneth Lee wrote: [...] > > > > > > > > > My more general question is do we want to grow VFIO to become > > > > > > > > > a more generic device driver API. This patchset adds a command > > > > > > > > > queue concept to it (i don't think it exist today but i have > > > > > > > > > not follow VFIO closely). > > > > > > > > > > > > > > > > The thing is, VFIO is the only place to support DMA from user land. If we don't > > > > > > > put it here, we have to create another similar facility to support the same. > > > > > > No it is not, network device, GPU, block device, ... they all do > > > > > > support DMA. The point i am trying to make here is that even in > > > > > Sorry, wait a minute, are we talking the same thing? I meant "DMA from user > > > > > land", not "DMA from kernel driver". To do that we have to manipulate the > > > > > IOMMU(Unit). I think it can only be done by default_domain or vfio domain. Or > > > > > the user space have to directly access the IOMMU. > > > > GPU do DMA in the sense that you pass to the kernel a valid > > > > virtual address (kernel driver do all the proper check) and > > > > then you can use the GPU to copy from or to that range of > > > > virtual address. Exactly how you want to use this compression > > > > engine. It does not rely on SVM but SVM going forward would > > > > still be the prefered option. > > > > > > > No, SVM is not the reason why we rely on Jean's SVM(SVA) series. We rely on > > > Jean's series because of multi-process (PASID or substream ID) support. > > > > > > But of couse, WarpDrive can still benefit from the SVM feature. > > We are getting side tracked here. PASID/ID do not require VFIO. > > > Yes, PASID itself do not require VFIO. But what if: > > 1. Support DMA from user space. > 2. The hardware makes use of standard IOMMU/SMMU for IO address translation. > 3. The IOMMU facility is shared by both kernel and user drivers. > 4. Support PASID with the current IOMMU facility I do not see how any of this means it has to be in VFIO. Other devices do just that. GPUs driver for instance share DMA engine (that copy data around) between kernel and user space. Sometime kernel use it to move things around. Evict some memory to make room for a new process is the common example. Same DMA engines is often use by userspace itself during rendering or compute (program moving things on there own). So they are already kernel driver that do all 4 of the above and are not in VFIO. > > > > > > your mechanisms the userspace must have a specific userspace > > > > > > drivers for each hardware and thus there are virtually no > > > > > > differences between having this userspace driver open a device > > > > > > file in vfio or somewhere else in the device filesystem. This is > > > > > > just a different path. > > > > > > > > > > > The basic problem WarpDrive want to solve it to avoid syscall. This is important > > > > > to accelerators. We have some data here: > > > > > https://www.slideshare.net/linaroorg/progress-and-demonstration-of-wrapdrive-a-accelerator-framework-sfo17317 > > > > > > > > > > (see page 3) > > > > > > > > > > The performance is different on using kernel and user drivers. > > > > Yes and example i point to is exactly that. You have a one time setup > > > > cost (creating command buffer binding PASID with command buffer and > > > > couple other setup steps). Then userspace no longer have to do any > > > > ioctl to schedule work on the GPU. It is all down from userspace and > > > > it use a doorbell to notify hardware when it should go look at command > > > > buffer for new thing to execute. > > > > > > > > My point stands on that. You have existing driver already doing so > > > > with no new framework and in your scheme you need a userspace driver. > > > > So i do not see the value add, using one path or the other in the > > > > userspace driver is litteraly one line to change. > > > > > > > Sorry, I'd got confuse here. I partially agree that the user driver is > > > redundance of kernel driver. (But for WarpDrive, the kernel driver is a full > > > driver include all preparation and setup stuff for the hardware, the user driver > > > is simply to send request and receive answer). Yes, it is just a choice of path. > > > But the user path is faster if the request come from use space. And to do that, > > > we need user land DMA support. Then why is it invaluable to let VFIO involved? > > Some drivers in the kernel already do exactly what you said. The user > > space emit commands without ever going into kernel by directly scheduling > > commands and ringing a doorbell. They do not need VFIO either and they > > can map userspace address into the DMA address space of the device and > > again they do not need VFIO for that. > Could you please directly point out which driver you refer to here? Thank > you. drivers/gpu/drm/amd/ Sub-directory of interest is amdkfd Because it is a big driver here is a highlevel overview of how it works (this is a simplification): - Process can allocate GPUs buffer (through ioclt) and map them into its address space (through mmap of device file at buffer object specific offset). - Process can map any valid range of virtual address space into device address space (IOMMU mapping). This must be regular memory ie not an mmap of a device file or any special file (this is the non PASID path) - Process can create a command queue and bind its process to it aka PASID, this is done through an ioctl. - Process can schedule commands onto queues it created from userspace without ioctl. For that it just write command into a ring buffer that it mapped during the command queue creation process and it rings a doorbell when commands are ready to be consume by the hardware. - Commands can reference (access) all 3 types of object above ie either full GPUs buffer, process regular memory maped as object (non PASID) and PASID memory all at the same time ie you can mix all of the above in same commands queue. - Kernel can evict, unbind any process command queues, unbind commands queue are still valid from process point of view but commands process schedules on them will not be executed until kernel re-bind the queue. - Kernel can schedule commands itself onto its dedicated command queues (kernel driver create its own command queues). - Kernel can control priorities between all the queues ie it can decides which queues should the hardware executed first next. I believe all of the above are the aspects that matters to you. The main reason i don't like creating a new driver infrastructure is that a lot of existing drivers will want to use some of the new features that are coming (memory topology, where to place process memory, pipeline devices, ...) and thus existing drivers are big (GPU drivers are the biggest of all the kernel drivers). So rewritting those existing drivers into VFIO or into any new infra- structure so that they can leverage new features is a no go from my point of view. I would rather see a set of helpers so that each features can be use either by new drivers or existing drivers. For instance a new way to expose memory topology. A new way to expose how you can pipe devices from one to another ... Hence i do not see any value in a whole new infra-structure in which drivers must be part of to leverage new features. Cheers, Jérôme