From mboxrd@z Thu Jan 1 00:00:00 1970 From: Knut Omang Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Date: Tue, 25 Apr 2017 08:30:03 +0200 Message-ID: <1493101803.3171.246.camel@oracle.com> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1491974532.7236.43.camel@kernel.crashing.org> <5ac22496-56ec-025d-f153-140001d2a7f9@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <1492169849.25766.3.camel@kernel.crashing.org> <630c1c63-ff17-1116-e069-2b8f93e50fa2@deltatee.com> <20170414190452.GA15679@bhelgaas-glaptop.roam.corp.google.com> <1492207643.25766.18.camel@kernel.crashing.org> <1492311719.25766.37.camel@kernel.crashing.org> <5e43818e-8c6b-8be8-23ff-b798633d2a73@deltatee.com> <1492381907.25766.49.camel@kernel.crashing.org> <1493019397.3171.118.camel@oracle.com> <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <9b6c0830-a728-c7ca-e6c6-2135f3f760ed-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Logan Gunthorpe , Benjamin Herrenschmidt , Dan Williams Cc: Jens Axboe , Keith Busch , "James E.J. Bottomley" , "Martin K. Petersen" , linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Steve Wise , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, Jason Gunthorpe , Jerome Glisse , Bjorn Helgaas , linux-scsi , linux-nvdimm , Max Gurtovoy , Christoph Hellwig List-Id: linux-nvdimm@lists.01.org T24gTW9uLCAyMDE3LTA0LTI0IGF0IDEwOjE0IC0wNjAwLCBMb2dhbiBHdW50aG9ycGUgd3JvdGU6 Cj4gCj4gT24gMjQvMDQvMTcgMDE6MzYgQU0sIEtudXQgT21hbmcgd3JvdGU6Cj4gPiBNeSBmaXJz dCByZWZsZXggd2hlbiByZWFkaW5nIHRoaXMgdGhyZWFkIHdhcyB0byB0aGluayB0aGF0IHRoaXMg d2hvbGUgZG9tYWluCj4gPiBsZW5kcyBpdCBzZWxmIGV4Y2VsbGVudGx5IHRvIHRlc3Rpbmcgdmlh IFFlbXUuIENvdWxkIGl0IGJlIHRoYXQgZG9pbmcgdGhpcyBpbsKgCj4gPiB0aGUgb3Bwb3NpdGUg ZGlyZWN0aW9uIG1pZ2h0IGJlIGEgc2FmZXIgYXBwcm9hY2ggaW4gdGhlIGxvbmcgcnVuIGV2ZW4g dGhvdWdowqAKPiA+IChzaWduaWZpY2FudCkgbW9yZSB3b3JrIHVwLWZyb250Pwo+IAo+IFRoYXQn cyBhbiBpbnRlcmVzdGluZyBpZGVhLiBXZSBkaWQgZG8gc29tZSB2ZXJ5IGxpbWl0ZWQgdGVzdGlu ZyBvbiBxZW11Cj4gd2l0aCBvbmUgaXRlcmF0aW9uIG9mIG91ciB3b3JrLiBIb3dldmVyLCBpdCdz IGRpZmZpY3VsdCBiZWNhdXNlIHRoZXJlIGlzCj4gbm8gc3VwcG9ydCBmb3IgYW55IFJETUEgZGV2 aWNlcyB3aGljaCBhcmUgYSBwYXJ0IG9mIG91ciBwcmltYXJ5IHVzZQo+IGNhc2UuIAoKWWVzLCB0 aGF0J3Mgd2h5IEkgdXNlZCAnc2lnbmlmaWNhbnQnLiBPbmUgZ29vZCB0aGluZyBpcyB0aGF0IGdp dmVuIHJlc291cmNlc8KgCml0IGNhbiBlYXNpbHkgYmUgZG9uZSBpbiBwYXJhbGxlbCB3aXRoIG90 aGVyIGRldmVsb3BtZW50LCBhbmQgd2lsbCBnaXZlIGFkZGl0aW9uYWwKaW5zaWdodCBvZiBzb21l IGZvcm0uCgo+IEkgYWxzbyBpbWFnaW5lIGl0IHdvdWxkIGJlIHF1aXRlIGRpZmZpY3VsdCB0byBk ZXZlbG9wIHRob3NlIG1vZGVscwo+IGdpdmVuIHRoZSBhcnJheSBvZiBoYXJkd2FyZSB0aGF0IG5l ZWRzIHRvIGJlIHN1cHBvcnRlZCBhbmQgdGhlIGRlZXAKPiBmdW5jdGlvbmFsIGtub3dsZWRnZSBy ZXF1aXJlZCB0byBmaWd1cmUgb3V0IGFwcHJvcHJpYXRlIHJlc3RyaWN0aW9ucy4KCkZyb20gbXkg bmFpdmUgcGVyc3BlY3RpdmUgaXQgc2VlbXMgaXQgbmVlZCBub3QgZXZlbiBiZSBhIGZ1bGwgbW9k ZWwgdG8gZ2V0IHNvbWUgYmVuZWZpdHMsCmp1c3QgbG93IGxldmVsIGZ1bmN0aW9uYWxpdHkgdGVz dHMgd2l0aCBzb21lIGluc3RhbmNlcyBvZiBhCmRldmljZSB0aGF0IG9mZmVycyBzb21lIE1NSU8g c3BhY2UgJ3BsYXlncm91bmQnLgoKT3IgbWF5YmUgeW91IGNhbiBsZXZlcmFnZSBzb21lIG9mIHRo ZSBhbHJlYWR5IGltcGxlbWVudGVkIGVtdWxhdGVkIGRldmljZXMgaW4gUWVtdT8KCktudXQKCj4g Cj4gTG9nYW4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K TGludXgtbnZkaW1tIG1haWxpbmcgbGlzdApMaW51eC1udmRpbW1AbGlzdHMuMDEub3JnCmh0dHBz Oi8vbGlzdHMuMDEub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtbnZkaW1tCg== From mboxrd@z Thu Jan 1 00:00:00 1970 From: knut.omang@oracle.com (Knut Omang) Date: Tue, 25 Apr 2017 08:30:03 +0200 Subject: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory In-Reply-To: <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1491974532.7236.43.camel@kernel.crashing.org> <5ac22496-56ec-025d-f153-140001d2a7f9@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <1492169849.25766.3.camel@kernel.crashing.org> <630c1c63-ff17-1116-e069-2b8f93e50fa2@deltatee.com> <20170414190452.GA15679@bhelgaas-glaptop.roam.corp.google.com> <1492207643.25766.18.camel@kernel.crashing.org> <1492311719.25766.37.camel@kernel.crashing.org> <5e43818e-8c6b-8be8-23ff-b798633d2a73@deltatee.com> <1492381907.25766.49.camel@kernel.crashing.org> <1493019397.3171.118.camel@oracle.com> <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> Message-ID: <1493101803.3171.246.camel@oracle.com> On Mon, 2017-04-24@10:14 -0600, Logan Gunthorpe wrote: > > On 24/04/17 01:36 AM, Knut Omang wrote: > > My first reflex when reading this thread was to think that this whole domain > > lends it self excellently to testing via Qemu. Could it be that doing this in? > > the opposite direction might be a safer approach in the long run even though? > > (significant) more work up-front? > > That's an interesting idea. We did do some very limited testing on qemu > with one iteration of our work. However, it's difficult because there is > no support for any RDMA devices which are a part of our primary use > case. Yes, that's why I used 'significant'. One good thing is that given resources? it can easily be done in parallel with other development, and will give additional insight of some form. > I also imagine it would be quite difficult to develop those models > given the array of hardware that needs to be supported and the deep > functional knowledge required to figure out appropriate restrictions. >>From my naive perspective it seems it need not even be a full model to get some benefits, just low level functionality tests with some instances of a device that offers some MMIO space 'playground'. Or maybe you can leverage some of the already implemented emulated devices in Qemu? Knut > > Logan From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: Message-ID: <1493101803.3171.246.camel@oracle.com> Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory From: Knut Omang To: Logan Gunthorpe , Benjamin Herrenschmidt , Dan Williams Cc: Bjorn Helgaas , Jason Gunthorpe , Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Keith Busch , linux-pci@vger.kernel.org, linux-scsi , linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm , "linux-kernel@vger.kernel.org" , Jerome Glisse Date: Tue, 25 Apr 2017 08:30:03 +0200 In-Reply-To: <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1491974532.7236.43.camel@kernel.crashing.org> <5ac22496-56ec-025d-f153-140001d2a7f9@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <1492169849.25766.3.camel@kernel.crashing.org> <630c1c63-ff17-1116-e069-2b8f93e50fa2@deltatee.com> <20170414190452.GA15679@bhelgaas-glaptop.roam.corp.google.com> <1492207643.25766.18.camel@kernel.crashing.org> <1492311719.25766.37.camel@kernel.crashing.org> <5e43818e-8c6b-8be8-23ff-b798633d2a73@deltatee.com> <1492381907.25766.49.camel@kernel.crashing.org> <1493019397.3171.118.camel@oracle.com> <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 List-ID: On Mon, 2017-04-24 at 10:14 -0600, Logan Gunthorpe wrote: >=20 > On 24/04/17 01:36 AM, Knut Omang wrote: > > My first reflex when reading this thread was to think that this whole d= omain > > lends it self excellently to testing via Qemu. Could it be that doing t= his in=C2=A0 > > the opposite direction might be a safer approach in the long run even t= hough=C2=A0 > > (significant) more work up-front? >=20 > That's an interesting idea. We did do some very limited testing on qemu > with one iteration of our work. However, it's difficult because there is > no support for any RDMA devices which are a part of our primary use > case.=20 Yes, that's why I used 'significant'. One good thing is that given resource= s=C2=A0 it can easily be done in parallel with other development, and will give add= itional insight of some form. > I also imagine it would be quite difficult to develop those models > given the array of hardware that needs to be supported and the deep > functional knowledge required to figure out appropriate restrictions. >>From my naive perspective it seems it need not even be a full model to get = some benefits, just low level functionality tests with some instances of a device that offers some MMIO space 'playground'. Or maybe you can leverage some of the already implemented emulated devices = in Qemu? Knut >=20 > Logan From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S980805AbdDYGbj convert rfc822-to-8bit (ORCPT ); Tue, 25 Apr 2017 02:31:39 -0400 Received: from userp1040.oracle.com ([156.151.31.81]:47216 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S941644AbdDYGbb (ORCPT ); Tue, 25 Apr 2017 02:31:31 -0400 Message-ID: <1493101803.3171.246.camel@oracle.com> Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory From: Knut Omang To: Logan Gunthorpe , Benjamin Herrenschmidt , Dan Williams Cc: Bjorn Helgaas , Jason Gunthorpe , Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Keith Busch , linux-pci@vger.kernel.org, linux-scsi , linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm , "linux-kernel@vger.kernel.org" , Jerome Glisse Date: Tue, 25 Apr 2017 08:30:03 +0200 In-Reply-To: <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1491974532.7236.43.camel@kernel.crashing.org> <5ac22496-56ec-025d-f153-140001d2a7f9@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <1492169849.25766.3.camel@kernel.crashing.org> <630c1c63-ff17-1116-e069-2b8f93e50fa2@deltatee.com> <20170414190452.GA15679@bhelgaas-glaptop.roam.corp.google.com> <1492207643.25766.18.camel@kernel.crashing.org> <1492311719.25766.37.camel@kernel.crashing.org> <5e43818e-8c6b-8be8-23ff-b798633d2a73@deltatee.com> <1492381907.25766.49.camel@kernel.crashing.org> <1493019397.3171.118.camel@oracle.com> <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.20.5 (3.20.5-1.fc24) Mime-Version: 1.0 Content-Transfer-Encoding: 8BIT X-Source-IP: userv0022.oracle.com [156.151.31.74] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 2017-04-24 at 10:14 -0600, Logan Gunthorpe wrote: > > On 24/04/17 01:36 AM, Knut Omang wrote: > > My first reflex when reading this thread was to think that this whole domain > > lends it self excellently to testing via Qemu. Could it be that doing this in  > > the opposite direction might be a safer approach in the long run even though  > > (significant) more work up-front? > > That's an interesting idea. We did do some very limited testing on qemu > with one iteration of our work. However, it's difficult because there is > no support for any RDMA devices which are a part of our primary use > case. Yes, that's why I used 'significant'. One good thing is that given resources  it can easily be done in parallel with other development, and will give additional insight of some form. > I also imagine it would be quite difficult to develop those models > given the array of hardware that needs to be supported and the deep > functional knowledge required to figure out appropriate restrictions. >>From my naive perspective it seems it need not even be a full model to get some benefits, just low level functionality tests with some instances of a device that offers some MMIO space 'playground'. Or maybe you can leverage some of the already implemented emulated devices in Qemu? Knut > > Logan