From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keith Busch Subject: Re: [PATCH v8 13/13] nvmet: Optionally use PCI P2P memory Date: Thu, 27 Sep 2018 11:12:30 -0600 Message-ID: <20180927171230.GE19589@localhost.localdomain> References: <20180927165420.5290-1-logang@deltatee.com> <20180927165420.5290-14-logang@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <20180927165420.5290-14-logang-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Logan Gunthorpe Cc: Jens Axboe , linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Steve Wise , linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, Christoph Hellwig , linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Alex Williamson , Jason Gunthorpe , =?iso-8859-1?B?Suly9G1l?= Glisse , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christian =?iso-8859-1?Q?K=F6nig?= List-Id: linux-rdma@vger.kernel.org On Thu, Sep 27, 2018 at 10:54:20AM -0600, Logan Gunthorpe wrote: > We create a configfs attribute in each nvme-fabrics target port to > enable p2p memory use. When enabled, the port will only then use the > p2p memory if a p2p memory device can be found which is behind the > same switch hierarchy as the RDMA port and all the block devices in > use. If the user enabled it and no devices are found, then the system > will silently fall back on using regular memory. > > If appropriate, that port will allocate memory for the RDMA buffers > for queues from the p2pmem device falling back to system memory should > anything fail. > > Ideally, we'd want to use an NVME CMB buffer as p2p memory. This would > save an extra PCI transfer as the NVME card could just take the data > out of it's own memory. However, at this time, only a limited number > of cards with CMB buffers seem to be available. > > Signed-off-by: Stephen Bates > Signed-off-by: Steve Wise > [hch: partial rewrite of the initial code] > Signed-off-by: Christoph Hellwig > Signed-off-by: Logan Gunthorpe I haven't the necessary hardware to try this out, but looking forward to it in the future. Looks good. Reviewed-by: Keith Busch