From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Gunthorpe Subject: Re: [PATCH v3 01/11] PCI/P2PDMA: Support peer-to-peer memory Date: Mon, 26 Mar 2018 10:41:29 -0600 Message-ID: <20180326164129.GF15530@mellanox.com> References: <20180312193525.2855-1-logang@deltatee.com> <20180312193525.2855-2-logang@deltatee.com> <59fd2f5d-177f-334a-a9c4-0f8a6ec7c303@codeaurora.org> <24d8e5c2-065d-8bde-3f5d-7f158be9c578@deltatee.com> <20180326121138.00005e30@huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <20180326121138.00005e30-hv44wF8Li93QT0dZR+AlfA@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Jonathan Cameron Cc: Jens Axboe , linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Alex Williamson , linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, Sinan Kaya , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Keith Busch , Eric Wehage , Christoph Hellwig List-Id: linux-rdma@vger.kernel.org On Mon, Mar 26, 2018 at 12:11:38PM +0100, Jonathan Cameron wrote: > On Tue, 13 Mar 2018 10:43:55 -0600 > Logan Gunthorpe wrote: > > > On 12/03/18 09:28 PM, Sinan Kaya wrote: > > > On 3/12/2018 3:35 PM, Logan Gunthorpe wrote: > > > Regarding the switch business, It is amazing how much trouble you went into > > > limit this functionality into very specific hardware. > > > > > > I thought that we reached to an agreement that code would not impose > > > any limits on what user wants. > > > > > > What happened to all the emails we exchanged? > > > > It turns out that root ports that support P2P are far less common than > > anyone thought. So it will likely have to be a white list. > > This came as a bit of a surprise to our PCIe architect. I don't think it is a hardware problem. I know Mellanox and Nvidia have been doing p2p on Intel root complexes for something like 5-6 years now.. I don't have the details, but it does seem to work. I have heard some chips give poor performance.. Also AMD GPU SLI uses P2P these days, so this isn't exactly a niche feature in Intel/AMD land. I think the main issue here is that there is some BIOS involvement to set things up properly. Eg in GPU land motherboards certify for 'crossfire' support. > His follow up was whether it was worth raising an ECR for the PCIe spec > to add a capability bit to allow this to be discovered. This might > long term avoid the need to maintain the white list for new devices. If it is primarily a BIOS issue then it should be an ACPI thing, right? Jason