From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from quartz.orcorp.ca ([184.70.90.242]:35921 "EHLO quartz.orcorp.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1763438AbdAETBV (ORCPT ); Thu, 5 Jan 2017 14:01:21 -0500 Date: Thu, 5 Jan 2017 12:01:13 -0700 From: Jason Gunthorpe To: Jerome Glisse Cc: "Deucher, Alexander" , "'linux-kernel@vger.kernel.org'" , "'linux-rdma@vger.kernel.org'" , "'linux-nvdimm@lists.01.org'" , "'Linux-media@vger.kernel.org'" , "'dri-devel@lists.freedesktop.org'" , "'linux-pci@vger.kernel.org'" , "Kuehling, Felix" , "Sagalovitch, Serguei" , "Blinzer, Paul" , "Koenig, Christian" , "Suthikulpanit, Suravee" , "Sander, Ben" , hch@infradead.org, david1.zhou@amd.com, qiang.yu@amd.com Subject: Re: Enabling peer to peer device transactions for PCIe devices Message-ID: <20170105190113.GA12587@obsidianresearch.com> References: <20170105183927.GA5324@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <20170105183927.GA5324@gmail.com> Sender: linux-pci-owner@vger.kernel.org List-ID: On Thu, Jan 05, 2017 at 01:39:29PM -0500, Jerome Glisse wrote: > 1) peer-to-peer because of userspace specific API like NVidia GPU > direct (AMD is pushing its own similar API i just can't remember > marketing name). This does not happen through a vma, this happens > through specific device driver call going through device specific > ioctl on both side (GPU and RDMA). So both kernel driver are aware > of each others. Today you can only do user-initiated RDMA operations in conjection with a VMA. We'd need a really big and strong reason to create an entirely new non-VMA based memory handle scheme for RDMA. So my inclination is to just completely push back on this idea. You need a VMA to do RMA. GPUs need to create VMAs for the memory they want to RDMA from, even if the VMA handle just causes SIGBUS for any CPU access. Jason