Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

From: "Christian König" <christian.koenig@amd.com>
To: Oded Gabbay <oded.gabbay@gmail.com>
Cc: linux-rdma <linux-rdma@vger.kernel.org>,
	"Christian König" <ckoenig.leichtzumerken@gmail.com>,
	sleybo@amazon.com, "Gal Pressman" <galpress@amazon.com>,
	dri-devel <dri-devel@lists.freedesktop.org>,
	"Christoph Hellwig" <hch@lst.de>,
	"moderated list:DMA BUFFER SHARING FRAMEWORK"
	<linaro-mm-sig@lists.linaro.org>,
	"Jason Gunthorpe" <jgg@ziepe.ca>,
	"Doug Ledford" <dledford@redhat.com>,
	"Tomer Tayar" <ttayar@habana.ai>,
	"amd-gfx list" <amd-gfx@lists.freedesktop.org>,
	"Greg KH" <gregkh@linuxfoundation.org>,
	"Alex Deucher" <alexander.deucher@amd.com>,
	"Leon Romanovsky" <leonro@nvidia.com>,
	"Oded Gabbay" <ogabbay@kernel.org>,
	"Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
	"open list:DMA BUFFER SHARING FRAMEWORK"
	<linux-media@vger.kernel.org>
Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF
Date: Tue, 22 Jun 2021 17:49:41 +0200	[thread overview]
Message-ID: <f59ca863-a1c0-0793-99c8-60674da0bfb6@amd.com> (raw)
In-Reply-To: <CAFCwf13w74qFDchEsdCAtpOvnYkzvcsU0RAdBLNGJq-e5srfHA@mail.gmail.com>

Am 22.06.21 um 17:40 schrieb Oded Gabbay:
> On Tue, Jun 22, 2021 at 6:31 PM Christian König
> <christian.koenig@amd.com> wrote:
>>
>>
>> Am 22.06.21 um 17:28 schrieb Jason Gunthorpe:
>>> On Tue, Jun 22, 2021 at 05:24:08PM +0200, Christian König wrote:
>>>
>>>>>> I will take two GAUDI devices and use one as an exporter and one as an
>>>>>> importer. I want to see that the solution works end-to-end, with real
>>>>>> device DMA from importer to exporter.
>>>>> I can tell you it doesn't. Stuffing physical addresses directly into
>>>>> the sg list doesn't involve any of the IOMMU code so any configuration
>>>>> that requires IOMMU page table setup will not work.
>>>> Sure it does. See amdgpu_vram_mgr_alloc_sgt:
>>>>
>>>>           amdgpu_res_first(res, offset, length, &cursor);
>>>            ^^^^^^^^^^
>>>
>>> I'm not talking about the AMD driver, I'm talking about this patch.
>>>
>>> +             bar_address = hdev->dram_pci_bar_start +
>>> +                             (pages[cur_page] - prop->dram_base_address);
>>> +             sg_dma_address(sg) = bar_address;
>> Yeah, that is indeed not working.
>>
>> Oded you need to use dma_map_resource() for this.
>>
>> Christian.
> Yes, of course.
> But will it be enough ?
> Jason said that supporting IOMMU isn't nice when we don't have struct pages.
> I fail to understand the connection, I need to dig into this.

Question is what you want to do with this?

A struct page is always needed if you want to do stuff like HMM with it, 
if you only want P2P between device I actually recommend to avoid it.

Christian.

>
> Oded
>
>>
>>
>>> Jason

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx