IOMMU and domain binding/unbinding logic

IOMMU and domain binding/unbinding logic

[Jerome Glisse]

Hi,

I was looking at IOMMUv2 and AMD APU kfd driver registration and initialization
(drivers/gpu/drm/amd/amdkfd/kfd_device.c). I am left puzzle with 2 things.

General IOMMU question, my understanding is that each device is associated with
one domain only (ignoring device that have several alias if such thing exist).
So i would assume that when some one wants to attach a new domain to a device
it would fail if the device is already attached to a domain (other than the
passthrough domain or some common dma domain). But i fail to see anywhere in
the code for such test (wether in drivers/iommu/iommu.c or in any iommu driver
code). On contrary if i look at driver code the first thing it does in the
attach callback is unbind the currently bound domain.

So if there are some active DMA mapping in current bound domain they are lost
and even worst once the driver call the dma_unmap api it will possibly free
new unrelated mapping. Which sounds really buggy.

Of course there are very few user of the iommu api, mostly for virtualization
and AMD kfd driver. In virtualization case i think the passthrough implicitly
force that no host driver are bound to the device and hence no domain. So no
issue there AFAICT.

In the kfd driver things are different, kfd driver initialization happens
after GPU driver initialization (drivers/gpu/drm/radeon/radeon_kms.c look
for radeon_kfd_device_probe() radeon_kfd_device_init()). At this point the
GPU driver already allocated a bunch of memory and mapped it through the
pci_map_page() (which endup in dma_map_page()). So assuming we are not in
passthrough there are live and working/in use mapping.

Now the issue is that kfd call amd_iommu_init_device() which allocates a
new iommu domain and replace currently bound domain with it. This new
domain is however different from previously (default domain) in that it
does not have PD_DMA_OPS_MASK domain flags set and as such if i am reading
code properly will lead to all dma_map_page() to fail (or any other dma-op
for that matter).

So how comes it still looks like it is working properly ? My best guest
it is because it is by default in passthrough, but i haven't checked that
theory.


I am sure i am missing something very obvious somewhere in the code but
after spending couple hour staring at it, it is still eluding me.

Cheers,
Jérôme

Re: IOMMU and domain binding/unbinding logic

[Joerg Roedel]

Hi Jerome,

On Thu, Mar 26, 2015 at 12:50:47PM -0400, Jerome Glisse wrote:
> So how comes it still looks like it is working properly ? My best guest
> it is because it is by default in passthrough, but i haven't checked that
> theory.

Your guess is correct, devices that might potentially use IOMMUv2
functionality are identity mapped by default. The reason is that IOMMU
hardware implements two dimensional paging, where the addresses in the
guest page-tables (the ones translating PASID requests) are translated
by the host page-tables. So to be able to use CPU page-tables (mm->pgd)
for the IOMMU, the host page-tables need to be identity mapped.

The other reason is, as you observed already, that host-only dma
mappings need to work in parallel.


	Joerg

Re: IOMMU and domain binding/unbinding logic

[Jerome Glisse]

On Thu, Mar 26, 2015 at 06:09:35PM +0100, Joerg Roedel wrote:
> Hi Jerome,
> 
> On Thu, Mar 26, 2015 at 12:50:47PM -0400, Jerome Glisse wrote:
> > So how comes it still looks like it is working properly ? My best guest
> > it is because it is by default in passthrough, but i haven't checked that
> > theory.
> 
> Your guess is correct, devices that might potentially use IOMMUv2
> functionality are identity mapped by default. The reason is that IOMMU
> hardware implements two dimensional paging, where the addresses in the
> guest page-tables (the ones translating PASID requests) are translated
> by the host page-tables. So to be able to use CPU page-tables (mm->pgd)
> for the IOMMU, the host page-tables need to be identity mapped.

Well this still doesn't explain how things works. So assume that you
boot computer and that the IOMMU bind to the GPU is not passthrough and
perform real remapping of bus address to physical address. You have :

  radeon_gpu_driver_init() {
    general_buffer = dma_map_page(gpu_dev, page);
    radeon_kfd_init() {
      amd_iommu_init_device(gpu_dev) {
        // bind new DOMAIN and detach old one, all previous dma address
        // are now invalid and will point to physical address directly,
        // if dma address where identity mapped at gpu_driver init time
        // than things are fine.
        //
        // Otherwise hw is accessing some random memory other then what
        // it believe general_buffer is at ie the first dma_map_page()
        // returned dma_addr != page_to_phys(page)
      }
    }

More over after amd_iommu_init_device(gpu_dev), any dma_map_page() should
fails as it will return ERR_PTR(-EINVAL) because get_domain(dev) (from
drivers/iommu/amd_iommu.c) will return ERR_PTR(-EBUSY) as the new domain
is not DMA capable.

Of course if dev.archdata.dma_ops is pointing to nommu_dma_ops then every
things is fine, but here i am wondering for the case where the ops are
actualy amd_iommu_dma_ops. In the first place ie at iommu initialization
or when hotpluging a new device (as in the hotpluging case it always set
dev.archdata.dma_ops to amd_iommu_dma_ops AFAICT.

Cheers,
Jérôme

Re: IOMMU and domain binding/unbinding logic

[Joerg Roedel]

Hi Jerome,

On Thu, Mar 26, 2015 at 01:58:58PM -0400, Jerome Glisse wrote:
> Of course if dev.archdata.dma_ops is pointing to nommu_dma_ops then every
> things is fine, but here i am wondering for the case where the ops are
> actualy amd_iommu_dma_ops. In the first place ie at iommu initialization
> or when hotpluging a new device (as in the hotpluging case it always set
> dev.archdata.dma_ops to amd_iommu_dma_ops AFAICT.

Yes, every device that is IOMMUv2 capable (ATS + PRI + PASID capability
present) is put into the passthrough domain at boot time. The dma_ops
point to nommu_dma_ops and after the device is used for IOMMUv2 mappings
the old DMA-API mappings remain valid.

For details you can have a look into the AMD IOMMU driver and how the
return value of pci_iommuv2_capable() is propageted.


	Joerg