Re: IOMMU and domain binding/unbinding logic

[Jerome Glisse <j.glisse-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

On Thu, Mar 26, 2015 at 06:09:35PM +0100, Joerg Roedel wrote:
> Hi Jerome,
> 
> On Thu, Mar 26, 2015 at 12:50:47PM -0400, Jerome Glisse wrote:
> > So how comes it still looks like it is working properly ? My best guest
> > it is because it is by default in passthrough, but i haven't checked that
> > theory.
> 
> Your guess is correct, devices that might potentially use IOMMUv2
> functionality are identity mapped by default. The reason is that IOMMU
> hardware implements two dimensional paging, where the addresses in the
> guest page-tables (the ones translating PASID requests) are translated
> by the host page-tables. So to be able to use CPU page-tables (mm->pgd)
> for the IOMMU, the host page-tables need to be identity mapped.

Well this still doesn't explain how things works. So assume that you
boot computer and that the IOMMU bind to the GPU is not passthrough and
perform real remapping of bus address to physical address. You have :

  radeon_gpu_driver_init() {
    general_buffer = dma_map_page(gpu_dev, page);
    radeon_kfd_init() {
      amd_iommu_init_device(gpu_dev) {
        // bind new DOMAIN and detach old one, all previous dma address
        // are now invalid and will point to physical address directly,
        // if dma address where identity mapped at gpu_driver init time
        // than things are fine.
        //
        // Otherwise hw is accessing some random memory other then what
        // it believe general_buffer is at ie the first dma_map_page()
        // returned dma_addr != page_to_phys(page)
      }
    }

More over after amd_iommu_init_device(gpu_dev), any dma_map_page() should
fails as it will return ERR_PTR(-EINVAL) because get_domain(dev) (from
drivers/iommu/amd_iommu.c) will return ERR_PTR(-EBUSY) as the new domain
is not DMA capable.

Of course if dev.archdata.dma_ops is pointing to nommu_dma_ops then every
things is fine, but here i am wondering for the case where the ops are
actualy amd_iommu_dma_ops. In the first place ie at iommu initialization
or when hotpluging a new device (as in the hotpluging case it always set
dev.archdata.dma_ops to amd_iommu_dma_ops AFAICT.

Cheers,
Jérôme