From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ozlabs.org (ozlabs.org [IPv6:2401:3900:2:1::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3vjCc222pyzDqpy for ; Tue, 14 Mar 2017 22:45:54 +1100 (AEDT) In-Reply-To: <20170221024131.47753-1-aik@ozlabs.ru> To: Alexey Kardashevskiy , linuxppc-dev@lists.ozlabs.org From: Michael Ellerman Cc: Alexey Kardashevskiy , Gavin Shan , David Gibson Subject: Re: [kernel] powerpc/powernv/ioda2: Update iommu table base on ownership change Message-Id: <3vjCc20kKWz9s3w@ozlabs.org> Date: Tue, 14 Mar 2017 22:45:54 +1100 (AEDT) List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Tue, 2017-02-21 at 02:41:31 UTC, Alexey Kardashevskiy wrote: > On POWERNV platform, in order to do DMA via IOMMU (i.e. 32bit DMA in > our case), a device needs an iommu_table pointer set via > set_iommu_table_base(). > > The codeflow is: > - pnv_pci_ioda2_setup_dma_pe() > - pnv_pci_ioda2_setup_default_config() > - pnv_ioda_setup_bus_dma() [1] > > pnv_pci_ioda2_setup_dma_pe() creates IOMMU groups, > pnv_pci_ioda2_setup_default_config() does default DMA setup, > pnv_ioda_setup_bus_dma() takes a bus PE (on IODA2, all physical function > PEs as bus PEs except NPU), walks through all underlying buses and > devices, adds all devices to an IOMMU group and sets iommu_table. > > On IODA2, when VFIO is used, it takes ownership over a PE which means it > removes all tables and creates new ones (with a possibility of sharing > them among PEs). So when the ownership is returned from VFIO to > the kernel, the iommu_table pointer written to a device at [1] is > stale and needs an update. > > This adds an "add_to_group" parameter to pnv_ioda_setup_bus_dma() > (in fact re-adds as it used to be there a while ago for different > reasons) to tell the helper if a device needs to be added to > an IOMMU group with an iommu_table update or just the latter. > > This calls pnv_ioda_setup_bus_dma(..., false) from > pnv_ioda2_release_ownership() so when the ownership is restored, > 32bit DMA can work again for a device. This does the same thing > on obtaining ownership as the iommu_table point is stale at this point > anyway and it is safer to have NULL there. > > We did not hit this earlier as all tested devices in recent years were > only using 64bit DMA; the rare exception for this is MPT3 SAS adapter > which uses both 32bit and 64bit DMA access and it has not been tested > with VFIO much. > > Cc: Gavin Shan > Signed-off-by: Alexey Kardashevskiy > Acked-by: Gavin Shan > Reviewed-by: David Gibson Applied to powerpc fixes, thanks. https://git.kernel.org/powerpc/c/db08e1d53034a54fe177ced70476fd cheers