* Re: IOMMU Query
2011-03-23 9:08 IOMMU Query Abhilash Kesavan
@ 2011-03-23 9:21 ` Roedel, Joerg
2011-03-23 22:18 ` Stepan Moskovchenko
1 sibling, 0 replies; 4+ messages in thread
From: Roedel, Joerg @ 2011-03-23 9:21 UTC (permalink / raw)
To: Abhilash Kesavan
Cc: stepanm@codeaurora.org, linux-arm-msm@vger.kernel.org,
kvm@vger.kernel.org, iommu@lists.linux-foundation.org,
pullip.cho@samsung.com
On Wed, Mar 23, 2011 at 05:08:45AM -0400, Abhilash Kesavan wrote:
> We are investigating if the generic IOMMU API (include/linux/iommu.h)
> can be used for the IOMMU on our arm-based boards.
Great :-)
> I noticed that msm's IOMMU code (arch/arm/mach-msm/iommu.c) uses the
> genric IOMMU API and have a few queries regarding the supported
> features. Following are some features/requirements of our system:
>
> 1) Multiple IOMMU's - consistency between parallel IOMMUs must be maintained
This is hidden behind the API. The IOMMU driver is responsible for
managing multiple IOMMUs. On x86 there is a unique device<->IOMMU
relationship which is used internally in the drivers. Users of the
IOMMU-API can ignore the fact that there are multiple IOMMUs in the
system.
> 2) Supports varying page sizes - 4K, 64K, 1M, 16M i.e. a sg list of
> contiguous physical memory blocks be made available. Thus, giving us
> mappings from the conservative 4K to large1M reducing TLB misses.
This is supported but also hidden in the IOMMU driver implementation.
the AMD-IOMMU driver implements multiple page-sizes already using this
API. The API user has a function which looks as follows:
int iommu_map(struct iommu_domain *domain,
unsigned long iova,
phys_addr_t paddr,
int gfp_order,
int prot);
this function needs to map the pysically contiguous memory starting at
paddr at the io-virtual address iova. The size of the mapping is
determinded by the gfp_order parameter. The size of the mapping is
basically (2**(gfp_order))*PAGE_SIZE. Note that PAGE_SIZE is the cpu
page-size.
The IOMMU driver has enough information from that function to map the
area with the best-fit page-sizes.
> 3) Multiple VA allocations can be associated with the same PA space.
I don't really understand what you mean here. Can you elabortate?
> Who manages the virtual address space of IO device that has IOMMU? I
> think the generic iommu.h API cannot handle virtual address space and
> memory allocation. Is it just a IOMMU handler? Should we have another
> virtual memory manager for IOMMU?
The layout of the virtual address space for the IO device needs to be
managed by the particular device driver. The IOMMU-API only provides
functions for mapping/unmapping memory.
Regards,
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: IOMMU Query
2011-03-23 9:08 IOMMU Query Abhilash Kesavan
2011-03-23 9:21 ` Roedel, Joerg
@ 2011-03-23 22:18 ` Stepan Moskovchenko
2011-03-24 6:10 ` Joerg Roedel
1 sibling, 1 reply; 4+ messages in thread
From: Stepan Moskovchenko @ 2011-03-23 22:18 UTC (permalink / raw)
To: Abhilash Kesavan; +Cc: joerg.roedel, linux-arm-msm, kvm, iommu, pullip.cho
On 3/23/2011 2:08 AM, Abhilash Kesavan wrote:
> Hi all,
>
Hello
> We are investigating if the generic IOMMU API (include/linux/iommu.h)
> can be used
> for the IOMMU on our arm-based boards. I noticed that msm's IOMMU code
> (arch/arm/mach-msm/iommu.c) uses the genric IOMMU API and have a few queries
> regarding the supported features. Following are some
> features/requirements of our
> system:
>
> 1) Multiple IOMMU's - consistency between parallel IOMMUs must be maintained
> 2) Supports varying page sizes - 4K, 64K, 1M, 16M i.e. a sg list of
> contiguous physical
> memory blocks be made available. Thus, giving us mappings from the
> conservative 4K
> to large1M reducing TLB misses.
> 3) Multiple VA allocations can be associated with the same PA space.
>
Consistency between multiple IOMMUs can be maintained. Map and unmap
operations happen on a 'domain' structure (which is intended to abstract
a page table), and a domain can be attached to one or more IOMMU
devices, assuming that all your IOMMUs use the same page table format.
If not, you will need to put more "higher-level" information into your
domain and then translate it into the HW-specific format for each IOMMU
that you have. But it sounds pretty unlikely that you would have
multiple IOMMUs in the same system with different page table formats.
Thus, I think your consistency requirement should be supported, as long
as your map/unmap function knows to iterate over all the IOMMUs attached
to that domain and do any hardware-specific maintenance that is needed
in addition to just updating the page table (such as invalidating
IOTLBs, etc). The MSM driver currently does this.
Varying mapping sizes can be supported, but you will likely need an
additional layer to organize arbitrarily-sized mappings into something
that the IOMMU layer can easily digest. The IOMMU API takes a page order
as an argument, and you can mix and match these (as the MSM driver does)
but the chunk breakdown would have to come from the layer above.
Multiple VA allocations for the same PA should also be possible, and
should really not be affected by the IOMMU API (nothing really prevents
you from passing the same PA to multiple calls to iommu_map()). Just
keep in mind that your multiple VA allocations need to be mapped with
consistent attributes as per ARM spec, at least on the CPU side (not
sure if your IOMMU HW also has this requirement).
> Who manages the virtual address space of IO device that has IOMMU? I think the
> generic iommu.h API cannot handle virtual address space and memory allocation.
> Is it just a IOMMU handler? Should we have another virtual memory
> manager for IOMMU?
>
The IOMMU layer itself does not handle virtual space management. Its
sole responsibility is to set up the mappings. You would need something
else to do the management for you. There have been some discussions
about it in the past...
> Can you please tell me if the current msm sysmmu code supports some/all of the
> above features, so that we can use it as reference. Also, any pointers on areas
> which would need work would be appreciated.
>
The MSM driver supports most of the features. The only thing that is
stubbed out right now is domain_has_cap. I believe the only place this
is currently used is to determine cache coherency capabilities. While
the MSM IOMMU does support cache coherent access, this needs to be
configured through the mapping attributes to be "fully correct". I hope
to have the has_cap stuff properly supported at a later time.
> Lastly, is there any driver making use of msm's IOMMU code - mainline
> or otherwise ?
> I wanted to check the usage of the same.
>
I don't think there is any usage of the MSM driver in mainline at this
time. You could look around for usage examples of the API in other
places (like KVM) but I doubt that will have the usage that you are
looking for. There are currently efforts to use it in MSM code, but that
code has not been sent out yet.
I hope this helps.
Steve
> Regards,
> Abhilash
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum
^ permalink raw reply [flat|nested] 4+ messages in thread