IOMMU Query

IOMMU Query

[Abhilash Kesavan]

Hi all,

We are investigating if the generic IOMMU API (include/linux/iommu.h)
can be used
for the IOMMU on our arm-based boards. I noticed that msm's IOMMU code
(arch/arm/mach-msm/iommu.c) uses the genric IOMMU API and have a few queries
regarding the supported features. Following are some
features/requirements of our
system:

1) Multiple IOMMU's  - consistency between parallel IOMMUs must be maintained
2) Supports varying page sizes - 4K, 64K, 1M, 16M i.e. a sg list of
contiguous physical
memory blocks be made available. Thus, giving us mappings from the
conservative 4K
to large1M reducing TLB misses.
3) Multiple VA allocations can be associated with the same PA space.

Who manages the virtual address space of IO device that has IOMMU? I think the
generic iommu.h API cannot handle virtual address space and memory allocation.
Is it just a IOMMU handler? Should we have another virtual memory
manager for IOMMU?

Can you please tell me if the current msm sysmmu code supports some/all of the
above features, so that we can use it as reference. Also, any pointers on areas
which would need work would be appreciated.

Lastly, is there any driver making use of msm's IOMMU code - mainline
or otherwise ?
I wanted to check the usage of the same.

Regards,
Abhilash

Re: IOMMU Query

[Roedel, Joerg]

On Wed, Mar 23, 2011 at 05:08:45AM -0400, Abhilash Kesavan wrote:
> We are investigating if the generic IOMMU API (include/linux/iommu.h)
> can be used for the IOMMU on our arm-based boards.

Great :-)

> I noticed that msm's IOMMU code (arch/arm/mach-msm/iommu.c) uses the
> genric IOMMU API and have a few queries regarding the supported
> features. Following are some features/requirements of our system:
> 
> 1) Multiple IOMMU's  - consistency between parallel IOMMUs must be maintained

This is hidden behind the API. The IOMMU driver is responsible for
managing multiple IOMMUs. On x86 there is a unique device<->IOMMU
relationship which is used internally in the drivers. Users of the
IOMMU-API can ignore the fact that there are multiple IOMMUs in the
system.

> 2) Supports varying page sizes - 4K, 64K, 1M, 16M i.e. a sg list of
> contiguous physical memory blocks be made available. Thus, giving us
> mappings from the conservative 4K to large1M reducing TLB misses.

This is supported but also hidden in the IOMMU driver implementation.
the AMD-IOMMU driver implements multiple page-sizes already using this
API. The API user has a function which looks as follows:

	int iommu_map(struct iommu_domain *domain,
		      unsigned long iova,
		      phys_addr_t paddr,
		      int gfp_order,
		      int prot);

this function needs to map the pysically contiguous memory starting at
paddr at the io-virtual address iova. The size of the mapping is
determinded by the gfp_order parameter. The size of the mapping is
basically (2**(gfp_order))*PAGE_SIZE. Note that PAGE_SIZE is the cpu
page-size.
The IOMMU driver has enough information from that function to map the
area with the best-fit page-sizes.

> 3) Multiple VA allocations can be associated with the same PA space.

I don't really understand what you mean here. Can you elabortate?

> Who manages the virtual address space of IO device that has IOMMU? I
> think the generic iommu.h API cannot handle virtual address space and
> memory allocation.  Is it just a IOMMU handler? Should we have another
> virtual memory manager for IOMMU?

The layout of the virtual address space for the IO device needs to be
managed by the particular device driver. The IOMMU-API only provides
functions for mapping/unmapping memory.

Regards,

	Joerg

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632

Re: IOMMU Query

[Stepan Moskovchenko]

On 3/23/2011 2:08 AM, Abhilash Kesavan wrote:
> Hi all,
>
Hello

> We are investigating if the generic IOMMU API (include/linux/iommu.h)
> can be used
> for the IOMMU on our arm-based boards. I noticed that msm's IOMMU code
> (arch/arm/mach-msm/iommu.c) uses the genric IOMMU API and have a few queries
> regarding the supported features. Following are some
> features/requirements of our
> system:
>
> 1) Multiple IOMMU's  - consistency between parallel IOMMUs must be maintained
> 2) Supports varying page sizes - 4K, 64K, 1M, 16M i.e. a sg list of
> contiguous physical
> memory blocks be made available. Thus, giving us mappings from the
> conservative 4K
> to large1M reducing TLB misses.
> 3) Multiple VA allocations can be associated with the same PA space.
>
Consistency between multiple IOMMUs can be maintained. Map and unmap 
operations happen on a 'domain' structure (which is intended to abstract 
a page table), and a domain can be attached to one or more IOMMU 
devices, assuming that all your IOMMUs use the same page table format. 
If not, you will need to put more "higher-level" information into your 
domain and then translate it into the HW-specific format for each IOMMU 
that you have. But it sounds pretty unlikely that you would have 
multiple IOMMUs in the same system with different page table formats. 
Thus, I think your consistency requirement should be supported, as long 
as your map/unmap function knows to iterate over all the IOMMUs attached 
to that domain and do any hardware-specific maintenance that is needed 
in addition to just updating the page table (such as invalidating 
IOTLBs, etc). The MSM driver currently does this.

Varying mapping sizes can be supported, but you will likely need an 
additional layer to organize arbitrarily-sized mappings into something 
that the IOMMU layer can easily digest. The IOMMU API takes a page order 
as an argument, and you can mix and match these (as the MSM driver does) 
but the chunk breakdown would have to come from the layer above.

Multiple VA allocations for the same PA should also be possible, and 
should really not be affected by the IOMMU API (nothing really prevents 
you from passing the same PA to multiple calls to iommu_map()). Just 
keep in mind that your multiple VA allocations need to be mapped with 
consistent attributes as per ARM spec, at least on the CPU side (not 
sure if your IOMMU HW also has this requirement).

> Who manages the virtual address space of IO device that has IOMMU? I think the
> generic iommu.h API cannot handle virtual address space and memory allocation.
> Is it just a IOMMU handler? Should we have another virtual memory
> manager for IOMMU?
>
The IOMMU layer itself does not handle virtual space management. Its 
sole responsibility is to set up the mappings. You would need something 
else to do the management for you. There have been some discussions 
about it in the past...

> Can you please tell me if the current msm sysmmu code supports some/all of the
> above features, so that we can use it as reference. Also, any pointers on areas
> which would need work would be appreciated.
>
The MSM driver supports most of the features. The only thing that is 
stubbed out right now is domain_has_cap. I believe the only place this 
is currently used is to determine cache coherency capabilities. While 
the MSM IOMMU does support cache coherent access, this needs to be 
configured through the mapping attributes to be "fully correct". I hope 
to have the has_cap stuff properly supported at a later time.

> Lastly, is there any driver making use of msm's IOMMU code - mainline
> or otherwise ?
> I wanted to check the usage of the same.
>
I don't think there is any usage of the MSM driver in mainline at this 
time. You could look around for usage examples of the API in other 
places (like KVM) but I doubt that will have the usage that you are 
looking for. There are currently efforts to use it in MSM code, but that 
code has not been sent out yet.

I hope this helps.
Steve

> Regards,
> Abhilash
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum

Re: IOMMU Query

[Joerg Roedel]

On Wed, Mar 23, 2011 at 03:18:04PM -0700, Stepan Moskovchenko wrote:
> Consistency between multiple IOMMUs can be maintained. Map and unmap  
> operations happen on a 'domain' structure (which is intended to abstract  
> a page table), and a domain can be attached to one or more IOMMU  
> devices, assuming that all your IOMMUs use the same page table format.

The IOMMU-API even supports it if your IOMMUs have different page-table
formats. The driver can just setup page-tables in every possible format
on map/unmap and use the appropriate one for each device. So this is
possible if wanted.

> The IOMMU layer itself does not handle virtual space management. Its  
> sole responsibility is to set up the mappings. You would need something  
> else to do the management for you. There have been some discussions  
> about it in the past...

Which is by the way the point of the IOMMU-API. If you want address
space management done the the DMA-API is the better choice to implement
for your IOMMU. The DMA-API manages the address space itself.
Something was needed that allows the driver to manage the address space
itself (for KVM device passthrough). This was not possible with the
DMA-API so the IOMMU-API was introduced.
Your IOMMU driver can also implement both APIs like the x86 IOMMU
drivers do (VT-d and AMD-Vi).

Regards,

	Joerg