IOMMU keeping mapping over long period of time

IOMMU keeping mapping over long period of time

[Jerome Glisse]

Hi,

For GPU we are entertaining the idea of keeping pci/dma mapping of page
around for the whole life span of GPU object (which can be as long as
the computer uptime). Here we are talking about possibly several GB of
data.

GPU for which we are interested have dma mask of 40bits at least, thus
they can work with high mem page. My main concern is when an iommu is
active, such amount of continuously mapped page might exhaust the iommu
address space. My understanding is that Intel & AMD iommu are capable of
growing their address space but is it somethings we can rely on ?

Cheers,
Jerome

Re: IOMMU keeping mapping over long period of time

[Andi Kleen]

Jerome Glisse <j.glisse@gmail.com> writes:
>
> GPU for which we are interested have dma mask of 40bits at least, thus
> they can work with high mem page. My main concern is when an iommu is
> active, such amount of continuously mapped page might exhaust the iommu
> address space. My understanding is that Intel & AMD iommu are capable of
> growing their address space but is it somethings we can rely on ?

I don't think relying on this is very wise. Better add some way to unmap
when space is running short.

-Andi

Re: IOMMU keeping mapping over long period of time

[Jerome Glisse]

On Tue, Oct 18, 2011 at 11:56 AM, Andi Kleen <andi@firstfloor.org> wrote:
> Jerome Glisse <j.glisse@gmail.com> writes:
>>
>> GPU for which we are interested have dma mask of 40bits at least, thus
>> they can work with high mem page. My main concern is when an iommu is
>> active, such amount of continuously mapped page might exhaust the iommu
>> address space. My understanding is that Intel & AMD iommu are capable of
>> growing their address space but is it somethings we can rely on ?
>
> I don't think relying on this is very wise. Better add some way to unmap
> when space is running short.
>
> -Andi
>

Maybe i should have add that the usecase involve the driver never knowing when
the GPU will make use of the mapped memory and this can happen anytime.

Cheers,
Jerome

Re: IOMMU keeping mapping over long period of time

[Benjamin Herrenschmidt]

On Tue, 2011-10-18 at 11:48 -0400, Jerome Glisse wrote:
> Hi,
> 
> For GPU we are entertaining the idea of keeping pci/dma mapping of page
> around for the whole life span of GPU object (which can be as long as
> the computer uptime). Here we are talking about possibly several GB of
> data.
> 
> GPU for which we are interested have dma mask of 40bits at least, thus
> they can work with high mem page. My main concern is when an iommu is
> active, such amount of continuously mapped page might exhaust the iommu
> address space. My understanding is that Intel & AMD iommu are capable of
> growing their address space but is it somethings we can rely on ?

This will be problematic on other platforms. Also keep in mind that
objects mapped in the iommu can no longer be swapped out.

It might be useful to have a more 'lazy' mechanism where objects are
kept mapped but with a callback to unmap them in case of iommu pressure.

The problem of course with such a scheme is that it requires iommu
object tracking and horrible locking issues.

Ben.