[Qemu-devel] vfio / iommu domain attributes

[Qemu-devel] vfio / iommu domain attributes

[Stuart Yoder]

In the vfio RFC thread there seemed to be convergence that some new
iommu_ops API is needed to set some platform specific aspects of an
iommu domain.

On Wed, Nov 30, 2011 at 10:58 AM, Alex Williamson
<alex.williamson@redhat.com> wrote:
[cut]
> In that case, you should definitely be following what Alexey is thinking
> about with an iommu_setup IOMMU API callback.  I think it's shaping up
> to do:
>
> x86:
>  - Report any IOVA range restrictions imposed by hw implementation
> POWER:
>  - Request IOVA window size, report size and base
> powerpc:
>  - Set domain attributes, probably report range as well.

Alex, Alexey I'm wondering if you've had any new thoughts on this over
the last week.

For Freescale, our iommu domain attributes would look something like:
    -domain iova base address
    -domain iova window size
    -domain enable/disable
    -number of subwindows
    -operation mapping table index
    -stash destination CPU
    -stash target (cache– L1, L2, L3)

These are all things that need to be set by the creator of the domain.

Since the domain attributes are going to be so different for each platform does
it make sense to define a new iommu_ops call back that just takes a void pointer
that can be implemented in a platform specific way?   For example:

    struct iommu_ops {
        [cut]
        int (*domain_set_attrs)(struct iommu_domain *domain,
                              void *attrs);
        int (*domain_get_attrs)(struct iommu_domain *domain,
                              void *attrs);
    }

Whatever this API winds up looking like it needs to be reflected in
the vfio interface to user space as well.

Thanks,
Stuart

Re: [Qemu-devel] vfio / iommu domain attributes

[Joerg Roedel]

On Wed, Dec 07, 2011 at 09:54:39AM -0600, Stuart Yoder wrote:
> Alex, Alexey I'm wondering if you've had any new thoughts on this over
> the last week.
> 
> For Freescale, our iommu domain attributes would look something like:
>     -domain iova base address
>     -domain iova window size

I agree with that.

>     -domain enable/disable
>     -number of subwindows
>     -operation mapping table index
>     -stash destination CPU
>     -stash target (cache– L1, L2, L3)

Why does the user of the IOMMU-API need to have control over these
things?

> These are all things that need to be set by the creator of the domain.
> 
> Since the domain attributes are going to be so different for each platform does
> it make sense to define a new iommu_ops call back that just takes a void pointer
> that can be implemented in a platform specific way?   For example:
> 
>     struct iommu_ops {
>         [cut]
>         int (*domain_set_attrs)(struct iommu_domain *domain,
>                               void *attrs);
>         int (*domain_get_attrs)(struct iommu_domain *domain,
>                               void *attrs);
>     }

A void pointer is certainly the worst choice for an interface. I think
it is better to have at least a set of common attributes. Somthing like
this:

iommu_domain_set_attr(struct iommu_domain *domain, enum attr_type, void *data)
iommu_domain_get_attr(struct iommu_domain *domain, enum attr_type, void *data)

The iova base/size options make sense for more IOMMUs than just
Freescale. For example it would allow to manage GART-like IOMMUs with
the IOMMU-API too.


	Joerg

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632

Re: [Qemu-devel] vfio / iommu domain attributes

[Stuart Yoder]

On Wed, Dec 7, 2011 at 10:38 AM, Joerg Roedel <joerg.roedel@amd.com> wrote:
> On Wed, Dec 07, 2011 at 09:54:39AM -0600, Stuart Yoder wrote:
>> Alex, Alexey I'm wondering if you've had any new thoughts on this over
>> the last week.
>>
>> For Freescale, our iommu domain attributes would look something like:
>>     -domain iova base address
>>     -domain iova window size
>
> I agree with that.
>
>>     -domain enable/disable
>>     -number of subwindows
>>     -operation mapping table index
>>     -stash destination CPU
>>     -stash target (cache– L1, L2, L3)
>
> Why does the user of the IOMMU-API need to have control over these
> things?

Our IOMMU complicates things in that it is used for more than just
memory protection
and address translation.  It has a concept of operation translation as well.
Some devices could do a 'write' transaction that when passing through the
iommu gets translated to a a 'write-with-stash'.   Stashed transactions
get pushed directly into some cache.

It's the entity creating and setting up the domain that will have the knowledge
of what cache is to be stashed to.    Right now software that uses stashing
is pinned to a CPU, but if in the future it's possible that we'll want to
work without pinning and may need to update stashing attributes on the
fly.

The overall iova window for the domain can be divided into a configurable
number of subwindows (a power of 2, up to 256), which means we can have
a contiguous iova region backed by up to 256 physically dis-contiguous
subwindows.    The creator of the iommu domain is in the best position
to know how many subwindows are needed (the fewer the better for
performance reasons).

So, in short, the above list of attributes are the attributes of our
iommu hardware
and  the knowlege of how they should be set is with the domain creator.

>> These are all things that need to be set by the creator of the domain.
>>
>> Since the domain attributes are going to be so different for each platform does
>> it make sense to define a new iommu_ops call back that just takes a void pointer
>> that can be implemented in a platform specific way?   For example:
>>
>>     struct iommu_ops {
>>         [cut]
>>         int (*domain_set_attrs)(struct iommu_domain *domain,
>>                               void *attrs);
>>         int (*domain_get_attrs)(struct iommu_domain *domain,
>>                               void *attrs);
>>     }
>
> A void pointer is certainly the worst choice for an interface. I think
> it is better to have at least a set of common attributes. Somthing like
> this:
>
> iommu_domain_set_attr(struct iommu_domain *domain, enum attr_type, void *data)
> iommu_domain_get_attr(struct iommu_domain *domain, enum attr_type, void *data)

That would be fine.

Stuart