RFC: iommu_ops -- limitation with current attach_dev() API

RFC: iommu_ops -- limitation with current attach_dev() API

[Stuart Yoder]

Wanted to get some input and feedback on a limitation we have run into
with the current
iommu_ops architecture regarding attaching (and detaching) devices.

The attach_dev API is defined as:

extern int iommu_attach_device(struct iommu_domain *domain,
                               struct device *dev);

The assumption is that a device targets at most one iommu domain.

We have devices that are capable of and need to access multiple
domains.    At the hardware level the device has multiple 'logical
I/O device numbers'  which is the index into our IOMMU tables.  This
means the device can target DMA operations to 2 different
address spaces.

One possible question is whether the device is really not one, but
multiple devices.   That's not the case.  It is literally one device with
multiple DMA address space targets. (analagous perhaps in a way
to a device with multiple interrupts?).

An example is a queue manager portal device on our SoCs which is represented
in our device tree as:

                qportal1: qman-portal@4000 {
                        cell-index = <0x1>;
                        compatible = "fsl,p4080-qman-portal", "fsl,qman-portal";
                        reg = <0x4000 0x4000 0x101000 0x1000>;
                        interrupts = <106 0x2 0 0>;
                        fsl,qman-channel-id = <0x1>;
                        fsl,liodn = <0x1 0x2>;
                };

The fsl,liodn property specifies the 2 io device numbers that allow
the distinction of 2 destination address spaces for DMA.

The first liodn allows the device to stash certain frame header
information directly to a CPU's cache.  The second is for normal
frame data.

There is only one "struct device" for this device, and thus we
now have an issue as iommu_attach_device() can't distinguish
which of the 2 address spaces are being configured.

Does anyone have any high level thoughts on how to solve
this problem?

In some internal discussions here at Freescale, here were some
of our initial ideas:

   -in 'struct device' define a new list of hardware/physical DMA 'domains'
    that represent each address space a device can target:

         struct list_head        iommu_hw_domain_head;

    That list would get populated when the device struct is
    initialized with an arch specific list of the hardware 'domains'.

   -define a  new iommu_ops function to add devices with this hardware
    domain info:

       extern int iommu_attach_device_ext(struct iommu_domain *domain,
                               struct device *dev,  void *iommu_hw_domain);


Regards,
Stuart

Re: RFC: iommu_ops -- limitation with current attach_dev() API

[Joerg Roedel]

On Tue, Jan 10, 2012 at 11:15:54AM -0600, Stuart Yoder wrote:
> We have devices that are capable of and need to access multiple
> domains.    At the hardware level the device has multiple 'logical
> I/O device numbers'  which is the index into our IOMMU tables.  This
> means the device can target DMA operations to 2 different
> address spaces.

Your IOMMU is a real challenge it seems ;-)

> In some internal discussions here at Freescale, here were some
> of our initial ideas:
> 
>    -in 'struct device' define a new list of hardware/physical DMA 'domains'
>     that represent each address space a device can target:
> 
>          struct list_head        iommu_hw_domain_head;

I think we really should introduce a dev->iommu pointer which can be
used by IOMMU drivers. Your hardware is on the third architecture
needing such a pointer and hiding it in dev->archdata.

>     That list would get populated when the device struct is
>     initialized with an arch specific list of the hardware 'domains'.
> 
>    -define a  new iommu_ops function to add devices with this hardware
>     domain info:
> 
>        extern int iommu_attach_device_ext(struct iommu_domain *domain,
>                                struct device *dev,  void *iommu_hw_domain);

How about just enumerating the device address spaces with an u32 and use
that to assign the domains. Or is an in-kernel descriptor for the
hardware-side really needed? If so, why?


	Joerg

Re: RFC: iommu_ops -- limitation with current attach_dev() API

[Scott Wood]

On Thu, Jan 12, 2012 at 03:55:00PM +0100, Joerg Roedel wrote:
> On Tue, Jan 10, 2012 at 11:15:54AM -0600, Stuart Yoder wrote:
> > We have devices that are capable of and need to access multiple
> > domains.    At the hardware level the device has multiple 'logical
> > I/O device numbers'  which is the index into our IOMMU tables.  This
> > means the device can target DMA operations to 2 different
> > address spaces.
> 
> Your IOMMU is a real challenge it seems ;-)

Yes. :-(

> >    -define a  new iommu_ops function to add devices with this hardware
> >     domain info:
> > 
> >        extern int iommu_attach_device_ext(struct iommu_domain *domain,
> >                                struct device *dev,  void *iommu_hw_domain);
> 
> How about just enumerating the device address spaces with an u32 and use
> that to assign the domains. Or is an in-kernel descriptor for the
> hardware-side really needed? If so, why?

A u32 would work well enough for our hardware, which has only one
numberspace for this (well, there are multiple IOMMUs, but you're
supposed to treat them as if they were one and point them all to the same
tables), but I'd hesitate to design the API that way -- we'd have less
complexity and confusion in interrupt config if we described IRQs as
pointers rather than ints, for example.

Ideally the hw domain would be a pointer to a struct that contains a
pointer to iommu ops, rather than tie it to what kind of bus dev sits on
(that both buses are "platform" doesn't mean they're anything like each
other), and dev would just be a place where you can find a list of IOMMU
resources (just like IRQs, which can each map to a different hw
controller).

That would be a significant overhaul, though -- letting it be a void
pointer would mean that bus types could choose this interpretation
gradually when and if they encounter the need, without having to invent
numberspace encodings or allocation.  How about an opaque unsigned long,
that could be an integer or a pointer?

-Scott