Re: [PATCH] documentation: iommu: add description of ARM System MMU binding

Re: [PATCH] documentation: iommu: add description of ARM System MMU binding

[Olav Haugan]

On 4/15/2013 6:13 AM, Will Deacon wrote:
> On Sat, Apr 13, 2013 at 10:02:09PM +0100, Olav Haugan wrote:
>> On 4/10/2013 10:37 AM, Will Deacon wrote:
>>> Ok, but then how does that interact with the IOMMU API in Linux? e.g.
>>> if a client asks for an iova <-> pa mapping on a device, then you
>>> need to lookup the StreamIDs for that device, no?
>>
>> Can you explain this use case a bit more. I am not clear what you mean
>> by a "client"? Are you referring to a client in Linux or are you
>> referring to the client being the core behind an IOMMU (such as a DMA
>> engine or a video core)?
> 
> Sure. By `client', I mean any use of the IOMMU API. It could be something
> like your LVD example later on, it could be KVM or it could be indirectly
> via dma-mapping. Basically, any user of <linux/iommu.h>.

Ok, so if I do a VA2PA on a domain (calling iommu_iova_to_phys()) you
don't need any information about StreamID (no need to look up the
StreamID). You would just walk the page table of the domain and find the
address (or use ATS on the IOMMU if supported by the HW).

>> I am not sure how StreamIDs relates to the IOMMU API in Linux :-)
> 
> That's the thing; they don't, at least not explicitly. The StreamIDs must
> all be handled inside the driver and it's actually what the driver would
> deal with, rather than devices.
> 
> For example, if I create an IOMMU domain containing a DMA controller, that
> means that I have associated a set of StreamIDs (i.e. those configured to
> identify the DMA controller's master ports) with that domain. That may have
> further consequences on which other devices can be put in the same domain
> (stream-matching may require some IDs to sit in the same address space/domain)
> and also effectively parameterises the domain, by pointing those StreamIDs
> at the same context bank.

Yes, this is clear to me.

>> I think it helps if I explain my use case to you so you understand how
>> we use the IOMMU and maybe you can explain your use case and how your
>> driver works so that we can get on the same page. BTW, our ARMv1 IOMMU
>> driver can be found @ [1]
>>
>> Our use case:
>>
>> 1) During boot IOMMU HW is probed, devices for each IOMMU and for each
>> context bank are created.
> 
> So you create a device for each context bank...
> 
>> 2) Video session starts.
>> 3) Linux Video Driver (LVD) creates one or more domain
>> (iommu_domain_alloc) and calls iommu_attach_device() for each context
>> bank it is going to use.
> 
> ...and attach the context bank to the domain dynamically?

Yes

> If so, doesn't that strongly tie your video driver to the SMMU?

Isn't this more or less what you are doing in DT where you associate
specific devices with an IOMMU (mmu-masters)?

>> The video driver selects context bank to attach to based on the use case
>> and does not know about StreamIDs really.
>>
>> BTW, when you call iommu_attach_device() what "device" do you pass as
>> argument? Do you pass the IOMMU as a device? How do you specify which
>> Context Bank to attach to?
> 
> This seems to be where we differ. My anticipated use-case is that a domain
> is allocated, which defines an empty address space. Masters are then
> attached to the domain by passing their struct device, so this may
> correspond to a DMA controller or a video core, in your case. The context
> bank is allocated dynamically when the first device is added to the
> domain, and then it is subsequently shared between all the masters in that
> domain.

I feel that there are some limitation with this. Maybe you can address
the following:

1) What if you want two context banks in one domain (shared page table)?
2) What if a master (device) needs to use 2 or more context banks?

>>> Why can't the secure world just hide those context banks by using 
>>> SMMU_SCR1.NSNUMCBO?
>>
>> Yes, that is a possibility for context banks. If you have 1 or more
>> "secure" context banks it also means that Linux does not have access
>> (read: can't have) to the global register space and this must be
>> communicated somehow (hence the "secure-id" IOMMU node property. The
>> secure-context property might be redundant. However, there might be use
>> cases were Linux would want to know about the secure context bank. One
>> such use case could be to register for page fault from the secure
>> context bank so that Linux can log a useful page fault message to the
>> kernel log.
> 
> Perhaps, but I don't think it's worth complicating the binding based on a
> potential nice-to-have fault message.
> 

However, global register space (including Stream Match Registers [SMR])
needs to be programmed by the secure environment for security or by
hyper-visor if you are in such an environment.

Thanks,

Olav Haugan

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH] documentation: iommu: add description of ARM System MMU binding

[Will Deacon]

Hi Olav,

On Tue, Apr 16, 2013 at 07:18:42PM +0100, Olav Haugan wrote:
> On 4/15/2013 6:13 AM, Will Deacon wrote:
> > If so, doesn't that strongly tie your video driver to the SMMU?
> 
> Isn't this more or less what you are doing in DT where you associate
> specific devices with an IOMMU (mmu-masters)?

No. The device-tree describes the *hardware*, as per usual. The StreamIDs
are fixed properties of the SoC and we can't change them from Linux, so we
describe all of the StreamIDs upstream of each SMMU so that we can program
the thing. There's no way to generic way to discover them.

> > This seems to be where we differ. My anticipated use-case is that a domain
> > is allocated, which defines an empty address space. Masters are then
> > attached to the domain by passing their struct device, so this may
> > correspond to a DMA controller or a video core, in your case. The context
> > bank is allocated dynamically when the first device is added to the
> > domain, and then it is subsequently shared between all the masters in that
> > domain.
> 
> I feel that there are some limitation with this. Maybe you can address
> the following:
> 
> 1) What if you want two context banks in one domain (shared page table)?

Why would a page table shared between devices require multiple context
banks? Multiple SMRs and S2CRs, sure, but they would ultimately point at the
same context bank (and hence same address space).

> 2) What if a master (device) needs to use 2 or more context banks?

If a master needs to be in two address spaces at once, then it will need to
attach it's StreamIDs to different domains. You can't place a single
StreamID in two address spaces (this is an architectural constraint).

> > Perhaps, but I don't think it's worth complicating the binding based on a
> > potential nice-to-have fault message.
> > 
> 
> However, global register space (including Stream Match Registers [SMR])
> needs to be programmed by the secure environment for security or by
> hyper-visor if you are in such an environment.

Of course, but that's not Linux (and KVM would likely have some additional
code if it wanted to do device passthrough, for example).

Will

Re: [PATCH] documentation: iommu: add description of ARM System MMU binding

[Olav Haugan]

Hi Will,

On 4/18/2013 12:01 PM, Will Deacon wrote:
> On Tue, Apr 16, 2013 at 07:18:42PM +0100, Olav Haugan wrote:
>> On 4/15/2013 6:13 AM, Will Deacon wrote:
>>> If so, doesn't that strongly tie your video driver to the SMMU?
>>
>> Isn't this more or less what you are doing in DT where you associate
>> specific devices with an IOMMU (mmu-masters)?
> 
> No. The device-tree describes the *hardware*, as per usual. The StreamIDs
> are fixed properties of the SoC and we can't change them from Linux, so we
> describe all of the StreamIDs upstream of each SMMU so that we can program
> the thing. There's no way to generic way to discover them.

I meant that you are putting phandles to bus masters in the device tree
that use/need the SMMU for VA2PA translation. Doesn't this put a
dependency on the bus master devices? How will this work? Lets say that
we have a device during bootup that needs to use the SMMU (such as a
display processor [DP]). Don't you have cyclic dependency? The SMMU
device needs the DP device (phandle) but the DP device needs the SMMU to
initialize its device driver?

>>> This seems to be where we differ. My anticipated use-case is that a domain
>>> is allocated, which defines an empty address space. Masters are then
>>> attached to the domain by passing their struct device, so this may
>>> correspond to a DMA controller or a video core, in your case. The context
>>> bank is allocated dynamically when the first device is added to the
>>> domain, and then it is subsequently shared between all the masters in that
>>> domain.
>>
>> I feel that there are some limitation with this. Maybe you can address
>> the following:
>>
>> 1) What if you want two context banks in one domain (shared page table)?
> 
> Why would a page table shared between devices require multiple context
> banks? Multiple SMRs and S2CRs, sure, but they would ultimately point at the
> same context bank (and hence same address space).

What if you want to have 2 different VMID's being generated? Also, what
about TLB management? If I have two context banks I can invalidate only
entries associated with 1 of the context banks (if VMID differ for the
two context banks).

>> 2) What if a master (device) needs to use 2 or more context banks?
> 
> If a master needs to be in two address spaces at once, then it will need to
> attach it's StreamIDs to different domains. You can't place a single
> StreamID in two address spaces (this is an architectural constraint).

Yes, you would have a separate domain. I am just wondering how I would
model this in DT using the bindings that you are proposing? How does it
work? The bindings specify bus masters to StreamIDs. So if I call attach
with the master device you will allocate a context bank and program the
StreamIDs as specified in the DT. So now if I want to have another
context bank associated with the same master device what do I do? I call
into attach with a new domain but with the same master device but the
master device is already attached to a context/domain.

Thanks,

Olav Haugan

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH] documentation: iommu: add description of ARM System MMU binding

[Will Deacon]

On Tue, Apr 23, 2013 at 11:54:53PM +0100, Olav Haugan wrote:
> Hi Will,

Hello again,

> On 4/18/2013 12:01 PM, Will Deacon wrote:
> > No. The device-tree describes the *hardware*, as per usual. The StreamIDs
> > are fixed properties of the SoC and we can't change them from Linux, so we
> > describe all of the StreamIDs upstream of each SMMU so that we can program
> > the thing. There's no way to generic way to discover them.
> 
> I meant that you are putting phandles to bus masters in the device tree
> that use/need the SMMU for VA2PA translation. Doesn't this put a
> dependency on the bus master devices? How will this work? Lets say that
> we have a device during bootup that needs to use the SMMU (such as a
> display processor [DP]). Don't you have cyclic dependency? The SMMU
> device needs the DP device (phandle) but the DP device needs the SMMU to
> initialize its device driver?

That's a problem you have with the driver model anyway. The SMMU must be
registered on the same bus as the device before it can be used and the
devices must be added to that bus before they can be attached to a domain.

Getting phandles has no dependencies on anything -- the only dependency is
that the device is added to the bus on which the SMMU sits, just like every
other IOMMU driver.

> > Why would a page table shared between devices require multiple context
> > banks? Multiple SMRs and S2CRs, sure, but they would ultimately point at the
> > same context bank (and hence same address space).
> 
> What if you want to have 2 different VMID's being generated? Also, what
> about TLB management? If I have two context banks I can invalidate only
> entries associated with 1 of the context banks (if VMID differ for the
> two context banks).

Huh? We allocate one VMID per domain. If you want more VMIDs, use more
domains. TLB invalidation is per-domain, so there's no issue there.

> > If a master needs to be in two address spaces at once, then it will need to
> > attach it's StreamIDs to different domains. You can't place a single
> > StreamID in two address spaces (this is an architectural constraint).
> 
> Yes, you would have a separate domain. I am just wondering how I would
> model this in DT using the bindings that you are proposing? How does it
> work? The bindings specify bus masters to StreamIDs. So if I call attach
> with the master device you will allocate a context bank and program the
> StreamIDs as specified in the DT. So now if I want to have another
> context bank associated with the same master device what do I do? I call
> into attach with a new domain but with the same master device but the
> master device is already attached to a context/domain.

Why would you want to place a StreamID into two domains? That doesn't make
any sense and isn't even supported by the architecture (things like
conflicting SMR entries may not even be reported).

Will

Re: [PATCH] documentation: iommu: add description of ARM System MMU binding

[Olav Haugan]

Hi Will,

On 4/24/2013 2:55 AM, Will Deacon wrote:
> On Tue, Apr 23, 2013 at 11:54:53PM +0100, Olav Haugan wrote:
>> On 4/18/2013 12:01 PM, Will Deacon wrote:
> Getting phandles has no dependencies on anything -- the only dependency is
> that the device is added to the bus on which the SMMU sits, just like every
> other IOMMU driver.
> 
>>> Why would a page table shared between devices require multiple context
>>> banks? Multiple SMRs and S2CRs, sure, but they would ultimately point at the
>>> same context bank (and hence same address space).
>>
>> What if you want to have 2 different VMID's being generated? Also, what
>> about TLB management? If I have two context banks I can invalidate only
>> entries associated with 1 of the context banks (if VMID differ for the
>> two context banks).
> 
> Huh? We allocate one VMID per domain. If you want more VMIDs, use more
> domains. TLB invalidation is per-domain, so there's no issue there.

If we allocate a new domain you would have another address space (page
table), no? I was just trying to figure out if your binding proposal
would allow for 2 context banks in one domain which it does not seem to
support.

>>> If a master needs to be in two address spaces at once, then it will need to
>>> attach it's StreamIDs to different domains. You can't place a single
>>> StreamID in two address spaces (this is an architectural constraint).
>>
>> Yes, you would have a separate domain. I am just wondering how I would
>> model this in DT using the bindings that you are proposing? How does it
>> work? The bindings specify bus masters to StreamIDs. So if I call attach
>> with the master device you will allocate a context bank and program the
>> StreamIDs as specified in the DT. So now if I want to have another
>> context bank associated with the same master device what do I do? I call
>> into attach with a new domain but with the same master device but the
>> master device is already attached to a context/domain.
> 
> Why would you want to place a StreamID into two domains? That doesn't make
> any sense and isn't even supported by the architecture (things like
> conflicting SMR entries may not even be reported).

I think you misunderstood me. I am talking about having for example 1
master with two (2) context banks so that StreamID "1" goes to CB0 and
StreamID "2" goes to CB1. This means that the master device is attached
to two different domains. How do I model this with the the device tree
bindings?



Olav Haugan

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH] documentation: iommu: add description of ARM System MMU binding

[Andreas Herrmann]

On Tue, May 07, 2013 at 04:26:02PM -0400, Olav Haugan wrote:
> Hi Will,
> 
> On 4/24/2013 2:55 AM, Will Deacon wrote:
> > On Tue, Apr 23, 2013 at 11:54:53PM +0100, Olav Haugan wrote:
> >> On 4/18/2013 12:01 PM, Will Deacon wrote:
> > Getting phandles has no dependencies on anything -- the only dependency is
> > that the device is added to the bus on which the SMMU sits, just like every
> > other IOMMU driver.
> > 
> >>> Why would a page table shared between devices require multiple context
> >>> banks? Multiple SMRs and S2CRs, sure, but they would ultimately point at the
> >>> same context bank (and hence same address space).
> >>
> >> What if you want to have 2 different VMID's being generated? Also, what
> >> about TLB management? If I have two context banks I can invalidate only
> >> entries associated with 1 of the context banks (if VMID differ for the
> >> two context banks).
> > 
> > Huh? We allocate one VMID per domain. If you want more VMIDs, use more
> > domains. TLB invalidation is per-domain, so there's no issue there.
> 
> If we allocate a new domain you would have another address space (page
> table), no? I was just trying to figure out if your binding proposal
> would allow for 2 context banks in one domain which it does not seem to
> support.
> 
> >>> If a master needs to be in two address spaces at once, then it will need to
> >>> attach it's StreamIDs to different domains. You can't place a single
> >>> StreamID in two address spaces (this is an architectural constraint).
> >>
> >> Yes, you would have a separate domain. I am just wondering how I would
> >> model this in DT using the bindings that you are proposing? How does it
> >> work? The bindings specify bus masters to StreamIDs. So if I call attach
> >> with the master device you will allocate a context bank and program the
> >> StreamIDs as specified in the DT. So now if I want to have another
> >> context bank associated with the same master device what do I do? I call
> >> into attach with a new domain but with the same master device but the
> >> master device is already attached to a context/domain.
> > 
> > Why would you want to place a StreamID into two domains? That doesn't make
> > any sense and isn't even supported by the architecture (things like
> > conflicting SMR entries may not even be reported).
> 
> I think you misunderstood me. I am talking about having for example 1
> master with two (2) context banks so that StreamID "1" goes to CB0 and
> StreamID "2" goes to CB1. This means that the master device is attached
> to two different domains. How do I model this with the the device tree
> bindings?

Hi Olav,

I think with the proposed device tree binding you can't model this.

Do we have such use cases already? Or what future use cases would
require this?

I can imagine of multiple contexts per device for stuff similar to
what PASID in PCI Express are used for.

But in such a case you probably want to have some configurable bits in
the StreamID (that should be set by an SMMU driver). And the DT
binding should contain the number of contexts that a master device can
support.


Andreas

Re: [PATCH] documentation: iommu: add description of ARM System MMU binding

[Will Deacon]

On Mon, May 13, 2013 at 10:07:24AM +0100, Andreas Herrmann wrote:
> On Tue, May 07, 2013 at 04:26:02PM -0400, Olav Haugan wrote:
> > I think you misunderstood me. I am talking about having for example 1
> > master with two (2) context banks so that StreamID "1" goes to CB0 and
> > StreamID "2" goes to CB1. This means that the master device is attached
> > to two different domains. How do I model this with the the device tree
> > bindings?
> 
> Hi Olav,
> 
> I think with the proposed device tree binding you can't model this.
> 
> Do we have such use cases already? Or what future use cases would
> require this?
> 
> I can imagine of multiple contexts per device for stuff similar to
> what PASID in PCI Express are used for.
> 
> But in such a case you probably want to have some configurable bits in
> the StreamID (that should be set by an SMMU driver). And the DT
> binding should contain the number of contexts that a master device can
> support.

See my reply to your other thread, but I don't think this should be solved
in the SMMU driver. I really think that the device/domain configuration
should be driven by the master device drivers, rather than the device-tree.

Will

Re: [PATCH] documentation: iommu: add description of ARM System MMU binding

[Olav Haugan]

Hi Andreas,

On 5/13/2013 2:07 AM, Andreas Herrmann wrote:
> On Tue, May 07, 2013 at 04:26:02PM -0400, Olav Haugan wrote:
>>
>>>>> If a master needs to be in two address spaces at once, then it will need to
>>>>> attach it's StreamIDs to different domains. You can't place a single
>>>>> StreamID in two address spaces (this is an architectural constraint).
>>>>
>>>> Yes, you would have a separate domain. I am just wondering how I would
>>>> model this in DT using the bindings that you are proposing? How does it
>>>> work? The bindings specify bus masters to StreamIDs. So if I call attach
>>>> with the master device you will allocate a context bank and program the
>>>> StreamIDs as specified in the DT. So now if I want to have another
>>>> context bank associated with the same master device what do I do? I call
>>>> into attach with a new domain but with the same master device but the
>>>> master device is already attached to a context/domain.
>>>
>>> Why would you want to place a StreamID into two domains? That doesn't make
>>> any sense and isn't even supported by the architecture (things like
>>> conflicting SMR entries may not even be reported).
>>
>> I think you misunderstood me. I am talking about having for example 1
>> master with two (2) context banks so that StreamID "1" goes to CB0 and
>> StreamID "2" goes to CB1. This means that the master device is attached
>> to two different domains. How do I model this with the the device tree
>> bindings?
> 
> Hi Olav,
> 
> I think with the proposed device tree binding you can't model this.
> 
> Do we have such use cases already? Or what future use cases would
> require this?
> 
> I can imagine of multiple contexts per device for stuff similar to
> what PASID in PCI Express are used for.
> 
> But in such a case you probably want to have some configurable bits in
> the StreamID (that should be set by an SMMU driver). And the DT
> binding should contain the number of contexts that a master device can
> support.
> 

Sorry for the late reply.

The only real use case at this time for a master having more than 1 CB
is for content protection I believe. I understand that this driver does
not currently support content protection or anything related to security.

Thanks,

Olav Haugan

-- 
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation