Re: [PATCH 1/2] irqchip/gicv3-its: Support share device ID

[Marc Zyngier <marc.zyngier@arm.com>]

On 24/04/15 17:18, Will Deacon wrote:
> On Wed, Apr 22, 2015 at 08:41:02PM +0100, Stuart Yoder wrote:
>>>> However, there is an improvement we envision as possible due to
>>>> the limited number of SMMU contexts (i.e. 64).  If there are
>>>> 64 SMMU context registers it means that there is a max of
>>>> 64 software contexts where things can be isolated.  But, say I have
>>>> an SRIOV card with 64 VFs, and I want to assign 8 of the VFs
>>>> to a KVM VM.  Those 8 PCI devices could share the same
>>>> streamID/ITS-device-ID since they all share the same isolation
>>>> context.
>>>>
>>>> What would be nice is at the time the 8 VFS are being added
>>>> to the IOMMU domain is for the pcidevid -> streamID mapping
>>>> table to be updated dynamically.  It simply lets us make
>>>> more efficient use of the limited streamIDs we have.
>>>>
>>>> I think it is this improvement that Minghuan had in mind
>>>> in this patch.
>>>
>>> Ok, but in this case it should be possible to use a single context bank for
>>> all of the VF streamIDs by configuring the appropriate SMR, no?
>>
>> Yes, but there are limited SMRs.  In our case there are only
>> 128 SMRs in SMMU-500 and we have potentially way more masters than
>> that.
> 
> Right, but you still only have 64 context banks at the end of the day, so do
> you really anticipate having more than 128 masters concurrently using the
> SMMU? If so, then we have devices sharing context banks so we could consider
> reusing SMRs across masters, but historically that's not been something that
> we've managed to solve.
> 
>>> Wouldn't
>>> that sort of thing be preferable to dynamic StreamID assignment? It would
>>> certainly make life easier for the MSIs.
>>
>> It would be preferable, but given only 128 total stream IDS and
>> 64 context registers it's potentially an issue.  On our LS2085 SoC it is
>> PCI and the fsl-mc bus (see description here:
>> https://lkml.org/lkml/2015/3/5/795) that potentially have way
>> more masters than streamIDS. So, for those busses we would essentially
>> view a streamID as a "context ID"-- each SMR is associated with
>> 1 context bank register.
>>
>> For PCI we have a programmable "PCI req ID"-to-"stream ID"
>> mapping table in the PCI controller that is dynamically
>> programmable.
>>
>> Looking at it like that means that we could have
>> any number of masters but only 64 "contexts"
>> and since the masters all all programmable it's
>> seems feasbile to envision doing some bus/vendor
>> specific set up when a device is added to an
>> IOMMU domain.   One thing that would need to be conveyed
>> to the SMMU driver if doing dynamic streamID setup
>> is what streamIDs are available to be used.
> 
> Ok, but this is going to make life difficult for the MSI people, I suspect.
> 
> Marc...?

We're really facing two conflicting requirements: in order to minimize
SMR usage, we want to alias multiple ReqIDs to a single StreamID, but in
order to efficiently deal with MSIs, we want to see discrete DeviceIDs
(the actual ReqIDs). I don't easily see how we reconcile the two.

We can deal with the aliasing, provided that we extend the level of
quirkiness that pci_for_each_dma_alias can deal with. But that doesn't
solve any form of hotplug/SR-IOV behaviour.

Somehow, we're going to end-up with grossly oversized ITTs, just to
accommodate for the fact that we have no idea how many MSIs we're going
to end-up needing. I'm not thrilled with that prospect.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...