Re: Xen on ARM vITS Handling Draft B (Was Re: Xen/arm: Virtual ITS command queue handling)

[Julien Grall <julien.grall@citrix.com>]

Hi Ian,

On 19/05/15 13:10, Ian Campbell wrote:
> On Fri, 2015-05-15 at 15:55 +0100, Julien Grall wrote:
> [...]
>>> Translation of certain commands can be expensive (XXX citation
>>> needed).
>>
>> The term "expensive" is subjective. I think we can end up to cheap
>> translation if we properly pre-allocate information (such as device,
>> LPIs...). We can have all the informations before the guest as boot or
>> during hotplug part. It wouldn't take more memory than it should use.
>>
>> During command translation, we would just need to enable the device/LPIs.
>>
>> The remaining expensive part would be the validation. I think we can
>> improve most of them of O(1) (such as collection checking) or O(log(n))
>> (such as device checking).
> [...]
>>> XXX need a solution for this.
>>
>> Command translation can be improved. It may be good too add a section
>> explaining how translation of command foo can be done.
> 
> I think that is covered by the spec, however if there are operations
> which form part of this which are potentially expensive we should
> outline in our design how this will be dealt with.
> 
> Perhaps you or Vijay could propose some additional text covering:
>       * What the potentially expensive operations during a translation
>         are.
>       * How we are going to deal with those operations, including:
>               * What data structure is used
>               * What start of day setup is required to enable this
>               * What operations are therefore required at translation
>                 time

I don't have much time to work on a proposal. I would be happy if Vijay
do it.

>>  I think
>> that limiting the number of batch/command sent per pass would allow a
>> small pass.
> 
> I think we have a few choices:
> 
>       * Limit to one batch per vits at a time
>       * Limit to some total number of batches per scheduling pass
>       * Time bound the scheduling procedure
>
> Do we have a preference?

Time bound may be difficult to implement. I think we would have to limit
batch per vITS (for code simplification) and total number of batch per
scheduling pass at the same time.

>>>   the underlying hardware to the guest.
>>> * Adds complexity to the guest layout, which is right now static. How
>>>   do you decide the number of vITS/root controller exposed:
>>>     * Hotplug is tricky
>>> * Toolstack needs greater knowledge of the host layout
>>> * Given that PCI passthrough doesn't allow migration, maybe we could
>>>   use the layout of the hardware.
>>>
>>> In 1 vITS for all pITS:
>>>
>>> * What to do with global commands? Inject to all pITS and then
>>>   synchronise on them all finishing.
>>> * Handling of out of order completion of commands queued with
>>>   different pITS, since the vITS must appear to complete in
>>>   order. Apart from the book keeping question it makes scheduling more
>>>   interesting:
>>>     * What if you have a pITS with slots available, and the guest command
>>>       queue contains commands which could go to the pITS, but behind ones
>>>       which are targetting another pITS which has no slots
>>>     * What if one pITS is very busy and another is mostly idle and a
>>>       guest submits one command to the busy one (contending with other
>>>       guest) followed by a load of commands targeting the idle one. Those
>>>       commands would be held up in this situation.
>>>     * Reasoning about fairness may be harder.
>>>
>>> XXX need a solution/decision here.
>>
>>> In addition the introduction of direct interrupt injection in version
>>> 4 GICs may imply a vITS per pITS. (Update: it seems not)
>>
>> Other items to add: NUMA and I/O NUMA. I don't know much about it but I
>> think the first solution would be more suitable.
> 
> first solution == ?

1 vITS per pITS.

Regards,

-- 
Julien Grall