Re: Xen on ARM vITS Handling Draft B (Was Re: Xen/arm: Virtual ITS command queue handling)

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-05-15 at 15:55 +0100, Julien Grall wrote:
[...]
> > Translation of certain commands can be expensive (XXX citation
> > needed).
> 
> The term "expensive" is subjective. I think we can end up to cheap
> translation if we properly pre-allocate information (such as device,
> LPIs...). We can have all the informations before the guest as boot or
> during hotplug part. It wouldn't take more memory than it should use.
> 
> During command translation, we would just need to enable the device/LPIs.
> 
> The remaining expensive part would be the validation. I think we can
> improve most of them of O(1) (such as collection checking) or O(log(n))
> (such as device checking).
[...]
> > XXX need a solution for this.
> 
> Command translation can be improved. It may be good too add a section
> explaining how translation of command foo can be done.

I think that is covered by the spec, however if there are operations
which form part of this which are potentially expensive we should
outline in our design how this will be dealt with.

Perhaps you or Vijay could propose some additional text covering:

      * What the potentially expensive operations during a translation
        are.
      * How we are going to deal with those operations, including:
              * What data structure is used
              * What start of day setup is required to enable this
              * What operations are therefore required at translation
                time

> > ### Filling the pITS Command Queue.
> > 
> > Various algorithms could be used here. For now a simple proposal is
> > to traverse the `pits.schedule_list` starting from where the last
> > refill finished (i.e not from the top of the list each time).
> > 
> > If a `vits_cq` has no pending commands then it is removed from the
> > list.
> > 
> > If a `vits_cq` has some pending commands then `min(pits-free-slots,
> > vits-outstanding, VITS_BATCH_SIZE)` will be taken from the vITS
> > command queue, translated and placed onto the pITS
> > queue. `vits_cq.progress` will be updated to reflect this.
> > 
> > Each `vits_cq` is handled in turn in this way until the pITS Command
> > Queue is full or there are no more outstanding commands.
> > 
> > There will likely need to be a data structure which shadows the pITS
> > Command Queue slots with references to the `vits_cq` which has a
> > command currently occupying that slot and corresponding the index into
> > the virtual command queue, for use when completing a command.
> > 
> > `VITS_BATCH_SIZE` should be small, TBD say 4 or 8.
> > 
> > Possible simplification: If we arrange that no guest ever has multiple
> > batches in flight (which can occur if we wrap around the list several
> > times) then we may be able to simplify the book keeping
> > required. However this may need some careful thought wrt fairness for
> > guests submitting frequent small batches of commands vs those sending
> > large batches.
> > 
> > XXX concern: Time spent filling the pITS queue could be significant if
> > guests are allowed to fill the ring completely.
> 
> I guess you sent this design before the end of the discussion?

Probably.

>  I think
> that limiting the number of batch/command sent per pass would allow a
> small pass.

I think we have a few choices:

      * Limit to one batch per vits at a time
      * Limit to some total number of batches per scheduling pass
      * Time bound the scheduling procedure

Do we have a preference?


> >   the underlying hardware to the guest.
> > * Adds complexity to the guest layout, which is right now static. How
> >   do you decide the number of vITS/root controller exposed:
> >     * Hotplug is tricky
> > * Toolstack needs greater knowledge of the host layout
> > * Given that PCI passthrough doesn't allow migration, maybe we could
> >   use the layout of the hardware.
> > 
> > In 1 vITS for all pITS:
> > 
> > * What to do with global commands? Inject to all pITS and then
> >   synchronise on them all finishing.
> > * Handling of out of order completion of commands queued with
> >   different pITS, since the vITS must appear to complete in
> >   order. Apart from the book keeping question it makes scheduling more
> >   interesting:
> >     * What if you have a pITS with slots available, and the guest command
> >       queue contains commands which could go to the pITS, but behind ones
> >       which are targetting another pITS which has no slots
> >     * What if one pITS is very busy and another is mostly idle and a
> >       guest submits one command to the busy one (contending with other
> >       guest) followed by a load of commands targeting the idle one. Those
> >       commands would be held up in this situation.
> >     * Reasoning about fairness may be harder.
> > 
> > XXX need a solution/decision here.
> 
> > In addition the introduction of direct interrupt injection in version
> > 4 GICs may imply a vITS per pITS. (Update: it seems not)
> 
> Other items to add: NUMA and I/O NUMA. I don't know much about it but I
> think the first solution would be more suitable.

first solution == ?

Ian.