Re: Xen/arm: Virtual ITS command queue handling

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-05-15 at 14:44 +0100, Julien Grall wrote:
> On 15/05/15 14:24, Ian Campbell wrote:
> > On Fri, 2015-05-15 at 18:44 +0530, Vijay Kilari wrote:
> >> On Fri, May 15, 2015 at 6:23 PM, Ian Campbell <ian.campbell@citrix.com> wrote:
> >>> On Fri, 2015-05-15 at 18:17 +0530, Vijay Kilari wrote:
> >>>> On Fri, May 15, 2015 at 5:33 PM, Julien Grall <julien.grall@citrix.com> wrote:
> >>>>> On 15/05/15 12:30, Ian Campbell wrote:
> >>>>>>> Handling of Single vITS and multipl pITS can be made simple.
> >>>>>>>
> >>>>>>> All ITS commands except SYNC & INVALL has device id which will
> >>>>>>> help us to know to which pITS it should be sent.
> >>>>>>>
> >>>>>>> SYNC & INVALL can be dropped by Xen on Guest request
> >>>>>>>  and let Xen append where ever SYNC & INVALL is required.
> >>>>>>> (Ex; Linux driver adds SYNC for required commands).
> >>>>>>> With this assumption, all ITS commands are mapped to pITS
> >>>>>>> and no need of synchronization across pITS
> >>>>>>
> >>>>>> You've ignored the second bullet its three sub-bullets, I think.
> >>>>>
> >>>>    Why can't we group the batch of commands based on pITS it has
> >>>> to be sent?.
> >>>
> >>> Are you suggesting that each batch we send should be synchronous? (i.e.
> >>> end with SYNC+INT) That doesn't seem at all desirable.
> >>
> >> Not only at the end of batch, SYNC can be appended based on every
> >> command within the batch.
> > 
> > Could be, but something to avoid I think?
> 
> That would slow down the ITS processing (SYNC is waiting that the
> previous command has executed).
> 
> Also, what about INTALL? Sending it everytime would be horrible for the
> performance because it flush the ITS cache.
> 
> >> Also to handle second bullet, where a batch of commands might be
> >> sent on multple pITS. In that case batch of ITS commands is split
> >> across pITS and we have
> >> to wait for all the pITS to complete. Managing this would be difficult.
> >> For this I propose, batch can be created/split such that each batch
> >> contains commands related to one pITS. But it leads to small batch of commands.
> 
> If I understand correctly, even with multiple pITS only a single batch
> per domain would be in-flight, right?
> 
> > That's not a bad idea, commonly I would expect commands for one device
> > to come in a short batch anyway. So long as the thing does cope if not I
> > think this might work.
> 
> This doesn't work well, we will need to read/validate twice a command.
> The first time to get the devID and notice we need to create a separate
> batch, the second time to effectively queue the command.
> 
> Given that validation is the part where the emulation will spend most of
> the time, we should avoid to do it twice.

Which can trivially be arranged by not doing it the dumb way. At worst
you remember the first translation which mismatched and use it again
next time.

Or you do translates in batches into a queue and then dequeue into the
physical command queue based on the target devices.

Thinking about global commands a bit, you could make those somewhat less
painful by remembering on a per `vits_cq` basis which pits devices it
has sent commands to since the last invalidate on that device and elide
any where the guest didn't touch that pits. Doesn't help against a
malicious guest in the worst case but does improve things in the common
case.

> Although, if we cache the validation we may send the wrong command/data
> if the guest decide to write in the command queue at the same time.

A guest which modifies its command queue after having advanced CWRITER
past that point deserves whatever it gets.

Ian.