From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ian Campbell Subject: Re: Xen on ARM vITS Handling Draft B (Was Re: Xen/arm: Virtual ITS command queue handling) Date: Tue, 19 May 2015 14:51:51 +0100 Message-ID: <1432043511.12989.132.camel@citrix.com> References: <1431442942.8263.175.camel@citrix.com> <1431690336.8943.97.camel@citrix.com> <555608C6.7000609@citrix.com> <1432037407.12989.103.camel@citrix.com> <555B3C7E.2030608@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <555B3C7E.2030608@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Julien Grall Cc: Vijay Kilari , Stefano Stabellini , Prasun Kapoor , manish.jaggi@caviumnetworks.com, Julien Grall , "xen-devel@lists.xen.org" , Stefano Stabellini List-Id: xen-devel@lists.xenproject.org On Tue, 2015-05-19 at 14:37 +0100, Julien Grall wrote: > Hi Ian, > > On 19/05/15 13:10, Ian Campbell wrote: > > On Fri, 2015-05-15 at 15:55 +0100, Julien Grall wrote: > > [...] > >>> Translation of certain commands can be expensive (XXX citation > >>> needed). > >> > >> The term "expensive" is subjective. I think we can end up to cheap > >> translation if we properly pre-allocate information (such as device, > >> LPIs...). We can have all the informations before the guest as boot or > >> during hotplug part. It wouldn't take more memory than it should use. > >> > >> During command translation, we would just need to enable the device/LPIs. > >> > >> The remaining expensive part would be the validation. I think we can > >> improve most of them of O(1) (such as collection checking) or O(log(n)) > >> (such as device checking). > > [...] > >>> XXX need a solution for this. > >> > >> Command translation can be improved. It may be good too add a section > >> explaining how translation of command foo can be done. > > > > I think that is covered by the spec, however if there are operations > > which form part of this which are potentially expensive we should > > outline in our design how this will be dealt with. > > > > Perhaps you or Vijay could propose some additional text covering: > > * What the potentially expensive operations during a translation > > are. > > * How we are going to deal with those operations, including: > > * What data structure is used > > * What start of day setup is required to enable this > > * What operations are therefore required at translation > > time > > I don't have much time to work on a proposal. I would be happy if Vijay > do it. OK, Vijay could you make a proposal here please. > > >> I think > >> that limiting the number of batch/command sent per pass would allow a > >> small pass. > > > > I think we have a few choices: > > > > * Limit to one batch per vits at a time > > * Limit to some total number of batches per scheduling pass > > * Time bound the scheduling procedure > > > > Do we have a preference? > > Time bound may be difficult to implement. Yes, I don't think that one is realistic. > I think we would have to limit > batch per vITS (for code simplification) and total number of batch per > scheduling pass at the same time. OK. > >>> the underlying hardware to the guest. > >>> * Adds complexity to the guest layout, which is right now static. How > >>> do you decide the number of vITS/root controller exposed: > >>> * Hotplug is tricky > >>> * Toolstack needs greater knowledge of the host layout > >>> * Given that PCI passthrough doesn't allow migration, maybe we could > >>> use the layout of the hardware. > >>> > >>> In 1 vITS for all pITS: > >>> > >>> * What to do with global commands? Inject to all pITS and then > >>> synchronise on them all finishing. > >>> * Handling of out of order completion of commands queued with > >>> different pITS, since the vITS must appear to complete in > >>> order. Apart from the book keeping question it makes scheduling more > >>> interesting: > >>> * What if you have a pITS with slots available, and the guest command > >>> queue contains commands which could go to the pITS, but behind ones > >>> which are targetting another pITS which has no slots > >>> * What if one pITS is very busy and another is mostly idle and a > >>> guest submits one command to the busy one (contending with other > >>> guest) followed by a load of commands targeting the idle one. Those > >>> commands would be held up in this situation. > >>> * Reasoning about fairness may be harder. > >>> > >>> XXX need a solution/decision here. > >> > >>> In addition the introduction of direct interrupt injection in version > >>> 4 GICs may imply a vITS per pITS. (Update: it seems not) > >> > >> Other items to add: NUMA and I/O NUMA. I don't know much about it but I > >> think the first solution would be more suitable. > > > > first solution == ? > > 1 vITS per pITS. Ah, yes.