Re: [RFC PATCH 21/24] ARM: vITS: handle INVALL command

[Julien Grall <julien.grall@arm.com>]

Hi Dario,

On 06/12/2016 22:39, Dario Faggioli wrote:
> On Tue, 2016-12-06 at 13:53 -0800, Stefano Stabellini wrote:
>> On Tue, 6 Dec 2016, Dario Faggioli wrote:
>>> Sorry if I can't be more useful than this for now. :-/
>>
>> We don't need scheduler support to implement interrupt migration. The
>> question was much simpler than that: moving a vCPU with interrupts
>> assigned to it is slower than moving a vCPU without interrupts
>> assigned
>> to it. You could say that the slowness is directly proportional do
>> the
>> number of interrupts assigned to the vCPU. Does the scheduler know
>> that?
>> Or blindly moves vCPUs around? Also see below.
>>
> Ah, ok, it is indeed a simpler question than I thought! :-)
>
> Answer: no, the scheduler does not use the information of how many or
> what interrupts are being routed to a vCPU in any way.
>
> Just for the sake of correctness and precision, it does not "blindly
> moves vCPUs around", as in, it follows some criteria for deciding
> whether or not to move a vCPU, and if yes, where to, but among those
> criteria, there is no trace of anything related to routed interrupts.
>
> Let me also add that the criteria are scheduler specific, so they're
> different, e.g., between Credit and Credit2.
>
> Starting considering routed interrupt as a migration criteria in Credit
> would be rather difficult. Credit use a 'best effort' approach for
> migrating vCPUs, which is hard to augment.
>
> Starting considering routed interrupt as a migration criteria in
> Credit2 would be much easier. Credit2's load balancer is specifically
> designed for being extendible with things like that. It would require
> some thinking, though, in order to figure out how important this
> particular aspect would be, wrt others that are considered.
>
> E.g., if I have pCPU 0 loaded at 75% and pCPU 1 loaded at 25%, vCPU A
> has a lot of routed interrupts, and moving it gives me perfect load
> balancing (i.e., load will become 50% on pCPU 0 and 50% on pCPU 1)
> should I move it or not?
> Well, it depends if whether or not we think that the overhead we save
> by not migrating outweights the benefit of a perfectly balanced system.
>
> Something like that...

This idea to migrate IRQ while migrating vCPU is already wrong to me as 
this is really expensive, I am talking in term of milliseconds easily as 
in some case Xen will have to interact with a shared command queue.

Spreading the load is much better and avoid to migrate directly IRQ that 
are disabled when the vCPU is been migrated.

I cannot see any reason to not do that as changing the interrupt 
affinity may be not taken into account directly. By that, I mean it 
might be possible to receive another interrupt on the wrong pCPU.

I really think we should make the vCPU migration much simpler (e.g avoid 
this big loop over interrupt). In fine, if we really expect the 
scheduler to migrate the vCPU on a different pCPU. We should also expect 
receiving the interrupt on the wrong pCPU may not happen often.

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel