Re: [PATCH 2/2] x86: implement multiple queues for smp function call IPIs

[Ingo Molnar <mingo@elte.hu>]

* Jeremy Fitzhardinge <jeremy@goop.org> wrote:

> This adds 8 queues for smp_call_function(), in order to avoid a 
> bottleneck on a single global lock and list for function calls.  When 
> initiating a function call, the sender chooses a queue based on its 
> own processor id (if there are more than 8 processors, they hash down 
> to 8 queues).  It then sends an IPI to the corresponding vector for 
> that queue to each target CPU.  The target CPUs use the vector number 
> to determine which queue they should scan for work.
>
> This should give smp_call_function the same performance 
> characteristics as the original x86-64 cross-cpu tlb flush code, which 
> used the same scheme.

heh, nice :-)

Before going into all the fine details an trying our luck in tip/master 
QA, i'm a bit worried about hw compatibility in general though. APICs 
have been flaky since the beginnings of times. We had erratas in the 
area of local timer IRQs(IPIs) overlapping with IPIs, etc. - so i'd not 
bet the farm on all APICs being able to handle a _lot_ more overlapped 
inter-CPU IPIs than we do currently. (which basically was just three of 
them until now, and now four with the new SMP cross-call IPIs)

So this _has_ to be approached defensively. It _should_ work, and i'm 
all in favor of utilizing hardware resources more fully, but it's an 
entirely new mode of operation for the hardware. I think a Kconfig 
option (which defaults to off), and a boot option to disable it would be 
nice, so that we can introduce this gently, at least initially. Then 
when we see that it's 100% trouble-free we can flip around the default.

Plus, would it be possible to shape this a bit more dynamically? I like 
8 as a nice round number, but i bet some folks would like to have 16, 
some would like to have 4 ... Perhaps even making it dynamic (so that we 
can turn it all off in the case of trouble with certain CPU/APIC 
versions).

Hm?

	Ingo