Re: [PATCH 2/2] x86: implement multiple queues for smp function call IPIs

[Jeremy Fitzhardinge <jeremy@goop.org>]

Ingo Molnar wrote:
>
> heh, nice :-)
>
> Before going into all the fine details an trying our luck in tip/master 
> QA, i'm a bit worried about hw compatibility in general though. APICs 
> have been flaky since the beginnings of times. We had erratas in the 
> area of local timer IRQs(IPIs) overlapping with IPIs, etc. - so i'd not 
> bet the farm on all APICs being able to handle a _lot_ more overlapped 
> inter-CPU IPIs than we do currently. (which basically was just three of 
> them until now, and now four with the new SMP cross-call IPIs)
>
> So this _has_ to be approached defensively. It _should_ work, and i'm 
> all in favor of utilizing hardware resources more fully, but it's an 
> entirely new mode of operation for the hardware. I think a Kconfig 
> option (which defaults to off), and a boot option to disable it would be 
> nice, so that we can introduce this gently, at least initially. Then 
> when we see that it's 100% trouble-free we can flip around the default.
>   

As Andi pointed out, this is more or less functionally identical to the 
code I ripped out of tlb_64.c, so this mode of operation has had lots of 
exposure on the 64-bit side.  Because the number of queues is a CONFIG 
variable, it would be relatively easy to make it a real config option, 
and/or use different numbers for 32 and 64-bit.  Choosing 1 as the 
number of queues will make it behave exactly as the current code does.

I'm not really familiar with all the ins and outs of apic bugs.  What's 
the issue you're concerned about?

> Plus, would it be possible to shape this a bit more dynamically? I like 
> 8 as a nice round number, but i bet some folks would like to have 16, 
> some would like to have 4 ... Perhaps even making it dynamic (so that we 
> can turn it all off in the case of trouble with certain CPU/APIC 
> versions).
>
> Hm?
>   

Sure, that's possible in a followup patch.  There's a pile of repeated 
boilerplate code which would need to be cleaned up to make it 
configurable and/or runtime changable.  Also, it would need some way to 
allocate a contiguous block of vectors; I'm not sure if the just-posted 
SGI patch allows that...

It also occurred to me that it might be more interesting to parameterise 
the queues - and the mapping of cpus->queues - in a more topology-aware 
way than simply NQUEUES=NCPUS/x, queue=cpuid % NQUEUES.  But I haven't 
given it much thought.

    J