Re: IRQ handling difference between i386 and x86_64

[Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>]

Arjan van de Ven wrote:
> On Sat, 2007-06-30 at 16:55 +0200, Krzysztof Oledzki wrote:
>> Hello,
>>
>> It seems that IRQ handling is somehow different between i386 and x86_64.
>>
>> In my Dell PowerEdge 1950 is it possible to enable interrupts spreading 
>> over all CPUs. This a single CPU, four CORE system (Quad-Core E5335 Xeon) 
>> so I think that interrupts migration may be useful. Unfortunately, it 
>> works only with 32-bit kernel. Booting it with x86_64 leads to situation, 
>> when all interrupts goes only to the first cpu matching a smp_affinity 
>> mask.
> 
> arguably that is the most efficient behavior... round robin of
> interrupts is the worst possible case in terms of performance
> 
> are you using irqbalance ? (www.irqbalance.org)

If you have not been using irqbalance, then setting more bits in
irq/smp_affinity will utilize hardware APIC routing. Sending interrupt
to more than one CPU will work in flat mode only.

In 32 bit kernel the IOAPIC is configured in flat mode while in 64 bit
it is in physical flat mode.  Physical flat mode will support
interrupt routing to only one CPU.

32-bit: Enabling APIC mode:  Flat.  Using 2 I/O APICs
64-bit: Setting APIC routing to physical flat

This is the reason for the observed interrupt routing behavior.  I
guess future kernels will use 'physical flat' mode to avoid hardware
bugs on various complex configurations and also make it scale up to
larger systems.  Also utilizing hardware routing to send interrupts to
many CPUs may not provide desired behavior.

irqbalance application will do the needful.  I agree with Arjan on the
fact that equal interrupt distribution need not provide best
performance.  You will run into complex routing issues only when you
have more NICs than actual number of CPU cores.

--Vaidy