Re: SMP Affinity Issue with IRQ#

[Robert Hancock <hancockr@shaw.ca>]

Khaled Al-Hamwi wrote:
> Hi list,
> 
> I am trying to evaluate the performance of a Linux box when working as
> IP forwarder.
> I have a hardware traffic generator IXIA T400 on one end.
> On the other end, I have a Linux box with kernel 2.6.16 and quad-core xeon CPU.
> 
> These two systems are connected through two crossover Gigabit cables and NICs.
> The traffic is sent from IXIA through one NIC to the forwarding machine
> and then it is forwarded back through the other NIC to IXIA.
> 
> I have two issues related to SMP affinity.
> The first one is that when I set the SMP affinity through
> /proc/irq/<IRQ#>/smp_affinity, it changes dynamically.
> Is there any load balancing in the system that changes the affinity
> after some time or after some packets are received?
> Is there a way to set it permanently?

Likely the irqbalance daemon is doing it. You should be able to stop it. 
(Or you have the kernel IRQ balancing config option enabled, which you 
likely shouldn't.)

> 
> The second issue is that changing the SMP affinity results in
> different delay and throughput measurements.
> I am assigning each NIC to a different CPU.
> If I used a different assignment but still each NIC is assigned to a
> different CPU, I am getting different performance results.
> I would expect that changing the assignment should yield the same
> performance results.
> The CPUs are identical and should have similar performance.
> Here, I have two examples of two different assignments:
> Example 1:
> /proc/irq/16/smp_affinity (eth0)  ->   CPU#1
> /proc/irq/20/smp_affinity (eth1)  ->   CPU#2
> Example 2:
> /proc/irq/16/smp_affinity (eth0)  ->   CPU#3
> /proc/irq/20/smp_affinity (eth1)  ->   CPU#2
> Which one of these two configurations can be used as a reference for
> performance evaluation?
> 
> Any ideas??

I believe the current Intel quad core CPUs are really two dual-core CPU 
dies in a common package. Splitting the network interrupts across the 
two physical CPU chips will likely give you worse performance.