relationship between cpu_affinity and Packet RX Processing

[gmate.amit@gmail.com (Kumar amit mehta)]

On Tue, Mar 26, 2013 at 09:31:01PM +0200, Rami Rosen wrote:
> Hi,
> 
> 1) You can be sure that it does not use more than one rx queue.
> 
> You can find more info about RX queues in
> http://media.wix.com/ugd//295986_4ef6dbdf11fd0a7f74f09741b4b5b2ee.pdf
> 
> (I wrote it, a shameless plug...)
> 
> 2) Does the irqbalance service, which is common in some distros, is
> running ? can you try to disable it and run traffic again and check
> /proc/interrupts afrer some time ?
> 
> 3) regardless of 2 (meaning not stopping irqbalance if it was active)
> can you please try "taskset 02 ping www.google.com" and tell us
> whether interrupts were incremented on the second CPU ? (please send
> /proc/interrupts | grep eth0 before and after sending, let's say, 5
> pings)?
>

My apologies for responding after a long time. I lost access to the machine on
which this VM was running. Now running this VM on my laptop, which seem to have
similar hardware configuration. Please find the details, which you had asked for
below:(comment in quotes)

amit at ubuntu:~$ cat /proc/cpuinfo |grep processor
processor	: 0
processor	: 1
"2 CPU"

amit at ubuntu:~$ cat /proc/interrupts |egrep 'CPU|eth0' 
           CPU0       CPU1       
 19:       7887         31   IO-APIC-fasteoi   eth0

amit at ubuntu:~$ ls -lah /sys/class/net/eth0/device/driver
lrwxrwxrwx 1 root root 0 Mar 29 01:13 /sys/class/net/eth0/device/driver ->
../../../../bus/pci/drivers/pcnet32

amit at ubuntu:~$ lsmod|grep -i pcnet32
pcnet32                40671  0 
"driver"

amit at ubuntu:~$ ls /sys/class/net/eth0/queues
rx-0  tx-0
"just one rx-queue"

amit at ubuntu:~$ sudo cat /proc/irq/19/smp_affinity
01
"tied to CPU0"

"disabling irqbalance"
amit at ubuntu:~$ status irqbalance
irqbalance start/running, process 1016

amit at ubuntu:~$ sudo stop irqbalance
irqbalance stop/waiting

amit at ubuntu:~$ taskset 02 ping www.google.com

"after sometime"
amit at ubuntu:~$ cat /proc/interrupts |egrep 'CPU|eth0' 
           CPU0       CPU1       
 19:       8046         31   IO-APIC-fasteoi   eth0

"No increase in interrupt counts on CPU1"

"Now lets use both CPUs"
root at ubuntu:/home/amit# echo "3" >/proc/irq/19/smp_affinity
root at ubuntu:/home/amit# cat /proc/irq/19/smp_affinity
03

"As before, start ping and expecting both CPUs to be used"
amit at ubuntu:~$ taskset 02 ping www.google.com

root at ubuntu:/home/amit# cat /proc/interrupts |egrep 'CPU|eth0'
	  CPU0       CPU1       
19:       9001         71   IO-APIC-fasteoi   eth0
                      ^^^^
and yes, indeed..
So multiple Rx queues seem to have nothing to do with CPU affinity, as in this
case I have a driver implementing a single Rx queue and I see that the rx packet
processing happens through both of the CPUs.

-Amit