relationship between cpu_affinity and Packet RX Processing

* relationship between cpu_affinity and Packet RX Processing
@ 2013-03-26 18:31 Kumar amit mehta
  2013-03-26 18:35 ` Rami Rosen
  2013-03-26 18:45 ` Arlie Stephens
  0 siblings, 2 replies; 10+ messages in thread
From: Kumar amit mehta @ 2013-03-26 18:31 UTC (permalink / raw)
  To: kernelnewbies

Hi All,

I was reading some stuff on interrupts and irq lines today and I thought I'll
expermient with the network rx path. To start with, I've a Virtual Machine
running 3.8 linux kernel. My machine has 4 CPU cores, network (eth) interface 
is driven by pcnet_32 AMD driver and is tied to IRQ line #19. I started some
network traffic and I notice that out of those 4 CPUs, only one of them is being
used and despite changing the CPU affinity, I still don't see the other cores
being used for this network traffic. So based on this behavior(please see the
logs below), I've these following queries:
i) Does it mean that this network card do not have multiple Rx Queues ?
ii) I think all the modern NICs must be implementing multiple Rx Queues and
hence Can someone please point me to the simplest of such implemenation in any
of the in-tree drivers ?
iii) I'm just doing a simple 'ping' to google with of big size packets, As I do 
not have a peer to use packetgen/netperf/iperf utilities.

ref: Comments in double quotes.
<logs>
$ uname -r
3.8.0-rc6-next-20130208

$ cat /proc/cpuinfo |grep processor
processor	: 0
processor	: 1
processor	: 2
processor	: 3
"Total 4 cpu cores"

$ cat /proc/interrupts|egrep 'eth0|CPU'
      	CPU0       CPU1       CPU2       CPU3       
19:     5103         74         33          5   IO-APIC-fasteoi 	eth0
"IRQ Line #19 for the network device"

$ lspci|grep -i ethernet
02:01.0 Ethernet controller: Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]
(rev 10)

"AMD NIC"
$ lspci -s 02:01.0 -vvv|grep module
	Kernel modules: pcnet32

$ lsmod|grep pcnet32
pcnet32                40671  0
"driver"

# whoami
root

# cat /proc/irq/19/smp_affinity
03

# cat /proc/irq/19/affinity_hint 
00

"I think smp_affinity is a bit map, therefore for all the four cores to be
utilized, all 4 bits should be set to 1, which leads to 15(0xf), hence Chaning
the cpu affinity"
# echo 15 > /proc/irq/19/smp_affinity
# cat /proc/irq/19/smp_affinity
15 

"started network traffic here and monitoring it" 
# cat /proc/interrupts|grep eth0
19:       5452         78         33          5   IO-APIC-fasteoi   eth0
# cat /proc/interrupts|grep eth0
19:       5488         78         35          5   IO-APIC-fasteoi   eth0
# cat /proc/interrupts|grep eth0
19:       5492         78         35          5   IO-APIC-fasteoi   eth0
# cat /proc/interrupts|grep eth0
19:       5500         78         35          5   IO-APIC-fasteoi   eth0
.................................
.........after some time.........
# cat /proc/interrupts|grep eth0
19:       6035         78         42          5   IO-APIC-fasteoi   eth0

"Most of the packets are still getting routed through CPU0 *Only*."
<logs>

-Amit

^ permalink raw reply	[flat|nested] 10+ messages in thread