Two Dual Core processors and NICS (not handling interrupts on one CPU/assigning a Two Dual Core processors and NICS (not handling interrupts on one CPU / assigning a CPU to a NIC)

Two Dual Core processors and NICS (not handling interrupts on one CPU/assigning a Two Dual Core processors and NICS (not handling interrupts on one CPU / assigning a CPU to a NIC)

[Mark Ryden]

Hello,


I have a machine with 2 dual core CPUs. This machine runs Fedora Core 6.
I have two Intel e1000 GigaBit network cards on this machine; I use bonding so
that the machine assigns the same IP address to both NICs ;
It seems to me that bonding is configured OK, bacuse when running:
"cat /proc/net/bonding/bond0"
I get:

Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 1
Permanent HW addr: .....

Slave Interface: eth1
MII Status: up
Link Failure Count: 1
Permanent HW addr: ....

(And the Permanent HW addr is diffenet in these two entries).

I send a large amount of packets to this machine (more than 20,000 in
a second).

cat /proc/interrupts shops something like this:
				CPU0       CPU1         CPU2         CPU3
 50:    3359337          0          0          0         PCI-MSI  eth0
 58:         49    3396136          0          0         PCI-MSI  eth1

 CPU0 and CPU1 are of the first CPU as far as I understand ; so
 this means as far as I understand that the second CPU (which has CPU3
and CPU4) does not handle
 interrupts of the arrived packets; Can I somehow change it so the second
 CPU will also handle network interrupts of receiving packets on the nic ?

Can I assign one CPU to  eth0 and the second CPU to eth1  ?

Regards,
Mark

Re: Two Dual Core processors and NICS (not handling interrupts on one CPU/assigning a Two Dual Core processors and NICS (not handling interrupts on one CPU / assigning a CPU to a NIC)

[Robert Iakobashvili]

Hi Mark,

On 1/15/07, Mark Ryden <markryde@gmail.com> wrote:
> I have a machine with 2 dual core CPUs. This machine runs Fedora Core 6.
> I have two Intel e1000 GigaBit network cards on this machine; I use bonding so
> that the machine assigns the same IP address to both NICs ;

> cat /proc/interrupts shops something like this:
>                                 CPU0       CPU1         CPU2         CPU3
>  50:    3359337          0          0          0         PCI-MSI  eth0
>  58:         49    3396136          0          0         PCI-MSI  eth1
>
>  CPU0 and CPU1 are of the first CPU as far as I understand ; so
>  this means as far as I understand that the second CPU (which has CPU3
> and CPU4) does not handle
>  interrupts of the arrived packets; Can I somehow change it so the second
>  CPU will also handle network interrupts of receiving packets on the nic ?
>
> Can I assign one CPU to  eth0 and the second CPU to eth1  ?

How it will help you?
Y can set smp-affinity mask for each irq in /proc/<irq-number>/
google for 'linux smp-affinity".

The subject in more details is discussed in:
http://linux-net.osdl.org/index.php/TODO#TCP
and thread
http://marc.theaimsgroup.com/?t=116695290600001&r=1&w=2, read from
bottom.


-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...................................................................
Navigare necesse est, vivere non est necesse
...................................................................
http://sourceforge.net/projects/curl-loader
A powerful open-source HTTP/S, FTP/S traffic
generating, loading and testing tool.

Re: Two Dual Core processors and NICS (not handling interrupts on one CPU/assigning a Two Dual Core processors and NICS (not handling interrupts on one CPU / assigning a CPU to a NIC)

[Auke Kok]

Mark Ryden wrote:
> Hello,
> 
> 
> I have a machine with 2 dual core CPUs. This machine runs Fedora Core 6.
> I have two Intel e1000 GigaBit network cards on this machine; I use 
> bonding so
> that the machine assigns the same IP address to both NICs ;
> It seems to me that bonding is configured OK, bacuse when running:
> "cat /proc/net/bonding/bond0"
> I get:
> 
> Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)
> 
> Bonding Mode: load balancing (round-robin)
> MII Status: up
> MII Polling Interval (ms): 100
> Up Delay (ms): 0
> Down Delay (ms): 0
> 
> Slave Interface: eth0
> MII Status: up
> Link Failure Count: 1
> Permanent HW addr: .....
> 
> Slave Interface: eth1
> MII Status: up
> Link Failure Count: 1
> Permanent HW addr: ....
> 
> (And the Permanent HW addr is diffenet in these two entries).
> 
> I send a large amount of packets to this machine (more than 20,000 in
> a second).
> 
> cat /proc/interrupts shops something like this:
>                 CPU0       CPU1         CPU2         CPU3
> 50:    3359337          0          0          0         PCI-MSI  eth0
> 58:         49    3396136          0          0         PCI-MSI  eth1
> 
> CPU0 and CPU1 are of the first CPU as far as I understand ; so
> this means as far as I understand that the second CPU (which has CPU3
> and CPU4) does not handle
> interrupts of the arrived packets; Can I somehow change it so the second
> CPU will also handle network interrupts of receiving packets on the nic ?
> 
> Can I assign one CPU to  eth0 and the second CPU to eth1  ?

you will most likely have better performance from the shared cache on the core 2 duo by 
keeping it the way that it is right now - packets that need to transverse the bridge now 
make the cpus happy because after receive the sending NIC already has the data in it's 
cache. Moving one of the NICs over to cpu2/cpu3 would cause a cascade of cache misses 
for every packet that passes across the two nics in the bridge.

Cheers,

Auke

Re: Two Dual Core processors and NICS (not handling interrupts on one CPU/assigning a Two Dual Core processors and NICS (not handling interrupts on one CPU / assigning a CPU to a NIC)

[Rick Jones]

Mark Ryden wrote:
> Hello,
> 
> 
> I have a machine with 2 dual core CPUs. This machine runs Fedora Core 6.
> I have two Intel e1000 GigaBit network cards on this machine; I use 
> bonding so
> that the machine assigns the same IP address to both NICs ;
> It seems to me that bonding is configured OK, bacuse when running:
> "cat /proc/net/bonding/bond0"
> I get:
> ...
> Permanent HW addr: ....
> 
> (And the Permanent HW addr is diffenet in these two entries).
> 
> I send a large amount of packets to this machine (more than 20,000 in
> a second).

Well, 20K a second is large in some contexts, but not in others :)
> 
> cat /proc/interrupts shops something like this:
>                 CPU0       CPU1         CPU2         CPU3
> 50:    3359337          0          0          0         PCI-MSI  eth0
> 58:         49    3396136          0          0         PCI-MSI  eth1
> 
> CPU0 and CPU1 are of the first CPU as far as I understand ; so this
> means as far as I understand that the second CPU (which has CPU3 and
> CPU4) does not handle interrupts of the arrived packets; Can I
> somehow change it so the second
> CPU will also handle network interrupts of receiving packets on the
> nic ?

Actually, those could be different chips - it depends on the CPUs I 
think, and I suppose the BIOS/OS.  On a Woodcrest system with which I've 
been playing, CPUs 0 and 2 appear to be on the same die, then 1 and 
three.   I ass-u-me-d the numbering was that way to get maximum 
processor cache when saying "numcpu=N" for something less than the 
number of cores in the system.

NUMA considerations might come into play if this is Opteron (well, any 
NUMA system really - larger IA64's, certain SPARC and Power systems 
etc...).  In broad handwaving terms, one is better-off with the NICs 
interrupts being handled by the topologically closest CPU.  (Not that 
some irqbalancer programs recognize that just yet :)

Now, if both CPU0 and CPU1 are saturated it might make sense to put some 
interrupts on 2 and/or 3.  One of those fun "it depends" situations.

rick jones