* SMP code / network stack
@ 2008-01-10 14:05 Jeba Anandhan
2008-01-10 14:45 ` Eric Dumazet
0 siblings, 1 reply; 5+ messages in thread
From: Jeba Anandhan @ 2008-01-10 14:05 UTC (permalink / raw)
To: netdev; +Cc: matthew.hattersley
Hi All,
If a server has multiple processors and N number of ethernet cards, is
it possible to handle transmission by each processor separately? .In
other words, each processor will be responsible for tx of few ethernet
cards?.
Example: Server has 4 processors and 8 ethernet cards. is it possible
for each processor for transmission using 2 ethernet cards only?. So
that, at a instant , data will be send out from 8 ethernet cards.
Thanks
Jeba
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: SMP code / network stack
2008-01-10 14:05 SMP code / network stack Jeba Anandhan
@ 2008-01-10 14:45 ` Eric Dumazet
2008-01-10 15:26 ` Jeba Anandhan
0 siblings, 1 reply; 5+ messages in thread
From: Eric Dumazet @ 2008-01-10 14:45 UTC (permalink / raw)
To: Jeba Anandhan; +Cc: netdev, matthew.hattersley
On Thu, 10 Jan 2008 14:05:46 +0000
Jeba Anandhan <jeba.anandhan@vaioni.com> wrote:
> Hi All,
>
> If a server has multiple processors and N number of ethernet cards, is
> it possible to handle transmission by each processor separately? .In
> other words, each processor will be responsible for tx of few ethernet
> cards?.
>
>
>
> Example: Server has 4 processors and 8 ethernet cards. is it possible
> for each processor for transmission using 2 ethernet cards only?. So
> that, at a instant , data will be send out from 8 ethernet cards.
Hi Jeba
Modern ethernet cards have a big TX queue, so that even one CPU is enough
to keep several cards busy in //
You can check /proc/interrupts and change /proc/irq/*/smp_affinities to direct IRQ to
particular cpus, but transmit is usually trigered by processes that might run on different
cpus.
If all ethernet cards are on the same IRQ, then you might have a problem...
Example on a dual processor :
# cat /proc/interrupts
CPU0 CPU1
0: 11472559 74291833 IO-APIC-edge timer
2: 0 0 XT-PIC cascade
8: 0 1 IO-APIC-edge rtc
81: 0 0 IO-APIC-level ohci_hcd
97: 1830022231 847 IO-APIC-level ehci_hcd, eth0
121: 163095662 166443627 IO-APIC-level libata
NMI: 0 0
LOC: 85887285 85887193
ERR: 0
MIS: 0
You can see eth0 is on IRQ 97
Then :
# cat /proc/irq/97/smp_affinity
00000001
# echo 2 >/proc/irq/97/smp_affinity
# grep 97 /proc/interrupts
97: 1830035216 2259 IO-APIC-level ehci_hcd, eth0
# sleep 10
# grep 97 /proc/interrupts
97: 1830035216 5482 IO-APIC-level ehci_hcd, eth0
You can see only CPU1 is now handling IRQ 97 (but CPU0 is allowed to give to eth0 some transmit work)
You might want to check /proc/net/softnet_stat too.
If your server is doing something very special (network trafic, no disk accesses or number crunching),
you might need to bind application processes to cpus, not only network irqs.
process A, using nic eth0 & eth1, bound to CPU 0 (process and IRQs)
process B, using nic eth2 & eth3, bound to CPU 1
process C, using nic eth4 & eth5, bound to CPU 2
process D, using nic eth6 & eth7, bound to CPU 3
Also, take a look at "ethtool -c ethX" command
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: SMP code / network stack
2008-01-10 14:45 ` Eric Dumazet
@ 2008-01-10 15:26 ` Jeba Anandhan
2008-01-10 17:46 ` Arnaldo Carvalho de Melo
0 siblings, 1 reply; 5+ messages in thread
From: Jeba Anandhan @ 2008-01-10 15:26 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev, matthew.hattersley
Hi Eric,
Thanks for the reply. I have one more doubt. For example, if we have 2
processor and 4 ethernet cards. Only CPU0 does all work through 8 cards.
If we set the affinity to each ethernet card as CPU number, will it be
efficient?.
Will this be default behavior?
# cat /proc/interrupts
CPU0 CPU1
0: 11472559 74291833 IO-APIC-edge timer
2: 0 0 XT-PIC cascade
8: 0 1 IO-APIC-edge rtc
81: 0 0 IO-APIC-level ohci_hcd
97: 1830022231 847 IO-APIC-level ehci_hcd, eth0
97: 3830012232 847 IO-APIC-level ehci_hcd, eth1
97: 5830052231 847 IO-APIC-level ehci_hcd, eth2
97: 6830032213 847 IO-APIC-level ehci_hcd, eth3
#sleep 10
# cat /proc/interrupts
CPU0 CPU1
0: 11472559 74291833 IO-APIC-edge timer
2: 0 0 XT-PIC cascade
8: 0 1 IO-APIC-edge rtc
81: 0 0 IO-APIC-level ohci_hcd
97: 2031409801 847 IO-APIC-level ehci_hcd, eth0
97: 4813981390 847 IO-APIC-level ehci_hcd, eth1
97: 7123982139 847 IO-APIC-level ehci_hcd, eth2
97: 8030193010 847 IO-APIC-level ehci_hcd, eth3
Instead of the above mentioned ,if we set the affinity for eth2 and
eth3.
the output will be
# cat /proc/interrupts
CPU0 CPU1
0: 11472559 74291833 IO-APIC-edge timer
2: 0 0 XT-PIC cascade
8: 0 1 IO-APIC-edge rtc
81: 0 0 IO-APIC-level ohci_hcd
97: 1830022231 847 IO-APIC-level ehci_hcd, eth0
97: 3830012232 847 IO-APIC-level ehci_hcd, eth1
97: 5830052231 923 IO-APIC-level ehci_hcd, eth2
97: 6830032213 1230 IO-APIC-level ehci_hcd, eth3
#sleep 10
# cat /proc/interrupts
CPU0 CPU1
0: 11472559 74291833 IO-APIC-edge timer
2: 0 0 XT-PIC cascade
8: 0 1 IO-APIC-edge rtc
81: 0 0 IO-APIC-level ohci_hcd
97: 2300022231 847 IO-APIC-level ehci_hcd, eth0
97: 4010212232 847 IO-APIC-level ehci_hcd, eth1
97: 5830052231 1847 IO-APIC-level ehci_hcd, eth2
97: 6830032213 2337 IO-APIC-level ehci_hcd, eth3
In this case, will the performance improves?.
Thanks
Jeba
On Thu, 2008-01-10 at 15:45 +0100, Eric Dumazet wrote:
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: SMP code / network stack
2008-01-10 15:26 ` Jeba Anandhan
@ 2008-01-10 17:46 ` Arnaldo Carvalho de Melo
2008-01-10 18:31 ` Kok, Auke
0 siblings, 1 reply; 5+ messages in thread
From: Arnaldo Carvalho de Melo @ 2008-01-10 17:46 UTC (permalink / raw)
To: Jeba Anandhan; +Cc: Eric Dumazet, netdev, matthew.hattersley
Em Thu, Jan 10, 2008 at 03:26:59PM +0000, Jeba Anandhan escreveu:
> Hi Eric,
> Thanks for the reply. I have one more doubt. For example, if we have 2
> processor and 4 ethernet cards. Only CPU0 does all work through 8 cards.
> If we set the affinity to each ethernet card as CPU number, will it be
> efficient?.
>
> Will this be default behavior?
>
> # cat /proc/interrupts
> CPU0 CPU1
> 0: 11472559 74291833 IO-APIC-edge timer
> 2: 0 0 XT-PIC cascade
> 8: 0 1 IO-APIC-edge rtc
> 81: 0 0 IO-APIC-level ohci_hcd
> 97: 1830022231 847 IO-APIC-level ehci_hcd, eth0
> 97: 3830012232 847 IO-APIC-level ehci_hcd, eth1
> 97: 5830052231 847 IO-APIC-level ehci_hcd, eth2
> 97: 6830032213 847 IO-APIC-level ehci_hcd, eth3
> #sleep 10
>
> # cat /proc/interrupts
> CPU0 CPU1
> 0: 11472559 74291833 IO-APIC-edge timer
> 2: 0 0 XT-PIC cascade
> 8: 0 1 IO-APIC-edge rtc
> 81: 0 0 IO-APIC-level ohci_hcd
> 97: 2031409801 847 IO-APIC-level ehci_hcd, eth0
> 97: 4813981390 847 IO-APIC-level ehci_hcd, eth1
> 97: 7123982139 847 IO-APIC-level ehci_hcd, eth2
> 97: 8030193010 847 IO-APIC-level ehci_hcd, eth3
>
>
> Instead of the above mentioned ,if we set the affinity for eth2 and
> eth3.
> the output will be
>
> # cat /proc/interrupts
> CPU0 CPU1
> 0: 11472559 74291833 IO-APIC-edge timer
> 2: 0 0 XT-PIC cascade
> 8: 0 1 IO-APIC-edge rtc
> 81: 0 0 IO-APIC-level ohci_hcd
> 97: 1830022231 847 IO-APIC-level ehci_hcd, eth0
> 97: 3830012232 847 IO-APIC-level ehci_hcd, eth1
> 97: 5830052231 923 IO-APIC-level ehci_hcd, eth2
> 97: 6830032213 1230 IO-APIC-level ehci_hcd, eth3
> #sleep 10
>
> # cat /proc/interrupts
> CPU0 CPU1
> 0: 11472559 74291833 IO-APIC-edge timer
> 2: 0 0 XT-PIC cascade
> 8: 0 1 IO-APIC-edge rtc
> 81: 0 0 IO-APIC-level ohci_hcd
> 97: 2300022231 847 IO-APIC-level ehci_hcd, eth0
> 97: 4010212232 847 IO-APIC-level ehci_hcd, eth1
> 97: 5830052231 1847 IO-APIC-level ehci_hcd, eth2
> 97: 6830032213 2337 IO-APIC-level ehci_hcd, eth3
>
> In this case, will the performance improves?.
ps ax | grep irqbalance
tells what?
If it is enabled please try:
service irqbalance stop
chkconfig irqbalance off
Then reset the smp_affinity entries to ff so and try again.
http://www.irqbalance.org/
- Arnaldo
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: SMP code / network stack
2008-01-10 17:46 ` Arnaldo Carvalho de Melo
@ 2008-01-10 18:31 ` Kok, Auke
0 siblings, 0 replies; 5+ messages in thread
From: Kok, Auke @ 2008-01-10 18:31 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Jeba Anandhan, Eric Dumazet, netdev,
matthew.hattersley
Arnaldo Carvalho de Melo wrote:
> Em Thu, Jan 10, 2008 at 03:26:59PM +0000, Jeba Anandhan escreveu:
>> Hi Eric,
>> Thanks for the reply. I have one more doubt. For example, if we have 2
>> processor and 4 ethernet cards. Only CPU0 does all work through 8 cards.
>> If we set the affinity to each ethernet card as CPU number, will it be
>> efficient?.
>>
>> Will this be default behavior?
>>
>> # cat /proc/interrupts
>> CPU0 CPU1
>> 0: 11472559 74291833 IO-APIC-edge timer
>> 2: 0 0 XT-PIC cascade
>> 8: 0 1 IO-APIC-edge rtc
>> 81: 0 0 IO-APIC-level ohci_hcd
>> 97: 1830022231 847 IO-APIC-level ehci_hcd, eth0
>> 97: 3830012232 847 IO-APIC-level ehci_hcd, eth1
>> 97: 5830052231 847 IO-APIC-level ehci_hcd, eth2
>> 97: 6830032213 847 IO-APIC-level ehci_hcd, eth3
another thing to try: if you don't need usb2 support, remove the ehci_hcd module -
this will give a slight less overhead servicing irq's in your system.
I take it that you have no MSI support in these ethernet cards?
Auke
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2008-01-10 18:32 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-01-10 14:05 SMP code / network stack Jeba Anandhan
2008-01-10 14:45 ` Eric Dumazet
2008-01-10 15:26 ` Jeba Anandhan
2008-01-10 17:46 ` Arnaldo Carvalho de Melo
2008-01-10 18:31 ` Kok, Auke
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).