* SMP code / network stack @ 2008-01-10 14:05 Jeba Anandhan 2008-01-10 14:45 ` Eric Dumazet 0 siblings, 1 reply; 5+ messages in thread From: Jeba Anandhan @ 2008-01-10 14:05 UTC (permalink / raw) To: netdev; +Cc: matthew.hattersley Hi All, If a server has multiple processors and N number of ethernet cards, is it possible to handle transmission by each processor separately? .In other words, each processor will be responsible for tx of few ethernet cards?. Example: Server has 4 processors and 8 ethernet cards. is it possible for each processor for transmission using 2 ethernet cards only?. So that, at a instant , data will be send out from 8 ethernet cards. Thanks Jeba ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: SMP code / network stack 2008-01-10 14:05 SMP code / network stack Jeba Anandhan @ 2008-01-10 14:45 ` Eric Dumazet 2008-01-10 15:26 ` Jeba Anandhan 0 siblings, 1 reply; 5+ messages in thread From: Eric Dumazet @ 2008-01-10 14:45 UTC (permalink / raw) To: Jeba Anandhan; +Cc: netdev, matthew.hattersley On Thu, 10 Jan 2008 14:05:46 +0000 Jeba Anandhan <jeba.anandhan@vaioni.com> wrote: > Hi All, > > If a server has multiple processors and N number of ethernet cards, is > it possible to handle transmission by each processor separately? .In > other words, each processor will be responsible for tx of few ethernet > cards?. > > > > Example: Server has 4 processors and 8 ethernet cards. is it possible > for each processor for transmission using 2 ethernet cards only?. So > that, at a instant , data will be send out from 8 ethernet cards. Hi Jeba Modern ethernet cards have a big TX queue, so that even one CPU is enough to keep several cards busy in // You can check /proc/interrupts and change /proc/irq/*/smp_affinities to direct IRQ to particular cpus, but transmit is usually trigered by processes that might run on different cpus. If all ethernet cards are on the same IRQ, then you might have a problem... Example on a dual processor : # cat /proc/interrupts CPU0 CPU1 0: 11472559 74291833 IO-APIC-edge timer 2: 0 0 XT-PIC cascade 8: 0 1 IO-APIC-edge rtc 81: 0 0 IO-APIC-level ohci_hcd 97: 1830022231 847 IO-APIC-level ehci_hcd, eth0 121: 163095662 166443627 IO-APIC-level libata NMI: 0 0 LOC: 85887285 85887193 ERR: 0 MIS: 0 You can see eth0 is on IRQ 97 Then : # cat /proc/irq/97/smp_affinity 00000001 # echo 2 >/proc/irq/97/smp_affinity # grep 97 /proc/interrupts 97: 1830035216 2259 IO-APIC-level ehci_hcd, eth0 # sleep 10 # grep 97 /proc/interrupts 97: 1830035216 5482 IO-APIC-level ehci_hcd, eth0 You can see only CPU1 is now handling IRQ 97 (but CPU0 is allowed to give to eth0 some transmit work) You might want to check /proc/net/softnet_stat too. If your server is doing something very special (network trafic, no disk accesses or number crunching), you might need to bind application processes to cpus, not only network irqs. process A, using nic eth0 & eth1, bound to CPU 0 (process and IRQs) process B, using nic eth2 & eth3, bound to CPU 1 process C, using nic eth4 & eth5, bound to CPU 2 process D, using nic eth6 & eth7, bound to CPU 3 Also, take a look at "ethtool -c ethX" command ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: SMP code / network stack 2008-01-10 14:45 ` Eric Dumazet @ 2008-01-10 15:26 ` Jeba Anandhan 2008-01-10 17:46 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 5+ messages in thread From: Jeba Anandhan @ 2008-01-10 15:26 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev, matthew.hattersley Hi Eric, Thanks for the reply. I have one more doubt. For example, if we have 2 processor and 4 ethernet cards. Only CPU0 does all work through 8 cards. If we set the affinity to each ethernet card as CPU number, will it be efficient?. Will this be default behavior? # cat /proc/interrupts CPU0 CPU1 0: 11472559 74291833 IO-APIC-edge timer 2: 0 0 XT-PIC cascade 8: 0 1 IO-APIC-edge rtc 81: 0 0 IO-APIC-level ohci_hcd 97: 1830022231 847 IO-APIC-level ehci_hcd, eth0 97: 3830012232 847 IO-APIC-level ehci_hcd, eth1 97: 5830052231 847 IO-APIC-level ehci_hcd, eth2 97: 6830032213 847 IO-APIC-level ehci_hcd, eth3 #sleep 10 # cat /proc/interrupts CPU0 CPU1 0: 11472559 74291833 IO-APIC-edge timer 2: 0 0 XT-PIC cascade 8: 0 1 IO-APIC-edge rtc 81: 0 0 IO-APIC-level ohci_hcd 97: 2031409801 847 IO-APIC-level ehci_hcd, eth0 97: 4813981390 847 IO-APIC-level ehci_hcd, eth1 97: 7123982139 847 IO-APIC-level ehci_hcd, eth2 97: 8030193010 847 IO-APIC-level ehci_hcd, eth3 Instead of the above mentioned ,if we set the affinity for eth2 and eth3. the output will be # cat /proc/interrupts CPU0 CPU1 0: 11472559 74291833 IO-APIC-edge timer 2: 0 0 XT-PIC cascade 8: 0 1 IO-APIC-edge rtc 81: 0 0 IO-APIC-level ohci_hcd 97: 1830022231 847 IO-APIC-level ehci_hcd, eth0 97: 3830012232 847 IO-APIC-level ehci_hcd, eth1 97: 5830052231 923 IO-APIC-level ehci_hcd, eth2 97: 6830032213 1230 IO-APIC-level ehci_hcd, eth3 #sleep 10 # cat /proc/interrupts CPU0 CPU1 0: 11472559 74291833 IO-APIC-edge timer 2: 0 0 XT-PIC cascade 8: 0 1 IO-APIC-edge rtc 81: 0 0 IO-APIC-level ohci_hcd 97: 2300022231 847 IO-APIC-level ehci_hcd, eth0 97: 4010212232 847 IO-APIC-level ehci_hcd, eth1 97: 5830052231 1847 IO-APIC-level ehci_hcd, eth2 97: 6830032213 2337 IO-APIC-level ehci_hcd, eth3 In this case, will the performance improves?. Thanks Jeba On Thu, 2008-01-10 at 15:45 +0100, Eric Dumazet wrote: ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: SMP code / network stack 2008-01-10 15:26 ` Jeba Anandhan @ 2008-01-10 17:46 ` Arnaldo Carvalho de Melo 2008-01-10 18:31 ` Kok, Auke 0 siblings, 1 reply; 5+ messages in thread From: Arnaldo Carvalho de Melo @ 2008-01-10 17:46 UTC (permalink / raw) To: Jeba Anandhan; +Cc: Eric Dumazet, netdev, matthew.hattersley Em Thu, Jan 10, 2008 at 03:26:59PM +0000, Jeba Anandhan escreveu: > Hi Eric, > Thanks for the reply. I have one more doubt. For example, if we have 2 > processor and 4 ethernet cards. Only CPU0 does all work through 8 cards. > If we set the affinity to each ethernet card as CPU number, will it be > efficient?. > > Will this be default behavior? > > # cat /proc/interrupts > CPU0 CPU1 > 0: 11472559 74291833 IO-APIC-edge timer > 2: 0 0 XT-PIC cascade > 8: 0 1 IO-APIC-edge rtc > 81: 0 0 IO-APIC-level ohci_hcd > 97: 1830022231 847 IO-APIC-level ehci_hcd, eth0 > 97: 3830012232 847 IO-APIC-level ehci_hcd, eth1 > 97: 5830052231 847 IO-APIC-level ehci_hcd, eth2 > 97: 6830032213 847 IO-APIC-level ehci_hcd, eth3 > #sleep 10 > > # cat /proc/interrupts > CPU0 CPU1 > 0: 11472559 74291833 IO-APIC-edge timer > 2: 0 0 XT-PIC cascade > 8: 0 1 IO-APIC-edge rtc > 81: 0 0 IO-APIC-level ohci_hcd > 97: 2031409801 847 IO-APIC-level ehci_hcd, eth0 > 97: 4813981390 847 IO-APIC-level ehci_hcd, eth1 > 97: 7123982139 847 IO-APIC-level ehci_hcd, eth2 > 97: 8030193010 847 IO-APIC-level ehci_hcd, eth3 > > > Instead of the above mentioned ,if we set the affinity for eth2 and > eth3. > the output will be > > # cat /proc/interrupts > CPU0 CPU1 > 0: 11472559 74291833 IO-APIC-edge timer > 2: 0 0 XT-PIC cascade > 8: 0 1 IO-APIC-edge rtc > 81: 0 0 IO-APIC-level ohci_hcd > 97: 1830022231 847 IO-APIC-level ehci_hcd, eth0 > 97: 3830012232 847 IO-APIC-level ehci_hcd, eth1 > 97: 5830052231 923 IO-APIC-level ehci_hcd, eth2 > 97: 6830032213 1230 IO-APIC-level ehci_hcd, eth3 > #sleep 10 > > # cat /proc/interrupts > CPU0 CPU1 > 0: 11472559 74291833 IO-APIC-edge timer > 2: 0 0 XT-PIC cascade > 8: 0 1 IO-APIC-edge rtc > 81: 0 0 IO-APIC-level ohci_hcd > 97: 2300022231 847 IO-APIC-level ehci_hcd, eth0 > 97: 4010212232 847 IO-APIC-level ehci_hcd, eth1 > 97: 5830052231 1847 IO-APIC-level ehci_hcd, eth2 > 97: 6830032213 2337 IO-APIC-level ehci_hcd, eth3 > > In this case, will the performance improves?. ps ax | grep irqbalance tells what? If it is enabled please try: service irqbalance stop chkconfig irqbalance off Then reset the smp_affinity entries to ff so and try again. http://www.irqbalance.org/ - Arnaldo ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: SMP code / network stack 2008-01-10 17:46 ` Arnaldo Carvalho de Melo @ 2008-01-10 18:31 ` Kok, Auke 0 siblings, 0 replies; 5+ messages in thread From: Kok, Auke @ 2008-01-10 18:31 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Jeba Anandhan, Eric Dumazet, netdev, matthew.hattersley Arnaldo Carvalho de Melo wrote: > Em Thu, Jan 10, 2008 at 03:26:59PM +0000, Jeba Anandhan escreveu: >> Hi Eric, >> Thanks for the reply. I have one more doubt. For example, if we have 2 >> processor and 4 ethernet cards. Only CPU0 does all work through 8 cards. >> If we set the affinity to each ethernet card as CPU number, will it be >> efficient?. >> >> Will this be default behavior? >> >> # cat /proc/interrupts >> CPU0 CPU1 >> 0: 11472559 74291833 IO-APIC-edge timer >> 2: 0 0 XT-PIC cascade >> 8: 0 1 IO-APIC-edge rtc >> 81: 0 0 IO-APIC-level ohci_hcd >> 97: 1830022231 847 IO-APIC-level ehci_hcd, eth0 >> 97: 3830012232 847 IO-APIC-level ehci_hcd, eth1 >> 97: 5830052231 847 IO-APIC-level ehci_hcd, eth2 >> 97: 6830032213 847 IO-APIC-level ehci_hcd, eth3 another thing to try: if you don't need usb2 support, remove the ehci_hcd module - this will give a slight less overhead servicing irq's in your system. I take it that you have no MSI support in these ethernet cards? Auke ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2008-01-10 18:32 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-01-10 14:05 SMP code / network stack Jeba Anandhan 2008-01-10 14:45 ` Eric Dumazet 2008-01-10 15:26 ` Jeba Anandhan 2008-01-10 17:46 ` Arnaldo Carvalho de Melo 2008-01-10 18:31 ` Kok, Auke
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).