Linux based router for Gigabit traffic

Linux based router for Gigabit traffic

[Anantha Kiran]

[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]

Hi
I am doing a project, in which i have to redirect traffic coming from
one ethernet card of a machine, to one of three remaining three
ethernet cards of same machine based on the src,dest IP and Port
values of the pkt. I wrote a net_hook module to do this which i
working fine for lower speeds like < 400 Mbps traffic. But project
goal is to deal with gigabit traffic. I have used gigabit ethernet
cards and Switch. But when the traffic rate is more than 400Mbps it is
dropping packets.  I did test, by increasing the transmit queue
length. But same problem is coming. I have found during the pkt drop
there is no memory or CPU is hundred percently utilised. So, what can
be the resource that is lacking while dropping of pkts is happend.

We have done a test, in which module will simply forward pkts from
"eth0" through "eth1". Same problem of pkt dropping at highier speed
is happening.

Now we thought , it can be due to PCI bus. But we are unable to find
what is configuration of PCI bus , our system is using. we have used
"lspci" to find that. But i am unable to interpret it. I am attaching
that output file. Can anybody tell me how to interpret that or give
some guidelines , like whom to mail, to find that.

Atlast we try to solve the problem in the following manner. Idea is
to, install a router for my machine which can deal with gigabit
traffic. If the router is able to work with gigabit traffic, then
definitely pkts dropping will not be due to PCI bus.

 So, canbdoy please, suggest some  opensource router for linux which
can deal with gigabit traffic. Or atleast some suggestions to solve
this problem.
-- 
K.AnanthaKiran
D-108/HALL-7

[-- Attachment #2: lspcioutput --]
[-- Type: application/octet-stream, Size: 6573 bytes --]

00:00.0 Host bridge: Intel Corp. 82845G/GL [Brookdale-G] Chipset Host Bridge (rev 01)
	Subsystem: Intel Corp. 82845G/GL [Brookdale-G] Chipset Host Bridge
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
	Latency: 0
	Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
	Capabilities: <available only to root>

00:02.0 VGA compatible controller: Intel Corp. 82845G/GL [Brookdale-G] Chipset Integrated Graphics Device (rev 01) (prog-if 00 [VGA])
	Subsystem: Intel Corp.: Unknown device 4c59
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at f0000000 (32-bit, prefetchable) [size=128M]
	Region 1: Memory at ffa80000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: <available only to root>

00:1d.0 USB Controller: Intel Corp. 82801DB USB (Hub #1) (rev 01) (prog-if 00 [UHCI])
	Subsystem: Intel Corp.: Unknown device 4c59
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 4: I/O ports at e800 [size=32]

00:1d.1 USB Controller: Intel Corp. 82801DB USB (Hub #2) (rev 01) (prog-if 00 [UHCI])
	Subsystem: Intel Corp.: Unknown device 4c59
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin B routed to IRQ 5
	Region 4: I/O ports at e880 [size=32]

00:1d.2 USB Controller: Intel Corp. 82801DB USB (Hub #3) (rev 01) (prog-if 00 [UHCI])
	Subsystem: Intel Corp.: Unknown device 4c59
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin C routed to IRQ 9
	Region 4: I/O ports at ec00 [size=32]

00:1d.7 USB Controller: Intel Corp. 82801DB USB EHCI Controller (rev 01) (prog-if 20 [EHCI])
	Subsystem: Intel Corp.: Unknown device 4c59
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin D routed to IRQ 10
	Region 0: Memory at ffa7fc00 (32-bit, non-prefetchable) [size=1K]
	Capabilities: <available only to root>

00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB PCI Bridge (rev 81) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR+
	Latency: 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: ff800000-ff8fffff
	Prefetchable memory behind bridge: e6a00000-e6afffff
	BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-

00:1f.0 ISA bridge: Intel Corp. 82801DB ISA Bridge (LPC) (rev 01)
	Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0

00:1f.1 IDE interface: Intel Corp. 82801DB ICH4 IDE (rev 01) (prog-if 8a [Master SecP PriP])
	Subsystem: Intel Corp.: Unknown device 4c59
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 9
	Region 0: I/O ports at <unassigned>
	Region 1: I/O ports at <unassigned>
	Region 2: I/O ports at <unassigned>
	Region 3: I/O ports at <unassigned>
	Region 4: I/O ports at ffa0 [size=16]
	Region 5: Memory at 10000000 (32-bit, non-prefetchable) [size=1K]

00:1f.3 SMBus: Intel Corp. 82801DB SMBus (rev 01)
	Subsystem: Intel Corp.: Unknown device 4c59
	Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Interrupt: pin B routed to IRQ 3
	Region 4: I/O ports at e000 [size=32]

00:1f.5 Multimedia audio controller: Intel Corp. 82801DB AC'97 Audio (rev 01)
	Subsystem: Intel Corp.: Unknown device 0303
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin B routed to IRQ 3
	Region 0: I/O ports at e400 [size=256]
	Region 1: I/O ports at e080 [size=64]
	Region 2: Memory at ffa7f800 (32-bit, non-prefetchable) [size=512]
	Region 3: Memory at ffa7f400 (32-bit, non-prefetchable) [size=256]
	Capabilities: <available only to root>

01:01.0 Ethernet controller: Intel Corp. 82540EM Gigabit Ethernet Controller (rev 02)
	Subsystem: Intel Corp. PRO/1000 MT Desktop Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 32 (63750ns min), cache line size 08
	Interrupt: pin A routed to IRQ 7
	Region 0: Memory at ff8e0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at ff8c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at dc00 [size=64]
	Expansion ROM at ff8a0000 [disabled] [size=128K]
	Capabilities: <available only to root>

01:08.0 Ethernet controller: Intel Corp. 82801BD PRO/100 VE (LOM) Ethernet Controller (rev 81)
	Subsystem: Intel Corp.: Unknown device 3009
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 32 (2000ns min, 14000ns max), cache line size 08
	Interrupt: pin A routed to IRQ 4
	Region 0: Memory at ff89f000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at d880 [size=64]
	Capabilities: <available only to root>

Re: Linux based router for Gigabit traffic

[Neil Horman]

Anantha Kiran wrote:

>Hi
>I am doing a project, in which i have to redirect traffic coming from
>one ethernet card of a machine, to one of three remaining three
>ethernet cards of same machine based on the src,dest IP and Port
>values of the pkt. I wrote a net_hook module to do this which i
>working fine for lower speeds like < 400 Mbps traffic. But project
>goal is to deal with gigabit traffic. I have used gigabit ethernet
>cards and Switch. But when the traffic rate is more than 400Mbps it is
>dropping packets.  I did test, by increasing the transmit queue
>length. But same problem is coming. I have found during the pkt drop
>there is no memory or CPU is hundred percently utilised. So, what can
>be the resource that is lacking while dropping of pkts is happend.
>
>We have done a test, in which module will simply forward pkts from
>"eth0" through "eth1". Same problem of pkt dropping at highier speed
>is happening.
>
>Now we thought , it can be due to PCI bus. But we are unable to find
>what is configuration of PCI bus , our system is using. we have used
>"lspci" to find that. But i am unable to interpret it. I am attaching
>that output file. Can anybody tell me how to interpret that or give
>some guidelines , like whom to mail, to find that.
>
>Atlast we try to solve the problem in the following manner. Idea is
>to, install a router for my machine which can deal with gigabit
>traffic. If the router is able to work with gigabit traffic, then
>definitely pkts dropping will not be due to PCI bus.
>
> So, canbdoy please, suggest some  opensource router for linux which
>can deal with gigabit traffic. Or atleast some suggestions to solve
>this problem.
>  
>
Quite simply a general purpose CPU system isn't  normally built to 
handle network traffic at gigabit rates, especially not from multiple 
ports at once.  If you really want a line rate gigabit router, you need 
custom ASIC to do hardware offload of that work.  Several networking 
companies make just such hardware (of course most only sell the chips, 
not full pci cards with integrated NICS).  Check out broadcom, AMCC or 
Intel, they all have hardware like what you're looking for.
Neil

Re: Linux based router for Gigabit traffic

[Matti Aarnio]

On Sun, Aug 22, 2004 at 06:02:59PM +0530, Anantha Kiran wrote:
> Hi
> I am doing a project, in which i have to redirect traffic coming from
> one ethernet card of a machine, to one of three remaining three
> ethernet cards of same machine based on the src,dest IP and Port
> values of the pkt. I wrote a net_hook module to do this which i
> working fine for lower speeds like < 400 Mbps traffic. But project
> goal is to deal with gigabit traffic. I have used gigabit ethernet
> cards and Switch. But when the traffic rate is more than 400Mbps it is
> dropping packets.  I did test, by increasing the transmit queue
> length. But same problem is coming. I have found during the pkt drop
> there is no memory or CPU is hundred percently utilised. So, what can
> be the resource that is lacking while dropping of pkts is happend.
> 
> We have done a test, in which module will simply forward pkts from
> "eth0" through "eth1". Same problem of pkt dropping at highier speed
> is happening.

The  lspci  output shows that you have a GE and a FE interfaces.
Getting thru more than 100 Mbps should not be possible with that..


01:01.0 Ethernet controller: Intel Corp. 82540EM Gigabit Ethernet Controller (rev 02)
        Subsystem: Intel Corp. PRO/1000 MT Desktop Adapter
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (63750ns min), cache line size 08

That card has a PCI bus of:
   PCI 2.3 32-bit 33/66 MHz

which means it can handle up to about 260 MB/s thru the bus.
If about 10% of that is transaction overhead, and then 50/50 goes
to network traffic, that gives upper limit of 115 MB/s which is
about 0.92 Gbit/s.

But if the bus runs at mere 33 MHz ...


01:08.0 Ethernet controller: Intel Corp. 82801BD PRO/100 VE (LOM) Ethernet Controller (rev 81)
        Subsystem: Intel Corp.: Unknown device 3009
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (2000ns min, 14000ns max), cache line size 08

"LOM" cards are something that are built into the system boards.
They communicate command and management traffic thru the PCI bus,
but payload traffic goes thru separate channel to main memory hub.

The LOM cards are usually giga-ethers, which means the pci-ids are
something most strange..   Also this is located into same PCI bus as
the PRO/1000 MT card above, and the bus can't alter speed in between
different cards -- which means that the bus is running at 33 MHz.

Possibly that LOM "card" really is some GE card, and it can run
full GE speed to main memory.   However it will eat into PCI bus
bandwidth for the command traffic, although the data traffic flows
thru a LOM side-channel.


The PCI bridge in between main hub, and the bus where those cards
are tells:

00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB PCI Bridge (rev 81) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR+
        Latency: 0
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=32


So, that bus is running at 33 MHz, and a 32 bit wide GE card in it
can get around 0.4 Gbit/s thru it (remembering that that bus has
also other traffic thru it, than just ethernet traffic itself.)


> Now we thought , it can be due to PCI bus. But we are unable to find
> what is configuration of PCI bus , our system is using. we have used
> "lspci" to find that. But i am unable to interpret it. I am attaching
> that output file. Can anybody tell me how to interpret that or give
> some guidelines , like whom to mail, to find that.
> 
> Atlast we try to solve the problem in the following manner. Idea is
> to, install a router for my machine which can deal with gigabit
> traffic. If the router is able to work with gigabit traffic, then
> definitely pkts dropping will not be due to PCI bus.

A motherboard with two or more LOM GE interfaces MIGHT do it.
Those are usually available only with 10/100/1000Base-T, not
with optical (gbic) mode.

Another approach is to have system with multiple PCI-X busses,
and add PCI-X GE cards.

For example:
  http://www.supermicro.com/products/motherboard/Xeon800/E7520/X6DH8-XG2.cfm


>  So, canbdoy please, suggest some  opensource router for linux which
> can deal with gigabit traffic. Or atleast some suggestions to solve
> this problem.
> -- 
> K.AnanthaKiran
> D-108/HALL-7

/Matti Aarnio

Re: Linux based router for Gigabit traffic

[Anantha Kiran]

Yes, we are generating the traffic by using a programme, which will
take the size of the pkt as argument and generates traffic to towards
the machine which we specified and prints the statistics like how many
pkts and with what speed.
That programme uses a dump file of pkts(collected over long time) 
from which it searches for a pkt of size what we specified , then
starts sending that pkt in infinite loop and accounting the time ,
number of pkts.

So, we have taken care that we are generating Gbps traffic. 


On Sun, 22 Aug 2004 12:32:04 -0400, Neil Horman <nhorman@redhat.com> wrote:
> 
> 
> alex@pilosoft.com wrote:
> 
> >>I am doing a project, in which i have to redirect traffic coming from
> >>one ethernet card of a machine, to one of three remaining three ethernet
> >>cards of same machine based on the src,dest IP and Port values of the
> >>pkt. I wrote a net_hook module to do this which i working fine for lower
> >>speeds like < 400 Mbps traffic. But project goal is to deal with gigabit
> >>traffic. I have used gigabit ethernet cards and Switch. But when the
> >>traffic rate is more than 400Mbps it is dropping packets.  I did test,
> >>by increasing the transmit queue length. But same problem is coming. I
> >>have found during the pkt drop there is no memory or CPU is hundred
> >>percently utilised. So, what can be the resource that is lacking while
> >>dropping of pkts is happend.
> >>
> >>
> >Are you using NAPI?
> >
> >What is the CPU utilization when packets start being dropped?
> >
> >I route full GE worth of traffic on a low-end machine without a problem.
> >
> >-alex
> >
> >-
> >To unsubscribe from this list: send the line "unsubscribe linux-net" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> Are you sure you're actually routing a gigabit worth of bandwidth
> through your machine, or just the small percent of frames that happen to
> be on your gigabit network?
> Neil
> 


-- 
K.AnanthaKiran
D-108/HALL-7

Re: Linux based router for Gigabit traffic

[Anantha Kiran]

No, we are not using NAPI for this. But we wrote a net_hook module
which catches the pkts before allowing it to go upper layers.

CPU utilization is 70+ while dropping.

Can u tell me the configuration of your low-end machine.Like PCI-bus,
and others.


On Sun, 22 Aug 2004 08:59:14 -0400 (EDT), alex@pilosoft.com
<alex@pilosoft.com> wrote:
> > I am doing a project, in which i have to redirect traffic coming from
> > one ethernet card of a machine, to one of three remaining three ethernet
> > cards of same machine based on the src,dest IP and Port values of the
> > pkt. I wrote a net_hook module to do this which i working fine for lower
> > speeds like < 400 Mbps traffic. But project goal is to deal with gigabit
> > traffic. I have used gigabit ethernet cards and Switch. But when the
> > traffic rate is more than 400Mbps it is dropping packets.  I did test,
> > by increasing the transmit queue length. But same problem is coming. I
> > have found during the pkt drop there is no memory or CPU is hundred
> > percently utilised. So, what can be the resource that is lacking while
> > dropping of pkts is happend.
> Are you using NAPI?
> 
> What is the CPU utilization when packets start being dropped?
> 
> I route full GE worth of traffic on a low-end machine without a problem.
> 
> -alex
> 
> 


-- 
K.AnanthaKiran
D-108/HALL-7

Re: Linux based router for Gigabit traffic

[David S. Miller]

On Sun, 22 Aug 2004 08:49:04 -0400
Neil Horman <nhorman@redhat.com> wrote:

> Quite simply a general purpose CPU system isn't  normally built to 
> handle network traffic at gigabit rates, especially not from multiple 
> ports at once.  If you really want a line rate gigabit router, you need 
> custom ASIC to do hardware offload of that work.  Several networking 
> companies make just such hardware (of course most only sell the chips, 
> not full pci cards with integrated NICS).  Check out broadcom, AMCC or 
> Intel, they all have hardware like what you're looking for.

Gigabit routing is possible with commodity hardware.  It's a software
problem for the cases that go fast enough currently.

Re: Linux based router for Gigabit traffic

[Mikael Abrahamsson]

On Sun, 22 Aug 2004, David S. Miller wrote:

> Gigabit routing is possible with commodity hardware.  It's a software
> problem for the cases that go fast enough currently.

To properly handle full duplex gigabit speeds at 300 bytes average packet
size we need approx 800k packets per second. 

So I guess the question should be rephrased into "how many packets per 
second can you get commodity architecture to do?" because that's of more 
interest than how many megabit/s we can handle (as it's usually the 
limiting factor).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Linux based router for Gigabit traffic

[urgrue]

On 2004.08.23 08:56, David S. Miller wrote:
> On Sun, 22 Aug 2004 08:49:04 -0400
> Neil Horman <nhorman@redhat.com> wrote:
> 
> > Quite simply a general purpose CPU system isn't  normally built to
> > handle network traffic at gigabit rates, especially not from
> multiple
> > ports at once.  If you really want a line rate gigabit router, you
> need
> > custom ASIC to do hardware offload of that work.  Several networking
> 
> Gigabit routing is possible with commodity hardware.  It's a software
> problem for the cases that go fast enough currently.
> -

Possible, but only up to 2 gbps. Correct me if I'm wrong, but one full-
duplex gigabit card alone can fill up the entire theoretical maximum 
bandwidth of the PCI bus, which is only 266MBps.
Four gigabit cards on a normal PCI bus is far more than PCI can handle.

Re: Linux based router for Gigabit traffic

[Neil Horman]

David S. Miller wrote:
> On Sun, 22 Aug 2004 08:49:04 -0400
> Neil Horman <nhorman@redhat.com> wrote:
> 
> 
>>Quite simply a general purpose CPU system isn't  normally built to 
>>handle network traffic at gigabit rates, especially not from multiple 
>>ports at once.  If you really want a line rate gigabit router, you need 
>>custom ASIC to do hardware offload of that work.  Several networking 
>>companies make just such hardware (of course most only sell the chips, 
>>not full pci cards with integrated NICS).  Check out broadcom, AMCC or 
>>Intel, they all have hardware like what you're looking for.
> 
> 
> Gigabit routing is possible with commodity hardware.  It's a software
> problem for the cases that go fast enough currently.
Really?  What are PCI bus transfer rates up to these days?  I havent 
looked into it in quite a while.
N3il

-- 
/***************************************************
  *Neil Horman
  *Software Engineer
  *Red Hat, Inc.
  *nhorman@redhat.com
  *gpg keyid: 1024D / 0x92A74FA1
  *http://pgp.mit.edu
  ***************************************************/

Re: Linux based router for Gigabit traffic

[urgrue]

On 2004.08.23 14:17, Neil Horman wrote:
> Really?  What are PCI bus transfer rates up to these days?  I havent 
> looked into it in quite a while.
> N3il

Your average 33mhz PCI bus has a paltry maximum bandwidth of 1 gbps. 
Full-duplex gigabit routing on more than one interface pretty much 
requires PCI Express. Here's a little table:
PCI32 33MHz = 133MB/s PCI32 66MHz = 266MB/s PCI64 33MHz = 266MB/s
PCI64 66MHz = 533MB/s
PCI-X 133MHz = 1066MB/s

Also gigabit routing on your average unmodified linux distribution 
requires a VERY hefty CPU to keep up, even just with two or three 
gigabit interfaces.

Re: Linux based router for Gigabit traffic

[Adam Lang]

Well, as was mentioned, the question is "what type of commodity hardware".
You can easily buy motherboards now with PCI X (express).

Then, you also have to consider USB network cards, which I assume are a
different bus system as well. It all matters if you are in a controlled
environment or you have to work with lowest common denominator.
----- Original Message ----- 
From: "Mikael Abrahamsson" <swmike@swm.pp.se>
To: <linux-admin@vger.kernel.org>
Cc: <linux-net@vger.kernel.org>
Sent: Monday, August 23, 2004 3:11 AM
Subject: Re: Linux based router for Gigabit traffic


> On Sun, 22 Aug 2004, David S. Miller wrote:
>
> > Gigabit routing is possible with commodity hardware.  It's a software
> > problem for the cases that go fast enough currently.
>
> To properly handle full duplex gigabit speeds at 300 bytes average packet
> size we need approx 800k packets per second.
>
> So I guess the question should be rephrased into "how many packets per
> second can you get commodity architecture to do?" because that's of more
> interest than how many megabit/s we can handle (as it's usually the
> limiting factor).
>
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-admin" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux based router for Gigabit traffic

[Stephen J. Smoogen]

On Sun, 22 Aug 2004 08:49:04 -0400, Neil Horman <nhorman@redhat.com> wrote:
> Anantha Kiran wrote:
> 
> >Hi
> >I am doing a project, in which i have to redirect traffic coming from
>
> Quite simply a general purpose CPU system isn't  normally built to
> handle network traffic at gigabit rates, especially not from multiple
> ports at once.  If you really want a line rate gigabit router, you need

Modern CPU's can handle gigabit and by the numbers 10 gigabit traffic
for what he is wanting.  (If you use PCI-X or Express) 

It is mainly getting the PCI bus and kernel 
interrupts to deal with that speed. We are able to push through 750 mbits
on E1000 cards through netfilter on HTTP loads. The main problem is
that the card generates an IRQ per packet and the soft irq takes up 
all the CPU load. On mixed network loads we are at about 400 mbits also..
but it isnt the bus that is loaded but the number of packets per second
that the card can handle.

To find out what the bottleneck is on the system, you need to make sure
your box has the correct tools ( a top that shows soft-irq, a newer mpstat
and other items). Also make sure that the motherboard Bus is PCI-X or better.

Re: Linux based router for Gigabit traffic

[Stephen J. Smoogen]

On Sun, 22 Aug 2004 23:55:29 +0530, Anantha Kiran
<ananth.kandukuri@gmail.com> wrote:
> No, we are not using NAPI for this. But we wrote a net_hook module
> which catches the pkts before allowing it to go upper layers.
> 
> CPU utilization is 70+ while dropping.
> 
> Can u tell me the configuration of your low-end machine.Like PCI-bus,
> and others.
> 

Ours are Intel(R) Xeon(TM) CPU 2.40GHz and the motherboard has a PCI-X
chipset on it. We have one Intel fibre optic Gig-E on the 02:01 PCI-X
Bus, and the other on the 03:01 PCI-X bus. I see a limit of about 750
mbits/sec on a fixed load (HTTP accesses for specific boxes), and 400
mbits/sec on a mixed load (ICMP/UDP/TCP). In each case, the maximum
looks to not be the bus speed as much as the soft-irq limit (which is
what mpstat says is using all the resources). The number of packets
per second that ifconfig and other tools gives me looks about the
same. so I am thinking it isnt bus width at the moment as much as
IRQ/packets per second.

The kernels are NAPI compiled.. but I really havent figure out if
using netfilters on these mixed loads 'disables' it.

-- 
Stephen J Smoogen.
Professional System Administrator

Re: Linux based router for Gigabit traffic

[Stephen Samuel]

An HTTP load is pretty much unidirectional (outbound), so that
700megabits is the total system load.

Anantha wants to  ROUTE 1gigabit of traffic -- That's
1gig in + 1gig out =2gig total.

if you divide your 700Megabit HTTP traffic by two, you'll
get a bit less than the 400Mb that he's claiming to have
gotten so far.

A quick overview:

Matti's bus bandwith calcs indicate that standard PCI won't
do the job in terms of theoretical bandwidth capability.
Once you switch up to PCI-Express, you'll have the
bus bandwith, but you may end up with A CPU limit
(the soft INT's).
   At that point, you might have to go to Real-Time Linux
   and, possibly, custom drivers.  If you can get away from
   the the kernel Task Switching overhead for each packet,
   you should be OK in terms of CPU capability.




Stephen J. Smoogen wrote:
 > On Sun, 22 Aug 2004 23:55:29 +0530, Anantha Kiran
 > <ananth.kandukuri@gmail.com> wrote:
 >
 >>No, we are not using NAPI for this. But we wrote a net_hook module
 >>which catches the pkts before allowing it to go upper layers.
 >>
 >>CPU utilization is 70+ while dropping.
 >>
 >>Can u tell me the configuration of your low-end machine.Like PCI-bus,
 >>and others.
 >>
 >
 >
 > Ours are Intel(R) Xeon(TM) CPU 2.40GHz and the motherboard has a PCI-X
 > chipset on it. We have one Intel fibre optic Gig-E on the 02:01 PCI-X
 > Bus, and the other on the 03:01 PCI-X bus. I see a limit of about 750
 > mbits/sec on a fixed load (HTTP accesses for specific boxes), and 400
 > mbits/sec on a mixed load (ICMP/UDP/TCP). In each case, the maximum
 > looks to not be the bus speed as much as the soft-irq limit (which is
 > what mpstat says is using all the resources). The number of packets
 > per second that ifconfig and other tools gives me looks about the
 > same. so I am thinking it isnt bus width at the moment as much as
 > IRQ/packets per second.
 >
 > The kernels are NAPI compiled.. but I really havent figure out if
 > using netfilters on these mixed loads 'disables' it.
 >


-- 
Stephen Samuel +1(604)876-0426                samuel@bcgreen.com
		   http://www.bcgreen.com/~samuel/
    Powerful committed communication. Transformation touching
      the jewel within each person and bringing it to light.

Re: Linux based router for Gigabit traffic

[neolozer]

Anantha Kiran <ananth.kandukuri <at> gmail.com> writes:

> 
> Hi
> I am doing a project, in which i have to redirect traffic coming from
> one ethernet card of a machine, to one of three remaining three
> ethernet cards of same machine based on the src,dest IP and Port
> values of the pkt. I wrote a net_hook module to do this which i
> working fine for lower speeds like < 400 Mbps traffic. But project
> goal is to deal with gigabit traffic. 
> 
>...
> 
>  So, canbdoy please, suggest some  opensource router for linux which
> can deal with gigabit traffic. Or atleast some suggestions to solve
> this problem.

I have not used any gigabit hardware but I know on 100Mbps cards throughput at 
all speeds is noticely improved by enabling full duplex. And that is what I 
suggest to you. I believe you set it with ifconfig but I have not done so 
through ifconfig since 2.2.x.x. Now-a-days most distros allow those changes to 
be made when you're setting up your box during the installation.

neolozer