Re: Linux based router for Gigabit traffic

[Matti Aarnio <matti.aarnio@zmailer.org>]

On Sun, Aug 22, 2004 at 06:02:59PM +0530, Anantha Kiran wrote:
> Hi
> I am doing a project, in which i have to redirect traffic coming from
> one ethernet card of a machine, to one of three remaining three
> ethernet cards of same machine based on the src,dest IP and Port
> values of the pkt. I wrote a net_hook module to do this which i
> working fine for lower speeds like < 400 Mbps traffic. But project
> goal is to deal with gigabit traffic. I have used gigabit ethernet
> cards and Switch. But when the traffic rate is more than 400Mbps it is
> dropping packets.  I did test, by increasing the transmit queue
> length. But same problem is coming. I have found during the pkt drop
> there is no memory or CPU is hundred percently utilised. So, what can
> be the resource that is lacking while dropping of pkts is happend.
> 
> We have done a test, in which module will simply forward pkts from
> "eth0" through "eth1". Same problem of pkt dropping at highier speed
> is happening.

The  lspci  output shows that you have a GE and a FE interfaces.
Getting thru more than 100 Mbps should not be possible with that..


01:01.0 Ethernet controller: Intel Corp. 82540EM Gigabit Ethernet Controller (rev 02)
        Subsystem: Intel Corp. PRO/1000 MT Desktop Adapter
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (63750ns min), cache line size 08

That card has a PCI bus of:
   PCI 2.3 32-bit 33/66 MHz

which means it can handle up to about 260 MB/s thru the bus.
If about 10% of that is transaction overhead, and then 50/50 goes
to network traffic, that gives upper limit of 115 MB/s which is
about 0.92 Gbit/s.

But if the bus runs at mere 33 MHz ...


01:08.0 Ethernet controller: Intel Corp. 82801BD PRO/100 VE (LOM) Ethernet Controller (rev 81)
        Subsystem: Intel Corp.: Unknown device 3009
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (2000ns min, 14000ns max), cache line size 08

"LOM" cards are something that are built into the system boards.
They communicate command and management traffic thru the PCI bus,
but payload traffic goes thru separate channel to main memory hub.

The LOM cards are usually giga-ethers, which means the pci-ids are
something most strange..   Also this is located into same PCI bus as
the PRO/1000 MT card above, and the bus can't alter speed in between
different cards -- which means that the bus is running at 33 MHz.

Possibly that LOM "card" really is some GE card, and it can run
full GE speed to main memory.   However it will eat into PCI bus
bandwidth for the command traffic, although the data traffic flows
thru a LOM side-channel.


The PCI bridge in between main hub, and the bus where those cards
are tells:

00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB PCI Bridge (rev 81) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR+
        Latency: 0
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=32


So, that bus is running at 33 MHz, and a 32 bit wide GE card in it
can get around 0.4 Gbit/s thru it (remembering that that bus has
also other traffic thru it, than just ethernet traffic itself.)


> Now we thought , it can be due to PCI bus. But we are unable to find
> what is configuration of PCI bus , our system is using. we have used
> "lspci" to find that. But i am unable to interpret it. I am attaching
> that output file. Can anybody tell me how to interpret that or give
> some guidelines , like whom to mail, to find that.
> 
> Atlast we try to solve the problem in the following manner. Idea is
> to, install a router for my machine which can deal with gigabit
> traffic. If the router is able to work with gigabit traffic, then
> definitely pkts dropping will not be due to PCI bus.

A motherboard with two or more LOM GE interfaces MIGHT do it.
Those are usually available only with 10/100/1000Base-T, not
with optical (gbic) mode.

Another approach is to have system with multiple PCI-X busses,
and add PCI-X GE cards.

For example:
  http://www.supermicro.com/products/motherboard/Xeon800/E7520/X6DH8-XG2.cfm


>  So, canbdoy please, suggest some  opensource router for linux which
> can deal with gigabit traffic. Or atleast some suggestions to solve
> this problem.
> -- 
> K.AnanthaKiran
> D-108/HALL-7

/Matti Aarnio