crappy/good gigabit chipsets?

crappy/good gigabit chipsets?

[Carl-Daniel Hailfinger]

Hi,

I'm trying to build a gigabit ethernet bridge with Linux 2.6 and
commodity hardware. The bridge should be able to filter traffic
(probably with ebtables), which will mostly be operations like
"drop traffic from this host", "drop traffic to this port", to
stop infected windows machines generating traffic until they are
reformatted with Linux.
The traffic source is a 100 MBit switched network with 2500 PCs
(1 GBit Uplink) and the usual frame sizes on a switched 100 MBit
network and the sink is a machine doing masquerading of the
internal machines to the outside world.

IIRC, TSO, hardware checksumming etc. are abolutely unneeded for
this type of operation, but a fast transfer of the packets to/
from memory is essential. Maybe I'm crazy to assume a current
machine with classic 32 bit PCI bus can handle such a load, but
since Linux performs stellar with a bunch of 100 MBit cards at
the same time, I see at least some hope.

Which gigabit chipsets do classic data transfer fast enough to
get a recommendation from the experts?

Which gigabit drivers are stable und well-written enough to
sustain fully loaded interfaces without breaking down?

Which chipsets should be avoided?

Thank you very much for your time.

Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/

Re: crappy/good gigabit chipsets?

[Lennert Buytenhek]

On Sat, Nov 27, 2004 at 07:17:57AM +0100, Carl-Daniel Hailfinger wrote:

> Hi,

Hello,


> Which gigabit chipsets do classic data transfer fast enough to
> get a recommendation from the experts?

I'm not much of an expert, but I do have some experience to share.

I tried r8169 (Realtek gigabit chipset, used in various el-cheapo
GigE cards) in a cheap Asus mainboard in a 32/33 slot, and when
throwing enough packets at it the card would just stall with the
driver spewing out various scary-looking PCI errors to the console.
Didn't have time to look into it further at the time (this was on
2.6.9-something.)  Might not be the NIC's fault.

	kernel: r8169: eth1: PCI error (status: 0x8000). Device disabled.
	kernel: r8169: eth1: PCI error (status: 0x8001). Device disabled.
	kernel: r8169: eth1: PCI error (status: 0x8005). Device disabled.
	kernel: r8169: eth1: PCI error (status: 0x8020). Device disabled.

I'm very happy with the e1000 cards.  In the cheap Asus board above,
they don't perform very well, but it would be impossible to make any
NIC perform well in a machine that has its three PCI slots on the
same 32/33 PCI bus as all the other on-board PCI devices.  At least
they've never locked up on me.

On an intel board with independent PCI buses, I see an e1000 'desktop'
NIC filling the pipe for any packet size > ~350 when the card sits on
its own PCI bus.  This is while running at 32/66, CPU being a 2.4GHz
Xeon.


cheers,
Lennert

Re: crappy/good gigabit chipsets?

[Francois Romieu]

Lennert Buytenhek <buytenh@wantstofly.org> :
[...]
> I'm not much of an expert, but I do have some experience to share.

AOL

> I tried r8169 (Realtek gigabit chipset, used in various el-cheapo
> GigE cards) in a cheap Asus mainboard in a 32/33 slot, and when
> driver spewing out various scary-looking PCI errors to the console.
> throwing enough packets at it the card would just stall with the

<r8169 maintainer hat on>

- which chipset on the motherboard ?
- do you have some (even gross) figures for the required packet rate ?
- is your computer with the r8169 adapter still available for testing ?

People, please report such behavior. Even if you do not have time to
further help fixing it, it really helps me to know own many hit the
issue. Bugzilla or mail are fine with me.

> Didn't have time to look into it further at the time (this was on
> 2.6.9-something.)  Might not be the NIC's fault.

Sensible. Without further documentation related to the recovery for
this specific issue (DAC related PCI errors are a different beast),
I assumed that the safer thing was to disable the device.

At least on x86, under load, I would not be surprised if it makes 
zero difference to ignore the PCI error. Before this check was introduced,
there was no identified complaint related to the stability of the driver
in the vanilla kernel.

More specifically, Google shows many people with r8169 issues. At the end
of the day, there are mostly related to old drivers, flaky hardware,
gcc 2.95.x, lovecraftian build of the kernel and such. Pending issues:
1) a complete failure on an amd64 laptop I'll be happy to see disappear in
   the "acpi killed me" category. Hopefully the things seem to improve
   on the acpi side.
2) Ben Greear experiences some troubles with a pcmcia adapter on its laptop.
   I have recently got the hardware to dig this issue.
3) gcc 2.95.4 related failure.
4) misc pci error reports on x86 since the code for it has been added.
   The pattern is not clear (chipset of the mobo ? cooling ? wtf ?).
5) DAC sucks on x86_64.

So far there are no known figures for the r8169 in the kind of setup that
Carl-Daniel plans to use.

> On an intel board with independent PCI buses, I see an e1000 'desktop'
> NIC filling the pipe for any packet size > ~350 when the card sits on
> its own PCI bus.  This is while running at 32/66, CPU being a 2.4GHz
> Xeon.

How do you fill it ?

--
Ueimor

Re: crappy/good gigabit chipsets?

[Lennert Buytenhek]

On Sat, Nov 27, 2004 at 02:47:06PM +0100, Francois Romieu wrote:

> > I tried r8169 (Realtek gigabit chipset, used in various el-cheapo
> > GigE cards) in a cheap Asus mainboard in a 32/33 slot, and when
> > driver spewing out various scary-looking PCI errors to the console.
> > throwing enough packets at it the card would just stall with the
> 
> <r8169 maintainer hat on>
> 
> - which chipset on the motherboard ?

Not sure, I'd have to open up the box.  Can't remember the mobo model off
the top of my hat either.  This is the lspci output (I swapped the r8169
for an e1000):

00:00.0 Host bridge: Silicon Integrated Systems [SiS] 651 Host (rev 02)
00:01.0 PCI bridge: Silicon Integrated Systems [SiS] Virtual PCI-to-PCI bridge (AGP)
00:02.0 ISA bridge: Silicon Integrated Systems [SiS] SiS962 [MuTIOL Media IO] (rev 25)
00:02.1 SMBus: Silicon Integrated Systems [SiS] SiS961/2 SMBus Controller
00:02.5 IDE interface: Silicon Integrated Systems [SiS] 5513 [IDE]
00:02.7 Multimedia audio controller: Silicon Integrated Systems [SiS] Sound Controller (rev a0)
00:03.0 USB Controller: Silicon Integrated Systems [SiS] USB 1.0 Controller (rev 0f)
00:03.1 USB Controller: Silicon Integrated Systems [SiS] USB 1.0 Controller (rev 0f)
00:03.2 USB Controller: Silicon Integrated Systems [SiS] USB 2.0 Controller
00:04.0 Ethernet controller: Silicon Integrated Systems [SiS] SiS900 PCI Fast Ethernet (rev 91)
00:0d.0 Ethernet controller: Intel Corp. 82541GI/PI Gigabit Ethernet Controller
01:00.0 VGA compatible controller: Silicon Integrated Systems [SiS] 65x/M650/740 PCI/AGP VGA Display Adapter


> - do you have some (even gross) figures for the required packet rate ?

I was blasting ~600kpps at it, and it couldn't handle that.  It seemed
to handle ~100kpps okay.  Didn't check anything else.

Half of the time the r8169 negotiates 100Mb/s with my other box' e1000
and half of the time it negotiates 1000Mb/s, even though there's only a
~1m cross-cable between the two, which is another reason why I swapped
it with the e1000 in the end.


> - is your computer with the r8169 adapter still available for testing ?

Sure, tell me what to do.


> People, please report such behavior. Even if you do not have time to
> further help fixing it, it really helps me to know own many hit the
> issue. Bugzilla or mail are fine with me.

I'm guilty and I'm sorry.  I wanted to do some more tests before sending
in a report but never got round to doing that testing.


> 4) misc pci error reports on x86 since the code for it has been added.
>    The pattern is not clear (chipset of the mobo ? cooling ? wtf ?).

I have a bunch of other PC that I can try this card in as well if that'd
help.  (Pity that my sparc doesn't take PCI cards.)


> > On an intel board with independent PCI buses, I see an e1000 'desktop'
> > NIC filling the pipe for any packet size > ~350 when the card sits on
> > its own PCI bus.  This is while running at 32/66, CPU being a 2.4GHz
> > Xeon.
> 
> How do you fill it ?

Non-cheesy hardware, dedicated PCI bus, and Robert Olsson's pktgen.  (Not
the one in the current 2.6 kernel but his new-and-updated version that's
somewhere on ftp://robur.slu.se/ )


--L