Ethernet, kernel bug/weakness

Ethernet, kernel bug/weakness

[Ruhland, Paul]

I've recently discovered what appears to be a bug or weakness in the kernel
that some may be interested in.

Hardware:
--
MPC850 @ 55Mhz
Bootloader:  PPCBOOT 1.1.4
Kernel:  linux-2.4.4-2001-07-23 from denx.de
Ethernet: LevelOne (LXT905) on SCC2

Conditions:
--
DOS type of attack on the ethernet port.

I originally discovered this when our QA department hooked a unit up to a
SmartBits Network Analyzer.  One test slams the UUTs ethernet port with
large amounts of IP packets...starting at the full bandwidth supported by
the network ( ie. 10Mbps/100Mbps ), or at configurable percentages of the
full bandwidth.  For our unit(s), the critical point is ~7-10% of 10Mpbs (
700Kbps - 1Mbps ).  The actual IP packets have valid ethernet/ip headers but
bogus data and a directed at the UUT's ethernet address.

I have since written a C program which can recreate the problem ( replacing
the SmartBits with a workstation, as not many people dont have them ).
Basically open a raw socket, providing the ip header and data yourself, loop
sending as fast as your machine permits to UUT.

Effect:
--
Once the ethernet traffic goes above the critical point described above, all
cpu cycles are spent servicing the ethernet traffic.  If the watchdog is
enabled, the unit will reboot since no other process is able to run (
including the one that kicks the dog ).  Even a kernel driver that kicks the
dog via timer interrupt will not be run.  Without watchdog enabled the unit
does...well, nothing...for all practical purposes it is dead.


Current Status
--
I've tried various things attempting to narrow the problem down.  Even
resetting the watchdog immediately upon entry to ethernet interrupt handler
doesn't stop the reboot ( in case where watchdog is enabled of course )...so
the problem appears to be at a higher level.  Some type of throttling of the
network load on system seems to be required but is not currently present in
the kernel.

Although this type of traffic doesn't occur under normal conditions,
sometimes bad things happen ( intentionally or accidentally ) and the
current behavior of the kernel is less than acceptable.

I currently don't have time to spend on this issue but thought folks should
be made aware to either verify, refute, or investigate and fix.       I will
be returning to this when time permits, but I will provide any other info (
if possible ) to anyone who inquires.

--
Paul Ruhland


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Ethernet, kernel bug/weakness

[Dan Malek]

Ruhland, Paul wrote:

> Effect:
> --
> Once the ethernet traffic goes above the critical point described above, all
> cpu cycles are spent servicing the ethernet traffic.

Then something is wrong with your hardware implementation or processor
initialization.  A 50 MHz 850 will easily process the 10 Mbit Ethernet and
IP stack with cycles to spare.  Ensure you have the caches enabled and all
pipeline debug modes disabled.  There could also be something wrong with
your PHY connection, generating lots of noise/errors that requires IP
retransmission, although this really shouldn't cause trouble.


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Ethernet, kernel bug/weakness

[Harald Welte]

On Thu, Mar 07, 2002 at 03:50:04PM -0500, Ruhland, Paul wrote:
> Effect:
> --
> Once the ethernet traffic goes above the critical point described above, all
> cpu cycles are spent servicing the ethernet traffic.

This is a common problem, also found with gigabit ethernet and large x86
boxes (or 100MBit and medium x86 boxes), as soon as you start shooting at them
with foll wire speed 20bytes IP packets.

It is well-known and about two years ago Jamal Hadi Selim and others were
solving the problem for a couple of example ethernet chips by using IRQ
mitigation.  This way the packets will get dropped on the network card
itself.

Seems like your driver isn't using this strategy for some strange reason.

Alternatively you can also try to detect high load within the driver and
switch to polling mode - this also has been implemented successfully in
some drivers, IIRC.

> Paul Ruhland

--
Live long and prosper
- Harald Welte / laforge@gnumonks.org               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M+
V-- PS++ PE-- Y++ PGP++ t+ 5-- !X !R tv-- b+++ !DI !D G+ e* h--- r++ y+(*)

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/