Kernel oops while routing

Kernel oops while routing

[Ricardo Scop]

Hi,

I'm doing some performance tests with a proprietary Linuxppc-based box
configured as a routing system. The processor is  MPC8255 @ 133 MHz (33Mhz on
the bus) and Linux revision is 2.4.15pre8 rsync'ed from MVista linuxppc_2_4
repository.

We are using two other Linux workstations to exercise the router, both
running Netpipe 2.4, one as a client application, the other as a server. Each
one is connected to a different fast ethernet port of our router  box
(100MHz, full-duplex mode) using cross cables.

We're achieving throughputs around 40 Mbps with this setup, which is enough
for our purposes.

But, when we try a 30 MBytes' block in Netpipe,  the kernel in our Linux box
crashes big time (trace bellow, including ksymoops decode). We also tryed
Linux 2.4.4 version, but then the performance slows down to around 13 Mbps.

My questions are:

Has anyone observed this kind of crash?
Is there any workaround?

Any pointers or suggestions will be greatly appreciated.

Regards,

~Ricardo
R SCOP Consulting.

Crash trace:
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Oops: kernel access of bad area, sig: 11
NIP: C00A9E78 XER: 00000000 LR: C00A9E54 SP: C1FADC00 REGS: c1fadb50 TRAP:
0300
   MSR: 00001032 EE: 0 PR: 0 FP: 0 ME: 1 IR/DR: 11
   TASK = c1fac000[3] 'ksoftirqd_CPU0' Last syscall: -1
   last math c1f30000 last altivec 00000000
   GPR00: 00000000 C1FADC00 C1FAC000 00000001 00009032 C1FADCE0 00000004
0000001F
   GPR08: C0154680 00000025 40601801 08000000 C1FADD98 1001F5F0 01FDF000
00000000
   GPR16: 00000001 007FFF00 FFFFFFFF 01FD816C 00001032 01FADCD0 00000000
C0003F80
   GPR24: C0004F90 00000400 C1FA5200 C0180000 000005FA 00000020 C0178E20
C0FC9594
   Call backtrace:
   00000000 C00A558C C00A5234 C0004EEC C0004FD8 C0003F80 C00E9FB4
   C00B5B14 C00B5F74 C00BCA94 C00AED38 C0016548 C0005034 C0003F80
   C0016548 C0016C20 C0006464
Warning (Oops_read): Code line not seen, dumping what data is available

>>???; c00a9e78 <alloc_skb+d4/204>   <=====

Trace; 00000000 Before first symbol
Trace; c00a558c <fcc_enet_rx+e4/220>
Trace; c00a5234 <fcc_enet_interrupt+3c/2b0>
Trace; c0004eec <ppc_irq_dispatch_handler+190/234>
Trace; c0004fd8 <do_IRQ+48/bc>
Trace; c0003f80 <ret_from_intercept+0/8>
Trace; c00e9fb4 <ip_conntrack_in+248/318>
Trace; c00b5b14 <nf_iterate+64/e4>
Trace; c00b5f74 <nf_hook_slow+100/1cc>
Trace; c00bca94 <ip_rcv+450/4b0>
Trace; c00aed38 <net_rx_action+2b0/3e0>
Trace; c0016548 <do_softirq+88/100>
Trace; c0005034 <do_IRQ+a4/bc>
Trace; c0003f80 <ret_from_intercept+0/8>
Trace; c0016548 <do_softirq+88/100>
Trace; c0016c20 <ksoftirqd+84/a8>
Trace; c0006464 <kernel_thread+34/40>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Kernel oops while routing

[Ricardo Scop]

Hi everybody,

Ok, time for an update on the problem bellow.

First of all, many thanks to Andy Lowe for his patch. The problem was
gone with it, but with some performance penalties that I would like to
discuss.

Even before applying the patch, we found out that the crash was caused
by a memory leak in the system. When the memoty was exhauted, the
kernel crashed. Furthermore, we discovered that
turning CONFIG...MMAP off decreased considerably the leakage rate, at
least in our test setup.

Then, applying the patch stopped the leak. But, just as I mentioned
before, the performance of our routing test decreased a lot. Best
throughput rates dropped to 15 Mbps, against 46 Mbps before patching.

I'm kind of lost with this performance variations. As far as I could
see, the patch did not insert much processing overhead, so...

Tips, commentaries, pointers on what to seek for... everything will be
appreciated.

[]'s, Scop                            mailto:scop@digitel.com.br

------------------------------------------------------------------
"What's money? A man is a success if he gets up in the morning and
goes to bed at night and in between does what he wants to do."
~Bob Dylan

Monday, November 26, 2001, 1:54:19 PM, Ricardo Scop wrote:

RS> Hi,

RS> I'm doing some performance tests with a proprietary Linuxppc-based box
RS> configured as a routing system. The processor is  MPC8255 @ 133 MHz (33Mhz on
RS> the bus) and Linux revision is 2.4.15pre8 rsync'ed from MVista linuxppc_2_4
RS> repository.

RS> We are using two other Linux workstations to exercise the router, both
RS> running Netpipe 2.4, one as a client application, the other as a server. Each
RS> one is connected to a different fast ethernet port of our router  box
RS> (100MHz, full-duplex mode) using cross cables.

RS> We're achieving throughputs around 40 Mbps with this setup, which is enough
RS> for our purposes.

RS> But, when we try a 30 MBytes' block in Netpipe,  the kernel in our Linux box
RS> crashes big time (trace bellow, including ksymoops decode). We also tryed
RS> Linux 2.4.4 version, but then the performance slows down to around 13 Mbps.

RS> My questions are:

RS> Has anyone observed this kind of crash?
RS> Is there any workaround?

RS> Any pointers or suggestions will be greatly appreciated.

RS> Regards,

RS> ~Ricardo
RS> R SCOP Consulting.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Kernel oops while routing

[Dan Malek]

Ricardo Scop wrote:


> I'm kind of lost with this performance variations. As far as I could
> see, the patch did not insert much processing overhead, so...

Perhaps if someone would post the patch for the rest of us to see we
could be of some assistance.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Kernel oops while routing

[Dan Malek]

Ricardo Scop wrote:


> So, we're offering this patch to the public list (with Andy's
> blessing :-). I can provide any other details about our tests, if
> required.

Cool.  Thanks.  This could also be a problem with all Ethernet drivers
(SCC, FCC, and FEC) on both 8xx and 8260.  I'll look at the others.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Kernel oops while routing

[Jean-Denis Boyer]

>      - in fcc_enet_start_xmit, after setting up another bd and
>      incrementing bdp, the next bd's tx-ready bit is tested in order
>      to stop the xmit queue if it is set, ok? But, sometimes, the CPM
>      may already have cleared this bit _and_ the corresponding
>      interrupt has not been serviced yet (because we're in a
>      spin_lock_irq); so, netif_stop_queue is not called in this case,
>      nor is tx_full set;
>
>      - next, the interrupt is serviced, but then curr_tx equals
>      dirty_tx _and_ tx_full is not set, so no sk_buffers are freed!

Yes! I totally agree with you, checking the ready bit in the buffer
descriptor is not guaranteed, even if the interrupts are masked, since the
CPM doesn't suspend its processing.

I have done many tests between two of our custom boards, that use an 8260
and a single FCC. I could effectively see a memory leak.

IMHO, I could suggest an easier patch, that would result in modifying only
one line of code, without changing the 'tx_full' logic. In function
fcc_enet_start_xmit, instead of checking the ready bit (which is bad), we
could only check if cur_tx has reached dirty_tx, and then call
netif_stop_queue. Does it make sense?


BTW, I worked hard last week in debugging the fcc_enet driver. It was not
handling correctly some transmission errors, resulting in the transmitter
completely stopping, without restarting. This is related to an errata
(CPM37) from Motorola about the 8260, concerning the way of restarting the
transmitter. If someone is interested, I can release a patch for that.



--------------------------------------------
 Jean-Denis Boyer, B.Eng., Technical Leader
 Mediatrix Telecom Inc.
 4229 Garlock Street
 Sherbrooke (Québec)
 J1L 2C8  CANADA
 (819)829-8749 x241
--------------------------------------------

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Kernel oops while routing

[Peter Desnoyers]

Could this be related to a problem we've been seeing on the 860T,
resulting in the following messages?

    NETDEV WATCHDOG: eth0: transmit timed out
    eth0: transmit timed out.
    Ring data dump: cur_tx c02f7130, dirty_tx c02f7130 cur_rx: c02f70d8
     tx: 16 buffers
      c02f7100: 1c00 004e 02a69442
      c02f7108: 1c00 004e 02a69542
      c02f7110: 1c00 004e 02a69642
      c02f7118: 1c00 004e 02a69742
      c02f7120: 1c00 004e 02a69842
      c02f7128: 1c00 004e 02a69942
      c02f7130: 1c00 0049 02a2e05e
      c02f7138: 1c00 0042 02a69c4e
      c02f7140: 1c04 0042 02aa2b3e
      c02f7148: 1c00 0036 02c8f6b2
      c02f7150: 1c00 0042 02aaaf2e
      c02f7158: 1c00 0042 02aa213e
      c02f7160: 1c00 0042 02aa223e
      c02f7168: 1c00 0042 02aa233e
      c02f7170: 1c00 0042 02a6914e
      c02f7178: 3c00 004e 02a69342
     rx: 32 buffers
      c02f7000: 8880 0040 002f6000
      [...]

This only seems to occur when receiving bulk data on a TCP connection -
we never saw it ftp-ing onto a system when we used an NFS filesystem,
but now that we're booting a ramdisk, an FTP get of more than 10-20k
results in a guaranteed transmitter timeout.

--
.....................................................................
 Peter Desnoyers            (781) 457-1165   pdesnoyers@chinook.com
 Chinook Communications     (617) 661-1979   pjd@fred.cambridge.ma.us
 100 Hayden Ave, Lexington MA 02421

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Kernel oops while routing

[Dan Malek]

Peter Desnoyers wrote:

> Could this be related to a problem we've been seeing on the 860T,
> resulting in the following messages?

Not likely.  In my experience this is the result of a hardware problem,
either noise on the lines between the 860 and PHY, or an incorectly
configured set of I/O pins.

Thanks.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Kernel oops while routing

[Jean-Denis Boyer]

>     NETDEV WATCHDOG: eth0: transmit timed out
> Could this be related to a problem we've been seeing on the 860T,

I don't think so, the driver on the 860T is really stable under heavy
traffic and transmission errors. I was not able to kill it. The FEC itself
is relatively simple to manage, compared to the complex FCC on the 8260.

Your problem seems to be related to a bug I've found in the past, in the
ethernet driver. It appeared when filling the transmit ring buffer. It was
easy to reproduce when connected to a 10baseT. On a 100baseT, I should use
the packet sockets :-E. The fix has been incorporated in version 2.4.11. The
tx_full flag was not set to 1.


--------------------------------------------
 Jean-Denis Boyer, B.Eng., Technical Leader
 Mediatrix Telecom Inc.
 4229 Garlock Street
 Sherbrooke (Québec)
 J1L 2C8  CANADA
 (819)829-8749 x241
--------------------------------------------

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re[2]: Kernel oops while routing

[Ricardo Scop]

Jean-Denis,


Wednesday, December 05, 2001, 3:01:28 PM, you wrote:


JDB> IMHO, I could suggest an easier patch, that would result in modifying only
JDB> one line of code, without changing the 'tx_full' logic. In function
JDB> fcc_enet_start_xmit, instead of checking the ready bit (which is bad), we
JDB> could only check if cur_tx has reached dirty_tx, and then call
JDB> netif_stop_queue. Does it make sense?

Make sense to me. I'll try it out.

JDB> BTW, I worked hard last week in debugging the fcc_enet driver. It was not
JDB> handling correctly some transmission errors, resulting in the transmitter
JDB> completely stopping, without restarting. This is related to an errata
JDB> (CPM37) from Motorola about the 8260, concerning the way of restarting the
JDB> transmitter. If someone is interested, I can release a patch for that.

I'm interested!

Ricardo Scop                            mailto:scop@vanet.com.br
R SCOP Consulting

------------------------------------------------------------------
"What's money? A man is a success if he gets up in the morning and
goes to bed at night and in between does what he wants to do."
~Bob Dylan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Kernel oops while routing

[Arto Vuori]

Jean-Denis Boyer wrote:
> IMHO, I could suggest an easier patch, that would result in modifying only
> one line of code, without changing the 'tx_full' logic. In function
> fcc_enet_start_xmit, instead of checking the ready bit (which is bad), we
> could only check if cur_tx has reached dirty_tx, and then call
> netif_stop_queue. Does it make sense?

I have fixed our version of the driver with very similar modifications.
Since that it has not leaked any memory while it has been under testing
for several months.

>
> BTW, I worked hard last week in debugging the fcc_enet driver. It was not
> handling correctly some transmission errors, resulting in the transmitter
> completely stopping, without restarting. This is related to an errata
> (CPM37) from Motorola about the 8260, concerning the way of restarting the
> transmitter. If someone is interested, I can release a patch for that.
>

I have noticed the same problem on old kernel versions. I think i
mentioned about it on the mailing list and i understood that it had been
already fixed on later versions, but i didn't check the status of the
latest driver version.

- Arto

--
Arto Vuori
email: avuori@ssh.com
mobile:	+358 40 754 5223

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/