* SKB corruption on heavy traffic
@ 2008-04-29 18:39 Franca, Jose (NSN - PT/Portugal - MiniMD)
2008-04-29 19:15 ` Scott Wood
0 siblings, 1 reply; 12+ messages in thread
From: Franca, Jose (NSN - PT/Portugal - MiniMD) @ 2008-04-29 18:39 UTC (permalink / raw)
To: linuxppc-dev, linuxppc-embedded
[-- Attachment #1: Type: text/plain, Size: 29183 bytes --]
Hello Community!
We are developing a MPC8247 based telecom board (512MB), using
linux 2.4 with some proprietary changes on IP stack and we are facing
some problems when we have heavy traffic on our Ethernet interfaces...
Namely:
- When sending a ping flood we have skb corruption after
some variable time (looks like memory problems). Resuming what seems to
be happening: after release (dealloc) of the skb, the kernel tries to
access it again. To verify this we have implemented a list of "to be
freed" skb to have a delay between the time they are freed by the stack
and the actual dealloc. In this delay we are able to verify that the skb
is changed.
- We have reports of DMA transfers not completed in
heavy traffic.
- After a while, and skb corrupted messages appearing in
the console, we start to have a scrambled characters. So, after some
seconds we don't understand what is written. This problem solves after
the crash or reboot. Which means that uart also seems to crack in heavy
load or at least some.
Below are some of our traces that show our problem... Someone has some
hints of what could be leading to this?? HHHHEEEEEELLLPP!!!!
Huge Thanks!
Filipe.
F:bcmhal_shape_modid_phyport_to_pnum L:1046 Invalid modid received 48
port 3
Apr 25 19:59:14 system: F:bcmhal_shape_modid_phyport_to_pnum L:1046
Invalid modid received 48 port 3
Apr 25 19:59:14 system: F:bcm_enet_rx L:1788 Packet received from an
invalid port 3 modid 48
F:bcmhal_shape_modid_phyport_to_pnum L:1046 Invalid modid received 48
port 3
Apr 25 20:02:39 system: F:bcmhal_shape_modid_phyport_to_pnum L:1046
Invalid modid received 48 port 3
Apr 25 20:02:39 system: F:bcm_enet_rx L:1788 Packet received from an
invalid port 3 modid 48
F:bcmhal_shape_modid_phyport_to_pnum L:1046 Invalid modid received 48
port 3
Apr 25 20:06:57 system: F:bcmhal_shape_modid_phyport_to_pnum L:1046
Invalid modid received 48 port 3
Apr 25 20:06:57 system: F:bcm_enet_rx L:1788 Packet received from an
invalid port 3 modid 48
Apr 25 21:54:06 comd: buf = f5 43 7d 43 bb ff 80 18 36 70 6e 91 00 00
01 01 08 0a 00 0e c1 6d 00 00 41 15 6e 29 01 07 04 e6
Apr 25 21:54:08 comd: buf = 00 00 00 00 00 00 00 00 00 00 00 00 43 49
47 47 07 05 20 71 00 00 00 00 00 00 00 00 00 00 00 03
Apr 25 21:54:09 system: skbuff.c:skb_clone:647: Bingle, got a freed
skb being cloned, return NULL!!!<1> skbuff.c:skb_clone:647: Bingle, got
a freed
skb being cloned, return NULL!!!<3>tcp_write_xmit: clone failed !
Apr 25 21:54:09 comd: buf = 00 6e 29 01 07 04 e6 55 7b 00 00 ff ff 33
2d ff ff 00 00 00 0f 33 2d 00 03 00 00 00 08 00 00 00
Apr 25 21:54:09 comd: buf = 00 00 00 00 00 00 00 00 00 00 00 00 53 49
44 41 2a 00 5a f9 00 00 00 00 00 00 00 00 00 00 00 01
Apr 25 21:54:11 comd: message data field too long, max=12288 fd=82
((nil)) source: shelf=2021 slot=2223 service=2425 port=2627
destination: s
helf=1819 slot=1a1b service=1c1d port=1e1f message type: 28292a2b
length: 2c2d2e2f fd out of sync, will be closed (nil).
Oops: Exception in kernel mode, sig: 7
NIP: C0034138 XER: 20000000 LR: C011B73C SP: C04E3D20 REGS: c04e3c70
TRAP: 0600 Not tainted
MSR: 00009032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
DAR: C1611DBF, DSISR: 00010920
TASK = c04e2000[2] 'keventd' Last syscall: 120
last math cb0f8000 last altivec 00000000
GPR00: C1611DBF C04E3D20 C04E2000 C1611DAB 00000000 00000190 C1611DAB
0003CAEC
GPR08: C0F98B04 D1C8201A C0F5C018 C0230000 22002024 1031E1BC 0FFD5000
1009B000
GPR16: 00000000 00000000 00000000 00000000 003FF000 00000001 00000000
C01E0000
GPR24: C01F0000 06D01B7F C04E3E18 00000001 C01D0000 0000158E 0000AC70
C3442DA0
TESCR1 0xf0010040: 00100000 TESCR2 0xf0010044: 00000000
ESR 0xf0010884: 00000000
PCI_EACR 0xf0010890: 0000000c
PCI_EDCR 0xf0010898: 00000000
PCI_ECCR 0xf00108a0: 00000000
GFPGA regs
0xd0000000 1901200803051610 0000000000090101
0xd0000100 007f7f7f7f7f7f7f 7f7f7f7f7f7f7f7f
0xd0000200 008dd3d3d3d3d3d3 d3d3d3d3d3d3d3d3
0xd0000280 00ffffff00000000 d3d3d3d3d3d3d3d3
0xd0000300 ea19b993faff0040 0000050000001600
0xd0000380 000f0f0f00000000 0000000000000000
0xd0000400 660f01ff0100a5a5 a5a5a5a5a5a5a5a5
0xd0000480 0001010100000000 a5a5a5a5a5a5a5a5
0xd0000500 0f01010101010101 0101010101010101
0xd0000580 003f3f3f00000000 0101010101010101
0xd0000600 1f9a030a24440100 0000000000000000
0xd0000680 00ffffff00000000 0000000000000000
0xd0000690 00ffffff00000000 0000000000000000
0xd00006a0 00ffffff00000000 0000000000000000
0xd00006b0 00e727c000000000 0000000000000000
Call backtrace:
C011D8CC C011B73C C011B7D0 C011B9C8 C011BC24 C01496CC C014A054
C014D380 C0156584 C0156B70 C0137B08 C0137FD8 C0120B9C C0120C58
C001B338 C002517C C000876C
Apr 25 21:54:12 system: skbuff.c:skb_clone:647: Bingle, got a freed
skb being cloned, return NULL!!!<1> skbuff.c:skb_clone:647: Bingle, got
a freed
skb being cloned, return NULL!!!<1> skbuff.c:skb_clone:647: Bingle, got
a freed skb being cloned, return NULL!!!<1> skbuff.c:skb_clone:647:
Bingle, go
t a freed skb being cloned, return NULL!!!9C C0120C58
Apr 25 21:54:12 system: C001B338 C002517C C000876C
Apr 25 21:54:12 system: skb_release_data:400 memory ffffee84 out of
range!
Apr 25 21:54:12 system: Call backtrace:
Apr 25 21:54:12 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:12 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:12 system: C001B338 C002517C C000876C
Apr 25 21:54:12 system: skb_release_data:400 memory ffffeea0 out of
range!
Apr 25 21:54:12 system: Call backtrace:
Apr 25 21:54:12 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:12 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:12 system: C001B338 C002517C C000876C
Apr 25 21:54:12 system: skb_release_data:400 memory ffffeec4 out of
range!
Apr 25 21:54:12 system: Call backtrace:
Apr 25 21:54:12 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:12 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:12 system: C001B338 C002517C C000876C
Apr 25 21:54:12 system: skb_release_data:400 memory ffffeef8 out of
range!
Apr 25 21:54:12 system: Call backtrace:
Apr 25 21:54:12 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:12 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:12 system: C001B338 C002517C C000876C
Apr 25 21:54:12 system: skb_release_data:400 memory ffffef34 out of
range!
Apr 25 21:54:12 system: Call backtrace:
Apr 25 21:54:12 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:12 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:12 system: C001B338 C002517C C000876C
Apr 25 21:54:12 system: skb_release_data:400 memory ffffef44 out of
range!
Apr 25 21:54:12 system: Call backtrace:
Apr 25 21:54:13 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:13 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:13 system: C001B338 C002517C C000876C
Apr 25 21:54:13 system: skb_release_data:400 memory ffffef54 out of
range!
Apr 25 21:54:13 system: Call backtrace:
Apr 25 21:54:13 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:13 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:13 system: C001B338 C002517C C000876C
Apr 25 21:54:13 system: skb_release_data:400 memory ffffef64 out of
range!
Apr 25 21:54:13 system: Call backtrace:
Apr 25 21:54:13 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:13 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:13 system: C001B338 C002517C C000876C
Apr 25 21:54:13 system: skb_release_data:400 memory ffffef74 out of
range!
Apr 25 21:54:13 system: Call backtrace:
Apr 25 21:54:13 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:13 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:13 system: C001B338 C002517C C000876C
Apr 25 21:54:13 system: skb_release_data:400 memory 41646472 out of
range!
Apr 25 21:54:13 system: Call backtrace:
Apr 25 21:54:13 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:13 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
linux-bcm-core (127): soc_counter_thread: DMA did not finish
buf32=cd380000
linux-bcm-core (132): soc_counter_thread: DMA did not finish
buf32=cd389800
linux-bcm-core (130): soc_counter_thread: DMA did not finish
buf32=cd382400
Apr 25 21:54:13 system: C001B338 C002517C C000876C
Apr 25 21:54:13 system: skb_release_data:400 memory 700a0000 out of
range!
Apr 25 21:54:13 system: Call backtrace:
Apr 25 21:54:13 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:13 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:13 system: C001B338 C002517C C000876C
Apr 25 21:54:13 system: skb_release_data:400 memory 6d654164 out of
range!
Apr 25 21:54:13 system: Call backtrace:
Apr 25 21:54:13 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:13 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:13 system: C001B338 C002517C C000876C
Apr 25 21:54:13 system: skb_release_data:400 memory 2025700a out of
range!
Apr 25 21:54:13 system: Call backtrace:
Apr 25 21:54:13 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:13 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:13 system: C001B338 C002517C C000876C
Apr 25 21:54:13 system: skb_release_data:400 memory 496e2025 out of
range!
Apr 25 21:54:13 system: Call backtrace:
Apr 25 21:54:13 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:13 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:13 system: C001B338 C002517C C000876C
Apr 25 21:54:13 system: skb_release_data:400 memory 70293a0a out of
range!
Apr 25 21:54:13 system: Call backtrace:
Apr 25 21:54:13 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:13 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:13 system: C001B338 C002517C C000876C
Apr 25 21:54:13 system: skb_release_data:400 memory 726f6c42 out of
range!
Apr 25 21:54:13 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:14 system: skb_release_data:400 memory 72616365 out of
range!
Apr 25 21:54:14 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:14 system: skb_release_data:400 memory 31000000 out of
range!
Apr 25 21:54:14 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:14 system: skb_release_data:400 memory 73202825 out of
range!
Apr 25 21:54:14 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:14 system: skb_release_data:400 memory 70526574 out of
range!
Apr 25 21:54:14 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:14 system: skb_release_data:400 memory 203d2025 out of
range!
Apr 25 21:54:14 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:14 system: skb_release_data:400 memory 726f6c42 out of
range!
Apr 25 21:54:14 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:14 system: skb_release_data:400 memory 72616365 out of
range!
Apr 25 21:54:14 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:14 system: skb_release_data:400 memory 32000000 out of
range!
Apr 25 21:54:14 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:14 system: skb_release_data:400 memory 61636b74 out of
range!
Apr 25 21:54:14 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:14 system: skb_release_data:400 memory 54657374 out of
range!
Apr 25 21:54:14 system: Call backtrace:
Apr 25 21:54:14 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:14 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:14 system: C001B338 C002517C C000876C
Apr 25 21:54:15 system: skb_release_data:400 memory 68756875 out of
range!
Apr 25 21:54:15 system: Call backtrace:
Apr 25 21:54:15 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:15 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:15 system: C001B338 C002517C C000876C
Apr 25 21:54:15 system: skb_release_data:400 memory 10015ddc out of
range!
Apr 25 21:54:15 system: Call backtrace:
Apr 25 21:54:15 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:15 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:15 system: C001B338 C002517C C000876C
Apr 25 21:54:15 system: skb_release_data:400 memory 0000ffff out of
range!
Apr 25 21:54:15 system: Call backtrace:
Apr 25 21:54:15 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:15 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:15 system: C001B338 C002517C C000876C
Apr 25 21:54:15 system: skb_release_data:400 memory 64437472 out of
range!
Apr 25 21:54:15 system: Call backtrace:
Apr 25 21:54:15 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:15 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:15 system: C001B338 C002517C C000876C
Apr 25 21:54:15 system: skb_release_data:400 memory 636b0000 out of
range!
Apr 25 21:54:15 system: Call backtrace:
Apr 25 21:54:15 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:15 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
linux-bcm-core (127): soc_counter_thread: DMA did not finish
buf32=cd380000
linux-bcm-core (132): soc_counter_thread: DMA did not finish
buf32=cd389800
linux-bcm-core (130): soc_counter_thread: DMA did not finish
buf32=cd382400
Apr 25 21:54:15 system: C001B338 C002517C C000876C
Apr 25 21:54:15 system: skb_release_data:400 memory 014a5374 out of
range!
Apr 25 21:54:15 system: Call backtrace:
Apr 25 21:54:15 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:15 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:15 system: C001B338 C002517C C000876C
Apr 25 21:54:15 system: skb_release_data:400 memory 7053796e out of
range!
Apr 25 21:54:15 system: Call backtrace:
Apr 25 21:54:15 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:15 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:15 system: C001B338 C002517C C000876C
Apr 25 21:54:15 system: skb_release_data:400 memory ee0e612c out of
range!
Apr 25 21:54:15 system: Call backtrace:
Apr 25 21:54:15 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:15 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:15 system: C001B338 C002517C C000876C
Apr 25 21:54:15 system: skb_release_data:400 memory 076dc419 out of
range!
Apr 25 21:54:15 system: Call backtrace:
Apr 25 21:54:15 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:15 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:15 system: C001B338 C002517C C000876C
Apr 25 21:54:15 system: skb_release_data:400 memory e963a535 out of
range!
Apr 25 21:54:15 system: Call backtrace:
Apr 25 21:54:15 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:15 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:15 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory 0edb8832 out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:16 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory e0d5e91e out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:16 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory 09b64c2b out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:16 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory e7b82d07 out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:16 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory 1db71064 out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:16 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory f3b97148 out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:16 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory 1adad47d out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:16 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory f4d4b551 out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:16 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory 136c9856 out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:16 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory fd62f97a out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:16 system: C001B338 C002517C C000876C
Apr 25 21:54:16 system: skb_release_data:400 memory 14015c4f out of
range!
Apr 25 21:54:16 system: Call backtrace:
Apr 25 21:54:16 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:16 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:17 system: C001B338 C002517C C000876C
Apr 25 21:54:17 system: skb_release_data:400 memory fa0f3d63 out of
range!
Apr 25 21:54:17 system: Call backtrace:
Apr 25 21:54:17 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:17 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:17 system: C001B338 C002517C C000876C
Apr 25 21:54:17 system: skb_release_data:400 memory 3b6e20c8 out of
range!
Apr 25 21:54:17 system: Call backtrace:
Apr 25 21:54:17 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:17 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:17 system: C001B338 C002517C C000876C
Apr 25 21:54:17 system: skb_release_data:400 memory d56041e4 out of
range!
Apr 25 21:54:17 system: Call backtrace:
Apr 25 21:54:17 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:17 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:17 system: C001B338 C002517C C000876C
Apr 25 21:54:17 system: skb_release_data:400 memory 3c03e4d1 out of
range!
Apr 25 21:54:17 system: Call backtrace:
Apr 25 21:54:17 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:17 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:17 system: C001B338 C002517C C000876C
Apr 25 21:54:17 system: skb_release_data:400 memory d20d85fd out of
range!
Apr 25 21:54:17 system: Call backtrace:
Apr 25 21:54:17 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:17 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
linux-bcm-core (127): soc_counter_thread: DMA did not finish
buf32=cd380000
linux-bcm-core (132): soc_counter_thread: DMA did not finish
buf32=cd389800
linux-bcm-core (130): soc_counter_thread: DMA did not finish
buf32=cd382400
Apr 25 21:54:17 system: C001B338 C002517C C000876C
Apr 25 21:54:17 system: skb_release_data:400 memory 35b5a8fa out of
range!
Apr 25 21:54:17 system: Call backtrace:
Apr 25 21:54:17 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:17 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:17 system: C001B338 C002517C C000876C
Apr 25 21:54:17 system: skb_release_data:400 memory dbbbc9d6 out of
range!
Apr 25 21:54:17 system: Call backtrace:
Apr 25 21:54:17 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:17 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:17 system: C001B338 C002517C C000876C
Apr 25 21:54:17 system: skb_release_data:400 memory 32d86ce3 out of
range!
Apr 25 21:54:17 system: Call backtrace:
Apr 25 21:54:17 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:17 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:17 system: C001B338 C002517C C000876C
Apr 25 21:54:17 system: skb_release_data:400 memory dcd60dcf out of
range!
Apr 25 21:54:17 system: Call backtrace:
Apr 25 21:54:17 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:17 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:17 system: C001B338 C002517C C000876C
Apr 25 21:54:17 system: skb_release_data:400 memory 26d930ac out of
range!
Apr 25 21:54:17 system: Call backtrace:
Apr 25 21:54:17 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:17 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:18 system: C001B338 C002517C C000876C
Apr 25 21:54:18 system: skb_release_data:400 memory 21b4f4b5 out of
range!
Apr 25 21:54:18 system: Call backtrace:
Apr 25 21:54:18 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:18 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:18 system: C001B338 C002517C C000876C
Apr 25 21:54:18 system: skb_release_data:400 memory cfba9599 out of
range!
Apr 25 21:54:18 system: Call backtrace:
Apr 25 21:54:18 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:18 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:18 system: C001B338 C002517C C000876C
Apr 25 21:54:18 system: skb_release_data:400 memory 2802b89e out of
range!
Apr 25 21:54:18 system: Call backtrace:
Apr 25 21:54:18 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:18 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:18 system: C001B338 C002517C C000876C
Apr 25 21:54:18 system: skb_release_data:400 memory 2f6f7c87 out of
range!
Apr 25 21:54:18 system: Call backtrace:
Apr 25 21:54:18 system: C011D8CC C011B720 C011B7D0 C011B9C8 C011BC24
C01496CC C014A054
Apr 25 21:54:18 system: C014D380 C0156584 C0156B70 C0137B08 C0137FD8
C0120B9C C0120C58
Apr 25 21:54:18 system: C001B338 C002517C C000876C
Apr 25 21:54:18 system: Oops: Exception in kernel mode, sig: 7
Apr 25 21:54:18 system: NIP: C0034138 XER: 20000000 LR: C011B73C SP:
C04E3D20 REGS: c04e3c70 TRAP: 0600 Not tainted
Apr 25 21:54:18 system: MSR: 00009032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
Apr 25 21:54:18 system: DAR: C1611DBF, DSISR: 00010920
Apr 25 21:54:18 system: TASK = c04e2000[2] 'keventd' Last syscall: 120
Apr 25 21:54:18 system: last math cb0f8000 last altivec 00000000
Apr 25 21:54:18 system: GPR00: C1611DBF C04E3D20 C04E2000 C1611DAB
00000000 00000190 C1611DAB 0003CAEC
Apr 25 21:54:18 system: GPR08: C0F98B04 D1C8201A C0F5C018 C0230000
22002024 1031E1BC 0FFD5000 1009B000
Apr 25 21:54:18 system: GPR16: 00000000 00000000 00000000 00000000
003FF000 00000001 00000000 C01E0000
Apr 25 21:54:18 system: GPR24: C01F0000 06D01B7F C04E3E18 00000001
C01D0000 0000158E 0000AC70 C3442DA0
Apr 25 21:54:18 system: TESCR1 0xf0010040: 00100000 TESCR2 0xf0010044:
00000000
Apr 25 21:54:18 system: ESR 0xf0010884: 00000000
Apr 25 21:54:18 system: PCI_EACR 0xf0010890: 0000000c
Apr 25 21:54:18 system: PCI_EDCR 0xf0010898: 00000000
Apr 25 21:54:18 system: PCI_ECCR 0xf00108a0: 00000000
[-- Attachment #2: Type: text/html, Size: 48110 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: SKB corruption on heavy traffic
2008-04-29 18:39 Franca, Jose (NSN - PT/Portugal - MiniMD)
@ 2008-04-29 19:15 ` Scott Wood
2008-04-30 8:43 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
0 siblings, 1 reply; 12+ messages in thread
From: Scott Wood @ 2008-04-29 19:15 UTC (permalink / raw)
To: Franca, Jose (NSN - PT/Portugal - MiniMD); +Cc: linuxppc-dev, linuxppc-embedded
On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - PT/Portugal - MiniMD) wrote:
> We are developing a MPC8247 based telecom board (512MB), using
> linux 2.4 with some proprietary changes on IP stack and we are facing
> some problems when we have heavy traffic on our Ethernet interfaces...
Do you see these problems without the proprietary changes, and with a
current kernel?
-Scott
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: SKB corruption on heavy traffic
2008-04-29 19:15 ` Scott Wood
@ 2008-04-30 8:43 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
2008-04-30 15:00 ` Scott Wood
0 siblings, 1 reply; 12+ messages in thread
From: Franca, Jose (NSN - PT/Portugal - MiniMD) @ 2008-04-30 8:43 UTC (permalink / raw)
To: ext Scott Wood; +Cc: linuxppc-dev, linuxppc-embedded
Hello!
Thank you for replying!
It't quite dificult to say if the problem exists without our changes, =
since the all software is dependent on this changes so to work with the =
hardware. I can't answer to that right now on that, but I forgot to add =
one thing: we have ring buffer full problems on our fcc_enet driver from =
time to time. So, I think the problem could be on linux configurations =
(related to hw) because there is a lot of posts on the web related to =
problems similar to this (none of them has really solved the bottom =
problem).=20
Regards,
Filipe=20
-----Original Message-----
From: ext Scott Wood [mailto:scottwood@freescale.com]=20
Sent: ter=E7a-feira, 29 de Abril de 2008 20:15
To: Franca, Jose (NSN - PT/Portugal - MiniMD)
Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
Subject: Re: SKB corruption on heavy traffic
On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - =
PT/Portugal - MiniMD) wrote:
> We are developing a MPC8247 based telecom board (512MB), using
> linux 2.4 with some proprietary changes on IP stack and we are facing
> some problems when we have heavy traffic on our Ethernet interfaces...
Do you see these problems without the proprietary changes, and with a
current kernel?
-Scott
^ permalink raw reply [flat|nested] 12+ messages in thread
* FW: SKB corruption on heavy traffic
@ 2008-04-30 9:07 Franca, Jose (NSN - PT/Portugal - MiniMD)
2008-04-30 12:25 ` Gerhard Pircher
2008-05-01 20:55 ` Myron.Dixon
0 siblings, 2 replies; 12+ messages in thread
From: Franca, Jose (NSN - PT/Portugal - MiniMD) @ 2008-04-30 9:07 UTC (permalink / raw)
To: linuxppc-dev, linuxppc-embedded
>From our latest debugs we found that the problem occurs mainly on skbuff =
code. After some variable time kfree or kalloc result in kernel oops.
-----Original Message-----
From: Franca, Jose (NSN - PT/Portugal - MiniMD)=20
Sent: quarta-feira, 30 de Abril de 2008 9:44
To: 'ext Scott Wood'
Cc: =09
Subject: RE: SKB corruption on heavy traffic
Hello!
Thank you for replying!
It't quite dificult to say if the problem exists without our changes, =
since the all software is dependent on this changes so to work with the =
hardware. I can't answer to that right now on that, but I forgot to add =
one thing: we have ring buffer full problems on our fcc_enet driver from =
time to time. So, I think the problem could be on linux configurations =
(related to hw) because there is a lot of posts on the web related to =
problems similar to this (none of them has really solved the bottom =
problem).=20
Regards,
Filipe=20
-----Original Message-----
From: ext Scott Wood [mailto:scottwood@freescale.com]=20
Sent: ter=E7a-feira, 29 de Abril de 2008 20:15
To: Franca, Jose (NSN - PT/Portugal - MiniMD)
Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
Subject: Re: SKB corruption on heavy traffic
On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - =
PT/Portugal - MiniMD) wrote:
> We are developing a MPC8247 based telecom board (512MB), using
> linux 2.4 with some proprietary changes on IP stack and we are facing
> some problems when we have heavy traffic on our Ethernet interfaces...
Do you see these problems without the proprietary changes, and with a
current kernel?
-Scott
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: FW: SKB corruption on heavy traffic
2008-04-30 9:07 FW: SKB corruption on heavy traffic Franca, Jose (NSN - PT/Portugal - MiniMD)
@ 2008-04-30 12:25 ` Gerhard Pircher
2008-04-30 13:03 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
2008-05-01 20:55 ` Myron.Dixon
1 sibling, 1 reply; 12+ messages in thread
From: Gerhard Pircher @ 2008-04-30 12:25 UTC (permalink / raw)
To: Franca, Jose (NSN - PT/Portugal - MiniMD), Scott Wood
Cc: linuxppc-dev, linuxppc-embedded
Hi,
I think I have the same problem here with all versions of the 2.6.x kernel
series (tested with kernel v2.6.8/14/16/18/25 on a PPC7455 machine with
different PCI network cards by transferring a big file over NFS/SCP). Data
corruption occurs under high load, but I don't get any kernel oops.
regards,
Gerhard
-------- Original-Nachricht --------
> Datum: Wed, 30 Apr 2008 10:07:15 +0100
> Von: "Franca, Jose (NSN - PT/Portugal - MiniMD)" <jose.franca@nsn.com>
> An: linuxppc-dev@ozlabs.org, linuxppc-embedded@ozlabs.org
> Betreff: FW: SKB corruption on heavy traffic
> >From our latest debugs we found that the problem occurs mainly on skbuff
> code. After some variable time kfree or kalloc result in kernel oops.
>
> -----Original Message-----
> From: Franca, Jose (NSN - PT/Portugal - MiniMD)
> Sent: quarta-feira, 30 de Abril de 2008 9:44
> To: 'ext Scott Wood'
> Cc:
> Subject: RE: SKB corruption on heavy traffic
>
> Hello!
>
> Thank you for replying!
> It't quite dificult to say if the problem exists without our
> changes, since the all software is dependent on this changes so to work
> with the hardware. I can't answer to that right now on that, but I forgot
> to add one thing: we have ring buffer full problems on our fcc_enet
> driver from time to time. So, I think the problem could be on linux
> configurations (related to hw) because there is a lot of posts on the web
> related to problems similar to this (none of them has really solved the
> bottom problem).
>
> Regards,
> Filipe
>
> -----Original Message-----
> From: ext Scott Wood [mailto:scottwood@freescale.com]
> Sent: terça-feira, 29 de Abril de 2008 20:15
> To: Franca, Jose (NSN - PT/Portugal - MiniMD)
> Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
> Subject: Re: SKB corruption on heavy traffic
>
> On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - PT/Portugal
> - MiniMD) wrote:
> > We are developing a MPC8247 based telecom board (512MB), using
> > linux 2.4 with some proprietary changes on IP stack and we are facing
> > some problems when we have heavy traffic on our Ethernet interfaces...
>
> Do you see these problems without the proprietary changes, and with a
> current kernel?
>
> -Scott
--
Psst! Geheimtipp: Online Games kostenlos spielen bei den GMX Free Games!
http://games.entertainment.gmx.net/de/entertainment/games/free
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: FW: SKB corruption on heavy traffic
2008-04-30 12:25 ` Gerhard Pircher
@ 2008-04-30 13:03 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
2008-04-30 19:20 ` Gerhard Pircher
0 siblings, 1 reply; 12+ messages in thread
From: Franca, Jose (NSN - PT/Portugal - MiniMD) @ 2008-04-30 13:03 UTC (permalink / raw)
To: ext Gerhard Pircher, Scott Wood; +Cc: linuxppc-dev, linuxppc-embedded
Hi!
There was a sugestion to change slab to slub alocation method... I =
don't know quite well yet what is necessary to do this, but it seems =
that the current implementation of slub is more commonly available on =
2.6 kernels, not in 2.4 that I use :(...
Any guesses or hints on this?
Regards!
Filipe.
-----Original Message-----
From: ext Gerhard Pircher [mailto:gerhard_pircher@gmx.net]=20
Sent: quarta-feira, 30 de Abril de 2008 13:26
To: Franca, Jose (NSN - PT/Portugal - MiniMD); Scott Wood
Cc: linuxppc-embedded@ozlabs.org; linuxppc-dev@ozlabs.org
Subject: Re: FW: SKB corruption on heavy traffic
Hi,
I think I have the same problem here with all versions of the 2.6.x =
kernel
series (tested with kernel v2.6.8/14/16/18/25 on a PPC7455 machine with
different PCI network cards by transferring a big file over NFS/SCP). =
Data
corruption occurs under high load, but I don't get any kernel oops.
regards,
Gerhard
-------- Original-Nachricht --------
> Datum: Wed, 30 Apr 2008 10:07:15 +0100
> Von: "Franca, Jose (NSN - PT/Portugal - MiniMD)" <jose.franca@nsn.com>
> An: linuxppc-dev@ozlabs.org, linuxppc-embedded@ozlabs.org
> Betreff: FW: SKB corruption on heavy traffic
> >From our latest debugs we found that the problem occurs mainly on =
skbuff
> code. After some variable time kfree or kalloc result in kernel oops.
>=20
> -----Original Message-----
> From: Franca, Jose (NSN - PT/Portugal - MiniMD)=20
> Sent: quarta-feira, 30 de Abril de 2008 9:44
> To: 'ext Scott Wood'
> Cc: =09
> Subject: RE: SKB corruption on heavy traffic
>=20
> Hello!
>=20
> Thank you for replying!
> It't quite dificult to say if the problem exists without our
> changes, since the all software is dependent on this changes so to =
work
> with the hardware. I can't answer to that right now on that, but I =
forgot=20
> to add one thing: we have ring buffer full problems on our fcc_enet
> driver from time to time. So, I think the problem could be on linux
> configurations (related to hw) because there is a lot of posts on the =
web=20
> related to problems similar to this (none of them has really solved =
the
> bottom problem).=20
>=20
> Regards,
> Filipe=20
>=20
> -----Original Message-----
> From: ext Scott Wood [mailto:scottwood@freescale.com]=20
> Sent: ter=E7a-feira, 29 de Abril de 2008 20:15
> To: Franca, Jose (NSN - PT/Portugal - MiniMD)
> Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
> Subject: Re: SKB corruption on heavy traffic
>=20
> On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - =
PT/Portugal
> - MiniMD) wrote:
> > We are developing a MPC8247 based telecom board (512MB), using
> > linux 2.4 with some proprietary changes on IP stack and we are =
facing
> > some problems when we have heavy traffic on our Ethernet =
interfaces...
>=20
> Do you see these problems without the proprietary changes, and with a
> current kernel?
>=20
> -Scott
--=20
Psst! Geheimtipp: Online Games kostenlos spielen bei den GMX Free Games! =
http://games.entertainment.gmx.net/de/entertainment/games/free
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: SKB corruption on heavy traffic
2008-04-30 8:43 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
@ 2008-04-30 15:00 ` Scott Wood
2008-04-30 15:07 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
0 siblings, 1 reply; 12+ messages in thread
From: Scott Wood @ 2008-04-30 15:00 UTC (permalink / raw)
To: Franca, Jose (NSN - PT/Portugal - MiniMD); +Cc: linuxppc-dev, linuxppc-embedded
On Wed, Apr 30, 2008 at 09:43:33AM +0100, Franca, Jose (NSN - PT/Portugal - MiniMD) wrote:
> It't quite dificult to say if the problem exists without our
> changes, since the all software is dependent on this changes so
> to work with the hardware. I can't answer to that right now on
> that, but I forgot to add one thing: we have ring buffer full
> problems on our fcc_enet driver from time to time. So, I think
> the problem could be on linux configurations (related to hw)
> because there is a lot of posts on the web related to problems
> similar to this (none of them has really solved the bottom
> problem).
One thing you could try is to enable slab debugging, rather than your
custom debugging code; this will poison the memory upon free, making it
more likely that whatever is accessing it will crash immediately rather
than just cause corruption. Or, if you want to mantain the delay, do the
poisoning yourself.
Of course, this won't help if the access is write-only, or if the
corruption is happening via DMA.
When you say you have a full ring buffer, do you mean tx or rx? Does it
recover gracefully? Do you think it's caused by something other than
sending data faster than the line can support (tx) or receiving packets
faster than the cpu can handle (rx)?
Are any of the other problem reports from recent, unmodified kernels (2.4
is very old)?
-Scott
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: SKB corruption on heavy traffic
2008-04-30 15:00 ` Scott Wood
@ 2008-04-30 15:07 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
0 siblings, 0 replies; 12+ messages in thread
From: Franca, Jose (NSN - PT/Portugal - MiniMD) @ 2008-04-30 15:07 UTC (permalink / raw)
To: ext Scott Wood; +Cc: linuxppc-dev, linuxppc-embedded
Ok... We will try to enable debug on slab! :)
Ring buffer full occurs on tx in spite of having free bd's in the ring,
it doesn't use them... And it does not recover.
The problems that I refered are from previous kernel versions to ours
(our base of development was 2.4.31) and even on 2.6 kernels.
Filipe
-----Original Message-----
From: ext Scott Wood [mailto:scottwood@freescale.com]=20
Sent: quarta-feira, 30 de Abril de 2008 16:01
To: Franca, Jose (NSN - PT/Portugal - MiniMD)
Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
Subject: Re: SKB corruption on heavy traffic
On Wed, Apr 30, 2008 at 09:43:33AM +0100, Franca, Jose (NSN -
PT/Portugal - MiniMD) wrote:
> It't quite dificult to say if the problem exists without our
> changes, since the all software is dependent on this changes so
> to work with the hardware. I can't answer to that right now on
> that, but I forgot to add one thing: we have ring buffer full
> problems on our fcc_enet driver from time to time. So, I think
> the problem could be on linux configurations (related to hw)
> because there is a lot of posts on the web related to problems
> similar to this (none of them has really solved the bottom
> problem).
One thing you could try is to enable slab debugging, rather than your
custom debugging code; this will poison the memory upon free, making it
more likely that whatever is accessing it will crash immediately rather
than just cause corruption. Or, if you want to mantain the delay, do
the
poisoning yourself.
Of course, this won't help if the access is write-only, or if the
corruption is happening via DMA.
When you say you have a full ring buffer, do you mean tx or rx? Does it
recover gracefully? Do you think it's caused by something other than
sending data faster than the line can support (tx) or receiving packets
faster than the cpu can handle (rx)?
Are any of the other problem reports from recent, unmodified kernels
(2.4
is very old)?
-Scott
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: RE: FW: SKB corruption on heavy traffic
2008-04-30 13:03 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
@ 2008-04-30 19:20 ` Gerhard Pircher
0 siblings, 0 replies; 12+ messages in thread
From: Gerhard Pircher @ 2008-04-30 19:20 UTC (permalink / raw)
To: Franca, Jose (NSN - PT/Portugal - MiniMD), scottwood
Cc: linuxppc-dev, linuxppc-embedded
-------- Original-Nachricht --------
> Datum: Wed, 30 Apr 2008 14:03:02 +0100
> Von: "Franca, Jose (NSN - PT/Portugal - MiniMD)" <jose.franca@nsn.com>
> An: "ext Gerhard Pircher" <gerhard_pircher@gmx.net>, "Scott Wood" <scottwood@freescale.com>
> CC: linuxppc-embedded@ozlabs.org, linuxppc-dev@ozlabs.org
> Betreff: RE: FW: SKB corruption on heavy traffic
> Hi!
>
> There was a sugestion to change slab to slub alocation method... I > don't know quite well yet what is necessary to do this, but it seems that
> the current implementation of slub is more commonly available on 2.6
> kernels, not in 2.4 that I use :(...
> Any guesses or hints on this?
I'm using SLUB with kernel v2.6.25 and I still get data corruption on high
load. On the other side it is a good idea to enable SLAB debugging, as
Scott suggested. Maybe that sheds some light on this issue (even if most
of the network drivers make use of DMA).
regards,
Gerhard
>
> Regards!
> Filipe.
>
> -----Original Message-----
> From: ext Gerhard Pircher [mailto:gerhard_pircher@gmx.net]
> Sent: quarta-feira, 30 de Abril de 2008 13:26
> To: Franca, Jose (NSN - PT/Portugal - MiniMD); Scott Wood
> Cc: linuxppc-embedded@ozlabs.org; linuxppc-dev@ozlabs.org
> Subject: Re: FW: SKB corruption on heavy traffic
>
> Hi,
>
> I think I have the same problem here with all versions of the 2.6.x
> kernel series (tested with kernel v2.6.8/14/16/18/25 on a PPC7455 machine
> with different PCI network cards by transferring a big file over
> NFS/SCP). Data corruption occurs under high load, but I don't get any
> kernel oops.
>
> regards,
>
> Gerhard
--
Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten
Browser-Versionen downloaden: http://www.gmx.net/de/go/browser
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: SKB corruption on heavy traffic
2008-04-30 9:07 FW: SKB corruption on heavy traffic Franca, Jose (NSN - PT/Portugal - MiniMD)
2008-04-30 12:25 ` Gerhard Pircher
@ 2008-05-01 20:55 ` Myron.Dixon
2008-05-03 14:07 ` Matvejchikov Ilya
1 sibling, 1 reply; 12+ messages in thread
From: Myron.Dixon @ 2008-05-01 20:55 UTC (permalink / raw)
To: Franca, Jose (NSN - PT/Portugal - MiniMD); +Cc: linuxppc-dev, linuxppc-embedded
We have experience a very similar problem using a 2.4.18 kernel on an =
8260 ppc processor.
We have a telecomunication product that for some time only used the fec =
for TCP/IP ethernet=20
traffic only and worked just fine.
After we upgraded our product to implement TDM data over IP we started =
to notice an occasional
kernel oops. We began to evaluate all of our products and determined =
that only some of the units
exhibted this behaviour at various rates of occurrence. Further =
evaluation revealed that the pointers
located in dpram pointing to the fec's buffer descriptors were some how =
getting corrupted.
Note that the 8260 has 4 internal scc/fccs and we use all four for =
various aspects of our application
and each shares dpram for pointers to buffer descriptors that reside in =
sdram. However, only the
fec that is used for IP experiences this buffer descriptor corruption =
and, then, apparently, only under=20
heavy traffic load. We spent about six months evaluating this problem =
including contacting freescale,=20
but never found a solution. We finally, decided to use an external =
ethernet chip on a daughter card=20
for our IP channel.
It is, however, our belief that our problem relates to a possible bug in =
the 8260 CPM, but have yet to
absolutely prove this.
If we are experiencing the same problem (and potentially others) and =
there is a solution we would be
very interested, as, we are not happy about the daughter card solution. =
Myron L. Dixon
Sr. Software Engineer
L-3 Communications, GNS
1519 Grundy's Lane
Bristol, PA 19007
Phone: 215 957 3739
Fax: 215 957 3790
email: Myron.Dixon@L-3Com.com
-----Original Message-----
From: linuxppc-dev-bounces+myron.dixon=3Dl-3com.com@ozlabs.org =
[mailto:linuxppc-dev-bounces+myron.dixon=3Dl-3com.com@ozlabs.org] On =
Behalf Of Franca, Jose (NSN - PT/Portugal - MiniMD)
Sent: Wednesday, April 30, 2008 5:07 AM
To: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
Subject: FW: SKB corruption on heavy traffic
>From our latest debugs we found that the problem occurs mainly on skbuff =
code. After some variable time kfree or kalloc result in kernel oops.
-----Original Message-----
From: Franca, Jose (NSN - PT/Portugal - MiniMD)
Sent: quarta-feira, 30 de Abril de 2008 9:44
To: 'ext Scott Wood'
Cc: =09
Subject: RE: SKB corruption on heavy traffic
Hello!
Thank you for replying!
It't quite dificult to say if the problem exists without our changes, =
since the all software is dependent on this changes so to work with the =
hardware. I can't answer to that right now on that, but I forgot to add =
one thing: we have ring buffer full problems on our fcc_enet driver from =
time to time. So, I think the problem could be on linux configurations =
(related to hw) because there is a lot of posts on the web related to =
problems similar to this (none of them has really solved the bottom =
problem).=20
Regards,
Filipe=20
-----Original Message-----
From: ext Scott Wood [mailto:scottwood@freescale.com]
Sent: ter=E7a-feira, 29 de Abril de 2008 20:15
To: Franca, Jose (NSN - PT/Portugal - MiniMD)
Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
Subject: Re: SKB corruption on heavy traffic
On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - =
PT/Portugal - MiniMD) wrote:
> We are developing a MPC8247 based telecom board (512MB), using linux=20
> 2.4 with some proprietary changes on IP stack and we are facing some=20
> problems when we have heavy traffic on our Ethernet interfaces...
Do you see these problems without the proprietary changes, and with a =
current kernel?
-Scott
_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@ozlabs.org
https://ozlabs.org/mailman/listinfo/linuxppc-dev
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: SKB corruption on heavy traffic
2008-05-01 20:55 ` Myron.Dixon
@ 2008-05-03 14:07 ` Matvejchikov Ilya
2008-05-05 8:54 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
0 siblings, 1 reply; 12+ messages in thread
From: Matvejchikov Ilya @ 2008-05-03 14:07 UTC (permalink / raw)
To: Myron.Dixon
Cc: linuxppc-dev, linuxppc-embedded,
Franca, Jose (NSN - PT/Portugal - MiniMD)
[-- Attachment #1: Type: text/plain, Size: 4998 bytes --]
Hi all!
The same problem! I've tried 2.4.xx and 2.6.xx kernels. Nothing changed!
BUT. After many days of fucking with fs_enet driver I've found a
stable (as I see) solution. The bugs I've had:
- kernel oopses
- SKB data corruption
- BDs status corruption
- SKB ring full message
- too many RX errors
- may be something else :)
For now I have a 2.4.35 fs_enet driver that works on heavy load
24/7... I don't know what happens with my 8260 board, but with this
code it be very stable. I supposed that there are some errors in 8260
CPM core, but errata don't know about it :)
I've append an attachment with my 2.4.35 kernel patch. Sorry for a big
file and not for only fs_enet file. Moreover I've used CPM111 errata
microcode and NAPI in fs_enet driver.
If you have any questions I'm glad to hear it.
2008/5/2 <Myron.Dixon@l-3com.com>:
> We have experience a very similar problem using a 2.4.18 kernel on an 8260 ppc processor.
> We have a telecomunication product that for some time only used the fec for TCP/IP ethernet
> traffic only and worked just fine.
>
> After we upgraded our product to implement TDM data over IP we started to notice an occasional
> kernel oops. We began to evaluate all of our products and determined that only some of the units
> exhibted this behaviour at various rates of occurrence. Further evaluation revealed that the pointers
> located in dpram pointing to the fec's buffer descriptors were some how getting corrupted.
> Note that the 8260 has 4 internal scc/fccs and we use all four for various aspects of our application
> and each shares dpram for pointers to buffer descriptors that reside in sdram. However, only the
> fec that is used for IP experiences this buffer descriptor corruption and, then, apparently, only under
> heavy traffic load. We spent about six months evaluating this problem including contacting freescale,
> but never found a solution. We finally, decided to use an external ethernet chip on a daughter card
> for our IP channel.
>
> It is, however, our belief that our problem relates to a possible bug in the 8260 CPM, but have yet to
> absolutely prove this.
>
> If we are experiencing the same problem (and potentially others) and there is a solution we would be
> very interested, as, we are not happy about the daughter card solution.
>
>
> Myron L. Dixon
> Sr. Software Engineer
> L-3 Communications, GNS
> 1519 Grundy's Lane
> Bristol, PA 19007
> Phone: 215 957 3739
> Fax: 215 957 3790
> email: Myron.Dixon@L-3Com.com
>
> -----Original Message-----
> From: linuxppc-dev-bounces+myron.dixon=l-3com.com@ozlabs.org [mailto:linuxppc-dev-bounces+myron.dixon=l-3com.com@ozlabs.org] On Behalf Of Franca, Jose (NSN - PT/Portugal - MiniMD)
> Sent: Wednesday, April 30, 2008 5:07 AM
> To: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
> Subject: FW: SKB corruption on heavy traffic
>
> From our latest debugs we found that the problem occurs mainly on skbuff code. After some variable time kfree or kalloc result in kernel oops.
>
> -----Original Message-----
> From: Franca, Jose (NSN - PT/Portugal - MiniMD)
> Sent: quarta-feira, 30 de Abril de 2008 9:44
> To: 'ext Scott Wood'
> Cc:
> Subject: RE: SKB corruption on heavy traffic
>
> Hello!
>
> Thank you for replying!
> It't quite dificult to say if the problem exists without our changes, since the all software is dependent on this changes so to work with the hardware. I can't answer to that right now on that, but I forgot to add one thing: we have ring buffer full problems on our fcc_enet driver from time to time. So, I think the problem could be on linux configurations (related to hw) because there is a lot of posts on the web related to problems similar to this (none of them has really solved the bottom problem).
>
> Regards,
> Filipe
>
> -----Original Message-----
> From: ext Scott Wood [mailto:scottwood@freescale.com]
> Sent: terça-feira, 29 de Abril de 2008 20:15
> To: Franca, Jose (NSN - PT/Portugal - MiniMD)
> Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
> Subject: Re: SKB corruption on heavy traffic
>
> On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - PT/Portugal - MiniMD) wrote:
> > We are developing a MPC8247 based telecom board (512MB), using linux
> > 2.4 with some proprietary changes on IP stack and we are facing some
> > problems when we have heavy traffic on our Ethernet interfaces...
>
> Do you see these problems without the proprietary changes, and with a current kernel?
>
> -Scott
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>
[-- Attachment #2: linux-2.4.35-m82-2.patch.gz --]
[-- Type: application/x-gzip, Size: 21860 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: SKB corruption on heavy traffic
2008-05-03 14:07 ` Matvejchikov Ilya
@ 2008-05-05 8:54 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
0 siblings, 0 replies; 12+ messages in thread
From: Franca, Jose (NSN - PT/Portugal - MiniMD) @ 2008-05-05 8:54 UTC (permalink / raw)
To: matvejchikov, Myron.Dixon; +Cc: linuxppc-dev, linuxppc-embedded
Hi!
Thanks for your contribution! We will try it right away! :))
As for patches, we also installed some patches provided by freescale =
and nothing so far... Still the same problems.
I will post the results as soon as I can...
Thanks U ALL!
Filipe.
-----Original Message-----
From: ext Matvejchikov Ilya [mailto:matvejchikov@gmail.com]=20
Sent: s=E1bado, 3 de Maio de 2008 15:08
To: Myron.Dixon@l-3com.com
Cc: Franca, Jose (NSN - PT/Portugal - MiniMD); linuxppc-dev@ozlabs.org; =
linuxppc-embedded@ozlabs.org
Subject: Re: SKB corruption on heavy traffic
Hi all!
The same problem! I've tried 2.4.xx and 2.6.xx kernels. Nothing changed!
BUT. After many days of fucking with fs_enet driver I've found a
stable (as I see) solution. The bugs I've had:
- kernel oopses
- SKB data corruption
- BDs status corruption
- SKB ring full message
- too many RX errors
- may be something else :)
For now I have a 2.4.35 fs_enet driver that works on heavy load
24/7... I don't know what happens with my 8260 board, but with this
code it be very stable. I supposed that there are some errors in 8260
CPM core, but errata don't know about it :)
I've append an attachment with my 2.4.35 kernel patch. Sorry for a big
file and not for only fs_enet file. Moreover I've used CPM111 errata
microcode and NAPI in fs_enet driver.
If you have any questions I'm glad to hear it.
2008/5/2 <Myron.Dixon@l-3com.com>:
> We have experience a very similar problem using a 2.4.18 kernel on an =
8260 ppc processor.
> We have a telecomunication product that for some time only used the =
fec for TCP/IP ethernet
> traffic only and worked just fine.
>
> After we upgraded our product to implement TDM data over IP we =
started to notice an occasional
> kernel oops. We began to evaluate all of our products and determined =
that only some of the units
> exhibted this behaviour at various rates of occurrence. Further =
evaluation revealed that the pointers
> located in dpram pointing to the fec's buffer descriptors were some =
how getting corrupted.
> Note that the 8260 has 4 internal scc/fccs and we use all four for =
various aspects of our application
> and each shares dpram for pointers to buffer descriptors that reside =
in sdram. However, only the
> fec that is used for IP experiences this buffer descriptor corruption =
and, then, apparently, only under
> heavy traffic load. We spent about six months evaluating this =
problem including contacting freescale,
> but never found a solution. We finally, decided to use an external =
ethernet chip on a daughter card
> for our IP channel.
>
> It is, however, our belief that our problem relates to a possible bug =
in the 8260 CPM, but have yet to
> absolutely prove this.
>
> If we are experiencing the same problem (and potentially others) and =
there is a solution we would be
> very interested, as, we are not happy about the daughter card =
solution.
>
>
> Myron L. Dixon
> Sr. Software Engineer
> L-3 Communications, GNS
> 1519 Grundy's Lane
> Bristol, PA 19007
> Phone: 215 957 3739
> Fax: 215 957 3790
> email: Myron.Dixon@L-3Com.com
>
> -----Original Message-----
> From: linuxppc-dev-bounces+myron.dixon=3Dl-3com.com@ozlabs.org =
[mailto:linuxppc-dev-bounces+myron.dixon=3Dl-3com.com@ozlabs.org] On =
Behalf Of Franca, Jose (NSN - PT/Portugal - MiniMD)
> Sent: Wednesday, April 30, 2008 5:07 AM
> To: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
> Subject: FW: SKB corruption on heavy traffic
>
> From our latest debugs we found that the problem occurs mainly on =
skbuff code. After some variable time kfree or kalloc result in kernel =
oops.
>
> -----Original Message-----
> From: Franca, Jose (NSN - PT/Portugal - MiniMD)
> Sent: quarta-feira, 30 de Abril de 2008 9:44
> To: 'ext Scott Wood'
> Cc:
> Subject: RE: SKB corruption on heavy traffic
>
> Hello!
>
> Thank you for replying!
> It't quite dificult to say if the problem exists without our =
changes, since the all software is dependent on this changes so to work =
with the hardware. I can't answer to that right now on that, but I =
forgot to add one thing: we have ring buffer full problems on our =
fcc_enet driver from time to time. So, I think the problem could be on =
linux configurations (related to hw) because there is a lot of posts on =
the web related to problems similar to this (none of them has really =
solved the bottom problem).
>
> Regards,
> Filipe
>
> -----Original Message-----
> From: ext Scott Wood [mailto:scottwood@freescale.com]
> Sent: ter=E7a-feira, 29 de Abril de 2008 20:15
> To: Franca, Jose (NSN - PT/Portugal - MiniMD)
> Cc: linuxppc-dev@ozlabs.org; linuxppc-embedded@ozlabs.org
> Subject: Re: SKB corruption on heavy traffic
>
> On Tue, Apr 29, 2008 at 07:39:07PM +0100, Franca, Jose (NSN - =
PT/Portugal - MiniMD) wrote:
> > We are developing a MPC8247 based telecom board (512MB), =
using linux
> > 2.4 with some proprietary changes on IP stack and we are facing =
some
> > problems when we have heavy traffic on our Ethernet interfaces...
>
> Do you see these problems without the proprietary changes, and with a =
current kernel?
>
> -Scott
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2008-05-05 8:54 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-04-30 9:07 FW: SKB corruption on heavy traffic Franca, Jose (NSN - PT/Portugal - MiniMD)
2008-04-30 12:25 ` Gerhard Pircher
2008-04-30 13:03 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
2008-04-30 19:20 ` Gerhard Pircher
2008-05-01 20:55 ` Myron.Dixon
2008-05-03 14:07 ` Matvejchikov Ilya
2008-05-05 8:54 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
-- strict thread matches above, loose matches on Subject: below --
2008-04-29 18:39 Franca, Jose (NSN - PT/Portugal - MiniMD)
2008-04-29 19:15 ` Scott Wood
2008-04-30 8:43 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
2008-04-30 15:00 ` Scott Wood
2008-04-30 15:07 ` Franca, Jose (NSN - PT/Portugal - MiniMD)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).