* Kernel Oops in UDP w/ ARM architecture
@ 2009-03-09 15:52 Ron Yorgason
2009-03-09 17:16 ` Eric Dumazet
0 siblings, 1 reply; 8+ messages in thread
From: Ron Yorgason @ 2009-03-09 15:52 UTC (permalink / raw)
To: netdev
I'm working on an embedded video streaming application using gstreamer
over RTP/UDP on a Freescale iMX27 ARM platform. I have one board
doing the video capture and compression, and streaming it across the
network to another board which does the decoding and display. I'm
stuck right now with a kernel oops we're getting. It usually occurs
within 2-6 hours, but sometimes it takes longer for it to happen. I
believe it always dies with the same address in the failure.
I'm using a 2.6.19.2 kernel release. I don't know if this problem has
already been found and fixed in a future release (I didn't see any
mention of it in the changelogs of the next few releases), but this is
a customized kernel and I don't know how feasible it would be to port
all the changes to a newer kernel. We haven't touched the networking
stack, so it's most likely this bug is in the stock release.
Unable to handle kernel paging request at virtual address c6f9202a
pgd = c6d7c000
[c6f9202a] *pgd=a6e0041e(bad)
Internal error: Oops: 1 [#3]
Modules linked in:
CPU: 0
PC is at udp_recvmsg+0x184/0x21c
LR is at 0xf2799669
pc : [<c024a3e0>] lr : [<f2799669>] Not tainted
sp : c6f9fd48 ip : 00000000 fp : c6f9fd80
r10: c6f9fea0 r9 : 00000000 r8 : 00000400
r7 : 00000400 r6 : c7a52200 r5 : c6f9ff20 r4 : c6291780
r3 : c6f9201e r2 : 00000000 r1 : 00000008 r0 : c6f9fea8
Flags: NzCv IRQs on FIQs on Mode SVC_32 Segment user
Control: 5317F
Table: A6D7C000 DAC: 00000015
Process gst-launch-0.10 (pid: 18165, stack limit = 0xc6f9e250)
Stack: (0xc6f9fd48 to 0xc6fa0000)
fd40: 00000001 00000000 00000000 00000000 c02fbb80 c6f9ff20
fd60: c6f9ff20 00000400 00000000 00000000 00000000 c6f9fda8 c6f9fd84 c0207468
fd80: c024a26c 00000000 00000000 c6f9fd90 00000010 c6f9fdb0 c7c4fac0 c6f9fe9c
fda0: c6f9fdac c0205ae0 c020742c 00000000 c02e06c8 00000001 00000000 00000001
fdc0: ffffffff 00000000 00000000 00000000 00000000 00000000 c7c4fac0 00000000
fde0: 00000000 c6c5d720 c7c4fac0 c006a3a4 c6f9fdf0 c6f9fdf0 c6f9e000 ffffffff
fe00: c6f9fe34 c7176b60 c7176b90 8511a8c0 c6f9fea8 00000408 c6f9fe44 c6f9fe28
fe20: c0209ff8 00000001 00000004 40ee9e04 40ee9e04 00000000 00000000 00000000
fe40: 00000400 c759bba0 00000000 00000000 c6f9ff20 00000500 00000000 00000000
fe60: 00000400 00000000 00000000 c03714a4 c6f9fef8 00000000 00000400 00093800
fe80: c6f9fea0 c76d45a0 c6f9e000 40ee9e84 c6f9ff70 c6f9fea0 c0206990 c0205a30
fea0: 03080002 c005d660 a0000093 00043887 c7d6a000 000002c0 c7d6a2c0 60000013
fec0: c6f9fedc c6f9fed0 c005dbc0 c005da94 c6f9ff34 c6f9fee0 c018455c c005db90
fee0: 485a7d2d 00046731 00000400 c6f9ff10 c6f9fefc c024a130 c0059780 c76d45a0
ff00: 0000541b c6f9ff20 c6f9ff14 c024ff7c c024a0a8 c6f9ff3c c6f9ff24 c02052cc
ff20: c6f9fea0 00000080 c6f9ff3c 00000001 00000000 00000000 c00a8cf8 00093c00
ff40: 00000000 00000001 40ee9e9c 0000000c 00093800 00000400 00000066 c0038f84
ff60: 404fa2f0 c6f9ffa4 c6f9ff74 c0206e9c c0206908 40ee9e84 40ee9ea0 0000000a
ff80: 00093800 00000400 00000000 40ee9e84 40ee9ea0 000001c4 00000000 c6f9ffa8
ffa0: c0038de0 c0206d10 000001c4 00093800 0000000c 40ee9dd4 40eea56c 00000002
ffc0: 000001c4 00093800 00000400 0000000a 40ee9ea0 40ee9e84 404fa2f0 000350d0
ffe0: 00000000 40ee9dd0 4020fe74 40210808 80000010 0000000c 033a0000 8c020000
Backtrace:
[<c024a25c>] (udp_recvmsg+0x0/0x21c) from [<c0207468>] (sock_common_recvmsg+0x4)
[<c020741c>] (sock_common_recvmsg+0x0/0x60) from [<c0205ae0>] (sock_recvmsg+0xc)
r5 = C7C4FAC0 r4 = C6F9FDB0
[<c0205a20>] (sock_recvmsg+0x0/0xec) from [<c0206990>] (sys_recvfrom+0x98/0xf0)
[<c02068f8>] (sys_recvfrom+0x0/0xf0) from [<c0206e9c>] (sys_socketcall+0x19c/0x)
[<c0206d00>] (sys_socketcall+0x0/0x1f0) from [<c0038de0>] (ret_fast_syscall+0x0)
r4 = 000001C4
Code: e28a0008 e1d330b0 e3a01008 e1ca30b2 (e5943020)
I did the disassembly to find out exactly where the failure occurs. I
put an asterisk by the address offset mentioned in the oops, but I
believe it's the next line down where it references the address where
it chokes.
00001ae4 <udp_recvmsg>:
1ae4: e1a0c00d mov ip, sp
1ae8: e92ddff0 stmdb sp!, {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
1aec: e24cb004 sub fp, ip, #4 ; 0x4
1af0: e24dd010 sub sp, sp, #16 ; 0x10
1af4: e59b000c ldr r0, [fp, #12]
1af8: e59b9008 ldr r9, [fp, #8]
1afc: e3500000 cmp r0, #0 ; 0x0
1b00: e1a08003 mov r8, r3
1b04: 13a03010 movne r3, #16 ; 0x10
1b08: e592a000 ldr sl, [r2]
1b0c: 15803000 strne r3, [r0]
1b10: e3190a02 tst r9, #8192 ; 0x2000
1b14: e1a05002 mov r5, r2
1b18: e1a06001 mov r6, r1
1b1c: 0a000004 beq 1b34 <udp_recvmsg+0x50>
1b20: e1a00001 mov r0, r1
1b24: e1a01002 mov r1, r2
1b28: e1a02008 mov r2, r8
1b2c: ebfffffe bl 0 <ip_recv_error>
1b30: ea00006e b 1cf0 <udp_recvmsg+0x20c>
1b34: e1a01009 mov r1, r9
1b38: e59b2004 ldr r2, [fp, #4]
1b3c: e24b302c sub r3, fp, #44 ; 0x2c
1b40: e1a00006 mov r0, r6
1b44: ebfffffe bl 0 <skb_recv_datagram>
1b48: e2504000 subs r4, r0, #0 ; 0x0
1b4c: e3a01008 mov r1, #8 ; 0x8
1b50: 0a000057 beq 1cb4 <udp_recvmsg+0x1d0>
1b54: e5943060 ldr r3, [r4, #96]
1b58: e2437008 sub r7, r3, #8 ; 0x8
1b5c: e1570008 cmp r7, r8
1b60: 85953018 ldrhi r3, [r5, #24]
1b64: 81a07008 movhi r7, r8
1b68: 83833020 orrhi r3, r3, #32 ; 0x20
1b6c: 85853018 strhi r3, [r5, #24]
1b70: e5d43074 ldrb r3, [r4, #116]
1b74: e203300c and r3, r3, #12 ; 0xc
1b78: e3530008 cmp r3, #8 ; 0x8
1b7c: 01a01003 moveq r1, r3
1b80: 0a000007 beq 1ba4 <udp_recvmsg+0xc0>
1b84: e5953018 ldr r3, [r5, #24]
1b88: e3130020 tst r3, #32 ; 0x20
1b8c: 0a000009 beq 1bb8 <udp_recvmsg+0xd4>
1b90: ebfffffe bl 0 <__skb_checksum_complete>
1b94: e3500000 cmp r0, #0 ; 0x0
1b98: 1a000047 bne 1cbc <udp_recvmsg+0x1d8>
1b9c: e1a00004 mov r0, r4
1ba0: e3a01008 mov r1, #8 ; 0x8
1ba4: e5952008 ldr r2, [r5, #8]
1ba8: e1a03007 mov r3, r7
1bac: ebfffffe bl 0 <skb_copy_datagram_iovec>
1bb0: e50b002c str r0, [fp, #-44]
1bb4: ea000004 b 1bcc <udp_recvmsg+0xe8>
1bb8: e5952008 ldr r2, [r5, #8]
1bbc: ebfffffe bl 0 <skb_copy_and_csum_datagram_iovec>
1bc0: e3700016 cmn r0, #22 ; 0x16
1bc4: e50b002c str r0, [fp, #-44]
1bc8: 0a00003b beq 1cbc <udp_recvmsg+0x1d8>
1bcc: e51b302c ldr r3, [fp, #-44]
1bd0: e3530000 cmp r3, #0 ; 0x0
1bd4: 1a000033 bne 1ca8 <udp_recvmsg+0x1c4>
1bd8: e594100c ldr r1, [r4, #12]
1bdc: e5962094 ldr r2, [r6, #148]
1be0: e50b1034 str r1, [fp, #-52]
1be4: e5943010 ldr r3, [r4, #16]
1be8: e3120b02 tst r2, #2048 ; 0x800
1bec: e50b3030 str r3, [fp, #-48]
1bf0: 0a00000f beq 1c34 <udp_recvmsg+0x150>
1bf4: e3510000 cmp r1, #0 ; 0x0
1bf8: 1a000001 bne 1c04 <udp_recvmsg+0x120>
1bfc: e24b0034 sub r0, fp, #52 ; 0x34
1c00: ebfffffe bl 0 <do_gettimeofday>
1c04: e51b3034 ldr r3, [fp, #-52]
1c08: e24bc034 sub ip, fp, #52 ; 0x34
1c0c: e584300c str r3, [r4, #12]
1c10: e51b3030 ldr r3, [fp, #-48]
1c14: e1a00005 mov r0, r5
1c18: e5843010 str r3, [r4, #16]
1c1c: e3a01001 mov r1, #1 ; 0x1
1c20: e3a0201d mov r2, #29 ; 0x1d
1c24: e3a03008 mov r3, #8 ; 0x8
1c28: e58dc000 str ip, [sp]
1c2c: ebfffffe bl 0 <put_cmsg>
1c30: ea000003 b 1c44 <udp_recvmsg+0x160>
1c34: e24b2034 sub r2, fp, #52 ; 0x34
1c38: e892000c ldmia r2, {r2, r3}
1c3c: e58620f8 str r2, [r6, #248]
1c40: e58630fc str r3, [r6, #252]
1c44: e35a0000 cmp sl, #0 ; 0x0
1c48: 0a00000a beq 1c78 <udp_recvmsg+0x194>
1c4c: e3a03002 mov r3, #2 ; 0x2
1c50: e1ca30b0 strh r3, [sl]
1c54: e594301c ldr r3, [r4, #28]
1c58: e28a0008 add r0, sl, #8 ; 0x8
1c5c: e1d330b0 ldrh r3, [r3]
1c60: e3a01008 mov r1, #8 ; 0x8
1c64: e1ca30b2 strh r3, [sl, #2]
* 1c68: e5943020 ldr r3, [r4, #32]
1c6c: e593300c ldr r3, [r3, #12]
1c70: e58a3004 str r3, [sl, #4]
1c74: ebfffffe bl 0 <__memzero>
1c78: e59f3078 ldr r3, [pc, #120] ; 1cf8 <.text+0x1cf8>
1c7c: e19630b3 ldrh r3, [r6, r3]
1c80: e3530000 cmp r3, #0 ; 0x0
1c84: 0a000002 beq 1c94 <udp_recvmsg+0x1b0>
1c88: e1a00005 mov r0, r5
1c8c: e1a01004 mov r1, r4
1c90: ebfffffe bl 0 <ip_cmsg_recv>
1c94: e3190020 tst r9, #32 ; 0x20
1c98: e50b702c str r7, [fp, #-44]
1c9c: 15943060 ldrne r3, [r4, #96]
1ca0: 12433008 subne r3, r3, #8 ; 0x8
1ca4: 150b302c strne r3, [fp, #-44]
1ca8: e1a00006 mov r0, r6
1cac: e1a01004 mov r1, r4
1cb0: ebfffffe bl 0 <skb_free_datagram>
1cb4: e51b002c ldr r0, [fp, #-44]
1cb8: ea00000c b 1cf0 <udp_recvmsg+0x20c>
1cbc: e59f3038 ldr r3, [pc, #56] ; 1cfc <.text+0x1cfc>
1cc0: e1a02009 mov r2, r9
1cc4: e593c000 ldr ip, [r3]
1cc8: e1a01004 mov r1, r4
1ccc: e59c300c ldr r3, [ip, #12]
1cd0: e1a00006 mov r0, r6
1cd4: e2833001 add r3, r3, #1 ; 0x1
1cd8: e58c300c str r3, [ip, #12]
1cdc: ebfffffe bl 0 <skb_kill_datagram>
1ce0: e59b2004 ldr r2, [fp, #4]
1ce4: e3520000 cmp r2, #0 ; 0x0
1ce8: 0affff91 beq 1b34 <udp_recvmsg+0x50>
1cec: e3e0000a mvn r0, #10 ; 0xa
1cf0: e24bd028 sub sp, fp, #40 ; 0x28
1cf4: e89daff0 ldmia sp, {r4, r5, r6, r7, r8, r9, sl, fp, sp, pc}
1cf8: 00000146 andeq r0, r0, r6, asr #2
1cfc: 00000000 andeq r0, r0, r0
In the udp_recvmsg() function, the fault occurs in this code:
/* Copy the address. */
if (sin)
{
sin->sin_family = AF_INET;
sin->sin_port = skb->h.uh->source;
sin->sin_addr.s_addr = skb->nh.iph->saddr; // <- failure accessing
memory at saddr
memset(sin->sin_zero, 0, sizeof(sin->sin_zero));
}
After reviewing the assembly and the source code, it looks like the
address "c6f9202a" is where it thinks saddr should be. Ideally, I'd
like to figure out how to solve the problem. From ifconfig, I'm
finding a few errors with overruns, so maybe the queue is wrapping
around and clobbering the sk_buffs.
eth0 Link encap:Ethernet HWaddr 00:00:D0:D0:DA:D2
inet addr:192.168.17.133 Bcast:192.168.17.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:440979642 errors:8 dropped:0 overruns:8 frame:0
TX packets:601998 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2838009823 (2.6 GiB) TX bytes:155320893 (148.1 MiB)
Base address:0xb000
I'd also be willing to settle for a short term solution of finding a
way to test whether it's safe to dereference that pointer, and
skipping that sk_buff if it's bad.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Kernel Oops in UDP w/ ARM architecture
2009-03-09 15:52 Kernel Oops in UDP w/ ARM architecture Ron Yorgason
@ 2009-03-09 17:16 ` Eric Dumazet
2009-03-09 17:46 ` Ron Yorgason
0 siblings, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2009-03-09 17:16 UTC (permalink / raw)
To: Ron Yorgason; +Cc: netdev
Ron Yorgason a écrit :
> I'm working on an embedded video streaming application using gstreamer
> over RTP/UDP on a Freescale iMX27 ARM platform. I have one board
> doing the video capture and compression, and streaming it across the
> network to another board which does the decoding and display. I'm
> stuck right now with a kernel oops we're getting. It usually occurs
> within 2-6 hours, but sometimes it takes longer for it to happen. I
> believe it always dies with the same address in the failure.
>
> I'm using a 2.6.19.2 kernel release. I don't know if this problem has
> already been found and fixed in a future release (I didn't see any
> mention of it in the changelogs of the next few releases), but this is
> a customized kernel and I don't know how feasible it would be to port
> all the changes to a newer kernel. We haven't touched the networking
> stack, so it's most likely this bug is in the stock release.
>
> Unable to handle kernel paging request at virtual address c6f9202a
> pgd = c6d7c000
> [c6f9202a] *pgd=a6e0041e(bad)
> Internal error: Oops: 1 [#3]
> Modules linked in:
> CPU: 0
> PC is at udp_recvmsg+0x184/0x21c
> LR is at 0xf2799669
> pc : [<c024a3e0>] lr : [<f2799669>] Not tainted
> sp : c6f9fd48 ip : 00000000 fp : c6f9fd80
> r10: c6f9fea0 r9 : 00000000 r8 : 00000400
> r7 : 00000400 r6 : c7a52200 r5 : c6f9ff20 r4 : c6291780
> r3 : c6f9201e r2 : 00000000 r1 : 00000008 r0 : c6f9fea8
> Flags: NzCv IRQs on FIQs on Mode SVC_32 Segment user
> Control: 5317F
> Table: A6D7C000 DAC: 00000015
> Process gst-launch-0.10 (pid: 18165, stack limit = 0xc6f9e250)
> Stack: (0xc6f9fd48 to 0xc6fa0000)
> fd40: 00000001 00000000 00000000 00000000 c02fbb80 c6f9ff20
> fd60: c6f9ff20 00000400 00000000 00000000 00000000 c6f9fda8 c6f9fd84 c0207468
> fd80: c024a26c 00000000 00000000 c6f9fd90 00000010 c6f9fdb0 c7c4fac0 c6f9fe9c
> fda0: c6f9fdac c0205ae0 c020742c 00000000 c02e06c8 00000001 00000000 00000001
> fdc0: ffffffff 00000000 00000000 00000000 00000000 00000000 c7c4fac0 00000000
> fde0: 00000000 c6c5d720 c7c4fac0 c006a3a4 c6f9fdf0 c6f9fdf0 c6f9e000 ffffffff
> fe00: c6f9fe34 c7176b60 c7176b90 8511a8c0 c6f9fea8 00000408 c6f9fe44 c6f9fe28
> fe20: c0209ff8 00000001 00000004 40ee9e04 40ee9e04 00000000 00000000 00000000
> fe40: 00000400 c759bba0 00000000 00000000 c6f9ff20 00000500 00000000 00000000
> fe60: 00000400 00000000 00000000 c03714a4 c6f9fef8 00000000 00000400 00093800
> fe80: c6f9fea0 c76d45a0 c6f9e000 40ee9e84 c6f9ff70 c6f9fea0 c0206990 c0205a30
> fea0: 03080002 c005d660 a0000093 00043887 c7d6a000 000002c0 c7d6a2c0 60000013
> fec0: c6f9fedc c6f9fed0 c005dbc0 c005da94 c6f9ff34 c6f9fee0 c018455c c005db90
> fee0: 485a7d2d 00046731 00000400 c6f9ff10 c6f9fefc c024a130 c0059780 c76d45a0
> ff00: 0000541b c6f9ff20 c6f9ff14 c024ff7c c024a0a8 c6f9ff3c c6f9ff24 c02052cc
> ff20: c6f9fea0 00000080 c6f9ff3c 00000001 00000000 00000000 c00a8cf8 00093c00
> ff40: 00000000 00000001 40ee9e9c 0000000c 00093800 00000400 00000066 c0038f84
> ff60: 404fa2f0 c6f9ffa4 c6f9ff74 c0206e9c c0206908 40ee9e84 40ee9ea0 0000000a
> ff80: 00093800 00000400 00000000 40ee9e84 40ee9ea0 000001c4 00000000 c6f9ffa8
> ffa0: c0038de0 c0206d10 000001c4 00093800 0000000c 40ee9dd4 40eea56c 00000002
> ffc0: 000001c4 00093800 00000400 0000000a 40ee9ea0 40ee9e84 404fa2f0 000350d0
> ffe0: 00000000 40ee9dd0 4020fe74 40210808 80000010 0000000c 033a0000 8c020000
> Backtrace:
> [<c024a25c>] (udp_recvmsg+0x0/0x21c) from [<c0207468>] (sock_common_recvmsg+0x4)
> [<c020741c>] (sock_common_recvmsg+0x0/0x60) from [<c0205ae0>] (sock_recvmsg+0xc)
> r5 = C7C4FAC0 r4 = C6F9FDB0
> [<c0205a20>] (sock_recvmsg+0x0/0xec) from [<c0206990>] (sys_recvfrom+0x98/0xf0)
> [<c02068f8>] (sys_recvfrom+0x0/0xf0) from [<c0206e9c>] (sys_socketcall+0x19c/0x)
> [<c0206d00>] (sys_socketcall+0x0/0x1f0) from [<c0038de0>] (ret_fast_syscall+0x0)
> r4 = 000001C4
> Code: e28a0008 e1d330b0 e3a01008 e1ca30b2 (e5943020)
>
>
> I did the disassembly to find out exactly where the failure occurs. I
> put an asterisk by the address offset mentioned in the oops, but I
> believe it's the next line down where it references the address where
> it chokes.
Yes I agree (R3 + offset) chokes, not (r4 + offset)
>
> 00001ae4 <udp_recvmsg>:
> 1ae4: e1a0c00d mov ip, sp
> 1ae8: e92ddff0 stmdb sp!, {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
> 1aec: e24cb004 sub fp, ip, #4 ; 0x4
> 1af0: e24dd010 sub sp, sp, #16 ; 0x10
> 1af4: e59b000c ldr r0, [fp, #12]
> 1af8: e59b9008 ldr r9, [fp, #8]
> 1afc: e3500000 cmp r0, #0 ; 0x0
> 1b00: e1a08003 mov r8, r3
> 1b04: 13a03010 movne r3, #16 ; 0x10
> 1b08: e592a000 ldr sl, [r2]
> 1b0c: 15803000 strne r3, [r0]
> 1b10: e3190a02 tst r9, #8192 ; 0x2000
> 1b14: e1a05002 mov r5, r2
> 1b18: e1a06001 mov r6, r1
> 1b1c: 0a000004 beq 1b34 <udp_recvmsg+0x50>
> 1b20: e1a00001 mov r0, r1
> 1b24: e1a01002 mov r1, r2
> 1b28: e1a02008 mov r2, r8
> 1b2c: ebfffffe bl 0 <ip_recv_error>
> 1b30: ea00006e b 1cf0 <udp_recvmsg+0x20c>
> 1b34: e1a01009 mov r1, r9
> 1b38: e59b2004 ldr r2, [fp, #4]
> 1b3c: e24b302c sub r3, fp, #44 ; 0x2c
> 1b40: e1a00006 mov r0, r6
> 1b44: ebfffffe bl 0 <skb_recv_datagram>
> 1b48: e2504000 subs r4, r0, #0 ; 0x0
> 1b4c: e3a01008 mov r1, #8 ; 0x8
> 1b50: 0a000057 beq 1cb4 <udp_recvmsg+0x1d0>
> 1b54: e5943060 ldr r3, [r4, #96]
> 1b58: e2437008 sub r7, r3, #8 ; 0x8
> 1b5c: e1570008 cmp r7, r8
> 1b60: 85953018 ldrhi r3, [r5, #24]
> 1b64: 81a07008 movhi r7, r8
> 1b68: 83833020 orrhi r3, r3, #32 ; 0x20
> 1b6c: 85853018 strhi r3, [r5, #24]
> 1b70: e5d43074 ldrb r3, [r4, #116]
> 1b74: e203300c and r3, r3, #12 ; 0xc
> 1b78: e3530008 cmp r3, #8 ; 0x8
> 1b7c: 01a01003 moveq r1, r3
> 1b80: 0a000007 beq 1ba4 <udp_recvmsg+0xc0>
> 1b84: e5953018 ldr r3, [r5, #24]
> 1b88: e3130020 tst r3, #32 ; 0x20
> 1b8c: 0a000009 beq 1bb8 <udp_recvmsg+0xd4>
> 1b90: ebfffffe bl 0 <__skb_checksum_complete>
> 1b94: e3500000 cmp r0, #0 ; 0x0
> 1b98: 1a000047 bne 1cbc <udp_recvmsg+0x1d8>
> 1b9c: e1a00004 mov r0, r4
> 1ba0: e3a01008 mov r1, #8 ; 0x8
> 1ba4: e5952008 ldr r2, [r5, #8]
> 1ba8: e1a03007 mov r3, r7
> 1bac: ebfffffe bl 0 <skb_copy_datagram_iovec>
> 1bb0: e50b002c str r0, [fp, #-44]
> 1bb4: ea000004 b 1bcc <udp_recvmsg+0xe8>
> 1bb8: e5952008 ldr r2, [r5, #8]
> 1bbc: ebfffffe bl 0 <skb_copy_and_csum_datagram_iovec>
> 1bc0: e3700016 cmn r0, #22 ; 0x16
> 1bc4: e50b002c str r0, [fp, #-44]
> 1bc8: 0a00003b beq 1cbc <udp_recvmsg+0x1d8>
> 1bcc: e51b302c ldr r3, [fp, #-44]
> 1bd0: e3530000 cmp r3, #0 ; 0x0
> 1bd4: 1a000033 bne 1ca8 <udp_recvmsg+0x1c4>
> 1bd8: e594100c ldr r1, [r4, #12]
> 1bdc: e5962094 ldr r2, [r6, #148]
> 1be0: e50b1034 str r1, [fp, #-52]
> 1be4: e5943010 ldr r3, [r4, #16]
> 1be8: e3120b02 tst r2, #2048 ; 0x800
> 1bec: e50b3030 str r3, [fp, #-48]
> 1bf0: 0a00000f beq 1c34 <udp_recvmsg+0x150>
> 1bf4: e3510000 cmp r1, #0 ; 0x0
> 1bf8: 1a000001 bne 1c04 <udp_recvmsg+0x120>
> 1bfc: e24b0034 sub r0, fp, #52 ; 0x34
> 1c00: ebfffffe bl 0 <do_gettimeofday>
> 1c04: e51b3034 ldr r3, [fp, #-52]
> 1c08: e24bc034 sub ip, fp, #52 ; 0x34
> 1c0c: e584300c str r3, [r4, #12]
> 1c10: e51b3030 ldr r3, [fp, #-48]
> 1c14: e1a00005 mov r0, r5
> 1c18: e5843010 str r3, [r4, #16]
> 1c1c: e3a01001 mov r1, #1 ; 0x1
> 1c20: e3a0201d mov r2, #29 ; 0x1d
> 1c24: e3a03008 mov r3, #8 ; 0x8
> 1c28: e58dc000 str ip, [sp]
> 1c2c: ebfffffe bl 0 <put_cmsg>
> 1c30: ea000003 b 1c44 <udp_recvmsg+0x160>
> 1c34: e24b2034 sub r2, fp, #52 ; 0x34
> 1c38: e892000c ldmia r2, {r2, r3}
> 1c3c: e58620f8 str r2, [r6, #248]
> 1c40: e58630fc str r3, [r6, #252]
> 1c44: e35a0000 cmp sl, #0 ; 0x0
>
>
> 1c48: 0a00000a beq 1c78 <udp_recvmsg+0x194>
> 1c4c: e3a03002 mov r3, #2 ; 0x2
> 1c50: e1ca30b0 strh r3, [sl]
> 1c54: e594301c ldr r3, [r4, #28]
> 1c58: e28a0008 add r0, sl, #8 ; 0x8
> 1c5c: e1d330b0 ldrh r3, [r3]
> 1c60: e3a01008 mov r1, #8 ; 0x8
> 1c64: e1ca30b2 strh r3, [sl, #2]
> * 1c68: e5943020 ldr r3, [r4, #32]
> 1c6c: e593300c ldr r3, [r3, #12]
> 1c70: e58a3004 str r3, [sl, #4]
> 1c74: ebfffffe bl 0 <__memzero>
> 1c78: e59f3078 ldr r3, [pc, #120] ; 1cf8 <.text+0x1cf8>
> 1c7c: e19630b3 ldrh r3, [r6, r3]
>
>
> 1c80: e3530000 cmp r3, #0 ; 0x0
> 1c84: 0a000002 beq 1c94 <udp_recvmsg+0x1b0>
> 1c88: e1a00005 mov r0, r5
> 1c8c: e1a01004 mov r1, r4
> 1c90: ebfffffe bl 0 <ip_cmsg_recv>
> 1c94: e3190020 tst r9, #32 ; 0x20
> 1c98: e50b702c str r7, [fp, #-44]
> 1c9c: 15943060 ldrne r3, [r4, #96]
> 1ca0: 12433008 subne r3, r3, #8 ; 0x8
> 1ca4: 150b302c strne r3, [fp, #-44]
> 1ca8: e1a00006 mov r0, r6
> 1cac: e1a01004 mov r1, r4
> 1cb0: ebfffffe bl 0 <skb_free_datagram>
> 1cb4: e51b002c ldr r0, [fp, #-44]
> 1cb8: ea00000c b 1cf0 <udp_recvmsg+0x20c>
> 1cbc: e59f3038 ldr r3, [pc, #56] ; 1cfc <.text+0x1cfc>
> 1cc0: e1a02009 mov r2, r9
> 1cc4: e593c000 ldr ip, [r3]
> 1cc8: e1a01004 mov r1, r4
> 1ccc: e59c300c ldr r3, [ip, #12]
> 1cd0: e1a00006 mov r0, r6
> 1cd4: e2833001 add r3, r3, #1 ; 0x1
> 1cd8: e58c300c str r3, [ip, #12]
> 1cdc: ebfffffe bl 0 <skb_kill_datagram>
> 1ce0: e59b2004 ldr r2, [fp, #4]
> 1ce4: e3520000 cmp r2, #0 ; 0x0
> 1ce8: 0affff91 beq 1b34 <udp_recvmsg+0x50>
> 1cec: e3e0000a mvn r0, #10 ; 0xa
> 1cf0: e24bd028 sub sp, fp, #40 ; 0x28
> 1cf4: e89daff0 ldmia sp, {r4, r5, r6, r7, r8, r9, sl, fp, sp, pc}
> 1cf8: 00000146 andeq r0, r0, r6, asr #2
> 1cfc: 00000000 andeq r0, r0, r0
>
>
> In the udp_recvmsg() function, the fault occurs in this code:
> /* Copy the address. */
> if (sin)
> {
> sin->sin_family = AF_INET;
> sin->sin_port = skb->h.uh->source;
> sin->sin_addr.s_addr = skb->nh.iph->saddr; // <- failure accessing
> memory at saddr
> memset(sin->sin_zero, 0, sizeof(sin->sin_zero));
> }
>
>
> After reviewing the assembly and the source code, it looks like the
> address "c6f9202a" is where it thinks saddr should be. Ideally, I'd
This address is not aligned to a word (multiple of 4), which seems strange...
Maybe ARM doesnt handle unaligned accesses ?
1c48: 0a00000a beq 1c78 <udp_recvmsg+0x194>
1c4c: e3a03002 mov r3, #2 ; 0x2
1c50: e1ca30b0 strh r3, [sl]
1c54: e594301c ldr r3, [r4, #28] skb->h.uh (udp hdr) OK
1c58: e28a0008 add r0, sl, #8 ; 0x8
1c5c: e1d330b0 ldrh r3, [r3]
1c60: e3a01008 mov r1, #8 ; 0x8
1c64: e1ca30b2 strh r3, [sl, #2]
* 1c68: e5943020 ldr r3, [r4, #32] skb->nh.iph (IP header) OK
1c6c: e593300c ldr r3, [r3, #12] but (R+12) is unaligned
1c70: e58a3004 str r3, [sl, #4]
1c74: ebfffffe bl 0 <__memzero>
1c78: e59f3078 ldr r3, [pc, #120] ; 1cf8 <.text+0x1cf8>
1c7c: e19630b3 ldrh r3, [r6, r3]
What is your NIC driver ?
> like to figure out how to solve the problem. From ifconfig, I'm
> finding a few errors with overruns, so maybe the queue is wrapping
> around and clobbering the sk_buffs.
>
> eth0 Link encap:Ethernet HWaddr 00:00:D0:D0:DA:D2
> inet addr:192.168.17.133 Bcast:192.168.17.255 Mask:255.255.255.0
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:440979642 errors:8 dropped:0 overruns:8 frame:0
> TX packets:601998 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:2838009823 (2.6 GiB) TX bytes:155320893 (148.1 MiB)
> Base address:0xb000
>
> I'd also be willing to settle for a short term solution of finding a
> way to test whether it's safe to dereference that pointer, and
> skipping that sk_buff if it's bad.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Kernel Oops in UDP w/ ARM architecture
2009-03-09 17:16 ` Eric Dumazet
@ 2009-03-09 17:46 ` Ron Yorgason
2009-03-09 18:21 ` Eric Dumazet
0 siblings, 1 reply; 8+ messages in thread
From: Ron Yorgason @ 2009-03-09 17:46 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
We're using the fec driver, found in drivers/net/fec.c. I modified
this driver slightly to get the MAC address from the redboot
configuration stored in flash memory, but it's otherwise untouched. I
can send my version of the file if that would help.
--Ron
On Mon, Mar 9, 2009 at 12:16 PM, Eric Dumazet <dada1@cosmosbay.com> wrote:
> Ron Yorgason a écrit :
> - Show quoted text -
>> I'm working on an embedded video streaming application using gstreamer
>> over RTP/UDP on a Freescale iMX27 ARM platform. I have one board
>> doing the video capture and compression, and streaming it across the
>> network to another board which does the decoding and display. I'm
>> stuck right now with a kernel oops we're getting. It usually occurs
>> within 2-6 hours, but sometimes it takes longer for it to happen. I
>> believe it always dies with the same address in the failure.
>>
>> I'm using a 2.6.19.2 kernel release. I don't know if this problem has
>> already been found and fixed in a future release (I didn't see any
>> mention of it in the changelogs of the next few releases), but this is
>> a customized kernel and I don't know how feasible it would be to port
>> all the changes to a newer kernel. We haven't touched the networking
>> stack, so it's most likely this bug is in the stock release.
>>
>> Unable to handle kernel paging request at virtual address c6f9202a
>> pgd = c6d7c000
>> [c6f9202a] *pgd=a6e0041e(bad)
>> Internal error: Oops: 1 [#3]
>> Modules linked in:
>> CPU: 0
>> PC is at udp_recvmsg+0x184/0x21c
>> LR is at 0xf2799669
>> pc : [<c024a3e0>] lr : [<f2799669>] Not tainted
>> sp : c6f9fd48 ip : 00000000 fp : c6f9fd80
>> r10: c6f9fea0 r9 : 00000000 r8 : 00000400
>> r7 : 00000400 r6 : c7a52200 r5 : c6f9ff20 r4 : c6291780
>> r3 : c6f9201e r2 : 00000000 r1 : 00000008 r0 : c6f9fea8
>> Flags: NzCv IRQs on FIQs on Mode SVC_32 Segment user
>> Control: 5317F
>> Table: A6D7C000 DAC: 00000015
>> Process gst-launch-0.10 (pid: 18165, stack limit = 0xc6f9e250)
>> Stack: (0xc6f9fd48 to 0xc6fa0000)
>> fd40: 00000001 00000000 00000000 00000000 c02fbb80 c6f9ff20
>> fd60: c6f9ff20 00000400 00000000 00000000 00000000 c6f9fda8 c6f9fd84 c0207468
>> fd80: c024a26c 00000000 00000000 c6f9fd90 00000010 c6f9fdb0 c7c4fac0 c6f9fe9c
>> fda0: c6f9fdac c0205ae0 c020742c 00000000 c02e06c8 00000001 00000000 00000001
>> fdc0: ffffffff 00000000 00000000 00000000 00000000 00000000 c7c4fac0 00000000
>> fde0: 00000000 c6c5d720 c7c4fac0 c006a3a4 c6f9fdf0 c6f9fdf0 c6f9e000 ffffffff
>> fe00: c6f9fe34 c7176b60 c7176b90 8511a8c0 c6f9fea8 00000408 c6f9fe44 c6f9fe28
>> fe20: c0209ff8 00000001 00000004 40ee9e04 40ee9e04 00000000 00000000 00000000
>> fe40: 00000400 c759bba0 00000000 00000000 c6f9ff20 00000500 00000000 00000000
>> fe60: 00000400 00000000 00000000 c03714a4 c6f9fef8 00000000 00000400 00093800
>> fe80: c6f9fea0 c76d45a0 c6f9e000 40ee9e84 c6f9ff70 c6f9fea0 c0206990 c0205a30
>> fea0: 03080002 c005d660 a0000093 00043887 c7d6a000 000002c0 c7d6a2c0 60000013
>> fec0: c6f9fedc c6f9fed0 c005dbc0 c005da94 c6f9ff34 c6f9fee0 c018455c c005db90
>> fee0: 485a7d2d 00046731 00000400 c6f9ff10 c6f9fefc c024a130 c0059780 c76d45a0
>> ff00: 0000541b c6f9ff20 c6f9ff14 c024ff7c c024a0a8 c6f9ff3c c6f9ff24 c02052cc
>> ff20: c6f9fea0 00000080 c6f9ff3c 00000001 00000000 00000000 c00a8cf8 00093c00
>> ff40: 00000000 00000001 40ee9e9c 0000000c 00093800 00000400 00000066 c0038f84
>> ff60: 404fa2f0 c6f9ffa4 c6f9ff74 c0206e9c c0206908 40ee9e84 40ee9ea0 0000000a
>> ff80: 00093800 00000400 00000000 40ee9e84 40ee9ea0 000001c4 00000000 c6f9ffa8
>> ffa0: c0038de0 c0206d10 000001c4 00093800 0000000c 40ee9dd4 40eea56c 00000002
>> ffc0: 000001c4 00093800 00000400 0000000a 40ee9ea0 40ee9e84 404fa2f0 000350d0
>> ffe0: 00000000 40ee9dd0 4020fe74 40210808 80000010 0000000c 033a0000 8c020000
>> Backtrace:
>> [<c024a25c>] (udp_recvmsg+0x0/0x21c) from [<c0207468>] (sock_common_recvmsg+0x4)
>> [<c020741c>] (sock_common_recvmsg+0x0/0x60) from [<c0205ae0>] (sock_recvmsg+0xc)
>> r5 = C7C4FAC0 r4 = C6F9FDB0
>> [<c0205a20>] (sock_recvmsg+0x0/0xec) from [<c0206990>] (sys_recvfrom+0x98/0xf0)
>> [<c02068f8>] (sys_recvfrom+0x0/0xf0) from [<c0206e9c>] (sys_socketcall+0x19c/0x)
>> [<c0206d00>] (sys_socketcall+0x0/0x1f0) from [<c0038de0>] (ret_fast_syscall+0x0)
>> r4 = 000001C4
>> Code: e28a0008 e1d330b0 e3a01008 e1ca30b2 (e5943020)
>>
>>
>> I did the disassembly to find out exactly where the failure occurs. I
>> put an asterisk by the address offset mentioned in the oops, but I
>> believe it's the next line down where it references the address where
>> it chokes.
>
> Yes I agree (R3 + offset) chokes, not (r4 + offset)
> - Show quoted text -
>>
>> 00001ae4 <udp_recvmsg>:
>> 1ae4: e1a0c00d mov ip, sp
>> 1ae8: e92ddff0 stmdb sp!, {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}
>> 1aec: e24cb004 sub fp, ip, #4 ; 0x4
>> 1af0: e24dd010 sub sp, sp, #16 ; 0x10
>> 1af4: e59b000c ldr r0, [fp, #12]
>> 1af8: e59b9008 ldr r9, [fp, #8]
>> 1afc: e3500000 cmp r0, #0 ; 0x0
>> 1b00: e1a08003 mov r8, r3
>> 1b04: 13a03010 movne r3, #16 ; 0x10
>> 1b08: e592a000 ldr sl, [r2]
>> 1b0c: 15803000 strne r3, [r0]
>> 1b10: e3190a02 tst r9, #8192 ; 0x2000
>> 1b14: e1a05002 mov r5, r2
>> 1b18: e1a06001 mov r6, r1
>> 1b1c: 0a000004 beq 1b34 <udp_recvmsg+0x50>
>> 1b20: e1a00001 mov r0, r1
>> 1b24: e1a01002 mov r1, r2
>> 1b28: e1a02008 mov r2, r8
>> 1b2c: ebfffffe bl 0 <ip_recv_error>
>> 1b30: ea00006e b 1cf0 <udp_recvmsg+0x20c>
>> 1b34: e1a01009 mov r1, r9
>> 1b38: e59b2004 ldr r2, [fp, #4]
>> 1b3c: e24b302c sub r3, fp, #44 ; 0x2c
>> 1b40: e1a00006 mov r0, r6
>> 1b44: ebfffffe bl 0 <skb_recv_datagram>
>> 1b48: e2504000 subs r4, r0, #0 ; 0x0
>> 1b4c: e3a01008 mov r1, #8 ; 0x8
>> 1b50: 0a000057 beq 1cb4 <udp_recvmsg+0x1d0>
>> 1b54: e5943060 ldr r3, [r4, #96]
>> 1b58: e2437008 sub r7, r3, #8 ; 0x8
>> 1b5c: e1570008 cmp r7, r8
>> 1b60: 85953018 ldrhi r3, [r5, #24]
>> 1b64: 81a07008 movhi r7, r8
>> 1b68: 83833020 orrhi r3, r3, #32 ; 0x20
>> 1b6c: 85853018 strhi r3, [r5, #24]
>> 1b70: e5d43074 ldrb r3, [r4, #116]
>> 1b74: e203300c and r3, r3, #12 ; 0xc
>> 1b78: e3530008 cmp r3, #8 ; 0x8
>> 1b7c: 01a01003 moveq r1, r3
>> 1b80: 0a000007 beq 1ba4 <udp_recvmsg+0xc0>
>> 1b84: e5953018 ldr r3, [r5, #24]
>> 1b88: e3130020 tst r3, #32 ; 0x20
>> 1b8c: 0a000009 beq 1bb8 <udp_recvmsg+0xd4>
>> 1b90: ebfffffe bl 0 <__skb_checksum_complete>
>> 1b94: e3500000 cmp r0, #0 ; 0x0
>> 1b98: 1a000047 bne 1cbc <udp_recvmsg+0x1d8>
>> 1b9c: e1a00004 mov r0, r4
>> 1ba0: e3a01008 mov r1, #8 ; 0x8
>> 1ba4: e5952008 ldr r2, [r5, #8]
>> 1ba8: e1a03007 mov r3, r7
>> 1bac: ebfffffe bl 0 <skb_copy_datagram_iovec>
>> 1bb0: e50b002c str r0, [fp, #-44]
>> 1bb4: ea000004 b 1bcc <udp_recvmsg+0xe8>
>> 1bb8: e5952008 ldr r2, [r5, #8]
>> 1bbc: ebfffffe bl 0 <skb_copy_and_csum_datagram_iovec>
>> 1bc0: e3700016 cmn r0, #22 ; 0x16
>> 1bc4: e50b002c str r0, [fp, #-44]
>> 1bc8: 0a00003b beq 1cbc <udp_recvmsg+0x1d8>
>> 1bcc: e51b302c ldr r3, [fp, #-44]
>> 1bd0: e3530000 cmp r3, #0 ; 0x0
>> 1bd4: 1a000033 bne 1ca8 <udp_recvmsg+0x1c4>
>> 1bd8: e594100c ldr r1, [r4, #12]
>> 1bdc: e5962094 ldr r2, [r6, #148]
>> 1be0: e50b1034 str r1, [fp, #-52]
>> 1be4: e5943010 ldr r3, [r4, #16]
>> 1be8: e3120b02 tst r2, #2048 ; 0x800
>> 1bec: e50b3030 str r3, [fp, #-48]
>> 1bf0: 0a00000f beq 1c34 <udp_recvmsg+0x150>
>> 1bf4: e3510000 cmp r1, #0 ; 0x0
>> 1bf8: 1a000001 bne 1c04 <udp_recvmsg+0x120>
>> 1bfc: e24b0034 sub r0, fp, #52 ; 0x34
>> 1c00: ebfffffe bl 0 <do_gettimeofday>
>> 1c04: e51b3034 ldr r3, [fp, #-52]
>> 1c08: e24bc034 sub ip, fp, #52 ; 0x34
>> 1c0c: e584300c str r3, [r4, #12]
>> 1c10: e51b3030 ldr r3, [fp, #-48]
>> 1c14: e1a00005 mov r0, r5
>> 1c18: e5843010 str r3, [r4, #16]
>> 1c1c: e3a01001 mov r1, #1 ; 0x1
>> 1c20: e3a0201d mov r2, #29 ; 0x1d
>> 1c24: e3a03008 mov r3, #8 ; 0x8
>> 1c28: e58dc000 str ip, [sp]
>> 1c2c: ebfffffe bl 0 <put_cmsg>
>> 1c30: ea000003 b 1c44 <udp_recvmsg+0x160>
>> 1c34: e24b2034 sub r2, fp, #52 ; 0x34
>> 1c38: e892000c ldmia r2, {r2, r3}
>> 1c3c: e58620f8 str r2, [r6, #248]
>> 1c40: e58630fc str r3, [r6, #252]
>> 1c44: e35a0000 cmp sl, #0 ; 0x0
>>
>>
>> 1c48: 0a00000a beq 1c78 <udp_recvmsg+0x194>
>> 1c4c: e3a03002 mov r3, #2 ; 0x2
>> 1c50: e1ca30b0 strh r3, [sl]
>> 1c54: e594301c ldr r3, [r4, #28]
>> 1c58: e28a0008 add r0, sl, #8 ; 0x8
>> 1c5c: e1d330b0 ldrh r3, [r3]
>> 1c60: e3a01008 mov r1, #8 ; 0x8
>> 1c64: e1ca30b2 strh r3, [sl, #2]
>> * 1c68: e5943020 ldr r3, [r4, #32]
>> 1c6c: e593300c ldr r3, [r3, #12]
>> 1c70: e58a3004 str r3, [sl, #4]
>> 1c74: ebfffffe bl 0 <__memzero>
>> 1c78: e59f3078 ldr r3, [pc, #120] ; 1cf8 <.text+0x1cf8>
>> 1c7c: e19630b3 ldrh r3, [r6, r3]
>>
>>
>> 1c80: e3530000 cmp r3, #0 ; 0x0
>> 1c84: 0a000002 beq 1c94 <udp_recvmsg+0x1b0>
>> 1c88: e1a00005 mov r0, r5
>> 1c8c: e1a01004 mov r1, r4
>> 1c90: ebfffffe bl 0 <ip_cmsg_recv>
>> 1c94: e3190020 tst r9, #32 ; 0x20
>> 1c98: e50b702c str r7, [fp, #-44]
>> 1c9c: 15943060 ldrne r3, [r4, #96]
>> 1ca0: 12433008 subne r3, r3, #8 ; 0x8
>> 1ca4: 150b302c strne r3, [fp, #-44]
>> 1ca8: e1a00006 mov r0, r6
>> 1cac: e1a01004 mov r1, r4
>> 1cb0: ebfffffe bl 0 <skb_free_datagram>
>> 1cb4: e51b002c ldr r0, [fp, #-44]
>> 1cb8: ea00000c b 1cf0 <udp_recvmsg+0x20c>
>> 1cbc: e59f3038 ldr r3, [pc, #56] ; 1cfc <.text+0x1cfc>
>> 1cc0: e1a02009 mov r2, r9
>> 1cc4: e593c000 ldr ip, [r3]
>> 1cc8: e1a01004 mov r1, r4
>> 1ccc: e59c300c ldr r3, [ip, #12]
>> 1cd0: e1a00006 mov r0, r6
>> 1cd4: e2833001 add r3, r3, #1 ; 0x1
>> 1cd8: e58c300c str r3, [ip, #12]
>> 1cdc: ebfffffe bl 0 <skb_kill_datagram>
>> 1ce0: e59b2004 ldr r2, [fp, #4]
>> 1ce4: e3520000 cmp r2, #0 ; 0x0
>> 1ce8: 0affff91 beq 1b34 <udp_recvmsg+0x50>
>> 1cec: e3e0000a mvn r0, #10 ; 0xa
>> 1cf0: e24bd028 sub sp, fp, #40 ; 0x28
>> 1cf4: e89daff0 ldmia sp, {r4, r5, r6, r7, r8, r9, sl, fp, sp, pc}
>> 1cf8: 00000146 andeq r0, r0, r6, asr #2
>> 1cfc: 00000000 andeq r0, r0, r0
>>
>>
>> In the udp_recvmsg() function, the fault occurs in this code:
>> /* Copy the address. */
>> if (sin)
>> {
>> sin->sin_family = AF_INET;
>> sin->sin_port = skb->h.uh->source;
>> sin->sin_addr.s_addr = skb->nh.iph->saddr; // <- failure accessing
>> memory at saddr
>> memset(sin->sin_zero, 0, sizeof(sin->sin_zero));
>> }
>>
>>
>> After reviewing the assembly and the source code, it looks like the
>> address "c6f9202a" is where it thinks saddr should be. Ideally, I'd
>
> This address is not aligned to a word (multiple of 4), which seems strange...
>
> Maybe ARM doesnt handle unaligned accesses ?
>
>
> 1c48: 0a00000a beq 1c78 <udp_recvmsg+0x194>
> 1c4c: e3a03002 mov r3, #2 ; 0x2
> 1c50: e1ca30b0 strh r3, [sl]
> 1c54: e594301c ldr r3, [r4, #28] skb->h.uh (udp hdr) OK
> 1c58: e28a0008 add r0, sl, #8 ; 0x8
> 1c5c: e1d330b0 ldrh r3, [r3]
> 1c60: e3a01008 mov r1, #8 ; 0x8
> 1c64: e1ca30b2 strh r3, [sl, #2]
> * 1c68: e5943020 ldr r3, [r4, #32] skb->nh.iph (IP header) OK
> 1c6c: e593300c ldr r3, [r3, #12] but (R+12) is unaligned
> 1c70: e58a3004 str r3, [sl, #4]
> 1c74: ebfffffe bl 0 <__memzero>
> 1c78: e59f3078 ldr r3, [pc, #120] ; 1cf8 <.text+0x1cf8>
> 1c7c: e19630b3 ldrh r3, [r6, r3]
>
> What is your NIC driver ?
> - Show quoted text -
>> like to figure out how to solve the problem. From ifconfig, I'm
>> finding a few errors with overruns, so maybe the queue is wrapping
>> around and clobbering the sk_buffs.
>>
>> eth0 Link encap:Ethernet HWaddr 00:00:D0:D0:DA:D2
>> inet addr:192.168.17.133 Bcast:192.168.17.255 Mask:255.255.255.0
>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>> RX packets:440979642 errors:8 dropped:0 overruns:8 frame:0
>> TX packets:601998 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:2838009823 (2.6 GiB) TX bytes:155320893 (148.1 MiB)
>> Base address:0xb000
>>
>> I'd also be willing to settle for a short term solution of finding a
>> way to test whether it's safe to dereference that pointer, and
>> skipping that sk_buff if it's bad.
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Kernel Oops in UDP w/ ARM architecture
2009-03-09 17:46 ` Ron Yorgason
@ 2009-03-09 18:21 ` Eric Dumazet
2009-03-09 19:18 ` Ron Yorgason
0 siblings, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2009-03-09 18:21 UTC (permalink / raw)
To: Ron Yorgason; +Cc: netdev
Please dont top post on this mailing list
Ron Yorgason a écrit :
> We're using the fec driver, found in drivers/net/fec.c. I modified
> this driver slightly to get the MAC address from the redboot
> configuration stored in flash memory, but it's otherwise untouched. I
> can send my version of the file if that would help.
>
> --Ron
>
>
Given that ARM seems to be picky about non aligned accesses, you might
try this patch. This should force IP header to be aligned.
diff -u linux-2.6.19/drivers/net/fec.c.old linux-2.6.19/drivers/net/fec.c
--- linux-2.6.19/drivers/net/fec.c.old
+++ linux-2.6.19/drivers/net/fec.c
@@ -641,13 +641,14 @@
* include that when passing upstream as it messes up
* bridging applications.
*/
- skb = dev_alloc_skb(pkt_len-4);
+ skb = dev_alloc_skb((pkt_len - 4) + 2);
if (skb == NULL) {
printk("%s: Memory squeeze, dropping packet.\n", dev->name);
fep->stats.rx_dropped++;
} else {
skb->dev = dev;
+ skb_reserve(skb, 2); /* Align IP on 16 byte boundaries */
skb_put(skb,pkt_len-4); /* Make room */
eth_copy_and_sum(skb, data, pkt_len-4, 0);
skb->protocol=eth_type_trans(skb,dev);
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Kernel Oops in UDP w/ ARM architecture
2009-03-09 18:21 ` Eric Dumazet
@ 2009-03-09 19:18 ` Ron Yorgason
2009-03-09 20:24 ` Eric Dumazet
0 siblings, 1 reply; 8+ messages in thread
From: Ron Yorgason @ 2009-03-09 19:18 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
On Mon, Mar 9, 2009 at 1:21 PM, Eric Dumazet <dada1@cosmosbay.com> wrote:
> Please dont top post on this mailing list
>
> Ron Yorgason a écrit :
>> We're using the fec driver, found in drivers/net/fec.c. I modified
>> this driver slightly to get the MAC address from the redboot
>> configuration stored in flash memory, but it's otherwise untouched. I
>> can send my version of the file if that would help.
>>
>> --Ron
>>
>>
>
> Given that ARM seems to be picky about non aligned accesses, you might
> try this patch. This should force IP header to be aligned.
>
> diff -u linux-2.6.19/drivers/net/fec.c.old linux-2.6.19/drivers/net/fec.c
> --- linux-2.6.19/drivers/net/fec.c.old
> +++ linux-2.6.19/drivers/net/fec.c
> @@ -641,13 +641,14 @@
> * include that when passing upstream as it messes up
> * bridging applications.
> */
> - skb = dev_alloc_skb(pkt_len-4);
> + skb = dev_alloc_skb((pkt_len - 4) + 2);
>
> if (skb == NULL) {
> printk("%s: Memory squeeze, dropping packet.\n", dev->name);
> fep->stats.rx_dropped++;
> } else {
> skb->dev = dev;
> + skb_reserve(skb, 2); /* Align IP on 16 byte boundaries */
> skb_put(skb,pkt_len-4); /* Make room */
> eth_copy_and_sum(skb, data, pkt_len-4, 0);
> skb->protocol=eth_type_trans(skb,dev);
>
>
Thanks for the patch. It looks like Freescale modified this section
of the code in our BSP, so I'm looking at how to best merge things in.
Also, since we're allocating an extra 2 bytes in dev_alloc_skb(), do
we need to account for those bytes in skb_put() and
eth_copy_and_sum(), or does the skb_reserve() call handle that?
Ron
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Kernel Oops in UDP w/ ARM architecture
2009-03-09 19:18 ` Ron Yorgason
@ 2009-03-09 20:24 ` Eric Dumazet
2009-03-09 20:57 ` Ron Yorgason
0 siblings, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2009-03-09 20:24 UTC (permalink / raw)
To: Ron Yorgason; +Cc: netdev
Ron Yorgason a écrit :
> On Mon, Mar 9, 2009 at 1:21 PM, Eric Dumazet <dada1@cosmosbay.com> wrote:
>> Please dont top post on this mailing list
>>
>> Ron Yorgason a écrit :
>>> We're using the fec driver, found in drivers/net/fec.c. I modified
>>> this driver slightly to get the MAC address from the redboot
>>> configuration stored in flash memory, but it's otherwise untouched. I
>>> can send my version of the file if that would help.
>>>
>>> --Ron
>>>
>>>
>> Given that ARM seems to be picky about non aligned accesses, you might
>> try this patch. This should force IP header to be aligned.
>>
>> diff -u linux-2.6.19/drivers/net/fec.c.old linux-2.6.19/drivers/net/fec.c
>> --- linux-2.6.19/drivers/net/fec.c.old
>> +++ linux-2.6.19/drivers/net/fec.c
>> @@ -641,13 +641,14 @@
>> * include that when passing upstream as it messes up
>> * bridging applications.
>> */
>> - skb = dev_alloc_skb(pkt_len-4);
>> + skb = dev_alloc_skb((pkt_len - 4) + 2);
>>
>> if (skb == NULL) {
>> printk("%s: Memory squeeze, dropping packet.\n", dev->name);
>> fep->stats.rx_dropped++;
>> } else {
>> skb->dev = dev;
>> + skb_reserve(skb, 2); /* Align IP on 16 byte boundaries */
>> skb_put(skb,pkt_len-4); /* Make room */
>> eth_copy_and_sum(skb, data, pkt_len-4, 0);
>> skb->protocol=eth_type_trans(skb,dev);
>>
>>
>
> Thanks for the patch. It looks like Freescale modified this section
> of the code in our BSP, so I'm looking at how to best merge things in.
> Also, since we're allocating an extra 2 bytes in dev_alloc_skb(), do
> we need to account for those bytes in skb_put() and
> eth_copy_and_sum(), or does the skb_reserve() call handle that?
We allocate 2 extra bytes, because skb_reserve() will advance skb->data pointer
by two bytes. Those two bytes plus 14 bytes ethernet header : total 16 bytes,
so that IP header is aligned on 16 byte boundaries.
No other changes are necessary.
Then, maybe this patch is not necessary at all on your platform, since crash happens
a *long* time after boot. It may be a shoot in the dark...
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Kernel Oops in UDP w/ ARM architecture
2009-03-09 20:24 ` Eric Dumazet
@ 2009-03-09 20:57 ` Ron Yorgason
2009-03-09 21:19 ` Eric Dumazet
0 siblings, 1 reply; 8+ messages in thread
From: Ron Yorgason @ 2009-03-09 20:57 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
On Mon, Mar 9, 2009 at 3:24 PM, Eric Dumazet <dada1@cosmosbay.com> wrote:
> Ron Yorgason a écrit :
> - Show quoted text -
>> On Mon, Mar 9, 2009 at 1:21 PM, Eric Dumazet <dada1@cosmosbay.com> wrote:
>>> Please dont top post on this mailing list
>>>
>>> Ron Yorgason a écrit :
>>>> We're using the fec driver, found in drivers/net/fec.c. I modified
>>>> this driver slightly to get the MAC address from the redboot
>>>> configuration stored in flash memory, but it's otherwise untouched. I
>>>> can send my version of the file if that would help.
>>>>
>>>> --Ron
>>>>
>>>>
>>> Given that ARM seems to be picky about non aligned accesses, you might
>>> try this patch. This should force IP header to be aligned.
>>>
>>> diff -u linux-2.6.19/drivers/net/fec.c.old linux-2.6.19/drivers/net/fec.c
>>> --- linux-2.6.19/drivers/net/fec.c.old
>>> +++ linux-2.6.19/drivers/net/fec.c
>>> @@ -641,13 +641,14 @@
>>> * include that when passing upstream as it messes up
>>> * bridging applications.
>>> */
>>> - skb = dev_alloc_skb(pkt_len-4);
>>> + skb = dev_alloc_skb((pkt_len - 4) + 2);
>>>
>>> if (skb == NULL) {
>>> printk("%s: Memory squeeze, dropping packet.\n", dev->name);
>>> fep->stats.rx_dropped++;
>>> } else {
>>> skb->dev = dev;
>>> + skb_reserve(skb, 2); /* Align IP on 16 byte boundaries */
>>> skb_put(skb,pkt_len-4); /* Make room */
>>> eth_copy_and_sum(skb, data, pkt_len-4, 0);
>>> skb->protocol=eth_type_trans(skb,dev);
>>>
>>>
>>
>> Thanks for the patch. It looks like Freescale modified this section
>> of the code in our BSP, so I'm looking at how to best merge things in.
>> Also, since we're allocating an extra 2 bytes in dev_alloc_skb(), do
>> we need to account for those bytes in skb_put() and
>> eth_copy_and_sum(), or does the skb_reserve() call handle that?
>
> We allocate 2 extra bytes, because skb_reserve() will advance skb->data pointer
> by two bytes. Those two bytes plus 14 bytes ethernet header : total 16 bytes,
> so that IP header is aligned on 16 byte boundaries.
>
> No other changes are necessary.
>
> Then, maybe this patch is not necessary at all on your platform, since crash happens
> a *long* time after boot. It may be a shoot in the dark...
>
>
>
The code as supplied by Freescale looks like this:
if ((pkt_len - 4) < fec_copy_threshold) {
skb = dev_alloc_skb(pkt_len);
} else {
skb = dev_alloc_skb(FEC_ENET_RX_FRSIZE);
}
if (skb == NULL) {
printk("%s: Memory squeeze, dropping packet.\n", dev->name);
fep->stats.rx_dropped++;
} else {
if ((pkt_len - 4) < fec_copy_threshold) {
skb_reserve(skb, 2); /*skip 2bytes, so ipheader is align 4bytes*/
skb_put(skb,pkt_len-4); /* Make room */
eth_copy_and_sum(skb, (unsigned char *)data,
pkt_len-4, 0);
} else {
struct sk_buff *pskb = fep->rx_skbuff[rx_index];
fep->rx_skbuff[rx_index] = skb;
skb->data = FEC_ADDR_ALIGNMENT(skb->data);
bdp->cbd_bufaddr = __pa(skb->data);
skb_put(pskb,pkt_len-4); /* Make room */
skb = pskb;
}
skb->dev = dev;
skb->protocol=eth_type_trans(skb,dev);
netif_rx(skb);
}
fec_copy_threshold was defined with this comment:
/*
* fec_copy_threshold controls the copy when recieving ethernet frame.
* If ethernet header aligns 4bytes, the ip header and upper
header will not aligns 4bytes.
* The resean is ethernet header is 14bytes.
* And the max size of tcp & ip header is 128bytes. Normally it is 40bytes.
* So I set the default value between 128 to 256.
*/
static int fec_copy_threshold = 192;
And the FEC_ADDR_ALIGNMENT uses these definitions:
...
#elif defined(CONFIG_ARCH_MXC)
#include <asm/arch/hardware.h>
#include <asm/arch/iim.h>
#include "fec.h"
#define FEC_ALIGNMENT (0x0F) /*FEC needs 128bits(32bytes) alignment*/
#else
...
#define FEC_ADDR_ALIGNMENT(x) ((unsigned char *)(((unsigned long )(x)
+ (FEC_ALIGNMENT)) & (~FEC_ALIGNMENT)))
It looks like for a packet smaller than 196 byes will get allocated
with its actual size, but skb_reserve() skips 2 bytes and now it's a
little short. For a larger packet, they just allocate 2k, they're not
skipping 2 bytes and they're aligning the data but not the ip headers.
However, everything works 99.999% of the time, so I assume there's
just some border condition that's not being covered correctly.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Kernel Oops in UDP w/ ARM architecture
2009-03-09 20:57 ` Ron Yorgason
@ 2009-03-09 21:19 ` Eric Dumazet
0 siblings, 0 replies; 8+ messages in thread
From: Eric Dumazet @ 2009-03-09 21:19 UTC (permalink / raw)
To: Ron Yorgason; +Cc: netdev
Ron Yorgason a écrit :
> On Mon, Mar 9, 2009 at 3:24 PM, Eric Dumazet <dada1@cosmosbay.com> wrote:
>> Ron Yorgason a écrit :
>> - Show quoted text -
>>> On Mon, Mar 9, 2009 at 1:21 PM, Eric Dumazet <dada1@cosmosbay.com> wrote:
>>>> Please dont top post on this mailing list
>>>>
>>>> Ron Yorgason a écrit :
>>>>> We're using the fec driver, found in drivers/net/fec.c. I modified
>>>>> this driver slightly to get the MAC address from the redboot
>>>>> configuration stored in flash memory, but it's otherwise untouched. I
>>>>> can send my version of the file if that would help.
>>>>>
>>>>> --Ron
>>>>>
>>>>>
>>>> Given that ARM seems to be picky about non aligned accesses, you might
>>>> try this patch. This should force IP header to be aligned.
>>>>
>>>> diff -u linux-2.6.19/drivers/net/fec.c.old linux-2.6.19/drivers/net/fec.c
>>>> --- linux-2.6.19/drivers/net/fec.c.old
>>>> +++ linux-2.6.19/drivers/net/fec.c
>>>> @@ -641,13 +641,14 @@
>>>> * include that when passing upstream as it messes up
>>>> * bridging applications.
>>>> */
>>>> - skb = dev_alloc_skb(pkt_len-4);
>>>> + skb = dev_alloc_skb((pkt_len - 4) + 2);
>>>>
>>>> if (skb == NULL) {
>>>> printk("%s: Memory squeeze, dropping packet.\n", dev->name);
>>>> fep->stats.rx_dropped++;
>>>> } else {
>>>> skb->dev = dev;
>>>> + skb_reserve(skb, 2); /* Align IP on 16 byte boundaries */
>>>> skb_put(skb,pkt_len-4); /* Make room */
>>>> eth_copy_and_sum(skb, data, pkt_len-4, 0);
>>>> skb->protocol=eth_type_trans(skb,dev);
>>>>
>>>>
>>> Thanks for the patch. It looks like Freescale modified this section
>>> of the code in our BSP, so I'm looking at how to best merge things in.
>>> Also, since we're allocating an extra 2 bytes in dev_alloc_skb(), do
>>> we need to account for those bytes in skb_put() and
>>> eth_copy_and_sum(), or does the skb_reserve() call handle that?
>> We allocate 2 extra bytes, because skb_reserve() will advance skb->data pointer
>> by two bytes. Those two bytes plus 14 bytes ethernet header : total 16 bytes,
>> so that IP header is aligned on 16 byte boundaries.
>>
>> No other changes are necessary.
>>
>> Then, maybe this patch is not necessary at all on your platform, since crash happens
>> a *long* time after boot. It may be a shoot in the dark...
>>
>>
>>
>
> The code as supplied by Freescale looks like this:
>
> if ((pkt_len - 4) < fec_copy_threshold) {
> skb = dev_alloc_skb(pkt_len);
> } else {
> skb = dev_alloc_skb(FEC_ENET_RX_FRSIZE);
> }
>
> if (skb == NULL) {
> printk("%s: Memory squeeze, dropping packet.\n", dev->name);
> fep->stats.rx_dropped++;
> } else {
> if ((pkt_len - 4) < fec_copy_threshold) {
> skb_reserve(skb, 2); /*skip 2bytes, so ipheader is align 4bytes*/
> skb_put(skb,pkt_len-4); /* Make room */
> eth_copy_and_sum(skb, (unsigned char *)data,
> pkt_len-4, 0);
> } else {
> struct sk_buff *pskb = fep->rx_skbuff[rx_index];
>
> fep->rx_skbuff[rx_index] = skb;
> skb->data = FEC_ADDR_ALIGNMENT(skb->data);
...
> bdp->cbd_bufaddr = __pa(skb->data);
...
> skb_put(pskb,pkt_len-4); /* Make room */
> skb = pskb;
> }
> skb->dev = dev;
> skb->protocol=eth_type_trans(skb,dev);
> netif_rx(skb);
> }
>
>
> fec_copy_threshold was defined with this comment:
> /*
> * fec_copy_threshold controls the copy when recieving ethernet frame.
> * If ethernet header aligns 4bytes, the ip header and upper
> header will not aligns 4bytes.
> * The resean is ethernet header is 14bytes.
> * And the max size of tcp & ip header is 128bytes. Normally it is 40bytes.
> * So I set the default value between 128 to 256.
> */
> static int fec_copy_threshold = 192;
>
>
> And the FEC_ADDR_ALIGNMENT uses these definitions:
>
> ...
> #elif defined(CONFIG_ARCH_MXC)
> #include <asm/arch/hardware.h>
> #include <asm/arch/iim.h>
> #include "fec.h"
> #define FEC_ALIGNMENT (0x0F) /*FEC needs 128bits(32bytes) alignment*/
> #else
> ...
>
> #define FEC_ADDR_ALIGNMENT(x) ((unsigned char *)(((unsigned long )(x)
> + (FEC_ALIGNMENT)) & (~FEC_ALIGNMENT)))
>
>
> It looks like for a packet smaller than 196 byes will get allocated
> with its actual size, but skb_reserve() skips 2 bytes and now it's a
> little short. For a larger packet, they just allocate 2k, they're not
> skipping 2 bytes and they're aligning the data but not the ip headers.
> However, everything works 99.999% of the time, so I assume there's
> just some border condition that's not being covered correctly.
>
>
This driver seems bugy, and not part of standard kernel tree.
It should not change skb->data without changing skb->tail.
Fortunatly, dev_alloc_skb() already provides an address with a 16bytes alignement
(unless patched by Freescale...)
I suggest you redefine fec_copy_threshold to 1600 to get back a working driver.
My patch is not necessary on this driver (skb_reserve() already called)
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2009-03-09 21:19 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-03-09 15:52 Kernel Oops in UDP w/ ARM architecture Ron Yorgason
2009-03-09 17:16 ` Eric Dumazet
2009-03-09 17:46 ` Ron Yorgason
2009-03-09 18:21 ` Eric Dumazet
2009-03-09 19:18 ` Ron Yorgason
2009-03-09 20:24 ` Eric Dumazet
2009-03-09 20:57 ` Ron Yorgason
2009-03-09 21:19 ` Eric Dumazet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).