Kernel crashing in tcp_sendmsg()

Kernel crashing in tcp_sendmsg()

[Bosko Radivojevic]

Hi All!

I have Atmel's AT91SAM9260 based system, Linux 2.6.33.4 kernel with
applied at91 patch. When the system is on a heavy load it happens
quite often to see kernel crashed during web server execution. It
seems the problem it is tcp related. I tried two different web servers
(thttpd and lighttpd) with the same results. I tried to debug the
problem, but without success. Any ideas or hitns how to proceed in
this situtation are more than welcome.

PS. I'm not subscribed to the mailing list so please cc me in replies. Thanks.

Crash report:

Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = c39a4000
[00000000] *pgd=239c1031, *pte=00000000, *ppte=00000000
Internal error: Oops: 817 [#1]
last sysfs file:
Modules linked in: eplcmod
CPU: 0    Not tainted  (2.6.33.4 #42)
PC is at __kprobes_text_end+0x860/0xa80
LR is at csum_partial_copy_from_user+0x18/0x3a4
pc : [<c01e2658>]    lr : [<c0130830>]    psr: 00000013
sp : c3989d68  ip : c3989db4  fp : c3989de8
r10: 4023c000  r9 : c3a5d990  r8 : 00000000
r7 : 000000ed  r6 : 000004c7  r5 : 00000000  r4 : fffffff2
r3 : 00000000  r2 : 000004c7  r1 : c3a48199  r0 : 4023c000
Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Control: 0005317f  Table: 239a4000  DAC: 00000015
Process thttpd (pid: 289, stack limit = 0xc3988260)
Stack: (0xc3989d68 to 0xc398a000)
9d60:                   c3a48199 000004c7 c3a5d920 c39ab8a0 000004c7 000000ed
9d80: 00000000 c01b5384 c3989db4 c3989f00 00000000 0000001f 000006b6 000005b4
9da0: c3988000 c00667dc 000005b4 00000040 000000ed 00000000 00000000 000005b4
9dc0: c3989e58 c3989ef8 00000002 c3a55340 000006b6 c3988000 c017f484 c3989e4c
9de0: c3989dec c017f58c c01b5074 00000000 c3846580 c027cd20 000007a3 c342e8c0
9e00: c0261a34 00000000 c3989e0c 00000000 00000000 c3989ef8 00000002 00000000
9e20: 00000000 00000040 c3989e58 c3989e58 c3989f80 fffffdee c3989ef8 00000002
9e40: c3989eec c3989e54 c00bcc90 c017f498 00000000 00000000 c3989e74 c3989e68
9e60: 00000000 00000001 ffffffff c3a55340 00000000 00000000 00000000 00000000
9e80: c3872040 00000000 00000000 00000000 00000000 c3989dec 0000001f 000007a3
9ea0: 00000000 000007a3 c3989ef8 c3988000 bec82920 c0022208 c3989ef8 00000010
9ec0: 00000000 00000002 00000001 000007a3 c3a55340 00000002 000007a3 c3989f80
9ee0: c3989f60 c3989ef0 c00bd364 c00bcbf8 c3989f80 c017f484 00046b88 000000ed
9f00: 4023c000 000006b6 00000092 c0022208 c3988000 00004fa6 c3989f34 c3989f28
9f20: c00267e8 c0026628 c3989fa4 c3989f38 c0021bec c00267a4 c3989ef8 00000002
9f40: bec82920 00000000 00000092 c0022208 000006b6 c3989f7c c3989f64 c00bd4c0
9f60: c00bd2c0 c3989f80 c3a55340 00000000 c3989fa4 c3989f80 c00bd5ac c00bd468
9f80: 00000000 00000000 00000000 0002a208 00022740 bec84db4 00000000 c3989fa8
9fa0: c0022044 c00bd570 0002a208 00022740 00000002 bec82920 00000002 00000000
9fc0: 0002a208 00022740 bec84db4 00022740 00044dd8 bec8253c 000006b6 bec82920
9fe0: 000217c8 bec81534 0000c6d0 4004b210 20000010 00000002 00000000 00000000
Backtrace:
[<c01b5064>] (tcp_sendmsg+0x0/0xaf0) from [<c017f58c>]
(sock_aio_write+0x108/0x118)
[<c017f488>] (sock_aio_write+0x4/0x118) from [<c00bcc90>]
(do_sync_readv_writev+0xa8/0xe8)
 r8:00000002 r7:c3989ef8 r6:fffffdee r5:c3989f80 r4:c3989e58
[<c00bcbe8>] (do_sync_readv_writev+0x0/0xe8) from [<c00bd364>]
(do_readv_writev+0xb4/0x1a8)
[<c00bd2b0>] (do_readv_writev+0x0/0x1a8) from [<c00bd4c0>]
(vfs_writev+0x68/0x74)
[<c00bd458>] (vfs_writev+0x0/0x74) from [<c00bd5ac>] (sys_writev+0x4c/0x80)
 r5:00000000 r4:c3a55340
[<c00bd560>] (sys_writev+0x0/0x80) from [<c0022044>]
(ret_fast_syscall+0x0/0x10)
 r6:bec84db4 r5:00022740 r4:0002a208
Code: 00000000 00000000 e3e0400d e59b5004 (e5854000)
---[ end trace c238dd9fcae91d1d ]---

snippets from objdump -d vmliux:

c0130818 <csum_partial_copy_from_user>:
c0130818:       e92d41f6        push    {r1, r2, r4, r5, r6, r7, r8, lr}
c013081c:       e3520008        cmp     r2, #8
c0130820:       3affffe3        bcc     c01307b4
<csum_partial_copy_nocheck+0x3b4>
c0130824:       e2933000        adds    r3, r3, #0
c0130828:       e3110003        tst     r1, #3
c013082c:       1bffffd0        blne    c0130774
<csum_partial_copy_nocheck+0x374>
c0130830:       e3100003        tst     r0, #3
c0130834:       1a00002f        bne     c01308f8
<csum_partial_copy_from_user+0xe0>
[..]

c0130400 <csum_partial_copy_nocheck>:
[..]
c01307b4:       e3320000        teq     r2, #0
c01307b8:       0affffeb        beq     c013076c
<csum_partial_copy_nocheck+0x36c>
c01307bc:       e3110001        tst     r1, #1
c01307c0:       0a00000c        beq     c01307f8
<csum_partial_copy_nocheck+0x3f8>
c01307c4:       e4f0c001        ldrbt   ip, [r0], #1
c01307c8:       e2422001        sub     r2, r2, #1
c01307cc:       e0b3340c        adcs    r3, r3, ip, lsl #8
c01307d0:       e4c1c001        strb    ip, [r1], #1
c01307d4:       e3120006        tst     r2, #6
c01307d8:       0a000008        beq     c0130800
<csum_partial_copy_nocheck+0x400>
c01307dc:       e4f08001        ldrbt   r8, [r0], #1
c01307e0:       e4f0c001        ldrbt   ip, [r0], #1
c01307e4:       e2422002        sub     r2, r2, #2
c01307e8:       e0b33008        adcs    r3, r3, r8
c01307ec:       e4c18001        strb    r8, [r1], #1
c01307f0:       e0b3340c        adcs    r3, r3, ip, lsl #8
c01307f4:       e4c1c001        strb    ip, [r1], #1
c01307f8:       e3120006        tst     r2, #6
c01307fc:       1afffff6        bne     c01307dc
<csum_partial_copy_nocheck+0x3dc>
c0130800:       e3120001        tst     r2, #1
c0130804:       0a000036        beq     c01308e4
<csum_partial_copy_from_user+0xcc>
c0130808:       e4f08001        ldrbt   r8, [r0], #1
c013080c:       e0b33008        adcs    r3, r3, r8
c0130810:       e4c18001        strb    r8, [r1], #1
c0130814:       ea000032        b       c01308e4
<csum_partial_copy_from_user+0xcc>
[..]

Kernel crashing in tcp_sendmsg()

[Russell King - ARM Linux]

On Fri, Jul 09, 2010 at 11:25:13AM +0200, Bosko Radivojevic wrote:
> I have Atmel's AT91SAM9260 based system, Linux 2.6.33.4 kernel with
> applied at91 patch. When the system is on a heavy load it happens
> quite often to see kernel crashed during web server execution. It
> seems the problem it is tcp related. I tried two different web servers
> (thttpd and lighttpd) with the same results. I tried to debug the
> problem, but without success. Any ideas or hitns how to proceed in
> this situtation are more than welcome.

csum_partial_copy_from_user's exception path is broken.  It has this
function prototype:

unsigned int csum_partial_copy_from_user(const char *src, char *dst,
  int len, int sum, int *err_ptr);

So, r0=src, r1=dst, r2=len, r3=sum, [sp]=err_ptr.

On function entry, it stacks registers like so:

                stmfd   sp!, {r1, r2, r4 - r8, lr}

On exception:

9001:           mov     r4, #-EFAULT
                ldr     r5, [fp, #4]            @ *err_ptr
                str     r4, [r5]

This only works if the code is using frame pointers.  This should fix it.
Please test and provide a Tested-by: line.

Subject: Fix csum_partial_copy_from_user()

Using the parent functions frame pointer to access our arguments is
completely wrong, whether or not we're building with frame pointers
or not.  What we should be using is the stack pointer to get at the
word above the registers we stacked ourselves.

Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
---
 arch/arm/lib/csumpartialcopyuser.S |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/arm/lib/csumpartialcopyuser.S b/arch/arm/lib/csumpartialcopyuser.S
index 59ff6fd..7d08b43 100644
--- a/arch/arm/lib/csumpartialcopyuser.S
+++ b/arch/arm/lib/csumpartialcopyuser.S
@@ -71,7 +71,7 @@
 		.pushsection .fixup,"ax"
 		.align	4
 9001:		mov	r4, #-EFAULT
-		ldr	r5, [fp, #4]		@ *err_ptr
+		ldr	r5, [sp, #8*4]		@ *err_ptr
 		str	r4, [r5]
 		ldmia	sp, {r1, r2}		@ retrieve dst, len
 		add	r2, r2, r1

Kernel crashing in tcp_sendmsg()

[Bosko Radivojevic]

On Fri, Jul 9, 2010 at 11:53 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> This only works if the code is using frame pointers. ?This should fix it.
> Please test and provide a Tested-by: line.

It works! Thanks. My colleague noticed a funny thing - non-stripped
thttpd/lighttpd binary is not crashing the kernel (without your patch,
of course). How is it related?

> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
> ---
> ?arch/arm/lib/csumpartialcopyuser.S | ? ?2 +-
> ?1 files changed, 1 insertions(+), 1 deletions(-)

Tested-by: Bosko Radivojevic <bosko.radivojevic@gmail.com>

Kernel crashing in tcp_sendmsg()

[Russell King - ARM Linux]

Hi,

Fixes generally don't get merged unless they're tested as working.
Can you please report back on whether this patch resolves your issue?

Thanks.

On Fri, Jul 09, 2010 at 10:53:29AM +0100, Russell King - ARM Linux wrote:
> On Fri, Jul 09, 2010 at 11:25:13AM +0200, Bosko Radivojevic wrote:
> > I have Atmel's AT91SAM9260 based system, Linux 2.6.33.4 kernel with
> > applied at91 patch. When the system is on a heavy load it happens
> > quite often to see kernel crashed during web server execution. It
> > seems the problem it is tcp related. I tried two different web servers
> > (thttpd and lighttpd) with the same results. I tried to debug the
> > problem, but without success. Any ideas or hitns how to proceed in
> > this situtation are more than welcome.
> 
> csum_partial_copy_from_user's exception path is broken.  It has this
> function prototype:
> 
> unsigned int csum_partial_copy_from_user(const char *src, char *dst,
>   int len, int sum, int *err_ptr);
> 
> So, r0=src, r1=dst, r2=len, r3=sum, [sp]=err_ptr.
> 
> On function entry, it stacks registers like so:
> 
>                 stmfd   sp!, {r1, r2, r4 - r8, lr}
> 
> On exception:
> 
> 9001:           mov     r4, #-EFAULT
>                 ldr     r5, [fp, #4]            @ *err_ptr
>                 str     r4, [r5]
> 
> This only works if the code is using frame pointers.  This should fix it.
> Please test and provide a Tested-by: line.
> 
> Subject: Fix csum_partial_copy_from_user()
> 
> Using the parent functions frame pointer to access our arguments is
> completely wrong, whether or not we're building with frame pointers
> or not.  What we should be using is the stack pointer to get at the
> word above the registers we stacked ourselves.
> 
> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
> ---
>  arch/arm/lib/csumpartialcopyuser.S |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/arch/arm/lib/csumpartialcopyuser.S b/arch/arm/lib/csumpartialcopyuser.S
> index 59ff6fd..7d08b43 100644
> --- a/arch/arm/lib/csumpartialcopyuser.S
> +++ b/arch/arm/lib/csumpartialcopyuser.S
> @@ -71,7 +71,7 @@
>  		.pushsection .fixup,"ax"
>  		.align	4
>  9001:		mov	r4, #-EFAULT
> -		ldr	r5, [fp, #4]		@ *err_ptr
> +		ldr	r5, [sp, #8*4]		@ *err_ptr
>  		str	r4, [r5]
>  		ldmia	sp, {r1, r2}		@ retrieve dst, len
>  		add	r2, r2, r1
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Kernel crashing in tcp_sendmsg()

[Gilles Chanteperdrix]

Russell King - ARM Linux wrote:
> Hi,
> 
> Fixes generally don't get merged unless they're tested as working.
> Can you please report back on whether this patch resolves your issue?

See:
http://lists.infradead.org/pipermail/linux-arm-kernel/2010-July/020086.html

-- 
					    Gilles.

Kernel crashing in tcp_sendmsg()

[Russell King - ARM Linux]

On Mon, Jul 26, 2010 at 01:16:05PM +0200, Gilles Chanteperdrix wrote:
> Russell King - ARM Linux wrote:
> > Hi,
> > 
> > Fixes generally don't get merged unless they're tested as working.
> > Can you please report back on whether this patch resolves your issue?
> 
> See:
> http://lists.infradead.org/pipermail/linux-arm-kernel/2010-July/020086.html

Oops, sorry, missed that.