Fw: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic

Fw: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic

[Andrew Morton]

Begin forwarded message:

Date: Thu, 22 Feb 2007 07:56:27 -0800
From: bugme-daemon@bugzilla.kernel.org
To: bugme-new@lists.osdl.org
Subject: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic


http://bugzilla.kernel.org/show_bug.cgi?id=8057

           Summary: slab corruption running ip6sic
    Kernel Version: 2.6.21-rc1
            Status: NEW
          Severity: normal
             Owner: yoshfuji@linux-ipv6.org
         Submitter: snakebyte@gmx.de


Most recent kernel where this bug did *NOT* occur: unknown
Distribution: gentoo
Hardware Environment: AMD-K6, 400MHz, 288MB Ram
Software Environment: ip6sic (http://ip6sic.sourceforge.net/)
Problem Description:

When running ip6sic against the loopback interface i get the following kernel
messages:

[  199.514486] Slab corruption: start=d0505554, len=156
[  199.514704] Redzone: 0x5a2cf071/0x5a2cf071.
[  199.514859] Last user: [<c0465813>](kfree_skbmem+0x33/0x80)
[  199.515301] 080: 6b 6b 6b 6b 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b
[  199.517096] Single bit error detected. Probably bad RAM.
[  199.517255] Run memtest86+ or a similar memory test tool.
[  199.517413] Prev obj: start=d05054ac, len=156
[  199.517568] Redzone: 0x170fc2a5/0x170fc2a5.
[  199.517719] Last user: [<c0465634>](skb_clone+0x34/0x1e0)
[  199.518088] 000: b4 52 50 d0 0c 52 50 d0 08 c6 4a d0 00 00 00 00
[  199.519851] 010: 00 00 00 00 00 00 00 00 5a 5a 5a 5a e0 a6 85 d0
[  199.521614] Next obj: start=d05055fc, len=156
[  199.521771] Redzone: 0x170fc2a5/0x170fc2a5.
[  199.521922] Last user: [<c0465634>](skb_clone+0x34/0x1e0)
[  199.522327] 000: 04 54
[  199.868486] Slab corruption: start=d0505554, len=156
[  199.869836] Redzone: 0x5a2cf071/0x5a2cf071.
[  199.870975] Last user: [<c0465813>](kfree_skbmem+0x33/0x80)
[  199.872820] 080: 6b 6b 6b 6b 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b
[  199.879166] Single bit error detected. Probably bad RAM.
[  199.880347] Run memtest86+ or a similar memory test tool.
[  199.881539] Prev obj: start=d05054ac, len=156
[  199.882679] Redzone: 0x170fc2a5/0x170fc2a5.
[  199.883983] Last user: [<c0465634>](skb_clone+0x34/0x1e0)
[  199.885807] 000: b4 52 50 d0 0c 52 50 d0 08 c6 4a d0 00 00 00 00
[  199.892047] 010: 00 00 00 00 00 00 00 00 5a 5a 5a 5a e0 a6 85 d0
[  199.898369] Next obj: start=d05055fc, len=156
[  199.899511] Redzone: 0x170fc2a5/0x170fc2a5.
[  199.900642] Last user: [<c0465634>](skb_clone+0x34/0x1e0)
[  199.902471] 000: 04 54 50 d0 b4 52 50 d0 08 c6 4a d0 00 00 00 00
[  199.908818] 010: 00 00 00 00 00 00 00 00 5a 5a 5a 5a ec a8 85 d0
[  200.947125] Slab corruption: start=d0505554, len=156
[  200.948445] Redzone: 0x5a2cf071/0x5a2cf071.
[  200.949578] Last user: [<c0465813>](kfree_skbmem+0x33/0x80)
[  200.951417] 080: 6b 6b 6b 6b 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b
[  200.957775] Single bit error detected. Probably bad RAM.
[  200.958960] Run memtest86+ or a similar memory test tool.
[  200.960155] Prev obj: start=d05054ac, len=156
[  200.961299] Redzone: 0x170fc2a5/0x170fc2a5.
[  200.962429] Last user: [<c0465634>](skb_clone+0x34/0x1e0)
[  200.964342] 000: b4 52 50 d0 0c 52 50 d0 08 c6 4a d0 00 00 00 00
[  200.970601] 010: 00 00 00 00 00 00 00 00 5a 5a 5a 5a e0 a6 85 d0
[  200.976934] Next obj: start=d05055fc, len=156
[  200.978074] Redzone: 0x170fc2a5/0x170fc2a5.
[  200.979200] Last user: [<c0465634>](skb_clone+0x34/0x1e0)
[  200.981021] 000: 04 54 50 d0 b4 52 50 d0 08 c6 4a d0 00 00 00 00
[  200.987374] 010: 00 00 00 00 00 00 00 00 5a 5a 5a 5a ec a8 85 d0

I compiled the kernel with network debugging, rebooted and startet ip6sic again:

[  141.573883] Slab corruption: start=d1a30b1c, len=156
[  141.575258] Redzone: 0x5a2cf071/0x5a2cf071.
[  141.576277] Last user: [<c0465813>](kfree_skbmem+0x33/0x80)
[  141.577909] 080: 6b 6b 6b 6b 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b
[  141.583277] Single bit error detected. Probably bad RAM.
[  141.584321] Run memtest86+ or a similar memory test tool.
[  141.585442] Prev obj: start=d1a30a74, len=156
[  141.586452] Redzone: 0x170fc2a5/0x170fc2a5.
[  141.587453] Last user: [<c04660cc>](__alloc_skb+0x2c/0x120)
[  141.589059] 000: cc 09 a3 d1 fc fa a2 d1 00 00 00 00 00 00 00 00
[  141.594442] 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[  141.600358] Next obj: start=d1a30bc4, len=156
[  141.601416] Redzone: 0x170fc2a5/0x170fc2a5.
[  141.602420] Last user: [<c0465634>](skb_clone+0x34/0x1e0)
[  141.604032] 000: 6c 0c a3 d1 34 05 a3 d1 40 6e 48 d0 00 00 00 00
[  141.609495] 010: 00 00 00 00 00 00 00 00 5a 5a 5a 5a e4 7d 24 d1
[  141.861710] Slab corruption: start=d1a30b1c, len=156
[  141.862832] Redzone: 0x5a2cf071/0x5a2cf071.
[  141.863844] Last user: [<c0465813>](kfree_skbmem+0x33/0x80)
[  141.865522] 080: 6b 6b 6b 6b 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b
[  141.870941] Single bit error detected. Probably bad RAM.
[  141.871993] Run memtest86+ or a similar memory test tool.
[  141.873052] Prev obj: start=d1a30a74, len=156
[  141.874068] Redzone: 0x170fc2a5/0x170fc2a5.
[  141.875143] Last user: [<c04660cc>](__alloc_skb+0x2c/0x120)
[  141.876758] 000: cc 09 a3 d1 fc fa a2 d1 00 00 00 00 00 00 00 00
[  141.882185] 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[  141.887653] Next obj: start=d1a30bc4, len=156
[  141.888668] Redzone: 0x170fc2a5/0x170fc2a5.
[  141.889671] Last user: [<c0465634>](skb_clone+0x34/0x1e0)
[  141.891274] 000: 6c 0c a3 d1 34 05 a3 d1 40 6e 48 d0 00 00 00 00
[  141.896735] 010: 00 00 00 00 00 00 00 00 5a 5a 5a 5a e4 7d 24 d1
[  142.160467] Slab corruption: start=d1a30b1c, len=156
[  142.161618] Redzone: 0x5a2cf071/0x5a2cf071.
[  142.162627] Last user: [<c0465813>](kfree_skbmem+0x33/0x80)
[  142.164243] 080: 6b 6b 6b 6b 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b
[  142.169686] Single bit error detected. Probably bad RAM.
[  142.170729] Run memtest86+ or a similar memory test tool.
[  142.171784] Prev obj: start=d1a30a74, len=156
[  142.172796] Redzone: 0x170fc2a5/0x170fc2a5.
[  142.173798] Last user: [<c04660cc>](__alloc_skb+0x2c/0x120)
[  142.175463] 000: cc 09 a3 d1 fc fa a2 d1 00 00 00 00 00 00 00 00
[  142.180869] 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[  142.186327] Next obj: start=d1a30bc4, len=156
[  142.187343] Redzone: 0x170fc2a5/0x170fc2a5.
[  142.188349] Last user: [<c0465634>](skb_clone+0x34/0x1e0)
[  142.189952] 000: 6c 0c a3 d1 34 05 a3 d1 40 6e 48 d0 00 00 00 00
[  142.195522] 010: 00 00 00 00 00 00 00 00 5a 5a 5a 5a e4 7d 24 d1

I ran memtester (userspace memtest) which gave no errors, so i did a memtest86+,
the first 1 1/2 passes gave no errors, but I'll keep it running for a few more
hours.

Steps to reproduce:
ip6sic -i lo -d ::1 -p 1000

In case there are some config options or patches i should try, just say so. I'll
attach a .config in a few hours, but I guess I should let memtest86+ run for a
few more passes.

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

Re: Fw: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic

[Jarek Poplawski]

On 22-02-2007 22:49, Andrew Morton wrote:
> 
> Begin forwarded message:
> 
> Date: Thu, 22 Feb 2007 07:56:27 -0800
> From: bugme-daemon@bugzilla.kernel.org
> To: bugme-new@lists.osdl.org
> Subject: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic
> 
> 
> http://bugzilla.kernel.org/show_bug.cgi?id=8057
> 
>            Summary: slab corruption running ip6sic
>     Kernel Version: 2.6.21-rc1
>             Status: NEW
>           Severity: normal
>              Owner: yoshfuji@linux-ipv6.org
>          Submitter: snakebyte@gmx.de
> 
> 
> Most recent kernel where this bug did *NOT* occur: unknown
> Distribution: gentoo
> Hardware Environment: AMD-K6, 400MHz, 288MB Ram
> Software Environment: ip6sic (http://ip6sic.sourceforge.net/)
> Problem Description:
> 
> When running ip6sic against the loopback interface i get the following kernel
> messages:
> 
> [  199.514486] Slab corruption: start=d0505554, len=156
> [  199.514704] Redzone: 0x5a2cf071/0x5a2cf071.
> [  199.514859] Last user: [<c0465813>](kfree_skbmem+0x33/0x80)
...

>From bugzilla:
...
> Is it possible that the handler frees the skb even if it is not supposed to do so?
> 
> 
> ------- Additional Comment #14 From Eric Sesterhenn 2007-02-28 04:33 -------
> 
> the ipcomp handler is xfrm6_rcv(), which calls xfrm6_rcv_spi(), which contrary
> to all other handlers returns -1 instead of 0 after calling kfree_skb() on the
> skb. Changing the return value to 0 in xfrm6_input.c:xfrm6_rcv_spi() fixes the
> problem.
> But I got no clue at all if this would be a correct fix

I think your diagnose is correct (all "return -1" should be
changed to "return 0" in xfrm6_input.c).

Regards,
Jarek P.

Re: Fw: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic

[Jarek Poplawski]

On Mon, Mar 12, 2007 at 11:24:03AM +0100, Jarek Poplawski wrote:
...
> I think your diagnose is correct (all "return -1" should be
> changed to "return 0" in xfrm6_input.c).

Sorry! Of course should be:

I think your diagnose is correct (all "return -1" should be
changed to "return 0" in xfrm6_rcv_spi()).

Jarek P.

Re: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic

[David Miller]

From: Jarek Poplawski <jarkao2@o2.pl>
Date: Mon, 12 Mar 2007 11:24:03 +0100

> > the ipcomp handler is xfrm6_rcv(), which calls xfrm6_rcv_spi(), which contrary
> > to all other handlers returns -1 instead of 0 after calling kfree_skb() on the
> > skb. Changing the return value to 0 in xfrm6_input.c:xfrm6_rcv_spi() fixes the
> > problem.
> > But I got no clue at all if this would be a correct fix
> 
> I think your diagnose is correct (all "return -1" should be
> changed to "return 0" in xfrm6_input.c).

Unfortunately, that won't work.

The return value logic for proto->handler() is different in
IPV6's ip6_input.c than it is for IPV4's ip_input.c.

IPv4 goes:

			ret = ipprot->handler(skb);
			if (ret < 0) {
				protocol = -ret;
				goto resubmit;
			}

whereas IPV6 goes:

		ret = ipprot->handler(&skb);
		if (ret > 0)
			goto resubmit;

There was a good reason why things were done differently for
this case, but I don't remember what that reason was.

Anyways, changing -1 to 0 in xfrm6_input.c will break everything
even though it might make this crash go away. :-)))

Re: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic

[Jarek Poplawski]

On Fri, Apr 20, 2007 at 04:35:15PM -0700, David Miller wrote:
> From: Jarek Poplawski <jarkao2@o2.pl>
> Date: Mon, 12 Mar 2007 11:24:03 +0100
> 
> > > the ipcomp handler is xfrm6_rcv(), which calls xfrm6_rcv_spi(), which contrary
> > > to all other handlers returns -1 instead of 0 after calling kfree_skb() on the
> > > skb. Changing the return value to 0 in xfrm6_input.c:xfrm6_rcv_spi() fixes the
> > > problem.
> > > But I got no clue at all if this would be a correct fix
> > 
> > I think your diagnose is correct (all "return -1" should be
> > changed to "return 0" in xfrm6_input.c).

I've corrected this, yet:

"Sorry! Of course should be:
I think your diagnose is correct (all "return -1" should be
changed to "return 0" in xfrm6_rcv_spi())."

It's just like Eric diagnosed:

xfrm6_rcv() calls tunnel6_rcv(), which calls handlers->handler()
and if handler() returns anything but 0, skb is kfreed. But
handler: xfrm6_tunnel_rcv() calls xfrm6_rcv_spi() and returns its
return without changing, which is only 1 and -1. It seems, in
every -1 case skb is kfreed by xfrm6_rcv_spi() or by functions
called by it, probably meaning skb was handled (delivered or
kfreed). The only path where skb is not kfreed returns 1.

tunnel6_rcv() treats both returns the same way - so some skbs
are kfreed 2 times.

Regards,
Jarek P.

Re: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic

[Jarek Poplawski]

On Mon, Apr 23, 2007 at 08:44:16AM +0200, Jarek Poplawski wrote:
> On Fri, Apr 20, 2007 at 04:35:15PM -0700, David Miller wrote:
> > From: Jarek Poplawski <jarkao2@o2.pl>
> > Date: Mon, 12 Mar 2007 11:24:03 +0100
> > 
> > > > the ipcomp handler is xfrm6_rcv(), which calls xfrm6_rcv_spi(), which contrary
> > > > to all other handlers returns -1 instead of 0 after calling kfree_skb() on the
> > > > skb. Changing the return value to 0 in xfrm6_input.c:xfrm6_rcv_spi() fixes the
> > > > problem.
> > > > But I got no clue at all if this would be a correct fix
> > > 
> > > I think your diagnose is correct (all "return -1" should be
> > > changed to "return 0" in xfrm6_input.c).
> 
> I've corrected this, yet:
> 
> "Sorry! Of course should be:
> I think your diagnose is correct (all "return -1" should be
> changed to "return 0" in xfrm6_rcv_spi())."
> 
> It's just like Eric diagnosed:
> 
> xfrm6_rcv() calls tunnel6_rcv(), which calls handlers->handler()
> and if handler() returns anything but 0, skb is kfreed. But
> handler: xfrm6_tunnel_rcv() calls xfrm6_rcv_spi() and returns its
> return without changing, which is only 1 and -1. It seems, in
> every -1 case skb is kfreed by xfrm6_rcv_spi() or by functions
> called by it, probably meaning skb was handled (delivered or
> kfreed). The only path where skb is not kfreed returns 1.
> 
> tunnel6_rcv() treats both returns the same way - so some skbs
> are kfreed 2 times.

OK, now I see this place is really "special" and there is more
than this. The same handler is used for 2 things, which expect
different error codes for similar things. 

My proposal is: maybe Eric could change this in
xfrm6_tunnel_rcv() from xfrm6_tunnel.c e.g. like this:

return xfrm6_rcv_spi(skb, spi) > 0 ? : 0;

and, if no errors in testing, he could resubmit this patch? 

Regards,
Jarek P.

Re: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic

[Herbert Xu]

Jarek Poplawski <jarkao2@o2.pl> wrote:
>
> My proposal is: maybe Eric could change this in
> xfrm6_tunnel_rcv() from xfrm6_tunnel.c e.g. like this:
> 
> return xfrm6_rcv_spi(skb, spi) > 0 ? : 0;
> 
> and, if no errors in testing, he could resubmit this patch? 

I agree, this is the right fix.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic

[Eric Sesterhenn / Snakebyte]

* Herbert Xu (herbert@gondor.apana.org.au) wrote:
> Jarek Poplawski <jarkao2@o2.pl> wrote:
> >
> > My proposal is: maybe Eric could change this in
> > xfrm6_tunnel_rcv() from xfrm6_tunnel.c e.g. like this:
> > 
> > return xfrm6_rcv_spi(skb, spi) > 0 ? : 0;
> > 
> > and, if no errors in testing, he could resubmit this patch? 
> 
> I agree, this is the right fix.


The fix proposed by Jarek indeed fixes the problem, tested on two boxes,
with an -rc5 kernel and a yesterdays git

Acked-by: Eric Sesterhenn <snakebyte@gmx.de>

--- linux-2.6/net/ipv6/xfrm6_tunnel.c.orig	2007-04-25 00:22:30.000000000 +0200
+++ linux-2.6/net/ipv6/xfrm6_tunnel.c	2007-04-25 00:22:45.000000000 +0200
@@ -261,7 +261,7 @@ static int xfrm6_tunnel_rcv(struct sk_bu
 	__be32 spi;
 
 	spi = xfrm6_tunnel_spi_lookup((xfrm_address_t *)&iph->saddr);
-	return xfrm6_rcv_spi(skb, spi);
+	return xfrm6_rcv_spi(skb, spi) > 0 ? : 0;
 }
 
 static int xfrm6_tunnel_err(struct sk_buff *skb, struct inet6_skb_parm *opt,

Re: [Bugme-new] [Bug 8057] New: slab corruption running ip6sic

[Jarek Poplawski]

On Wed, Apr 25, 2007 at 10:27:59AM +0200, Eric Sesterhenn / Snakebyte wrote:
> * Herbert Xu (herbert@gondor.apana.org.au) wrote:
> > Jarek Poplawski <jarkao2@o2.pl> wrote:
> > >
> > > My proposal is: maybe Eric could change this in
> > > xfrm6_tunnel_rcv() from xfrm6_tunnel.c e.g. like this:
> > > 
> > > return xfrm6_rcv_spi(skb, spi) > 0 ? : 0;
> > > 
> > > and, if no errors in testing, he could resubmit this patch? 
> > 
> > I agree, this is the right fix.
> 
> 
> The fix proposed by Jarek indeed fixes the problem, tested on two boxes,
> with an -rc5 kernel and a yesterdays git
> 
> Acked-by: Eric Sesterhenn <snakebyte@gmx.de>
> 
> --- linux-2.6/net/ipv6/xfrm6_tunnel.c.orig	2007-04-25 00:22:30.000000000 +0200
> +++ linux-2.6/net/ipv6/xfrm6_tunnel.c	2007-04-25 00:22:45.000000000 +0200
> @@ -261,7 +261,7 @@ static int xfrm6_tunnel_rcv(struct sk_bu
>  	__be32 spi;
>  
>  	spi = xfrm6_tunnel_spi_lookup((xfrm_address_t *)&iph->saddr);
> -	return xfrm6_rcv_spi(skb, spi);
> +	return xfrm6_rcv_spi(skb, spi) > 0 ? : 0;
>  }
>  
>  static int xfrm6_tunnel_err(struct sk_buff *skb, struct inet6_skb_parm *opt,

I think the main idea of fixing the problem plus testing is Eric's only
credit and I've only proposed some change in placement of cosmetic value.

So, Eric, considering all massive work you've done with rather feeble
support, please, fix the comment and sign this patch (at least I'm not
going to).

I also please Andrew to change the assignement of this patch in -mm.

Thanks & regards,
Jarek P.