netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] ipcomp: double free at ipcomp_destroy()
@ 2010-02-14 14:44 Alexey Dobriyan
  2010-02-15  0:18 ` Herbert Xu
  0 siblings, 1 reply; 12+ messages in thread
From: Alexey Dobriyan @ 2010-02-14 14:44 UTC (permalink / raw)
  To: davem; +Cc: netdev, herbert

Consider using ipcomp with tunnel mode:

	pfkey_add -> xfrm_state_init -> x->type->init_state() == ipcomp4_init_state

1. If ipcomp_tunnel_attach() fails, xfrm_state private data (x->data) are freed
   first time (synchronously), but stale pointer is left.
2. xfrm_state_init() failed, all right, we're going to do error unwind
   but this time asynchronously and we're going to double free x->data
   asynchronously.

Fix by clearing x->data pointer, so second time it'll be fine.

Note, second time can happen in quite arbitrary time, double free
messages were seen in completely irrelevant functions, e. g. 

	INFO: Allocated in icmp_sk_init
	INFO: Freed in icmp_sk_exit

	[<ffffffff810b5ceb>] kfree+0xab/0x140
	[<ffffffff810719ae>] free_sect_attrs	(!)
	[<ffffffff81072353>] free_module

The only common thing was kmalloc-16 cache.

Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
---

 net/xfrm/xfrm_ipcomp.c |    1 +
 1 file changed, 1 insertion(+)

--- a/net/xfrm/xfrm_ipcomp.c
+++ b/net/xfrm/xfrm_ipcomp.c
@@ -332,6 +332,7 @@ void ipcomp_destroy(struct xfrm_state *x)
 	ipcomp_free_data(ipcd);
 	mutex_unlock(&ipcomp_resource_mutex);
 	kfree(ipcd);
+	x->data = NULL;
 }
 EXPORT_SYMBOL_GPL(ipcomp_destroy);
 

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-14 14:44 [PATCH] ipcomp: double free at ipcomp_destroy() Alexey Dobriyan
@ 2010-02-15  0:18 ` Herbert Xu
  2010-02-15  7:32   ` Alexey Dobriyan
  0 siblings, 1 reply; 12+ messages in thread
From: Herbert Xu @ 2010-02-15  0:18 UTC (permalink / raw)
  To: Alexey Dobriyan; +Cc: davem, netdev

On Sun, Feb 14, 2010 at 04:44:15PM +0200, Alexey Dobriyan wrote:
> Consider using ipcomp with tunnel mode:
> 
> 	pfkey_add -> xfrm_state_init -> x->type->init_state() == ipcomp4_init_state
> 
> 1. If ipcomp_tunnel_attach() fails, xfrm_state private data (x->data) are freed
>    first time (synchronously), but stale pointer is left.
> 2. xfrm_state_init() failed, all right, we're going to do error unwind
>    but this time asynchronously and we're going to double free x->data
>    asynchronously.

Sorry, I don't see the async path, where is it?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-15  0:18 ` Herbert Xu
@ 2010-02-15  7:32   ` Alexey Dobriyan
  2010-02-15  8:08     ` Herbert Xu
  0 siblings, 1 reply; 12+ messages in thread
From: Alexey Dobriyan @ 2010-02-15  7:32 UTC (permalink / raw)
  To: Herbert Xu; +Cc: davem, netdev

On Mon, Feb 15, 2010 at 2:18 AM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> On Sun, Feb 14, 2010 at 04:44:15PM +0200, Alexey Dobriyan wrote:
>> Consider using ipcomp with tunnel mode:
>>
>>       pfkey_add -> xfrm_state_init -> x->type->init_state() == ipcomp4_init_state
>>
>> 1. If ipcomp_tunnel_attach() fails, xfrm_state private data (x->data) are freed
>>    first time (synchronously), but stale pointer is left.
>> 2. xfrm_state_init() failed, all right, we're going to do error unwind
>>    but this time asynchronously and we're going to double free x->data
>>    asynchronously.
>
> Sorry, I don't see the async path, where is it?

pfkey_add
pfkey_msg2xfrm_state
xfrm_init_state         [fails]
xfrm_state_put
__xfrm_state_destroy    [puts xfrm_state into GC list, schedule work]
xfrm_state_gc_task
xfrm_state_gc_destroy
x->type->destructor

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-15  7:32   ` Alexey Dobriyan
@ 2010-02-15  8:08     ` Herbert Xu
  2010-02-15  8:10       ` Herbert Xu
  2010-02-16  6:00       ` Herbert Xu
  0 siblings, 2 replies; 12+ messages in thread
From: Herbert Xu @ 2010-02-15  8:08 UTC (permalink / raw)
  To: Alexey Dobriyan; +Cc: davem, netdev

On Mon, Feb 15, 2010 at 09:32:33AM +0200, Alexey Dobriyan wrote:
>
> pfkey_add
> pfkey_msg2xfrm_state
> xfrm_init_state         [fails]
> xfrm_state_put
> __xfrm_state_destroy    [puts xfrm_state into GC list, schedule work]
> xfrm_state_gc_task
> xfrm_state_gc_destroy
> x->type->destructor

Doh, I was looking at the buggy xfrm_state_clone path (which
incidently needs to be fixed to use xfrm_state_put).

How about just getting rid of the ipcomp_destroy call in ipcomp's
init_state functions? Calling the destructor twice is a bit iffy
even if we can make it work by setting various things to NULL and
testing for it.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-15  8:08     ` Herbert Xu
@ 2010-02-15  8:10       ` Herbert Xu
  2010-02-15 15:50         ` Alexey Dobriyan
  2010-02-15 17:28         ` Alexey Dobriyan
  2010-02-16  6:00       ` Herbert Xu
  1 sibling, 2 replies; 12+ messages in thread
From: Herbert Xu @ 2010-02-15  8:10 UTC (permalink / raw)
  To: Alexey Dobriyan; +Cc: davem, netdev

On Mon, Feb 15, 2010 at 04:08:46PM +0800, Herbert Xu wrote:
> 
> How about just getting rid of the ipcomp_destroy call in ipcomp's
> init_state functions? Calling the destructor twice is a bit iffy
> even if we can make it work by setting various things to NULL and
> testing for it.

Something like this (totally untested):

diff --git a/net/ipv4/ipcomp.c b/net/ipv4/ipcomp.c
index 38fbf04..544ce08 100644
--- a/net/ipv4/ipcomp.c
+++ b/net/ipv4/ipcomp.c
@@ -124,16 +124,12 @@ static int ipcomp4_init_state(struct xfrm_state *x)
 	if (x->props.mode == XFRM_MODE_TUNNEL) {
 		err = ipcomp_tunnel_attach(x);
 		if (err)
-			goto error_tunnel;
+			goto out;
 	}
 
 	err = 0;
 out:
 	return err;
-
-error_tunnel:
-	ipcomp_destroy(x);
-	goto out;
 }
 
 static const struct xfrm_type ipcomp_type = {
diff --git a/net/ipv6/ipcomp6.c b/net/ipv6/ipcomp6.c
index 2f2a5ca..002e6ee 100644
--- a/net/ipv6/ipcomp6.c
+++ b/net/ipv6/ipcomp6.c
@@ -154,16 +154,12 @@ static int ipcomp6_init_state(struct xfrm_state *x)
 	if (x->props.mode == XFRM_MODE_TUNNEL) {
 		err = ipcomp6_tunnel_attach(x);
 		if (err)
-			goto error_tunnel;
+			goto out;
 	}
 
 	err = 0;
 out:
 	return err;
-error_tunnel:
-	ipcomp_destroy(x);
-
-	goto out;
 }
 
 static const struct xfrm_type ipcomp6_type =

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-15  8:10       ` Herbert Xu
@ 2010-02-15 15:50         ` Alexey Dobriyan
  2010-02-16  5:29           ` Herbert Xu
  2010-02-15 17:28         ` Alexey Dobriyan
  1 sibling, 1 reply; 12+ messages in thread
From: Alexey Dobriyan @ 2010-02-15 15:50 UTC (permalink / raw)
  To: Herbert Xu; +Cc: davem, netdev

On Mon, Feb 15, 2010 at 04:10:52PM +0800, Herbert Xu wrote:
> On Mon, Feb 15, 2010 at 04:08:46PM +0800, Herbert Xu wrote:
> > 
> > How about just getting rid of the ipcomp_destroy call in ipcomp's
> > init_state functions? Calling the destructor twice is a bit iffy
> > even if we can make it work by setting various things to NULL and
> > testing for it.
> 
> Something like this (totally untested):
> 
> diff --git a/net/ipv4/ipcomp.c b/net/ipv4/ipcomp.c
> index 38fbf04..544ce08 100644
> --- a/net/ipv4/ipcomp.c
> +++ b/net/ipv4/ipcomp.c
> @@ -124,16 +124,12 @@ static int ipcomp4_init_state(struct xfrm_state *x)
>  	if (x->props.mode == XFRM_MODE_TUNNEL) {
>  		err = ipcomp_tunnel_attach(x);
>  		if (err)
> -			goto error_tunnel;
> +			goto out;
>  	}
>  
>  	err = 0;
>  out:
>  	return err;
> -
> -error_tunnel:
> -	ipcomp_destroy(x);
> -	goto out;
>  }
>  
>  static const struct xfrm_type ipcomp_type = {
> diff --git a/net/ipv6/ipcomp6.c b/net/ipv6/ipcomp6.c
> index 2f2a5ca..002e6ee 100644
> --- a/net/ipv6/ipcomp6.c
> +++ b/net/ipv6/ipcomp6.c
> @@ -154,16 +154,12 @@ static int ipcomp6_init_state(struct xfrm_state *x)
>  	if (x->props.mode == XFRM_MODE_TUNNEL) {
>  		err = ipcomp6_tunnel_attach(x);
>  		if (err)
> -			goto error_tunnel;
> +			goto out;
>  	}
>  
>  	err = 0;
>  out:
>  	return err;
> -error_tunnel:
> -	ipcomp_destroy(x);
> -
> -	goto out;

Then checks for NULL ->tunnel, and NULL ->data should be removed.
In fact, I don't quite understand why destruction was done this way,
so simply cleared ->data pointer.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-15  8:10       ` Herbert Xu
  2010-02-15 15:50         ` Alexey Dobriyan
@ 2010-02-15 17:28         ` Alexey Dobriyan
  2010-02-16  5:24           ` Herbert Xu
  1 sibling, 1 reply; 12+ messages in thread
From: Alexey Dobriyan @ 2010-02-15 17:28 UTC (permalink / raw)
  To: Herbert Xu; +Cc: davem, netdev

On Mon, Feb 15, 2010 at 04:10:52PM +0800, Herbert Xu wrote:
> On Mon, Feb 15, 2010 at 04:08:46PM +0800, Herbert Xu wrote:
> > 
> > How about just getting rid of the ipcomp_destroy call in ipcomp's
> > init_state functions? Calling the destructor twice is a bit iffy
> > even if we can make it work by setting various things to NULL and
> > testing for it.
> 
> Something like this (totally untested):

OK, it survives beating here.

> --- a/net/ipv4/ipcomp.c
> +++ b/net/ipv4/ipcomp.c
> @@ -124,16 +124,12 @@ static int ipcomp4_init_state(struct xfrm_state *x)
>  	if (x->props.mode == XFRM_MODE_TUNNEL) {
>  		err = ipcomp_tunnel_attach(x);
>  		if (err)
> -			goto error_tunnel;
> +			goto out;
>  	}
>  
>  	err = 0;
>  out:
>  	return err;
> -
> -error_tunnel:
> -	ipcomp_destroy(x);
> -	goto out;
>  }
>  
>  static const struct xfrm_type ipcomp_type = {
> diff --git a/net/ipv6/ipcomp6.c b/net/ipv6/ipcomp6.c
> index 2f2a5ca..002e6ee 100644
> --- a/net/ipv6/ipcomp6.c
> +++ b/net/ipv6/ipcomp6.c
> @@ -154,16 +154,12 @@ static int ipcomp6_init_state(struct xfrm_state *x)
>  	if (x->props.mode == XFRM_MODE_TUNNEL) {
>  		err = ipcomp6_tunnel_attach(x);
>  		if (err)
> -			goto error_tunnel;
> +			goto out;
>  	}
>  
>  	err = 0;
>  out:
>  	return err;
> -error_tunnel:
> -	ipcomp_destroy(x);
> -
> -	goto out;
>  }
>  
>  static const struct xfrm_type ipcomp6_type =
> 
> Cheers,
> -- 
> Visit Openswan at http://www.openswan.org/
> Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-15 17:28         ` Alexey Dobriyan
@ 2010-02-16  5:24           ` Herbert Xu
  2010-02-16 23:14             ` David Miller
  0 siblings, 1 reply; 12+ messages in thread
From: Herbert Xu @ 2010-02-16  5:24 UTC (permalink / raw)
  To: Alexey Dobriyan; +Cc: davem, netdev

On Mon, Feb 15, 2010 at 07:28:10PM +0200, Alexey Dobriyan wrote:
>
> OK, it survives beating here.

Thanks a lot for testing! I'll do the clean-up you suggested in
another patch.  Let's get this fixed first.

ipcomp: Avoid duplicate calls to ipcomp_destroy

When ipcomp_tunnel_attach fails we will call ipcomp_destroy twice.
This may lead to double-frees on certain structures.

As there is no reason to explicitly call ipcomp_destroy, this patch
removes it from ipcomp*.c and lets the standard xfrm_state destruction
take place.

This is based on the discovery and patch by Alexey Dobriyan.

Tested-by: Alexey Dobriyan <adobriyan@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

diff --git a/net/ipv4/ipcomp.c b/net/ipv4/ipcomp.c
index 38fbf04..544ce08 100644
--- a/net/ipv4/ipcomp.c
+++ b/net/ipv4/ipcomp.c
@@ -124,16 +124,12 @@ static int ipcomp4_init_state(struct xfrm_state *x)
 	if (x->props.mode == XFRM_MODE_TUNNEL) {
 		err = ipcomp_tunnel_attach(x);
 		if (err)
-			goto error_tunnel;
+			goto out;
 	}
 
 	err = 0;
 out:
 	return err;
-
-error_tunnel:
-	ipcomp_destroy(x);
-	goto out;
 }
 
 static const struct xfrm_type ipcomp_type = {
diff --git a/net/ipv6/ipcomp6.c b/net/ipv6/ipcomp6.c
index 2f2a5ca..002e6ee 100644
--- a/net/ipv6/ipcomp6.c
+++ b/net/ipv6/ipcomp6.c
@@ -154,16 +154,12 @@ static int ipcomp6_init_state(struct xfrm_state *x)
 	if (x->props.mode == XFRM_MODE_TUNNEL) {
 		err = ipcomp6_tunnel_attach(x);
 		if (err)
-			goto error_tunnel;
+			goto out;
 	}
 
 	err = 0;
 out:
 	return err;
-error_tunnel:
-	ipcomp_destroy(x);
-
-	goto out;
 }
 
 static const struct xfrm_type ipcomp6_type =

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-15 15:50         ` Alexey Dobriyan
@ 2010-02-16  5:29           ` Herbert Xu
  0 siblings, 0 replies; 12+ messages in thread
From: Herbert Xu @ 2010-02-16  5:29 UTC (permalink / raw)
  To: Alexey Dobriyan; +Cc: davem, netdev

On Mon, Feb 15, 2010 at 05:50:49PM +0200, Alexey Dobriyan wrote:
>
> Then checks for NULL ->tunnel, and NULL ->data should be removed.
> In fact, I don't quite understand why destruction was done this way,
> so simply cleared ->data pointer.

I had a look and I don't think we can do that (unless I misunderstood
what you meant).

The NULL ipcd check in ipcomp_destroy is to handle the case where
ipcomp_init_state fails before it even gets to allocating ipcd.

While the NULL tunnel check in xfrm_state_delete_tunnel is for the
case where we failed before successfully attaching a tunnel.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-15  8:08     ` Herbert Xu
  2010-02-15  8:10       ` Herbert Xu
@ 2010-02-16  6:00       ` Herbert Xu
  2010-02-16 23:14         ` David Miller
  1 sibling, 1 reply; 12+ messages in thread
From: Herbert Xu @ 2010-02-16  6:00 UTC (permalink / raw)
  To: Alexey Dobriyan; +Cc: davem, netdev

On Mon, Feb 15, 2010 at 04:08:46PM +0800, Herbert Xu wrote:
>
> Doh, I was looking at the buggy xfrm_state_clone path (which
> incidently needs to be fixed to use xfrm_state_put).

Here's a fix for that problem.

xfrm: Fix xfrm_state_clone leak

xfrm_state_clone calls kfree instead of xfrm_state_put to free
a failed state.  Depending on the state of the failed state, it
can cause leaks to things like module references.

All states should be freed by xfrm_state_put past the point of
xfrm_init_state.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index b36cc34..f445ea1 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -1102,7 +1102,7 @@ static struct xfrm_state *xfrm_state_clone(struct xfrm_state *orig, int *errp)
 	int err = -ENOMEM;
 	struct xfrm_state *x = xfrm_state_alloc(net);
 	if (!x)
-		goto error;
+		goto out;
 
 	memcpy(&x->id, &orig->id, sizeof(x->id));
 	memcpy(&x->sel, &orig->sel, sizeof(x->sel));
@@ -1160,16 +1160,10 @@ static struct xfrm_state *xfrm_state_clone(struct xfrm_state *orig, int *errp)
 	return x;
 
  error:
+	xfrm_state_put(x);
+out:
 	if (errp)
 		*errp = err;
-	if (x) {
-		kfree(x->aalg);
-		kfree(x->ealg);
-		kfree(x->calg);
-		kfree(x->encap);
-		kfree(x->coaddr);
-	}
-	kfree(x);
 	return NULL;
 }
 
Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-16  5:24           ` Herbert Xu
@ 2010-02-16 23:14             ` David Miller
  0 siblings, 0 replies; 12+ messages in thread
From: David Miller @ 2010-02-16 23:14 UTC (permalink / raw)
  To: herbert; +Cc: adobriyan, netdev

From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Tue, 16 Feb 2010 13:24:30 +0800

> On Mon, Feb 15, 2010 at 07:28:10PM +0200, Alexey Dobriyan wrote:
>>
>> OK, it survives beating here.
> 
> Thanks a lot for testing! I'll do the clean-up you suggested in
> another patch.  Let's get this fixed first.
> 
> ipcomp: Avoid duplicate calls to ipcomp_destroy
> 
> When ipcomp_tunnel_attach fails we will call ipcomp_destroy twice.
> This may lead to double-frees on certain structures.
> 
> As there is no reason to explicitly call ipcomp_destroy, this patch
> removes it from ipcomp*.c and lets the standard xfrm_state destruction
> take place.
> 
> This is based on the discovery and patch by Alexey Dobriyan.
> 
> Tested-by: Alexey Dobriyan <adobriyan@gmail.com>
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Applied to net-2.6

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] ipcomp: double free at ipcomp_destroy()
  2010-02-16  6:00       ` Herbert Xu
@ 2010-02-16 23:14         ` David Miller
  0 siblings, 0 replies; 12+ messages in thread
From: David Miller @ 2010-02-16 23:14 UTC (permalink / raw)
  To: herbert; +Cc: adobriyan, netdev

From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Tue, 16 Feb 2010 14:00:51 +0800

> On Mon, Feb 15, 2010 at 04:08:46PM +0800, Herbert Xu wrote:
>>
>> Doh, I was looking at the buggy xfrm_state_clone path (which
>> incidently needs to be fixed to use xfrm_state_put).
> 
> Here's a fix for that problem.
> 
> xfrm: Fix xfrm_state_clone leak
> 
> xfrm_state_clone calls kfree instead of xfrm_state_put to free
> a failed state.  Depending on the state of the failed state, it
> can cause leaks to things like module references.
> 
> All states should be freed by xfrm_state_put past the point of
> xfrm_init_state.
> 
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Applied to net-2.6

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2010-02-16 23:14 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-02-14 14:44 [PATCH] ipcomp: double free at ipcomp_destroy() Alexey Dobriyan
2010-02-15  0:18 ` Herbert Xu
2010-02-15  7:32   ` Alexey Dobriyan
2010-02-15  8:08     ` Herbert Xu
2010-02-15  8:10       ` Herbert Xu
2010-02-15 15:50         ` Alexey Dobriyan
2010-02-16  5:29           ` Herbert Xu
2010-02-15 17:28         ` Alexey Dobriyan
2010-02-16  5:24           ` Herbert Xu
2010-02-16 23:14             ` David Miller
2010-02-16  6:00       ` Herbert Xu
2010-02-16 23:14         ` David Miller

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).