Re: BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0

Re: BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0

[Justin P. Mattock]

On 06/23/2010 07:16 AM, John W. Linville wrote:
> On Tue, Jun 22, 2010 at 04:16:53PM -0700, Justin Mattock wrote:
>> I remember ipsec was able to work cleanly on my machines probably
>> about 4/6 months ago
>> now I get this:
>
> <snip>
>
> Perhaps netdev would be a more appropriate list than linux-wireless
> for this?
>
> John

alright added the cc's..
almost done bisecting..hopefully this points to
the right area..(if not I'll re-bisect until I get this).

as for the troubled machines I've a macbookpro2,2(no proprietary
stuff) and an imac9,1 with broadcom-sta(blah) both
seem to hit this right when the ipsec transaction starts
with ssh, vncviewer..(last good kernel I have is
2.6.33-rc5-01007-ge9449d8).

Justin P. Mattock

Re: BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0

[Justin P. Mattock]

o.k. the bisect is pointing to the below results..
(I tried git revert xxx but this commit is too big
so I'll(hopefully)manually revert it on the latest HEAD to
see if this is the actual problem im experiencing)



80c802f3073e84c956846e921e8a0b02dfa3755f is the first bad commit
commit 80c802f3073e84c956846e921e8a0b02dfa3755f
Author: Timo Teräs <timo.teras@iki.fi>
Date:   Wed Apr 7 00:30:05 2010 +0000

     xfrm: cache bundles instead of policies for outgoing flows

     __xfrm_lookup() is called for each packet transmitted out of
     system. The xfrm_find_bundle() does a linear search which can
     kill system performance depending on how many bundles are
     required per policy.

     This modifies __xfrm_lookup() to store bundles directly in
     the flow cache. If we did not get a hit, we just create a new
     bundle instead of doing slow search. This means that we can now
     get multiple xfrm_dst's for same flow (on per-cpu basis).

     Signed-off-by: Timo Teras <timo.teras@iki.fi>
     Signed-off-by: David S. Miller <davem@davemloft.net>

:040000 040000 d8e60f5fa4c1329f450d9c7cdf98b34e6a177f22 
9f576e68e5bf4ce357d7f0305aee5f410250dfe2 M      include
:040000 040000 f2876df688ee36907af7b4123eea96592faaed3e 
a3f6f6f94f0309106856cd99b38ec90b024eb016 M      net



Justin P. Mattock

Re: BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0

[Eric Dumazet]

Le mercredi 23 juin 2010 à 10:00 -0700, Justin P. Mattock a écrit :
> o.k. the bisect is pointing to the below results..
> (I tried git revert xxx but this commit is too big
> so I'll(hopefully)manually revert it on the latest HEAD to
> see if this is the actual problem im experiencing)
> 
> 
> 
> 80c802f3073e84c956846e921e8a0b02dfa3755f is the first bad commit
> commit 80c802f3073e84c956846e921e8a0b02dfa3755f
> Author: Timo Teräs <timo.teras@iki.fi>
> Date:   Wed Apr 7 00:30:05 2010 +0000
> 
>      xfrm: cache bundles instead of policies for outgoing flows
> 
>      __xfrm_lookup() is called for each packet transmitted out of
>      system. The xfrm_find_bundle() does a linear search which can
>      kill system performance depending on how many bundles are
>      required per policy.
> 
>      This modifies __xfrm_lookup() to store bundles directly in
>      the flow cache. If we did not get a hit, we just create a new
>      bundle instead of doing slow search. This means that we can now
>      get multiple xfrm_dst's for same flow (on per-cpu basis).
> 
>      Signed-off-by: Timo Teras <timo.teras@iki.fi>
>      Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> :040000 040000 d8e60f5fa4c1329f450d9c7cdf98b34e6a177f22 
> 9f576e68e5bf4ce357d7f0305aee5f410250dfe2 M      include
> :040000 040000 f2876df688ee36907af7b4123eea96592faaed3e 
> a3f6f6f94f0309106856cd99b38ec90b024eb016 M      net
> 
> 

Thanks a lot for bisecting Jutin, this is really appreciated.

crash is in xfrm_bundle_ok()

if (xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
	return 0;

xdst->pols[0] contains a NULL pointer

Re: BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0

[Timo Teräs]

On 06/23/2010 08:29 PM, Eric Dumazet wrote:
> Le mercredi 23 juin 2010 à 10:00 -0700, Justin P. Mattock a écrit :
>> o.k. the bisect is pointing to the below results..
>> (I tried git revert xxx but this commit is too big
>> so I'll(hopefully)manually revert it on the latest HEAD to
>> see if this is the actual problem im experiencing)
>>
>> 80c802f3073e84c956846e921e8a0b02dfa3755f is the first bad commit
>> commit 80c802f3073e84c956846e921e8a0b02dfa3755f
>> Author: Timo Teräs <timo.teras@iki.fi>
>> Date:   Wed Apr 7 00:30:05 2010 +0000
>>
>>      xfrm: cache bundles instead of policies for outgoing flows
>>
>>      __xfrm_lookup() is called for each packet transmitted out of
>>      system. The xfrm_find_bundle() does a linear search which can
>>      kill system performance depending on how many bundles are
>>      required per policy.
>>
>>      This modifies __xfrm_lookup() to store bundles directly in
>>      the flow cache. If we did not get a hit, we just create a new
>>      bundle instead of doing slow search. This means that we can now
>>      get multiple xfrm_dst's for same flow (on per-cpu basis).
>>
>>      Signed-off-by: Timo Teras <timo.teras@iki.fi>
>>      Signed-off-by: David S. Miller <davem@davemloft.net>
>>
>> :040000 040000 d8e60f5fa4c1329f450d9c7cdf98b34e6a177f22 
>> 9f576e68e5bf4ce357d7f0305aee5f410250dfe2 M      include
>> :040000 040000 f2876df688ee36907af7b4123eea96592faaed3e 
>> a3f6f6f94f0309106856cd99b38ec90b024eb016 M      net
> 
> Thanks a lot for bisecting Jutin, this is really appreciated.
> 
> crash is in xfrm_bundle_ok()
> 
> if (xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
> 	return 0;
> 
> xdst->pols[0] contains a NULL pointer

That does not really make sense, if we get this far; there's a valid
xfrm_state with the bundle. This means that there existed a policy with
it too.

I'll take a deeper look at this tomorrow. Would it be possible to see
your xfrm policies?

- Timo

Re: BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0

[Justin P. Mattock]

On 06/23/10 11:10, Timo Teräs wrote:
> On 06/23/2010 08:29 PM, Eric Dumazet wrote:
>    
>> Le mercredi 23 juin 2010 à 10:00 -0700, Justin P. Mattock a écrit :
>>      
>>> o.k. the bisect is pointing to the below results..
>>> (I tried git revert xxx but this commit is too big
>>> so I'll(hopefully)manually revert it on the latest HEAD to
>>> see if this is the actual problem im experiencing)
>>>
>>> 80c802f3073e84c956846e921e8a0b02dfa3755f is the first bad commit
>>> commit 80c802f3073e84c956846e921e8a0b02dfa3755f
>>> Author: Timo Teräs<timo.teras@iki.fi>
>>> Date:   Wed Apr 7 00:30:05 2010 +0000
>>>
>>>       xfrm: cache bundles instead of policies for outgoing flows
>>>
>>>       __xfrm_lookup() is called for each packet transmitted out of
>>>       system. The xfrm_find_bundle() does a linear search which can
>>>       kill system performance depending on how many bundles are
>>>       required per policy.
>>>
>>>       This modifies __xfrm_lookup() to store bundles directly in
>>>       the flow cache. If we did not get a hit, we just create a new
>>>       bundle instead of doing slow search. This means that we can now
>>>       get multiple xfrm_dst's for same flow (on per-cpu basis).
>>>
>>>       Signed-off-by: Timo Teras<timo.teras@iki.fi>
>>>       Signed-off-by: David S. Miller<davem@davemloft.net>
>>>
>>> :040000 040000 d8e60f5fa4c1329f450d9c7cdf98b34e6a177f22
>>> 9f576e68e5bf4ce357d7f0305aee5f410250dfe2 M      include
>>> :040000 040000 f2876df688ee36907af7b4123eea96592faaed3e
>>> a3f6f6f94f0309106856cd99b38ec90b024eb016 M      net
>>>        
>> Thanks a lot for bisecting Jutin, this is really appreciated.
>>
>> crash is in xfrm_bundle_ok()
>>
>> if (xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
>> 	return 0;
>>
>> xdst->pols[0] contains a NULL pointer
>>      
> That does not really make sense, if we get this far; there's a valid
> xfrm_state with the bundle. This means that there existed a policy with
> it too.
>
> I'll take a deeper look at this tomorrow. Would it be possible to see
> your xfrm policies?
>
> - Timo
>
>    

@Eric sure no problem doing the bisect...

as for the xfrm policy here is the link that I used to setup ipsec:
http://www.linuxfromscratch.org/hints/downloads/files/ipsec.txt
(just change the keys if doing real world work..(but for me just testing).

below is a temporary fix for me to get this working, tcpdump reports 
everything is doing what it should be
11:16:32.496166 IP xxxxx > xxxxx: AH(spi=0x00000200,seq=0x1090): 
ESP(spi=0x00000201,seq=0x1090), length 56
11:16:32.496212 IP xxxxx > xxxxx: AH(spi=0x00000200,seq=0x1091): 
ESP(spi=0x00000201,seq=0x1091), length 56
11:16:32.496259 IP xxxxx > xxxxx: AH(spi=0x00000200,seq=0x1092): 
ESP(spi=0x00000201,seq=0x1092), length 56

(tested a few mins ago, but not the right fix..)

 From a280d9048c8f8f5756f9f0dcc608b4499645593e Mon Sep 17 00:00:00 2001
From: Justin P. Mattock <justinmattock@gmail.com>
Date: Wed, 23 Jun 2010 11:10:28 -0700
Subject: [PATCH] fix ipsec
  Signed-off-by: Justin P. Mattock <justinmattock@gmail.com>

---
  net/xfrm/xfrm_policy.c |    2 --
  1 files changed, 0 insertions(+), 2 deletions(-)

diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
index 4bf27d9..701d69a 100644
--- a/net/xfrm/xfrm_policy.c
+++ b/net/xfrm/xfrm_policy.c
@@ -2300,8 +2300,6 @@ int xfrm_bundle_ok(struct xfrm_policy *pol, struct 
xfrm_dst *first,
              return 0;
          if (xdst->xfrm_genid != dst->xfrm->genid)
              return 0;
-        if (xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
-            return 0;

          if (strict && fl &&
              !(dst->xfrm->outer_mode->flags & XFRM_MODE_FLAG_TUNNEL) &&
-- 
1.7.1.rc1.21.gf3bd6


I've got the machine ready for any patches.. let me know  o.k...

Justin P. Mattock

Re: BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0

[Timo Teräs]

On 06/23/2010 09:20 PM, Justin P. Mattock wrote:
> On 06/23/10 11:10, Timo Teräs wrote:
>> On 06/23/2010 08:29 PM, Eric Dumazet wrote:
>>   
>>> Le mercredi 23 juin 2010 à 10:00 -0700, Justin P. Mattock a écrit :
>>>     
>>>> o.k. the bisect is pointing to the below results..
>>>> (I tried git revert xxx but this commit is too big
>>>> so I'll(hopefully)manually revert it on the latest HEAD to
>>>> see if this is the actual problem im experiencing)
>>>>
>>>> 80c802f3073e84c956846e921e8a0b02dfa3755f is the first bad commit
>>>> commit 80c802f3073e84c956846e921e8a0b02dfa3755f
>>>> Author: Timo Teräs<timo.teras@iki.fi>
>>>> Date:   Wed Apr 7 00:30:05 2010 +0000
>>>>
>>>>       xfrm: cache bundles instead of policies for outgoing flows
>>>>
>>>>       __xfrm_lookup() is called for each packet transmitted out of
>>>>       system. The xfrm_find_bundle() does a linear search which can
>>>>       kill system performance depending on how many bundles are
>>>>       required per policy.
>>>>
>>>>       This modifies __xfrm_lookup() to store bundles directly in
>>>>       the flow cache. If we did not get a hit, we just create a new
>>>>       bundle instead of doing slow search. This means that we can now
>>>>       get multiple xfrm_dst's for same flow (on per-cpu basis).
>>>>
>>>>       Signed-off-by: Timo Teras<timo.teras@iki.fi>
>>>>       Signed-off-by: David S. Miller<davem@davemloft.net>
>>>>
>>>> :040000 040000 d8e60f5fa4c1329f450d9c7cdf98b34e6a177f22
>>>> 9f576e68e5bf4ce357d7f0305aee5f410250dfe2 M      include
>>>> :040000 040000 f2876df688ee36907af7b4123eea96592faaed3e
>>>> a3f6f6f94f0309106856cd99b38ec90b024eb016 M      net
>>>>        
>>> Thanks a lot for bisecting Jutin, this is really appreciated.
>>>
>>> crash is in xfrm_bundle_ok()
>>>
>>> if (xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
>>>     return 0;
>>>
>>> xdst->pols[0] contains a NULL pointer
>>>      
>> That does not really make sense, if we get this far; there's a valid
>> xfrm_state with the bundle. This means that there existed a policy with
>> it too.
>>
>> I'll take a deeper look at this tomorrow. Would it be possible to see
>> your xfrm policies?
>>
>> - Timo
>>
>>    
> 
> @Eric sure no problem doing the bisect...
> 
> as for the xfrm policy here is the link that I used to setup ipsec:
> http://www.linuxfromscratch.org/hints/downloads/files/ipsec.txt
> (just change the keys if doing real world work..(but for me just testing).
> 
> below is a temporary fix for me to get this working, tcpdump reports
> everything is doing what it should be
> 11:16:32.496166 IP xxxxx > xxxxx: AH(spi=0x00000200,seq=0x1090):
> ESP(spi=0x00000201,seq=0x1090), length 56
> 11:16:32.496212 IP xxxxx > xxxxx: AH(spi=0x00000200,seq=0x1091):
> ESP(spi=0x00000201,seq=0x1091), length 56
> 11:16:32.496259 IP xxxxx > xxxxx: AH(spi=0x00000200,seq=0x1092):
> ESP(spi=0x00000201,seq=0x1092), length 56
> 
> (tested a few mins ago, but not the right fix..)

Yes, that would break some other obscure scenarios.

Looks like it's ah inside esp. So you get chain of bundles. And only the
first bundle gets a policy. Should have thought of that. Does the below
fix it for you?

diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
index 4bf27d9..af1c173 100644
--- a/net/xfrm/xfrm_policy.c
+++ b/net/xfrm/xfrm_policy.c
@@ -2300,7 +2300,8 @@ int xfrm_bundle_ok(struct xfrm_policy *pol, struct
xfrm_dst *first,
 			return 0;
 		if (xdst->xfrm_genid != dst->xfrm->genid)
 			return 0;
-		if (xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
+		if (xdst->num_pols > 0 &&
+		    xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
 			return 0;

 		if (strict && fl &&

Re: BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0

[Justin P. Mattock]

On 06/23/2010 01:34 PM, Timo Teräs wrote:
> On 06/23/2010 09:20 PM, Justin P. Mattock wrote:
>> On 06/23/10 11:10, Timo Teräs wrote:
>>> On 06/23/2010 08:29 PM, Eric Dumazet wrote:
>>>
>>>> Le mercredi 23 juin 2010 à 10:00 -0700, Justin P. Mattock a écrit :
>>>>
>>>>> o.k. the bisect is pointing to the below results..
>>>>> (I tried git revert xxx but this commit is too big
>>>>> so I'll(hopefully)manually revert it on the latest HEAD to
>>>>> see if this is the actual problem im experiencing)
>>>>>
>>>>> 80c802f3073e84c956846e921e8a0b02dfa3755f is the first bad commit
>>>>> commit 80c802f3073e84c956846e921e8a0b02dfa3755f
>>>>> Author: Timo Teräs<timo.teras@iki.fi>
>>>>> Date:   Wed Apr 7 00:30:05 2010 +0000
>>>>>
>>>>>        xfrm: cache bundles instead of policies for outgoing flows
>>>>>
>>>>>        __xfrm_lookup() is called for each packet transmitted out of
>>>>>        system. The xfrm_find_bundle() does a linear search which can
>>>>>        kill system performance depending on how many bundles are
>>>>>        required per policy.
>>>>>
>>>>>        This modifies __xfrm_lookup() to store bundles directly in
>>>>>        the flow cache. If we did not get a hit, we just create a new
>>>>>        bundle instead of doing slow search. This means that we can now
>>>>>        get multiple xfrm_dst's for same flow (on per-cpu basis).
>>>>>
>>>>>        Signed-off-by: Timo Teras<timo.teras@iki.fi>
>>>>>        Signed-off-by: David S. Miller<davem@davemloft.net>
>>>>>
>>>>> :040000 040000 d8e60f5fa4c1329f450d9c7cdf98b34e6a177f22
>>>>> 9f576e68e5bf4ce357d7f0305aee5f410250dfe2 M      include
>>>>> :040000 040000 f2876df688ee36907af7b4123eea96592faaed3e
>>>>> a3f6f6f94f0309106856cd99b38ec90b024eb016 M      net
>>>>>
>>>> Thanks a lot for bisecting Jutin, this is really appreciated.
>>>>
>>>> crash is in xfrm_bundle_ok()
>>>>
>>>> if (xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
>>>>      return 0;
>>>>
>>>> xdst->pols[0] contains a NULL pointer
>>>>
>>> That does not really make sense, if we get this far; there's a valid
>>> xfrm_state with the bundle. This means that there existed a policy with
>>> it too.
>>>
>>> I'll take a deeper look at this tomorrow. Would it be possible to see
>>> your xfrm policies?
>>>
>>> - Timo
>>>
>>>
>>
>> @Eric sure no problem doing the bisect...
>>
>> as for the xfrm policy here is the link that I used to setup ipsec:
>> http://www.linuxfromscratch.org/hints/downloads/files/ipsec.txt
>> (just change the keys if doing real world work..(but for me just testing).
>>
>> below is a temporary fix for me to get this working, tcpdump reports
>> everything is doing what it should be
>> 11:16:32.496166 IP xxxxx>  xxxxx: AH(spi=0x00000200,seq=0x1090):
>> ESP(spi=0x00000201,seq=0x1090), length 56
>> 11:16:32.496212 IP xxxxx>  xxxxx: AH(spi=0x00000200,seq=0x1091):
>> ESP(spi=0x00000201,seq=0x1091), length 56
>> 11:16:32.496259 IP xxxxx>  xxxxx: AH(spi=0x00000200,seq=0x1092):
>> ESP(spi=0x00000201,seq=0x1092), length 56
>>
>> (tested a few mins ago, but not the right fix..)
>
> Yes, that would break some other obscure scenarios.
>
> Looks like it's ah inside esp. So you get chain of bundles. And only the
> first bundle gets a policy. Should have thought of that. Does the below
> fix it for you?
>
> diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
> index 4bf27d9..af1c173 100644
> --- a/net/xfrm/xfrm_policy.c
> +++ b/net/xfrm/xfrm_policy.c
> @@ -2300,7 +2300,8 @@ int xfrm_bundle_ok(struct xfrm_policy *pol, struct
> xfrm_dst *first,
>   			return 0;
>   		if (xdst->xfrm_genid != dst->xfrm->genid)
>   			return 0;
> -		if (xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
> +		if (xdst->num_pols>  0&&
> +		    xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
>   			return 0;
>
>   		if (strict&&  fl&&
>
>


yeah this works.. I can see AH and ESP showing up with tcpdump..
looks good..

Reported-Bisected-By: Justin P. Mattock <justinmattock@gmail.com>

cheers,

Justin P. Mattock

[PATCH] xfrm: check bundle policy existance before dereferencing it

[Timo Teräs]

Fix the bundle validation code to not assume having a valid policy.
When we have multiple transformations for a xfrm policy, the bundle
instance will be a chain of bundles with only the first one having
the policy reference. When policy_genid is bumped it will expire the
first bundle in the chain which is equivalent of expiring the whole
chain.

Reported-bisected-and-tested-by: Justin P. Mattock <justinmattock@gmail.com>
Signed-off-by: Timo Teräs <timo.teras@iki.fi>
---
 net/xfrm/xfrm_policy.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
index 4bf27d9..af1c173 100644
--- a/net/xfrm/xfrm_policy.c
+++ b/net/xfrm/xfrm_policy.c
@@ -2300,7 +2300,8 @@ int xfrm_bundle_ok(struct xfrm_policy *pol, struct xfrm_dst *first,
 			return 0;
 		if (xdst->xfrm_genid != dst->xfrm->genid)
 			return 0;
-		if (xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
+		if (xdst->num_pols > 0 &&
+		    xdst->policy_genid != atomic_read(&xdst->pols[0]->genid))
 			return 0;
 
 		if (strict && fl &&
-- 
1.7.0.4

Re: [PATCH] xfrm: check bundle policy existance before dereferencing it

[David Miller]

From: Timo Teräs <timo.teras@iki.fi>
Date: Thu, 24 Jun 2010 08:45:19 +0300

> Fix the bundle validation code to not assume having a valid policy.
> When we have multiple transformations for a xfrm policy, the bundle
> instance will be a chain of bundles with only the first one having
> the policy reference. When policy_genid is bumped it will expire the
> first bundle in the chain which is equivalent of expiring the whole
> chain.
> 
> Reported-bisected-and-tested-by: Justin P. Mattock <justinmattock@gmail.com>
> Signed-off-by: Timo Teräs <timo.teras@iki.fi>

Applied.