Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Sat, 23 Apr 2011 07:51:56 GMT bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=33842
> 
>            Summary: NULL pointer dereference in ip_fragment

oops in ip_defragment().  Kernel is 2.6.39-rc4.  There are some
screenshots attached to the report.


>            Product: Networking
>            Version: 2.5
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>         AssignedTo: acme@ghostprotocols.net
>         ReportedBy: tom@dbservice.com
>         Regression: No
> 
> 
> The host is using the ath9k driver. eth0+wlan0 are bridged. Shortly after I
> start using the wireless network with my macbook, the bug triggers. No idea if
> it's wireless related, because there's also a rtl8169_rx_interrupt entry in the
> stacktrace.
> 
> This is a transcript, since I don't (have/know of) any way to get the backtrace
> out of a crashed box.
> 
> IP: ip_fragment+0x52/0x840
> Call Trace:
>   <IRQ>
>   br_parse_ip_options
>   br_flood_deliver
>   br_parse_ip_options
>   br_nf_dev_queue_xmit
>   br_nf_post_routing
>   nf_iterate
> 
> then also:
>   lots of br_flood_deliver
>   lots of br_*_finish
>   one ? rtl8169_interrupt
>   one ? ath9k_ioread32
> 

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Bandan Das]

https://bugzilla.kernel.org/show_bug.cgi?id=33842

I believe  Eric's recent change to br_parse_ip_options() 
didn't make it to 2.6.39-rc4:

bridge: reset IPCB in br_parse_ip_options
commit f8e9881c2aef1e982e5abc25c046820cd0b7cf64

diff --git a/net/bridge/br_netfilter.c b/net/bridge/br_netfilter.c
index 008ff6c..b353f7c 100644
--- a/net/bridge/br_netfilter.c
+++ b/net/bridge/br_netfilter.c
@@ -249,11 +249,9 @@  static int br_parse_ip_options(struct sk_buff *skb)
           goto drop;
           }
 
-       /* Zero out the CB buffer if no options present */
-       if (iph->ihl == 5) {
-          memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
+          memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
+          if (iph->ihl == 5)
              return 0;
-             }
 
        opt->optlen = iph->ihl*4 - sizeof(struct iphdr);
        if (ip_options_compile(dev_net(dev), opt, skb))



Tomas, could you please try a kernel that has the above 
mentioned change and see if the crash re-occurs ?

Thanks,
Bandan

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[David Miller]

From: Bandan Das <bandan.das@stratus.com>
Date: Tue, 26 Apr 2011 16:31:54 -0400

> https://bugzilla.kernel.org/show_bug.cgi?id=33842
> 
> I believe  Eric's recent change to br_parse_ip_options() 
> didn't make it to 2.6.39-rc4:
> 
> bridge: reset IPCB in br_parse_ip_options
> commit f8e9881c2aef1e982e5abc25c046820cd0b7cf64
> 
> diff --git a/net/bridge/br_netfilter.c b/net/bridge/br_netfilter.c
> index 008ff6c..b353f7c 100644
> --- a/net/bridge/br_netfilter.c
> +++ b/net/bridge/br_netfilter.c
> @@ -249,11 +249,9 @@  static int br_parse_ip_options(struct sk_buff *skb)
>            goto drop;
>            }
>  
> -       /* Zero out the CB buffer if no options present */
> -       if (iph->ihl == 5) {
> -          memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
> +          memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
> +          if (iph->ihl == 5)
>               return 0;
> -             }
>  
>         opt->optlen = iph->ihl*4 - sizeof(struct iphdr);
>         if (ip_options_compile(dev_net(dev), opt, skb))
> 
> 
> 
> Tomas, could you please try a kernel that has the above 
> mentioned change and see if the crash re-occurs ?

This patch is mangled by your email client, tab characters
have been turned into spaces, so it won't be usable by anyone.

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Eric Dumazet]

Le mardi 26 avril 2011 à 13:46 -0700, David Miller a écrit :

> This patch is mangled by your email client, tab characters
> have been turned into spaces, so it won't be usable by anyone.

Thats strange, I thought it was already in linux-2.6 anyway ?

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Bandan Das]

On  0, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le mardi 26 avril 2011 à 13:46 -0700, David Miller a écrit :
> 
> > This patch is mangled by your email client, tab characters
> > have been turned into spaces, so it won't be usable by anyone.
> 
> Thats strange, I thought it was already in linux-2.6 anyway ?
> 
Umm.. I could be wrong! I just did a quick grep for your name in the 
2.6.39-rc4 changelog : 
http://www.kernel.org/pub/linux/kernel/v2.6/testing/ChangeLog-2.6.39-rc4

and didn't find it there.

--
Bandan

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Eric Dumazet]

Le mardi 26 avril 2011 à 16:59 -0400, Bandan Das a écrit :
> On  0, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > Le mardi 26 avril 2011 à 13:46 -0700, David Miller a écrit :
> > 
> > > This patch is mangled by your email client, tab characters
> > > have been turned into spaces, so it won't be usable by anyone.
> > 
> > Thats strange, I thought it was already in linux-2.6 anyway ?
> > 
> Umm.. I could be wrong! I just did a quick grep for your name in the 
> 2.6.39-rc4 changelog : 
> http://www.kernel.org/pub/linux/kernel/v2.6/testing/ChangeLog-2.6.39-rc4
> 
> and didn't find it there.

Then it will be in rc5, dont worry ;)

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Bandan Das]

> > Umm.. I could be wrong! I just did a quick grep for your name in the 
> > 2.6.39-rc4 changelog : 
> > http://www.kernel.org/pub/linux/kernel/v2.6/testing/ChangeLog-2.6.39-rc4
> > 
> > and didn't find it there.
> 
> Then it will be in rc5, dont worry ;)
> 
> 
Yeah, I just rechecked and this is already in Linus' tree. So, Tomas you can
either try pulling in those changes or you can apply this patch and see
if it makes any difference. Thanks!


diff --git a/net/bridge/br_netfilter.c b/net/bridge/br_netfilter.c
index 008ff6c..f3bc322 100644
--- a/net/bridge/br_netfilter.c
+++ b/net/bridge/br_netfilter.c
@@ -249,11 +249,9 @@ static int br_parse_ip_options(struct sk_buff *skb)
 		goto drop;
 	}
 
-	/* Zero out the CB buffer if no options present */
-	if (iph->ihl == 5) {
-		memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
+	memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
+	if (iph->ihl == 5)
 		return 0;
-	}
 
 	opt->optlen = iph->ihl*4 - sizeof(struct iphdr);
 	if (ip_options_compile(dev_net(dev), opt, skb))

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Eric Dumazet]

Le mardi 26 avril 2011 à 17:19 -0400, Bandan Das a écrit :
> > 
> Yeah, I just rechecked and this is already in Linus' tree. So, Tomas you can
> either try pulling in those changes or you can apply this patch and see
> if it makes any difference. Thanks!

Better pull Linus tree because there is another patch involved.

(commits c65353daf137dd41f3ede3baf62d561fca076228
ip: ip_options_compile() resilient to NULL skb route)

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Tomas Carnecky]

On 4/26/11 11:24 PM, Eric Dumazet wrote:
> Le mardi 26 avril 2011 à 17:19 -0400, Bandan Das a écrit :
>> Yeah, I just rechecked and this is already in Linus' tree. So, Tomas you can
>> either try pulling in those changes or you can apply this patch and see
>> if it makes any difference. Thanks!
> Better pull Linus tree because there is another patch involved.
>
> (commits c65353daf137dd41f3ede3baf62d561fca076228
> ip: ip_options_compile() resilient to NULL skb route

Still getting that error (on rc4-00245-g4175242, which includes that 
commit).

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Eric Dumazet]

Le mercredi 27 avril 2011 à 09:22 +0200, Tomas Carnecky a écrit :
> On 4/26/11 11:24 PM, Eric Dumazet wrote:
> > Le mardi 26 avril 2011 à 17:19 -0400, Bandan Das a écrit :
> >> Yeah, I just rechecked and this is already in Linus' tree. So, Tomas you can
> >> either try pulling in those changes or you can apply this patch and see
> >> if it makes any difference. Thanks!
> > Better pull Linus tree because there is another patch involved.
> >
> > (commits c65353daf137dd41f3ede3baf62d561fca076228
> > ip: ip_options_compile() resilient to NULL skb route
> 
> Still getting that error (on rc4-00245-g4175242, which includes that 
> commit).
> 

Could you send us a complete trace ?

One way to get one is to use netconsole (provided you have another
machine )

grep NETCONSOLE .config
CONFIG_NETCONSOLE=y

Add on your boot command

netconsole=4444@192.168.20.108/eth0,4444@192.168.20.112/00:1e:0b:ec:c3:e4

messages sent to host 192.168.20.112 udp port 4444, mac addr 00:1e:0b:ec:c3:e4, on eth0

then on 192.168.20.112 start a netcat listening on udp port 4444 to get
a copy of messages.

netcat -l -u -p 4444

Complete doc on Documentation/networking/netconsole.txt

Thanks !

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Tomas Carnecky]

On 4/27/11 9:41 AM, Eric Dumazet wrote:
> netconsole=4444@192.168.20.108/eth0,4444@192.168.20.112/00:1e:0b:ec:c3:e4
I'm not having any luck with the netconsole. The last message I see on 
the target host is "Freeing unused kernel memory: 100k freed". I don't 
see any messages after that. Do I need to configure the same IP address 
in netconsole as is later configured by userspace? I set netconsole to 
use 192.168.0.50 while the init scripts set br0 to have 192.168.0.82. 
And would netconsole even work, as the bug is in the networking code itself?

Re: [Bugme-new] [Bug 33842] New: NULL pointer dereference in ip_fragment

[Eric Dumazet]

Le mercredi 27 avril 2011 à 13:52 +0200, Tomas Carnecky a écrit :
> On 4/27/11 9:41 AM, Eric Dumazet wrote:
> > netconsole=4444@192.168.20.108/eth0,4444@192.168.20.112/00:1e:0b:ec:c3:e4
> I'm not having any luck with the netconsole. The last message I see on 
> the target host is "Freeing unused kernel memory: 100k freed". I don't 
> see any messages after that. Do I need to configure the same IP address 
> in netconsole as is later configured by userspace? I set netconsole to 
> use 192.168.0.50 while the init scripts set br0 to have 192.168.0.82. 
> And would netconsole even work, as the bug is in the networking code itself?
> 
> 

It should work yes, even for a bug in networking stack.

Yes, you should take the source address you're supposed to have once
machine running. I suspect it could work with another IP address, but
using the normal one makes sure you dont hit some anti spoofing rule in
your LAN.

To check if netconsole works (after boot), you can try

dmesg -n 8
modprobe pktgen

You should see on remote machine :
pktgen: Packet Generator for packet performance testing. Version: 2.74