Fw: crash in ipt_do_table

Fw: crash in ipt_do_table

[Andrew Morton]

Begin forwarded message:

Date: Mon, 14 Aug 2006 22:36:37 +0100
From: Chris Lightfoot <chris@ex-parrot.com>
To: linux-kernel@vger.kernel.org
Subject: crash in ipt_do_table


We recently saw this oops on a 2.6.17.6 machine (dual
Xeon, e1000, 3ware 9xxx disk controllers):

BUG: unable to handle kernel paging request at virtual address 4e50cff2
 printing eip:
f8a770c5
*pde = 00000000
Oops: 0000 [#1]
SMP
Modules linked in: xt_tcpudp iptable_filter ip_tables x_tables w83781d hwmon_vid i2c_isa i2c_i801 nfsd exportfs lockd sunrpc e1000 e100 mii dummy
CPU:    0
EIP:    0060:[<f8a770c5>]    Not tainted VLI
EFLAGS: 00010212   (2.6.17.6-sph1 #1)
EIP is at ipt_do_table+0xa9/0x2fc [ip_tables]
eax: 464c457f   ebx: d9435ac0   ecx: 00000003   edx: e4b5c810
esi: 4e50cf9f   edi: 00000000   ebp: 46744586   esp: f6915d88
ds: 007b   es: 007b   ss: 0068
Process nfsd (pid: 1016, threadinfo=f6915000 task=f798c560)
Stack: 00000000 464c457f dfc5a000 f8a7a880 00000000 e4b5c810 00000108 00000000
       f6915e20 c03abff8 80000000 c02301b1 f8a5d073 f6915e60 00000003 00000000
       dfc5a000 f8a5d600 00000000 c022921e 00000003 f6915e60 00000000 dfc5a000
Call Trace:
 <c02301b1> dst_output+0x0/0xd
 <f8a5d073> ipt_local_out_hook+0x53/0x58 [iptable_filter]
 <c022921e> nf_iterate+0x3f/0x5f
 <c02301b1> dst_output+0x0/0xd
 <c0229285> nf_hook_slow+0x47/0xa7
 <c02301b1> dst_output+0x0/0xd
 <c02323cd> ip_push_pending_frames+0x30a/0x3e0
 <c02301b1> dst_output+0x0/0xd
 <c0248a09> udp_push_pending_frames+0x1fe/0x21f
 <c024908e> udp_sendpage+0xcf/0xe9
 <f8aa83b8> svc_sendto+0xf5/0x20c [sunrpc]
 <c01b29f6> _atomic_dec_and_lock+0x2e/0x48
 <f8aa88d6> svc_udp_sendto+0x10/0x23 [sunrpc]
 <f8aa97d7> svc_send+0xa0/0xd2 [sunrpc]
 <f8aa7cf7> svc_process+0x439/0x61a [sunrpc]
 <f8a1e38d> nfsd+0x18f/0x2e8 [nfsd]
 <f8a1e1fe> nfsd+0x0/0x2e8 [nfsd]
 <c0100e2d> kernel_thread_helper+0x5/0xb
Code: ff ff 21 e0 8b 40 10 8b 4c 24 38 8b 44 83 34 89 44 24 04 89 c6 89 c5 03 74 8b 0c 03 6c 8b 20 0f b7 7c 24 1a 8b 54 24 14 89 3c 24 <0f> b6 5e 53 8b 42 0c 8b 0e f6 c3 08 8b 56 08 74 0c 21 d0 39 c8
EIP: [<f8a770c5>] ipt_do_table+0xa9/0x2fc [ip_tables] SS:ESP 0068:f6915d88
 <0>Kernel panic - not syncing: Fatal exception in interrupt

the corresponding code is:

        movzbl  83(%esi), %ebx  # <variable>.invflags, <variable>.invflags
        movl    12(%edx), %eax  # <variable>.saddr, <variable>.saddr
        movl    (%esi), %ecx    # <variable>.src.s_addr, <variable>.src.s_addr
        testb   $8, %bl         #, <variable>.invflags
        movl    8(%esi), %edx   # <variable>.smsk.s_addr, <variable>.smsk.s_addr
        je      .L18            #,
        andl    %edx, %eax      # <variable>.smsk.s_addr, <variable>.saddr
        cmpl    %ecx, %eax      # <variable>.src.s_addr, <variable>.saddr
        je      .L52            #,
        jmp     .L19            #

.config is here:
    http://ex-parrot.com/~chris/tmp/20060814/config

This looks rather like the report in,
    http://lkml.org/lkml/2006/7/25/88
though the generated code is slightly different.

This has only happened once so far, so I'm not (yet) aware
of any way to reproduce it. Unfortunately I don't have a
copy of the iptables rules themselves at the time of the
crash -- on that system they're created dynamically and
the specific setup doesn't survive a reboot.

There didn't seem to be any resolution of the report of a
similar problem from July; any advice would be
appreciated. I'm not on the list so please cc replies if
possible.

-- 
Tigers don't go out on rainy nights /
They've no need to whet their appetites
 (`Hunting Tigers out in Indiah', the Bonzo Dog Doo-Dah Band)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Fw: crash in ipt_do_table

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

Andrew Morton wrote:
> From: Chris Lightfoot <chris@ex-parrot.com>
>
> EIP is at ipt_do_table+0xa9/0x2fc [ip_tables]
>
> This has only happened once so far, so I'm not (yet) aware
> of any way to reproduce it. Unfortunately I don't have a
> copy of the iptables rules themselves at the time of the
> crash -- on that system they're created dynamically and
> the specific setup doesn't survive a reboot.

Any chance you're also changing your ruleset dynamically? If yes
this patch might help.


[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1329 bytes --]

[NETFILTER]: ip_tables: fix table locking in ipt_do_table

table->private might change because of ruleset changes, don't use it without
holding the lock.

Signed-off-by: Patrick McHardy <kaber@trash.net>

---
commit b7534132e3970ec95ea058a701193a71cb1bcc13
tree c90921385346dd0f61d637c126f2c757261aa2c0
parent 32ce9bc41528c327b1353713b2108d2213128dee
author Patrick McHardy <kaber@trash.net> Tue, 15 Aug 2006 15:28:16 +0200
committer Patrick McHardy <kaber@trash.net> Tue, 15 Aug 2006 15:28:16 +0200

 net/ipv4/netfilter/ip_tables.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/net/ipv4/netfilter/ip_tables.c b/net/ipv4/netfilter/ip_tables.c
index f316ff5..048514f 100644
--- a/net/ipv4/netfilter/ip_tables.c
+++ b/net/ipv4/netfilter/ip_tables.c
@@ -230,7 +230,7 @@ ipt_do_table(struct sk_buff **pskb,
 	const char *indev, *outdev;
 	void *table_base;
 	struct ipt_entry *e, *back;
-	struct xt_table_info *private = table->private;
+	struct xt_table_info *private;
 
 	/* Initialization */
 	ip = (*pskb)->nh.iph;
@@ -247,6 +247,7 @@ ipt_do_table(struct sk_buff **pskb,
 
 	read_lock_bh(&table->lock);
 	IP_NF_ASSERT(table->valid_hooks & (1 << hook));
+	private = table->private;
 	table_base = (void *)private->entries[smp_processor_id()];
 	e = get_entry(table_base, private->hook_entry[hook]);
 

Re: Fw: crash in ipt_do_table

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 744 bytes --]

Patrick McHardy wrote:
> Andrew Morton wrote:
> 
>>From: Chris Lightfoot <chris@ex-parrot.com>
>>
>>EIP is at ipt_do_table+0xa9/0x2fc [ip_tables]
>>
>>This has only happened once so far, so I'm not (yet) aware
>>of any way to reproduce it. Unfortunately I don't have a
>>copy of the iptables rules themselves at the time of the
>>crash -- on that system they're created dynamically and
>>the specific setup doesn't survive a reboot.
> 
> 
> Any chance you're also changing your ruleset dynamically? If yes
> this patch might help.
> 
> 
> 
> ------------------------------------------------------------------------
> 
> [NETFILTER]: ip_tables: fix table locking in ipt_do_table


The same bug is present in arp_tables, this patch covers both.


[-- Attachment #2: x --]
[-- Type: text/plain, Size: 2246 bytes --]

[NETFILTER]: ip_tables: fix table locking in ipt_do_table

table->private might change because of ruleset changes, don't use it without
holding the lock.

Signed-off-by: Patrick McHardy <kaber@trash.net>

---
commit 338fe5c67e8fb799c9e3470331db6f3c60a31b1e
tree 2dc15d63244ed18a8035ae483ae2d722e7fbcf62
parent 32ce9bc41528c327b1353713b2108d2213128dee
author Patrick McHardy <kaber@trash.net> Tue, 15 Aug 2006 16:06:57 +0200
committer Patrick McHardy <kaber@trash.net> Tue, 15 Aug 2006 16:06:57 +0200

 net/ipv4/netfilter/arp_tables.c |    3 ++-
 net/ipv4/netfilter/ip_tables.c  |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/net/ipv4/netfilter/arp_tables.c b/net/ipv4/netfilter/arp_tables.c
index df4854c..8d1d7a6 100644
--- a/net/ipv4/netfilter/arp_tables.c
+++ b/net/ipv4/netfilter/arp_tables.c
@@ -236,7 +236,7 @@ unsigned int arpt_do_table(struct sk_buf
 	struct arpt_entry *e, *back;
 	const char *indev, *outdev;
 	void *table_base;
-	struct xt_table_info *private = table->private;
+	struct xt_table_info *private;
 
 	/* ARP header, plus 2 device addresses, plus 2 IP addresses.  */
 	if (!pskb_may_pull((*pskb), (sizeof(struct arphdr) +
@@ -248,6 +248,7 @@ unsigned int arpt_do_table(struct sk_buf
 	outdev = out ? out->name : nulldevname;
 
 	read_lock_bh(&table->lock);
+	private = table->private;
 	table_base = (void *)private->entries[smp_processor_id()];
 	e = get_entry(table_base, private->hook_entry[hook]);
 	back = get_entry(table_base, private->underflow[hook]);
diff --git a/net/ipv4/netfilter/ip_tables.c b/net/ipv4/netfilter/ip_tables.c
index f316ff5..048514f 100644
--- a/net/ipv4/netfilter/ip_tables.c
+++ b/net/ipv4/netfilter/ip_tables.c
@@ -230,7 +230,7 @@ ipt_do_table(struct sk_buff **pskb,
 	const char *indev, *outdev;
 	void *table_base;
 	struct ipt_entry *e, *back;
-	struct xt_table_info *private = table->private;
+	struct xt_table_info *private;
 
 	/* Initialization */
 	ip = (*pskb)->nh.iph;
@@ -247,6 +247,7 @@ ipt_do_table(struct sk_buff **pskb,
 
 	read_lock_bh(&table->lock);
 	IP_NF_ASSERT(table->valid_hooks & (1 << hook));
+	private = table->private;
 	table_base = (void *)private->entries[smp_processor_id()];
 	e = get_entry(table_base, private->hook_entry[hook]);
 

Re: Fw: crash in ipt_do_table

[Chris Lightfoot]

On Tue, Aug 15, 2006 at 04:01:06PM +0200, Patrick McHardy wrote:
> Andrew Morton wrote:
> > From: Chris Lightfoot <chris@ex-parrot.com>
> >
> > EIP is at ipt_do_table+0xa9/0x2fc [ip_tables]
> >
> > This has only happened once so far, so I'm not (yet) aware
> > of any way to reproduce it. Unfortunately I don't have a
> > copy of the iptables rules themselves at the time of the
> > crash -- on that system they're created dynamically and
> > the specific setup doesn't survive a reboot.
> 
> Any chance you're also changing your ruleset dynamically? If yes
> this patch might help.

yes, we are. Thanks for the patch -- I will apply it and
see what happens.

-- 
``The fishy glitter in his eye became intensified. He looked like
  a halibut which had been asked by another halibut to lend it a
  couple of quid till next Wednesday.'' (P G Wodehouse)

Re: crash in ipt_do_table

[David Miller]

From: Patrick McHardy <kaber@trash.net>
Date: Tue, 15 Aug 2006 16:01:06 +0200

> Any chance you're also changing your ruleset dynamically? If yes
> this patch might help.

I've applied this.

Re: crash in ipt_do_table

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

David Miller wrote:
> From: Patrick McHardy <kaber@trash.net>
> Date: Tue, 15 Aug 2006 16:01:06 +0200
> 
> 
>>Any chance you're also changing your ruleset dynamically? If yes
>>this patch might help.
> 
> 
> I've applied this.

Thanks, but it seems you applied the first patch I sent, which was
missing the same fix for arp_tables. This patch contains the missing
bits.

I'm going to send the (combined) patch to -stable as well, this bug
seems to have hit quite a few people.


[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1470 bytes --]

[NETFILTER]: arp_tables: fix table locking in arpt_do_table

table->private might change because of ruleset changes, don't use it without
holding the lock.

Signed-off-by: Patrick McHardy <kaber@trash.net>

---
commit 338fe5c67e8fb799c9e3470331db6f3c60a31b1e
tree 2dc15d63244ed18a8035ae483ae2d722e7fbcf62
parent 32ce9bc41528c327b1353713b2108d2213128dee
author Patrick McHardy <kaber@trash.net> Tue, 15 Aug 2006 16:06:57 +0200
committer Patrick McHardy <kaber@trash.net> Tue, 15 Aug 2006 16:06:57 +0200

 net/ipv4/netfilter/arp_tables.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/net/ipv4/netfilter/arp_tables.c b/net/ipv4/netfilter/arp_tables.c
index df4854c..8d1d7a6 100644
--- a/net/ipv4/netfilter/arp_tables.c
+++ b/net/ipv4/netfilter/arp_tables.c
@@ -236,7 +236,7 @@ unsigned int arpt_do_table(struct sk_buf
 	struct arpt_entry *e, *back;
 	const char *indev, *outdev;
 	void *table_base;
-	struct xt_table_info *private = table->private;
+	struct xt_table_info *private;
 
 	/* ARP header, plus 2 device addresses, plus 2 IP addresses.  */
 	if (!pskb_may_pull((*pskb), (sizeof(struct arphdr) +
@@ -248,6 +248,7 @@ unsigned int arpt_do_table(struct sk_buf
 	outdev = out ? out->name : nulldevname;
 
 	read_lock_bh(&table->lock);
+	private = table->private;
 	table_base = (void *)private->entries[smp_processor_id()];
 	e = get_entry(table_base, private->hook_entry[hook]);
 	back = get_entry(table_base, private->underflow[hook]);

Re: crash in ipt_do_table

[David Miller]

From: Patrick McHardy <kaber@trash.net>
Date: Fri, 18 Aug 2006 07:45:59 +0200

> Thanks, but it seems you applied the first patch I sent, which was
> missing the same fix for arp_tables. This patch contains the missing
> bits.
> 
> I'm going to send the (combined) patch to -stable as well, this bug
> seems to have hit quite a few people.

I just sent a round of patches to Greg for his 2.6.18 tree, which
included the first part, so I'll push this second part to him next.

Feel free to queue the whole thing up for -stable, thanks a lot.