kernel BUG at kernel/sched.c:2833!

kernel BUG at kernel/sched.c:2833!

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2116 bytes --]

Hello,

------------[ cut here ]------------
kernel BUG at kernel/sched.c:2833!
invalid operand: 0000 [#1]
PREEMPT
Modules linked in:
CPU:    0
EIP:    0060:[sub_preempt_count+53/64]    Not tainted VLI
EFLAGS: 00010206   (2.6.14.2)
EIP is at sub_preempt_count+0x35/0x40
eax: dccff000   ebx: dcf502a4   ecx: 00000000   edx: 000000ff
esi: dcf501e8   edi: 00000100   ebp: dccffc9c   esp: dccffc9c
ds: 007b   es: 007b   ss: 0068
Process conntrack (pid: 373, threadinfo=dccff000 task=dccaa5c0)
Stack: dd284278 c011f1f9 dcf502a4 dcf501e8 00000100 dd284278 c03564f5 df2efb60
        00000175 4378576a 00000000 00000001 dcf501e8 df2efb60 df2efb60 dd284260
        dd56a600 c0311968 df2efb60 dd284260 00000000 000000d0 dd56a600 dccffd48
Call Trace:
  [local_bh_enable+25/144] local_bh_enable+0x19/0x90
  [ctnetlink_dump_table+149/272] ctnetlink_dump_table+0x95/0x110
  [netlink_dump+88/512] netlink_dump+0x58/0x200
  [netlink_recvmsg+558/576] netlink_recvmsg+0x22e/0x240
  [netlink_sendskb+50/96] netlink_sendskb+0x32/0x60
  [sock_recvmsg+254/288] sock_recvmsg+0xfe/0x120
  [update_atime+149/176] update_atime+0x95/0xb0
  [sock_sendmsg+229/256] sock_sendmsg+0xe5/0x100
  [autoremove_wake_function+0/96] autoremove_wake_function+0x0/0x60
  [sys_recvmsg+323/512] sys_recvmsg+0x143/0x200
  [lru_cache_add_active+57/112] lru_cache_add_active+0x39/0x70
  [do_anonymous_page+250/352] do_anonymous_page+0xfa/0x160
  [do_no_page+105/816] do_no_page+0x69/0x330
  [copy_from_user+70/144] copy_from_user+0x46/0x90
  [sys_socketcall+591/608] sys_socketcall+0x24f/0x260
  [do_page_fault+0/1501] do_page_fault+0x0/0x5dd
  [syscall_call+7/11] syscall_call+0x7/0xb
Code: 89 e5 3b 50 14 7f 24 81 fa fe 00 00 00 76 0c b8 00 f0 ff ff 21 e0 29 50 14 c9 c3 80 78 14 00 75 ee 0f 0b 15 0b 37 8e 3b c0 eb e4 <0f> 0b 11 0b 37 8e 3b c0 eb d2 90 55 89 e5 8b 45 08 8b 50 04 89


AFAIK there were some small fixes that went into 2.6.15-rc1 which were 
supposed to fix such problems with conntrack -L. Shuldn't we send them to 
-stable to make 2.6.14.x also usable?


Best regards,


 				Krzysztof Olędzki

Re: kernel BUG at kernel/sched.c:2833!

[Pablo Neira]

[-- Attachment #1: Type: text/plain, Size: 417 bytes --]

Krzysztof Oledzki wrote:
> EIP is at sub_preempt_count+0x35/0x40
> 
> AFAIK there were some small fixes that went into 2.6.15-rc1 which were
> supposed to fix such problems with conntrack -L. Shuldn't we send them
> to -stable to make 2.6.14.x also usable?

I think so, at the least Yasuyuki's:
[NETFILTER] refcount leak of proto when ctnetlink dumping tuple

That fixes the problem that you're reporting.

-- 
Pablo

[-- Attachment #2: 02-ctnl-refcnt.patch --]
[-- Type: text/plain, Size: 1343 bytes --]

[NETFILTER] refcount leak of proto when ctnetlink dumping tuple

Signed-off-by: Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp>

---
commit 3a4486b6419a1f25324bb4280d51f5c77b1117f7
tree 88b1831d06e21417baca01d1632131d96e3be611
parent 61a002f080c6473da94f28314502ff0f15fe3625
author Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp> Fri, 04 Nov 2005 14:35:27 +0900
committer Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp> Fri, 04 Nov 2005 14:35:27 +0900

 net/ipv4/netfilter/ip_conntrack_netlink.c |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/net/ipv4/netfilter/ip_conntrack_netlink.c b/net/ipv4/netfilter/ip_conntrack_netlink.c
--- a/net/ipv4/netfilter/ip_conntrack_netlink.c
+++ b/net/ipv4/netfilter/ip_conntrack_netlink.c
@@ -58,14 +58,17 @@ ctnetlink_dump_tuples_proto(struct sk_bu
 			    const struct ip_conntrack_tuple *tuple)
 {
 	struct ip_conntrack_protocol *proto;
+	int ret = 0;
 
 	NFA_PUT(skb, CTA_PROTO_NUM, sizeof(u_int8_t), &tuple->dst.protonum);
 
 	proto = ip_conntrack_proto_find_get(tuple->dst.protonum);
-	if (proto && proto->tuple_to_nfattr)
-		return proto->tuple_to_nfattr(skb, tuple);
+	if (likely(proto && proto->tuple_to_nfattr)) {
+		ret = proto->tuple_to_nfattr(skb, tuple);
+		ip_conntrack_proto_put(proto);
+	}
 
-	return 0;
+	return ret;
 
 nfattr_failure:
 	return -1;

Re: kernel BUG at kernel/sched.c:2833!

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2401 bytes --]



On Mon, 14 Nov 2005, Pablo Neira wrote:

> Krzysztof Oledzki wrote:
>> EIP is at sub_preempt_count+0x35/0x40
>>
>> AFAIK there were some small fixes that went into 2.6.15-rc1 which were
>> supposed to fix such problems with conntrack -L. Shuldn't we send them
>> to -stable to make 2.6.14.x also usable?
>
> I think so, at the least Yasuyuki's:
> [NETFILTER] refcount leak of proto when ctnetlink dumping tuple
>
> That fixes the problem that you're reporting.

What about this one?

------------[ cut here ]------------
kernel BUG at kernel/sched.c:2833!
invalid operand: 0000 [#2]
PREEMPT
Modules linked in:
CPU:    0
EIP:    0060:[sub_preempt_count+53/64]    Not tainted VLI
EFLAGS: 00010206   (2.6.14.2)
EIP is at sub_preempt_count+0x35/0x40
eax: daa35000   ebx: daecaec0   ecx: 00000000   edx: 000000ff
esi: daecae04   edi: 00000e5d   ebp: daa35c9c   esp: daa35c9c
ds: 007b   es: 007b   ss: 0068
Process conntrack (pid: 1403, threadinfo=daa35000 task=dc3ca0b0)
Stack: ded31fb8 c011f1f9 daecaec0 daecae04 00000e5d ded31fb8 c0356565 da1cd260
        0000057b 43790384 00000000 00000001 daecae04 da1cd260 da1cd260 ded31fa0
        dbc5dc00 c03119a8 da1cd260 ded31fa0 00000000 000000d0 dbc5dc00 daa35d48
Call Trace:
  [local_bh_enable+25/144] local_bh_enable+0x19/0x90
  [ctnetlink_dump_table+149/272] ctnetlink_dump_table+0x95/0x110
  [netlink_dump+88/512] netlink_dump+0x58/0x200
  [netlink_recvmsg+558/576] netlink_recvmsg+0x22e/0x240
  [sock_recvmsg+254/288] sock_recvmsg+0xfe/0x120
  [update_atime+149/176] update_atime+0x95/0xb0
  [do_generic_mapping_read+789/1520] do_generic_mapping_read+0x315/0x5f0
  [autoremove_wake_function+0/96] autoremove_wake_function+0x0/0x60
  [sys_recvmsg+323/512] sys_recvmsg+0x143/0x200
  [current_fs_time+81/112] current_fs_time+0x51/0x70
  [inode_update_time+82/224] inode_update_time+0x52/0xe0
  [pipe_writev+742/1248] pipe_writev+0x2e6/0x4e0
  [pipe_write+55/64] pipe_write+0x37/0x40
  [copy_from_user+70/144] copy_from_user+0x46/0x90
  [sys_socketcall+591/608] sys_socketcall+0x24f/0x260
  [sys_write+81/128] sys_write+0x51/0x80
  [syscall_call+7/11] syscall_call+0x7/0xb
Code: 89 e5 3b 50 14 7f 24 81 fa fe 00 00 00 76 0c b8 00 f0 ff ff 21 e0 29 50 14 c9 c3 80 78 14 0
15 0b 17 8f 3b c0 eb e4 <0f> 0b 11 0b 17 8f 3b c0 eb d2 90 55 89 e5 8b 45 08 8b 50 04 89


Best regards,

 				Krzysztof Olędzki

Re: kernel BUG at kernel/sched.c:2833!

[Pablo Neira]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

Krzysztof Oledzki wrote:
> On Mon, 14 Nov 2005, Pablo Neira wrote:
> 
>> Krzysztof Oledzki wrote:
>>
>>> EIP is at sub_preempt_count+0x35/0x40
>>>
>>> AFAIK there were some small fixes that went into 2.6.15-rc1 which were
>>> supposed to fix such problems with conntrack -L. Shuldn't we send them
>>> to -stable to make 2.6.14.x also usable?
>>
>> I think so, at the least Yasuyuki's:
>> [NETFILTER] refcount leak of proto when ctnetlink dumping tuple
>>
>> That fixes the problem that you're reporting.
> 
> What about this one?

I realised that Yasuyuki's patch is incomplete. Could you give a try to
the patch attached. Thanks.

-- 
Pablo

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1401 bytes --]

[NETFILTER] Fix ip_conntrack_proto_find_get badness

The function ip_conntrack_proto_find_get always returns a valid pointer. The 
generic protocol helper is returned when no specific protocol helper is found.
This patch fixes as well a missing putting at dump_protoinfo, when no to_attr
is found.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>

diff --git a/net/ipv4/netfilter/ip_conntrack_netlink.c b/net/ipv4/netfilter/ip_conntrack_netlink.c
index f5e5e31..b797ef9 100644
--- a/net/ipv4/netfilter/ip_conntrack_netlink.c
+++ b/net/ipv4/netfilter/ip_conntrack_netlink.c
@@ -59,11 +59,13 @@ ctnetlink_dump_tuples_proto(struct sk_bu
 
 	NFA_PUT(skb, CTA_PROTO_NUM, sizeof(u_int8_t), &tuple->dst.protonum);
 
+	/* If no protocol helper is found, this function will return the
+	 * generic protocol helper, so proto won't *ever* be NULL */
 	proto = ip_conntrack_proto_find_get(tuple->dst.protonum);
-	if (likely(proto && proto->tuple_to_nfattr)) {
+	if (likely(proto->tuple_to_nfattr))
 		ret = proto->tuple_to_nfattr(skb, tuple);
-		ip_conntrack_proto_put(proto);
-	}
+
+	ip_conntrack_proto_put(proto);
 
 	return ret;
 
@@ -128,9 +130,11 @@ ctnetlink_dump_protoinfo(struct sk_buff 
 
 	struct nfattr *nest_proto;
 	int ret;
-	
-	if (!proto || !proto->to_nfattr)
+
+	if (!proto->to_nfattr) {
+		ip_conntrack_proto_put(proto);
 		return 0;
+	}
 	
 	nest_proto = NFA_NEST(skb, CTA_PROTOINFO);
 

Re: kernel BUG at kernel/sched.c:2833!

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]

On Thu, Nov 17, 2005 at 01:36:35AM +0100, Pablo Neira wrote:
> [NETFILTER] Fix ip_conntrack_proto_find_get badness

thanks, applied.

-- 
- Harald Welte <laforge@netfilter.org>                 http://netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: kernel BUG at kernel/sched.c:2833!

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2529 bytes --]

y

On Thu, 17 Nov 2005, Pablo Neira wrote:

> Krzysztof Oledzki wrote:
>> On Mon, 14 Nov 2005, Pablo Neira wrote:
>>
>>> Krzysztof Oledzki wrote:
>>>
>>>> EIP is at sub_preempt_count+0x35/0x40
>>>>
>>>> AFAIK there were some small fixes that went into 2.6.15-rc1 which were
>>>> supposed to fix such problems with conntrack -L. Shuldn't we send them
>>>> to -stable to make 2.6.14.x also usable?
>>>
>>> I think so, at the least Yasuyuki's:
>>> [NETFILTER] refcount leak of proto when ctnetlink dumping tuple
>>>
>>> That fixes the problem that you're reporting.
>>
>> What about this one?
>
> I realised that Yasuyuki's patch is incomplete. Could you give a try to
> the patch attached. Thanks.

Didn't help:

  ------------[ cut here ]------------
kernel BUG at kernel/sched.c:2833!
invalid operand: 0000 [#55]
PREEMPT
Modules linked in:
CPU:    0
EIP:    0060:[<c01166a5>]    Not tainted VLI
EFLAGS: 00010206   (2.6.14.2)
EIP is at sub_preempt_count+0x35/0x40
eax: dc4a1000   ebx: de57d82c   ecx: 00000000   edx: 000000ff
esi: de57d770   edi: 000006ff   ebp: dc4a1c9c   esp: dc4a1c9c
ds: 007b   es: 007b   ss: 0068
Process conntrack (pid: 2601, threadinfo=dc4a1000 task=dddf35c0)
Stack: ded58fb8 c011f1f9 de57d82c de57d770 000006ff ded58fb8 c0356565 dc4808e0
        00000a29 437cfdd2 00000000 00000001 de57d770 dc4808e0 dc4808e0 ded58fa0
        dcae7c00 c03119a8 dc4808e0 ded58fa0 00000000 000000d0 dcae7c00 dc4a1d48
Call Trace:
  [<c011f1f9>] local_bh_enable+0x19/0x90
  [<c0356565>] ctnetlink_dump_table+0x95/0x110
  [<c03119a8>] netlink_dump+0x58/0x200
  [<c031172e>] netlink_recvmsg+0x22e/0x240
  [<c02e04fe>] sock_recvmsg+0xfe/0x120
  [<c0177e78>] update_atime+0x58/0xb0
  [<c013c1f5>] do_generic_mapping_read+0x315/0x5f0
  [<c012f620>] autoremove_wake_function+0x0/0x60
  [<c02e1f93>] sys_recvmsg+0x143/0x200
  [<c011eff1>] current_fs_time+0x51/0x70
  [<c0177f22>] inode_update_time+0x52/0xe0
  [<c0168cd6>] pipe_writev+0x2e6/0x4e0
  [<c0101469>] __switch_to+0x19/0x1f0
  [<c0216836>] copy_from_user+0x46/0x90
  [<c02e229f>] sys_socketcall+0x24f/0x260
  [<c0102be1>] syscall_call+0x7/0xb
Code: 89 e5 3b 50 14 7f 24 81 fa fe 00 00 00 76 0c b8 00 f0 ff ff 21 e0 29 50 14 c9 c3 80 78 14 00 75 ee 0f 0b 15 0b 17 8f 3b c0 eb e4 <0f> 0b 11 0b 17 8f 3b c0 eb d2 90 55 89 e5 8b 45 08 8b 50 04 89

It seems it is even worse. :(

Need about 3 seconds to trigger this bug with:

# while true; do conntrack -L|wc -l; done

Best regards,

 				Krzysztof Olędzki

Re: kernel BUG at kernel/sched.c:2833!

[Pablo Neira]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

Harald Welte wrote:
> On Thu, Nov 17, 2005 at 01:36:35AM +0100, Pablo Neira wrote:
> 
>>[NETFILTER] Fix ip_conntrack_proto_find_get badness
> 
> thanks, applied.

Sorry Harald, that patch is incomplete :(. Please, revert it. Attached
the final version.

BTW, you have two trees at people.netfilter.org, which one should I use?

-- 
Pablo

[-- Attachment #2: proto_find.patch --]
[-- Type: text/plain, Size: 2249 bytes --]

Remove proto == NULL checking since ip_conntrack_[nat_]proto_find_get always
returns a valid pointer. Fix missing ip_conntrack_proto_put in some paths.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>

Index: netfilter-2.6.14.git/net/ipv4/netfilter/ip_conntrack_netlink.c
===================================================================
--- netfilter-2.6.14.git.orig/net/ipv4/netfilter/ip_conntrack_netlink.c	2005-11-18 04:10:01.000000000 +0100
+++ netfilter-2.6.14.git/net/ipv4/netfilter/ip_conntrack_netlink.c	2005-11-18 04:12:09.000000000 +0100
@@ -59,11 +59,13 @@ ctnetlink_dump_tuples_proto(struct sk_bu
 
 	NFA_PUT(skb, CTA_PROTO_NUM, sizeof(u_int8_t), &tuple->dst.protonum);
 
+	/* If no protocol helper is found, this function will return the
+	 * generic protocol helper, so proto won't *ever* be NULL */
 	proto = ip_conntrack_proto_find_get(tuple->dst.protonum);
-	if (likely(proto && proto->tuple_to_nfattr)) {
+	if (likely(proto->tuple_to_nfattr))
 		ret = proto->tuple_to_nfattr(skb, tuple);
-		ip_conntrack_proto_put(proto);
-	}
+	
+	ip_conntrack_proto_put(proto);
 
 	return ret;
 
@@ -128,9 +130,11 @@ ctnetlink_dump_protoinfo(struct sk_buff 
 
 	struct nfattr *nest_proto;
 	int ret;
-	
-	if (!proto || !proto->to_nfattr)
+
+	if (!proto->to_nfattr) {
+		ip_conntrack_proto_put(proto);
 		return 0;
+	}
 	
 	nest_proto = NFA_NEST(skb, CTA_PROTOINFO);
 
@@ -527,10 +531,10 @@ ctnetlink_parse_tuple_proto(struct nfatt
 
 	proto = ip_conntrack_proto_find_get(tuple->dst.protonum);
 
-	if (likely(proto && proto->nfattr_to_tuple)) {
+	if (likely(proto && proto->nfattr_to_tuple))
 		ret = proto->nfattr_to_tuple(tb, tuple);
-		ip_conntrack_proto_put(proto);
-	}
+	
+	ip_conntrack_proto_put(proto);
 	
 	return ret;
 }
@@ -596,8 +600,6 @@ static int ctnetlink_parse_nat_proto(str
 		return -EINVAL;
 
 	npt = ip_nat_proto_find_get(ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple.dst.protonum);
-	if (!npt)
-		return 0;
 
 	if (!npt->nfattr_to_range) {
 		ip_nat_proto_put(npt);
@@ -957,8 +959,6 @@ ctnetlink_change_protoinfo(struct ip_con
 	nfattr_parse_nested(tb, CTA_PROTOINFO_MAX, attr);
 
 	proto = ip_conntrack_proto_find_get(npt);
-	if (!proto)
-		return -EINVAL;
 
 	if (proto->from_nfattr)
 		err = proto->from_nfattr(tb, ct);

Re: kernel BUG at kernel/sched.c:2833!

[Pablo Neira]

Pablo Neira wrote:
> @@ -527,10 +531,10 @@ ctnetlink_parse_tuple_proto(struct nfatt
>  
>  	proto = ip_conntrack_proto_find_get(tuple->dst.protonum);
>  
> -	if (likely(proto && proto->nfattr_to_tuple)) {
> +	if (likely(proto && proto->nfattr_to_tuple))
>  		ret = proto->nfattr_to_tuple(tb, tuple);

Still not good. Forgot to remove that proto != NULL checking. It's too
late. I'll resend a patch tomorrow in the morning.

-- 
Pablo

Re: kernel BUG at kernel/sched.c:2833!

[Harald Welte]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, Nov 18, 2005 at 04:19:01AM +0100, Pablo Neira wrote:
> Harald Welte wrote:
> > On Thu, Nov 17, 2005 at 01:36:35AM +0100, Pablo Neira wrote:
> > 
> >>[NETFILTER] Fix ip_conntrack_proto_find_get badness
> > 
> > thanks, applied.
> 
> Sorry Harald, that patch is incomplete :(. Please, revert it. Attached
> the final version.

ok, thanks.

> BTW, you have two trees at people.netfilter.org, which one should I use?

netfilter-2.6.14 is a convenience symlink to netfilter-2.6, so it
doesn't really matter ;)

- -- 
- - Harald Welte <laforge@netfilter.org>                 http://netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDfZLlXaXGVTD0i/8RArhAAJ9ctfHsQrmU/GR8SqC432WOY4LWUACfWnxt
HEKRrHwG8HftuofIc5B0HoQ=
=hr/D
-----END PGP SIGNATURE-----

Re: kernel BUG at kernel/sched.c:2833!

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 559 bytes --]



On Fri, 18 Nov 2005, Pablo Neira wrote:

> Pablo Neira wrote:
>> @@ -527,10 +531,10 @@ ctnetlink_parse_tuple_proto(struct nfatt
>>
>>  	proto = ip_conntrack_proto_find_get(tuple->dst.protonum);
>>
>> -	if (likely(proto && proto->nfattr_to_tuple)) {
>> +	if (likely(proto && proto->nfattr_to_tuple))
>>  		ret = proto->nfattr_to_tuple(tb, tuple);
>
> Still not good. Forgot to remove that proto != NULL checking. It's too
> late. I'll resend a patch tomorrow in the morning.

I'm still waiting... ;)

Best regards,

 				Krzysztof Olędzki

Re: kernel BUG at kernel/sched.c:2833!

[Pablo Neira]

Hi,

Krzysztof Oledzki wrote:
>> I realised that Yasuyuki's patch is incomplete. Could you give a try to
>> the patch attached. Thanks.
> 
> Didn't help:
> 
>  ------------[ cut here ]------------
> kernel BUG at kernel/sched.c:2833!
> invalid operand: 0000 [#55]
> PREEMPT
> Modules linked in:

Did you install any extra module from pom-ng?

> CPU:    0
> EIP:    0060:[<c01166a5>]    Not tainted VLI
> EFLAGS: 00010206   (2.6.14.2)
> EIP is at sub_preempt_count+0x35/0x40
> 
> Need about 3 seconds to trigger this bug with:
> 
> # while true; do conntrack -L|wc -l; done

I can't reproduce that problem that you're reporting. I don't see how
the ctnetlink code can trigger that at the moment :(

The only bugfix that was in my pending queue is:
[PATCH] relax ip_conntrack_[nat_]proto_find_get checkings

-- 
Pablo

Re: kernel BUG at kernel/sched.c:2833!

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1044 bytes --]



On Mon, 21 Nov 2005, Pablo Neira wrote:

> Hi,
>
> Krzysztof Oledzki wrote:
>>> I realised that Yasuyuki's patch is incomplete. Could you give a try to
>>> the patch attached. Thanks.
>>
>> Didn't help:
>>
>>  ------------[ cut here ]------------
>> kernel BUG at kernel/sched.c:2833!
>> invalid operand: 0000 [#55]
>> PREEMPT
>> Modules linked in:
>
> Did you install any extra module from pom-ng?
>
>> CPU:    0
>> EIP:    0060:[<c01166a5>]    Not tainted VLI
>> EFLAGS: 00010206   (2.6.14.2)
>> EIP is at sub_preempt_count+0x35/0x40
>>
>> Need about 3 seconds to trigger this bug with:
>>
>> # while true; do conntrack -L|wc -l; done
>
> I can't reproduce that problem that you're reporting. I don't see how
> the ctnetlink code can trigger that at the moment :(
What information should I provide to help tracking this bug?

> The only bugfix that was in my pending queue is:
> [PATCH] relax ip_conntrack_[nat_]proto_find_get checkings
OK. I'll test this patch.

Best regards,

 				Krzysztof Olędzki

Re: kernel BUG at kernel/sched.c:2833!

[Pablo Neira]

Krzysztof Oledzki wrote:
>> I can't reproduce that problem that you're reporting. I don't see how
>> the ctnetlink code can trigger that at the moment :(
> 
> What information should I provide to help tracking this bug?

Any extra patch applied to your vanilla kernel? architecture? whatever
that can give me a clue on what's wrong. Send me such info in private if
you want.

>> The only bugfix that was in my pending queue is:
>> [PATCH] relax ip_conntrack_[nat_]proto_find_get checkings
> 
> OK. I'll test this patch.

Fine.

-- 
Pablo

Re: kernel BUG at kernel/sched.c:2833!

[Patrick McHardy]

Pablo Neira wrote:
> Krzysztof Oledzki wrote:
> 
>>>I can't reproduce that problem that you're reporting. I don't see how
>>>the ctnetlink code can trigger that at the moment :(
>>
>>What information should I provide to help tracking this bug?
> 
> Any extra patch applied to your vanilla kernel? architecture? whatever
> that can give me a clue on what's wrong. Send me such info in private if
> you want.

Please don't send information related to the bug in private if not
necessary, the initial bug report is public so other people might
be looking into it or notice that they are affected by the same
problem.

Re: kernel BUG at kernel/sched.c:2833!

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3573 bytes --]



On Mon, 21 Nov 2005, Pablo Neira wrote:

> Krzysztof Oledzki wrote:
>>> I can't reproduce that problem that you're reporting. I don't see how
>>> the ctnetlink code can trigger that at the moment :(
>>
>> What information should I provide to help tracking this bug?
>
> Any extra patch applied to your vanilla kernel?

From pom-ng:
  ROUTE TARPIT TTL iprange ipv4options policy comment time set u32 random unclean

From http://www.ipp2p.org: (pom-ng contains much older version):
  ipp2p-0.8.0

And my set of patches (patch-ole-2.6.14-o5.gz):
  + 0020: iptables-PreroutingFilter-2611
  + 0030: IPv6-OptionalSIT-26
  + 0050: VGACanDo64KB-Option-26
  + 0060: 3c59x-ShowEthID-26
  + 0061: tulip-8021q-2.6.13.2-szpajder
  + 0070: 2.6.14-libata-passthru
  + 0100: netfilter-ctnl-refcnt
  + 0101: proto_find
  + 0110: bonding-vlan
  + 0120: workaround-for-pnp-device-interrupt

You can find it here:
  ftp://ftp.ans.pl/pub/patches/patch-ole-2.6.14-broken-out/

AFAIK there is nothing special here.

> architecture?
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 8
model name      : AMD Athlon(tm) XP 2400+
stepping        : 1
cpu MHz         : 1994.966
cache size      : 256 KB

It is:
Linux version 2.6.14.2 (root@gate) (gcc version 3.3.5) #1 PREEMPT Mon Nov 21 22:02:40 CET 2005

Do you need my config?

> whatever that can give me a clue on what's wrong.
OK. Thank you for helping me.

>>> The only bugfix that was in my pending queue is:
>>> [PATCH] relax ip_conntrack_[nat_]proto_find_get checkings
>>
>> OK. I'll test this patch.
>
> Fine.

Tested. Still wrong:

------------[ cut here ]------------
kernel BUG at kernel/sched.c:2833!
invalid operand: 0000 [#1]
PREEMPT
Modules linked in:
CPU:    0
EIP:    0060:[sub_preempt_count+53/64]    Not tainted VLI
EFLAGS: 00010206   (2.6.14.2)
EIP is at sub_preempt_count+0x35/0x40
eax: dd3b2000   ebx: dd2c8f4c   ecx: 00000000   edx: 000000ff
esi: dd2c8e90   edi: 00000418   ebp: dd3b2c9c   esp: dd3b2c9c
ds: 007b   es: 007b   ss: 0068
Process conntrack (pid: 372, threadinfo=dd3b2000 task=dd3b15c0)
Stack: dedac8b8 c011f1f9 dd2c8f4c dd2c8e90 00000418 dedac8b8 c0356565 c155e6c0
        00000174 438236f7 00000000 00000001 dd2c8e90 c155e6c0 c155e6c0 dedac8a0
        dd4e0000 c03119a8 c155e6c0 dedac8a0 00000000 000000d0 dd4e0000 dd3b2d48
Call Trace:
  [local_bh_enable+25/144] local_bh_enable+0x19/0x90
  [ctnetlink_dump_table+149/272] ctnetlink_dump_table+0x95/0x110
  [netlink_dump+88/512] netlink_dump+0x58/0x200
  [netlink_recvmsg+558/576] netlink_recvmsg+0x22e/0x240
  [sock_recvmsg+254/288] sock_recvmsg+0xfe/0x120
  [update_atime+149/176] update_atime+0x95/0xb0
  [do_generic_mapping_read+789/1520] do_generic_mapping_read+0x315/0x5f0
  [autoremove_wake_function+0/96] autoremove_wake_function+0x0/0x60
  [sys_recvmsg+323/512] sys_recvmsg+0x143/0x200
  [current_fs_time+81/112] current_fs_time+0x51/0x70
  [inode_update_time+82/224] inode_update_time+0x52/0xe0
  [pipe_writev+742/1248] pipe_writev+0x2e6/0x4e0
  [pipe_write+55/64] pipe_write+0x37/0x40
  [copy_from_user+70/144] copy_from_user+0x46/0x90
  [sys_socketcall+591/608] sys_socketcall+0x24f/0x260
  [do_IRQ+89/128] do_IRQ+0x59/0x80
  [syscall_call+7/11] syscall_call+0x7/0xb
Code: 89 e5 3b 50 14 7f 24 81 fa fe 00 00 00 76 0c b8 00 f0 ff ff 21 e0 29 50 14 c9 c3 80 78 14 00 75 ee 0f 0b 15 0b 97 8e 3b c0 eb e4 <0f> 0b 11 0b 97 8e 3b c0 eb d2 90 55 89 e5 8b 45 08 8b 50 04 89


Best regards,


 				Krzysztof Olędzki

Re: kernel BUG at kernel/sched.c:2833!

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 966 bytes --]



On Mon, 21 Nov 2005, Pablo Neira wrote:

> Hi,
>
> Krzysztof Oledzki wrote:
>>> I realised that Yasuyuki's patch is incomplete. Could you give a try to
>>> the patch attached. Thanks.
>>
>> Didn't help:
>>
>>  ------------[ cut here ]------------
>> kernel BUG at kernel/sched.c:2833!
>> invalid operand: 0000 [#55]
>> PREEMPT
>> Modules linked in:
>
> Did you install any extra module from pom-ng?
>
>> CPU:    0
>> EIP:    0060:[<c01166a5>]    Not tainted VLI
>> EFLAGS: 00010206   (2.6.14.2)
>> EIP is at sub_preempt_count+0x35/0x40
>>
>> Need about 3 seconds to trigger this bug with:
>>
>> # while true; do conntrack -L|wc -l; done
>
> I can't reproduce that problem that you're reporting.
Ah. One idea: maybe to reproduce this problem you need many active 
conntracks with differenet protocols (tcp/udp/icmp/esp...)? This router 
handles 5K-10K connections, SNAT, DNAT, IPSec, etc...

Best regards,

 				Krzysztof Olędzki

Re: kernel BUG at kernel/sched.c:2833!

[Pablo Neira Ayuso]

Hi,

Krzysztof Oledzki wrote:
> On Mon, 21 Nov 2005, Pablo Neira wrote:
> 
>> Krzysztof Oledzki wrote:
>>
>>>> I can't reproduce that problem that you're reporting. I don't see how
>>>> the ctnetlink code can trigger that at the moment :(
>>>
>>> What information should I provide to help tracking this bug?
>>
>> Any extra patch applied to your vanilla kernel?
> 
> From pom-ng:
>  ROUTE TARPIT TTL iprange ipv4options policy comment time set u32 random
> unclean

A lot of patches from pom-ng. I tried with a vanilla 2.6.14.2 and I'm
not able to reproduce the problem, always with preemption and SMP
enabled but nothing.

Could you try to reproduce it without the extra patches? That will help
to know if any of those are spotting that error.

BTW, iprange is already in mainline.

-- 
Pablo

Re: kernel BUG at kernel/sched.c:2833!

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1039 bytes --]



On Fri, 25 Nov 2005, Pablo Neira Ayuso wrote:

> Hi,
>
> Krzysztof Oledzki wrote:
>> On Mon, 21 Nov 2005, Pablo Neira wrote:
>>
>>> Krzysztof Oledzki wrote:
>>>
>>>>> I can't reproduce that problem that you're reporting. I don't see how
>>>>> the ctnetlink code can trigger that at the moment :(
>>>>
>>>> What information should I provide to help tracking this bug?
>>>
>>> Any extra patch applied to your vanilla kernel?
>>
>> From pom-ng:
>>  ROUTE TARPIT TTL iprange ipv4options policy comment time set u32 random
>> unclean
>
> A lot of patches from pom-ng. I tried with a vanilla 2.6.14.2 and I'm
> not able to reproduce the problem, always with preemption and SMP
> enabled but nothing.
OK.

> Could you try to reproduce it without the extra patches? That will help
> to know if any of those are spotting that error.
This will take some time because I really need most of them. Will do my 
best. ;)

> BTW, iprange is already in mainline.
OK. Thank you ;)

Best regards,

 			Krzysztof Olędzki