Kernel Panic removing psnap or p8022 module in 2.4.28

Kernel Panic removing psnap or p8022 module in 2.4.28

[Tom Dickson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've attached the config file used to compile the vanilla 2.4.28 kernel
that produces the following panic:

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBt1u/2dxAfYNwANIRAnv/AJ4ugdX1btB8jpV42hzDzRSBb7IHFwCdEFBk
ZJBG04V2+KKCW16/CDFUKis=
=92+9
-----END PGP SIGNATURE-----

Re: Kernel Panic removing psnap or p8022 module in 2.4.28 (appletalk dependencies)

[Tom Dickson]

[-- Attachment #1: Type: text/plain, Size: 2866 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Next time I promise to finish the report before hitting send.

I'm using a Gentoo Linux system with the vanilla 2.4.28 sources installed. The
attached config file is what I'm using.

appletalk.o now has a dependency it didn't have in 2.4.26, which are the modules
psnap.o and p8022.o, which I can't find how to disable.

If I start netatalk, it modprobes appletalk.o, which loads psnap.o and p8022.o. If
I then stop appletalk, it removes appletalk.o (which works) and then psnap.o and
p8022.o (which kernel panics).

I can also get the kernel panic by modprobing p8022.o and then rmmodding it.

The kernel panic is as follows:

Oops: 0000
CPU:    0
EIP:    0010:[<c0238490>]    Not tainted
EFLAGS: 00010286
eax: 00000004   ebx: cc84a3c0   ecx: 41b75c48   edx: c1363b28
esi: cb6d65e0   edi: 00000400   ebp: 00013b56   esp: c030bf18
ds: 0018   es: 0018   ss: 0018
Process swapper (pid: 0. stackpage=c030b000)
Stack: c030bfa0 0000e401 cc841cfd cbcbb400 00000000 00000040 c0238623 cb6d65e0
~       c0324d6c c0324ca8 00013b56 00000046 c0238734 c0324ca8 c030bf54 0000012c
~       00000001 c0324b70 fffffffb c011c775 c0324b70 c03249a0 00000005 c134ec60
Call Trace:    [<cc841cfd>] [<c0238623>] [<c0238734>] [<c011c775>] [<c010a5fa>]
~  [<c0106f30>] [<c010c978>] [<c0106f30>] [<c0106f53>] [<c0106fe2>] [<c0105000>]

Code: 66 39 3b 74 60 8b 5b 10 85 db 75 f4 85 d2 74 36 8b 7a 0c 85
~ <0>Kernel panic: Aiee, killing interrupt hander!
In interrupt handler - not syncing

This is a problem because we can't shutdown the machine without this occurring. It
seems that if we disable ipchains in the kernel, it goes away. It also occurs with
the 2.4.27 kernel.

A change we see is in the linux-2.4.28/net/Makefile

mod-subdirs :=  ipv4/netfilter ipv6/netfilter ipx irda bluetooth atm netlink sched
core sctp 802

whereas 2.4.26 has:

mod-subdirs :=  ipv4/netfilter ipv6/netfilter ipx irda bluetooth atm netlink sched
core sctp

This also happens on a number of other machines; mainly RedHat 7.3 machines with
this kernel.

I'm trying to get a ksymoops now, if I get it to work, I'll reply to this post.

You can see an image of the kernel panic (if such things excite you) at:

http://schnecke.bombcar.com/random/kernelpanic.jpg

Note that there is a short delay between removing the module and the panic. You
can also get the panic by removing appletalk, (which works) and then removing
psnap and waiting a few seconds.

Thank you!

- -Tom Dickson

Tom Dickson wrote:
| I've attached the config file used to compile the vanilla 2.4.28 kernel
| that produces the following panic:
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBt2Ew2dxAfYNwANIRApVUAJ9IfJH7PJZLMsS8kidCD4myMyuVrgCdH2HP
BFEJHz1RlLf9k3hAbBX1FLk=
=7kKd
-----END PGP SIGNATURE-----

[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 5473 bytes --]

Re: Kernel Panic removing psnap or p8022 module in 2.4.28 (appletalk dependencies)

[Marcelo Tosatti]

On Wed, Dec 08, 2004 at 12:16:48PM -0800, Tom Dickson wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Next time I promise to finish the report before hitting send.
> 
> I'm using a Gentoo Linux system with the vanilla 2.4.28 sources installed. 
> The
> attached config file is what I'm using.
> 
> appletalk.o now has a dependency it didn't have in 2.4.26, which are the 
> modules
> psnap.o and p8022.o, which I can't find how to disable.

appletalk.o requires psnap.o and p8022.o, before 2.4.27 they were always builtin 
the kernel if appletalk was selected. 

So the change to have them as modules just uncovers the bug.

> If I start netatalk, it modprobes appletalk.o, which loads psnap.o and 
> p8022.o. If
> I then stop appletalk, it removes appletalk.o (which works) and then 
> psnap.o and
> p8022.o (which kernel panics).
> 
> I can also get the kernel panic by modprobing p8022.o and then rmmodding it.
> 
> The kernel panic is as follows:
> 
> Oops: 0000
> CPU:    0
> EIP:    0010:[<c0238490>]    Not tainted
> EFLAGS: 00010286
> eax: 00000004   ebx: cc84a3c0   ecx: 41b75c48   edx: c1363b28
> esi: cb6d65e0   edi: 00000400   ebp: 00013b56   esp: c030bf18
> ds: 0018   es: 0018   ss: 0018
> Process swapper (pid: 0. stackpage=c030b000)
> Stack: c030bfa0 0000e401 cc841cfd cbcbb400 00000000 00000040 c0238623 
> cb6d65e0
> ~       c0324d6c c0324ca8 00013b56 00000046 c0238734 c0324ca8 c030bf54 
> 0000012c
> ~       00000001 c0324b70 fffffffb c011c775 c0324b70 c03249a0 00000005 
> c134ec60
> Call Trace:    [<cc841cfd>] [<c0238623>] [<c0238734>] [<c011c775>] 
> [<c010a5fa>]
> ~  [<c0106f30>] [<c010c978>] [<c0106f30>] [<c0106f53>] [<c0106fe2>] 
> [<c0105000>]
> 
> Code: 66 39 3b 74 60 8b 5b 10 85 db 75 f4 85 d2 74 36 8b 7a 0c 85
> ~ <0>Kernel panic: Aiee, killing interrupt hander!
> In interrupt handler - not syncing
> 
> This is a problem because we can't shutdown the machine without this 
> occurring. It
> seems that if we disable ipchains in the kernel, it goes away. It also 
> occurs with
> the 2.4.27 kernel.

The problem is that p8022 does not unregister the packet type it has registered 
on startup, so once a p8022 packet is received netif_receive_skb() tries to access
an address from the module which is now unloaded, boom.

Can you please try the following untested patch 


--- a/net/802/p8022.c.orig	2004-12-08 14:19:38.000000000 -0200
+++ b/net/802/p8022.c	2004-12-08 14:21:40.000000000 -0200
@@ -97,7 +97,14 @@
 	return 0;
 }
 
+static void __exit p8022_exit(void)
+{
+	dev_remove_pack(&p8022_packet_type);
+	return;
+}
+
 module_init(p8022_init);
+module_exit(p8022_exit);
 
 struct datalink_proto *register_8022_client(unsigned char type, int (*rcvfunc)(struct sk_buff *, struct net_device *, struct packet_type *))
 {

Re: Kernel Panic removing psnap or p8022 module in 2.4.28 (appletalk dependencies)

[Tom Dickson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thank you for your quick response! We've applied this patch to three of our
systems, and it seems to be working. We're going to run multiple reboot tests
tonight, and will report on the results tomorrow.

I will also attempt to see if psnap rmmoding can cause a similar instance.

Marcelo Tosatti wrote:
| On Wed, Dec 08, 2004 at 12:16:48PM -0800, Tom Dickson wrote:
|
|>-----BEGIN PGP SIGNED MESSAGE-----
|>Hash: SHA1
|>
|>Next time I promise to finish the report before hitting send.
|>
|>I'm using a Gentoo Linux system with the vanilla 2.4.28 sources installed.
|>The
|>attached config file is what I'm using.
|>
|>appletalk.o now has a dependency it didn't have in 2.4.26, which are the
|>modules
|>psnap.o and p8022.o, which I can't find how to disable.
|
|
| appletalk.o requires psnap.o and p8022.o, before 2.4.27 they were always builtin
| the kernel if appletalk was selected.
|
| So the change to have them as modules just uncovers the bug.
|
|
|>If I start netatalk, it modprobes appletalk.o, which loads psnap.o and
|>p8022.o. If
|>I then stop appletalk, it removes appletalk.o (which works) and then
|>psnap.o and
|>p8022.o (which kernel panics).
|>
|>I can also get the kernel panic by modprobing p8022.o and then rmmodding it.
|>
|>The kernel panic is as follows:
|>
|>Oops: 0000
|>CPU:    0
|>EIP:    0010:[<c0238490>]    Not tainted
|>EFLAGS: 00010286
|>eax: 00000004   ebx: cc84a3c0   ecx: 41b75c48   edx: c1363b28
|>esi: cb6d65e0   edi: 00000400   ebp: 00013b56   esp: c030bf18
|>ds: 0018   es: 0018   ss: 0018
|>Process swapper (pid: 0. stackpage=c030b000)
|>Stack: c030bfa0 0000e401 cc841cfd cbcbb400 00000000 00000040 c0238623
|>cb6d65e0
|>~       c0324d6c c0324ca8 00013b56 00000046 c0238734 c0324ca8 c030bf54
|>0000012c
|>~       00000001 c0324b70 fffffffb c011c775 c0324b70 c03249a0 00000005
|>c134ec60
|>Call Trace:    [<cc841cfd>] [<c0238623>] [<c0238734>] [<c011c775>]
|>[<c010a5fa>]
|>~  [<c0106f30>] [<c010c978>] [<c0106f30>] [<c0106f53>] [<c0106fe2>]
|>[<c0105000>]
|>
|>Code: 66 39 3b 74 60 8b 5b 10 85 db 75 f4 85 d2 74 36 8b 7a 0c 85
|>~ <0>Kernel panic: Aiee, killing interrupt hander!
|>In interrupt handler - not syncing
|>
|>This is a problem because we can't shutdown the machine without this
|>occurring. It
|>seems that if we disable ipchains in the kernel, it goes away. It also
|>occurs with
|>the 2.4.27 kernel.
|
|
| The problem is that p8022 does not unregister the packet type it has registered
| on startup, so once a p8022 packet is received netif_receive_skb() tries to access
| an address from the module which is now unloaded, boom.
|
| Can you please try the following untested patch
|
|
| --- a/net/802/p8022.c.orig	2004-12-08 14:19:38.000000000 -0200
| +++ b/net/802/p8022.c	2004-12-08 14:21:40.000000000 -0200
| @@ -97,7 +97,14 @@
|  	return 0;
|  }
|
| +static void __exit p8022_exit(void)
| +{
| +	dev_remove_pack(&p8022_packet_type);
| +	return;
| +}
| +
|  module_init(p8022_init);
| +module_exit(p8022_exit);
|
|  struct datalink_proto *register_8022_client(unsigned char type, int
(*rcvfunc)(struct sk_buff *, struct net_device *, struct packet_type *))
|  {
|
| .
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBt53D2dxAfYNwANIRAn5XAJ9b2R0/rfWpDG/GGAv4bhGyAWMCowCfSPie
0yE1PKPGToG6VgFMx7veMS4=
=1WKd
-----END PGP SIGNATURE-----

Re: Kernel Panic removing psnap or p8022 module in 2.4.28 (appletalk dependencies)

[Tom Dickson]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Here is the ksymoops. The first one (ksymoops.txt) is the ksymoops run without the
modules loaded.

The second one (ksymoops2.txt) is with appletalk.o psnap.o and p8022.o loaded.
Only one line is different:

gentoogato root # diff ksymoops.txt ksymoops2.txt
34c34
< >>ebx; cc84a3c0 <.data.end+1b41/????>
- ---
| >>ebx; cc84a3c0 <[p8022]p8022_packet_type+0/0>

I hope this is helpful. Let me know if there is anything more I can run.

Tom Dickson wrote:
| Next time I promise to finish the report before hitting send.

SNIP

| Thank you!
|
| -Tom Dickson
|
| Tom Dickson wrote:
| | I've attached the config file used to compile the vanilla 2.4.28 kernel
| | that produces the following panic:
| |
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD4DBQFBt2nD2dxAfYNwANIRAtpzAJ9HNC0iCJoDs4iNNueDPgyIIOyj0QCYppIj
hQIAFgjqVePU65hBPEAlqg==
=Ci8h
-----END PGP SIGNATURE-----

[-- Attachment #2: ksymoops.gz --]
[-- Type: application/x-gzip, Size: 1481 bytes --]