Kernel panic

Kernel panic

[Allshouse, Brian M (Sabre)]

I'm having problems with kernel panics. I set up my firewall with the
latest and greatest version of iptables and the latest stable kernel. I'm
running slackware 8.1, and ever since I put it on the network for testing I
get kernel panics that crash the machine I tried the previous kernel
version (2.4.18) and also tried the latest patches for Iptables. I posted 
this problem here a few weeks ago and someone suggested using ksymoops to 
track down what is causing the kernel panic. I finally have been able to do
that, but not being a kernel programmer I don't know what to make of it. I'm
pasting the output from ksymoops below. If anyone knows what to make of this
or where I can find out what this means I would greatly appreciate it. I'm
on 
the verge of missing my deadline to have this box in place. Thanks.

<-------------------------Start ksymoops
output----------------------------------------->

ksymoops 2.4.9 on i586 2.4.20.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.20/ (default)
     -m /usr/src/linux/System.map (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c0272a86>]   Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010202
eax:  00000080  ebx:  00000000  ecx:  7354e5cc  edx:  000007ff
esi:  0000008c  edi:  ffffffea  ebp:  c0351e5c  esp:  c0351e0c
ds:  0018  es:  0018 ss:  0018
Process swapper (pid: 0, stackpage=c0351000)
Stack: 00000000 00000002 5750e58c 00000000 00000000 000004e8 00000000
00000000
       cb8fb990 cb8fb990 00000000 7d56e58c 00000000 fd010015 cb919920
5750e58c
       7354e48c 00000002 00000000 00000000 7354e58c c0272fbc cbe1cee0
5750e58c
Call Trace:   [<c0272fbc>] [<c0290c4b>] [<c026cf4a>] [<c0290f8f>]
[<c029aa8f>]
 [<c026745b>] [<c02674ee>] [<c0267604>] [<c0119c5a>] [<c0109aae>]
[<c0106d00>]
 [<c010bf38>] [<c0106d00>] [<c0106d23>] [<c0106d87>] [<c0105000>]
[<c0105027>]
Code: 03 00 83 f8 7f 0f 84 f0 03 00 00 8d 7d ec 8d 5d e4 a1 bc


>>EIP; c0272a86 <ip_route_input_slow+12e/558>   <=====

>>ebp; c0351e5c <init_task_union+1e5c/2000>
>>esp; c0351e0c <init_task_union+1e0c/2000>

Trace; c0272fbc <ip_route_input+10c/114>
Trace; c0290c4b <arp_process+1a3/44c>
Trace; c026cf4a <nf_hook_slow+132/188>
Trace; c0290f8f <arp_rcv+9b/c4>
Trace; c029aa8f <unix_dgram_sendmsg+35f/364>
Trace; c026745b <netif_receive_skb+14f/178>
Trace; c02674ee <process_backlog+6a/110>
Trace; c0267604 <net_rx_action+70/114>
Trace; c0119c5a <do_softirq+5a/a4>
Trace; c0109aae <do_IRQ+96/a8>
Trace; c0106d00 <default_idle+0/28>
Trace; c010bf38 <call_do_IRQ+5/d>
Trace; c0106d00 <default_idle+0/28>
Trace; c0106d23 <default_idle+23/28>
Trace; c0106d87 <cpu_idle+3f/54>
Trace; c0105000 <_stext+0/0>
Trace; c0105027 <rest_init+27/28>

Code;  c0272a86 <ip_route_input_slow+12e/558>
00000000 <_EIP>:
Code;  c0272a86 <ip_route_input_slow+12e/558>   <=====
   0:   03 00                     add    (%eax),%eax   <=====
Code;  c0272a88 <ip_route_input_slow+130/558>
   2:   83 f8 7f                  cmp    $0x7f,%eax
Code;  c0272a8b <ip_route_input_slow+133/558>
   5:   0f 84 f0 03 00 00         je     3fb <_EIP+0x3fb> c0272e81
<ip_route_input_slow+529/558>
Code;  c0272a91 <ip_route_input_slow+139/558>
   b:   8d 7d ec                  lea    0xffffffec(%ebp),%edi
Code;  c0272a94 <ip_route_input_slow+13c/558>
   e:   8d 5d e4                  lea    0xffffffe4(%ebp),%ebx
Code;  c0272a97 <ip_route_input_slow+13f/558>
  11:   a1 bc 00 00 00            mov    0xbc,%eax

 <0>Kernel panic: Aiee, killing interrupt handler!

1 warning issued.  Results may not be reliable.

<----------------------------------End ksymoops
output------------------------------------------->



Sincerely,

Brian Allshouse
UNIX Systems Administrator
Sabre Systems Inc.
mailto:allshousebm@navair.navy.mil
(301) 342-7034

kernel panic

[Samuel Díaz García]

Iptables 1.3.5
kernel 2.6.15.6

Any help?

Mar 30 00:31:01 fraile kernel: [17181150.312000] ip_nat_pptp version 3.0 
unloaded
Mar 30 00:31:01 fraile kernel: [17181150.492000] ctnetlink: 
unregistering from nfnetlink.
Mar 30 00:31:01 fraile kernel: [17181150.524000] ip_conntrack_pptp 
version 3.1 unloaded
Mar 30 00:31:01 fraile kernel: [17181150.688000] ctnetlink v0.90: 
registering with nfnetlink.
Mar 30 00:31:01 fraile kernel: [17181150.688000] Unable to handle kernel 
paging request at virtual address e0c76e54
Mar 30 00:31:01 fraile kernel: [17181150.688000]  printing eip:
Mar 30 00:31:01 fraile kernel: [17181150.688000] c012a309
Mar 30 00:31:01 fraile kernel: [17181150.688000] *pde = 1c9ac067
Mar 30 00:31:01 fraile kernel: [17181150.688000] Oops: 0000 [#1]
Mar 30 00:31:01 fraile kernel: [17181150.688000] Modules linked in: 
ip_conntrack_netlink ip_conntrack_netbios_ns ip_conntrack_irc 
ip_conntrack_h323 ip_conntrack_ftp ip_conntrack_amanda ipt_ipp2p 
parport_pc lp parport ipt_mac ipt_connlimit iptable_filter 
ipt_MASQUERADE iptable_nat ip_nat ipt_layer7 ipt_state ipt_MARK ipt_mark 
ipt_CONNMARK ip_conntrack nfnetlink ipt_comment iptable_mangle ip_tables 
police autofs4 it87 hwmon_vid hwmon eeprom i2c_isa bluetooth sunrpc 
dm_mod video thermal processor fan container button battery ac ohci_hcd 
shpchp i2c_sis630 i2c_sis96x i2c_core snd_intel8x0 snd_ac97_codec 
snd_ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq 
snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd soundcore 
snd_page_alloc sis900 8139too 3c59x mii floppy ext3 jbd aacraid sd_mod 
scsi_mod
Mar 30 00:31:01 fraile kernel: [17181150.688000] CPU:    0
Mar 30 00:31:01 fraile kernel: [17181150.688000] EIP: 
0060:[<c012a309>]    Tainted: GF     VLI
Mar 30 00:31:01 fraile kernel: [17181150.688000] EFLAGS: 00010282 
(2.6.15.6-ArcosCom)
Mar 30 00:31:01 fraile kernel: [17181150.688000] EIP is at 
notifier_chain_register+0x19/0x50
Mar 30 00:31:01 fraile kernel: [17181150.688000] eax: e0c76e4c   ebx: 
e0c114c0   ecx: ffffffff   edx: 00000000
Mar 30 00:31:01 fraile kernel: [17181150.688000] esi: e0c40e4c   edi: 
0805e218   ebp: d174e000   esp: d174ff8c
Mar 30 00:31:02 fraile kernel: [17181150.688000] ds: 007b   es: 007b 
ss: 0068
Mar 30 00:31:02 fraile kernel: [17181150.688000] Process modprobe (pid: 
4432, threadinfo=d174e000 task=d363f570)
Mar 30 00:31:02 fraile kernel: [17181150.688000] Stack: 00000000 
0805e1f8 e0c2a05c e0c3f4c4 e0c2a0db e0c40f00 c0138a87 b7e9c008
Mar 30 00:31:02 fraile kernel: [17181150.688000]        0805e1f8 
b7e9c008 0805e1f8 c0103055 b7e9c008 00034380 0805e1f8 0805e1f8
Mar 30 00:31:02 fraile kernel: [17181150.688000]        0805e218 
bfd130b8 ffffffda 0000007b 0000007b 00000080 ffffe410 00000073
Mar 30 00:31:02 fraile kernel: [17181150.688000] Call Trace:
Mar 30 00:31:02 fraile kernel: [17181150.688000]  [<e0c2a05c>] 
ctnetlink_init+0x5c/0xdb [ip_conntrack_netlink]
Mar 30 00:31:02 fraile kernel: [17181150.688000]  [<c0138a87>] 
sys_init_module+0xc7/0x1d0
Mar 30 00:31:02 fraile kernel: [17181150.688000]  [<c0103055>] 
syscall_call+0x7/0xb
Mar 30 00:31:03 fraile kernel: [17181150.688000] Code: 00 e8 5c 54 24 00 
b8 fe fd ff ff c3 90 90 90 90 90 90 56 89 d6 53 89 c3 b8 18 3c 3d c0 e8 
70 6b 24 00 8b 03 85 c0 74 1a 8b 56 08 <3b> 50 08 7f 12 89 f6 8d 58 04 
8b 40 04 85 c0 74 06 39 50 08 7d
Mar 30 00:31:03 fraile kernel: [17181150.688000]  ip_conntrack_pptp 
version 3.1 loaded
Mar 30 00:31:03 fraile kernel: [17181150.840000] ip_nat_pptp version 3.0 
loaded
Mar 30 00:31:30 fraile kernel: [17181179.176000] ip_nat_pptp version 3.0 
unloaded
Mar 30 00:31:30 fraile kernel: [17181179.388000] ip_conntrack_pptp 
version 3.1 unloaded
Mar 30 00:31:30 fraile kernel: [17181179.556000] ip_conntrack_pptp 
version 3.1 loaded


-- 
    Samuel Díaz García

Re: kernel panic

[Samuel Díaz García]

Any idea at least?

Samuel Díaz García escribió:
> Iptables 1.3.5
> kernel 2.6.15.6
> 
> Any help?
> 
> Mar 30 00:31:01 fraile kernel: [17181150.312000] ip_nat_pptp version 3.0 
> unloaded
> Mar 30 00:31:01 fraile kernel: [17181150.492000] ctnetlink: 
> unregistering from nfnetlink.
> Mar 30 00:31:01 fraile kernel: [17181150.524000] ip_conntrack_pptp 
> version 3.1 unloaded
> Mar 30 00:31:01 fraile kernel: [17181150.688000] ctnetlink v0.90: 
> registering with nfnetlink.
> Mar 30 00:31:01 fraile kernel: [17181150.688000] Unable to handle kernel 
> paging request at virtual address e0c76e54
> Mar 30 00:31:01 fraile kernel: [17181150.688000]  printing eip:
> Mar 30 00:31:01 fraile kernel: [17181150.688000] c012a309
> Mar 30 00:31:01 fraile kernel: [17181150.688000] *pde = 1c9ac067
> Mar 30 00:31:01 fraile kernel: [17181150.688000] Oops: 0000 [#1]
> Mar 30 00:31:01 fraile kernel: [17181150.688000] Modules linked in: 
> ip_conntrack_netlink ip_conntrack_netbios_ns ip_conntrack_irc 
> ip_conntrack_h323 ip_conntrack_ftp ip_conntrack_amanda ipt_ipp2p 
> parport_pc lp parport ipt_mac ipt_connlimit iptable_filter 
> ipt_MASQUERADE iptable_nat ip_nat ipt_layer7 ipt_state ipt_MARK ipt_mark 
> ipt_CONNMARK ip_conntrack nfnetlink ipt_comment iptable_mangle ip_tables 
> police autofs4 it87 hwmon_vid hwmon eeprom i2c_isa bluetooth sunrpc 
> dm_mod video thermal processor fan container button battery ac ohci_hcd 
> shpchp i2c_sis630 i2c_sis96x i2c_core snd_intel8x0 snd_ac97_codec 
> snd_ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq 
> snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd soundcore 
> snd_page_alloc sis900 8139too 3c59x mii floppy ext3 jbd aacraid sd_mod 
> scsi_mod
> Mar 30 00:31:01 fraile kernel: [17181150.688000] CPU:    0
> Mar 30 00:31:01 fraile kernel: [17181150.688000] EIP: 
> 0060:[<c012a309>]    Tainted: GF     VLI
> Mar 30 00:31:01 fraile kernel: [17181150.688000] EFLAGS: 00010282 
> (2.6.15.6-ArcosCom)
> Mar 30 00:31:01 fraile kernel: [17181150.688000] EIP is at 
> notifier_chain_register+0x19/0x50
> Mar 30 00:31:01 fraile kernel: [17181150.688000] eax: e0c76e4c   ebx: 
> e0c114c0   ecx: ffffffff   edx: 00000000
> Mar 30 00:31:01 fraile kernel: [17181150.688000] esi: e0c40e4c   edi: 
> 0805e218   ebp: d174e000   esp: d174ff8c
> Mar 30 00:31:02 fraile kernel: [17181150.688000] ds: 007b   es: 007b ss: 
> 0068
> Mar 30 00:31:02 fraile kernel: [17181150.688000] Process modprobe (pid: 
> 4432, threadinfo=d174e000 task=d363f570)
> Mar 30 00:31:02 fraile kernel: [17181150.688000] Stack: 00000000 
> 0805e1f8 e0c2a05c e0c3f4c4 e0c2a0db e0c40f00 c0138a87 b7e9c008
> Mar 30 00:31:02 fraile kernel: [17181150.688000]        0805e1f8 
> b7e9c008 0805e1f8 c0103055 b7e9c008 00034380 0805e1f8 0805e1f8
> Mar 30 00:31:02 fraile kernel: [17181150.688000]        0805e218 
> bfd130b8 ffffffda 0000007b 0000007b 00000080 ffffe410 00000073
> Mar 30 00:31:02 fraile kernel: [17181150.688000] Call Trace:
> Mar 30 00:31:02 fraile kernel: [17181150.688000]  [<e0c2a05c>] 
> ctnetlink_init+0x5c/0xdb [ip_conntrack_netlink]
> Mar 30 00:31:02 fraile kernel: [17181150.688000]  [<c0138a87>] 
> sys_init_module+0xc7/0x1d0
> Mar 30 00:31:02 fraile kernel: [17181150.688000]  [<c0103055>] 
> syscall_call+0x7/0xb
> Mar 30 00:31:03 fraile kernel: [17181150.688000] Code: 00 e8 5c 54 24 00 
> b8 fe fd ff ff c3 90 90 90 90 90 90 56 89 d6 53 89 c3 b8 18 3c 3d c0 e8 
> 70 6b 24 00 8b 03 85 c0 74 1a 8b 56 08 <3b> 50 08 7f 12 89 f6 8d 58 04 
> 8b 40 04 85 c0 74 06 39 50 08 7d
> Mar 30 00:31:03 fraile kernel: [17181150.688000]  ip_conntrack_pptp 
> version 3.1 loaded
> Mar 30 00:31:03 fraile kernel: [17181150.840000] ip_nat_pptp version 3.0 
> loaded
> Mar 30 00:31:30 fraile kernel: [17181179.176000] ip_nat_pptp version 3.0 
> unloaded
> Mar 30 00:31:30 fraile kernel: [17181179.388000] ip_conntrack_pptp 
> version 3.1 unloaded
> Mar 30 00:31:30 fraile kernel: [17181179.556000] ip_conntrack_pptp 
> version 3.1 loaded
> 
> 

-- 
    Samuel Díaz García

kernel panic

[Miha Verlic]

Hello,

today one of the servers I administer crashed for the third time due to 
kernel panic. Folks at colocation site provided me with the following 
"screenshot": http://miha.krneki.org/netfilter/kernelpanic.jpg

I'm not sure whether this is related with netfilter/iptables, tg3 or 
something else, but the screenshot is full of netfilter messagess, so I 
tried this list first.

Funny thing is that /proc/sys/kernel/panic was set to 60 seconds, yet 
the machine did not reboot itself after the crash and colocation people 
said machine was not responding to keyboard, so they could not provide 
the whole trace.

Another interesting thing - machine is heavily loaded (apache+php+mysql) 
- pushing around 35-45mbit/s througout the whole day. After midnight 
utilization gets very low and yet all 3 crashes occured at around 2AM 
local time. There are no cron scripts at that time that would cause any 
cpu/network hammering, system is virtually idle.

Server is Dual Opteron 250 with 4gb ECC RAM and Adaptec 2015S SCSI 
controller (2x RAID1).

At the time of crash, system was using:
OS: (32bit) slackware-based with:
glibc 2.3.6 (NPTL), gcc 3.3.6,
iptables 1.3.4 (upgraded to 1.3.5 after crash)
kernel 2.6.17.8 with the following patches:
   - grsecurity 2.1.9 (200608012135)
   - dpt_i2o-2.5.0-2426 driver from adaptec
SMP and PAE are enabled, config file is located at:
http://miha.krneki.org/netfilter/config-2.6.17.8+d1

If anyone has any clue, don't hesitate to contact me.

Cheers
--
Miha

kernel panic

[Jacques Rodary]

Hi Miha
I first didn't see your post, and posted independently on the same
subject (with a typo). I have the same message than yours, but not
always, but it always ends with "Fatal exception in interrupt", includes
sometimes "common_interrupt+0x1a/0x20" and sometimes
"ip_nat_fn..[iptables_nat]". And I never had this before I used NAT on a
SMP box (2 dual-core Xeons).
Also in my logs I found these kernel errors: e1000: eth0: e1000_up:
Unable to allocate interrupt Error: -16 ...:  5 Time(s), but it's
perhaps unrelated.
Finally I can obtain this kernel panic by trying to connect to vncserver
from outside.

Have you got any answer? Thanks in advance.
	Jacques Rodary
P.S.: You Can see my config in the post Kernet (sic) panic with NAT.

kernel panic

[senthilkumaar2021]

HI


I am using iptables version 1.43 and getting the following error in 
machine while using squid in tproxy.we are using Linux machine as bridge too

i was able to capture part of the error only please help me in solving 
the issue


  [<ffffffffa03933c2>] ? nf_nat_fn+0x138/0x14e [iptable_nat]
  [<ffffffffa0393585>] ? nf_nat_in+0x2f/0x6e [iptable_nat]
  [<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/
0x2c4 [bridge]
  [<ffffffffa027edfa>] br_nf_pre_routing_finish+0x50/
0x2c4 [bridge]
  [<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/0x2c4 [bridge]
  [<ffffffff81339a50>] ? nf_hook_slow+0x68/0xc8
  [<ffffffffa027edaa>] ? br_nf_pre_routing_finish+0x0/0x2c4 [bridge]
  [<ffffffffa027f616>] br_nf_pre_routing+0x5a8/0x5c7 [bridge]
  [<ffffffff813399ab>] nf_iterate+0x48/0x85
  [<ffffffffa027a931>] ? br_handle_frame_finish+0x0/0x154 [bridge]
  [<ffffffff81339a50>] nf_hook_slow+0x68/0xc8
  [<ffffffffa027a931>] ? br_handle_frame_finish+0x0/0x154 [bridge]
  [<ffffffffa027ac36>] br_handle_frame+0x1b1/0x1db [bridge]
  [<ffffffff8131d54b>] netif_receive_skb+0x316/0x434
  [<ffffffff8131dbfb>] napi_gro_receive+0x6e/0x83
  [<ffffffffa0125bfe>] e1000_receive_skb+0x5c/0x65 [e1000e]
  [<ffffffffa0125de8>] e1000_clean_rx_irq+0x1e1/0x28f [e1000e]
  [<ffffffffa012730e>] e1000_clean+0x99/0x24a [e1000e]
  [<ffffffff813bcfc5>] ? _spin_unlock_irqrestore+0x2c/0x43
  [<ffffffff8131ba62>] net_rx_action+0xb8/0x1b4
  [<ffffffff8104ed43>] __do_softirq+0x99/0x152
  [<ffffffff8101284c>] call_softirq+0x1c/0x30
  [<ffffffff81013a02>] do_softirq+0x52/0xb9
  [<ffffffff8104e969>] irq_exit+0x53/0x8d
  [<ffffffff81013d1a>] do_IRQ+0x135/0x157
  [<ffffffff81011f93>] ret_from_intr+0x0/0x2e
  <EOI> [<ffffffff81017e20>] ? mwait_idle+0x9e/0xc7
  [<ffffffff81017e17>] ? mwait_idle+0x95/0xc7
  [<ffffffff813bfd20>] ? atomic_notifier_call_chain+0x13/0x15
  [<ffffffff810102f4>] ? enter_idle+0x27/0x2>

  Regards
senthil

kernel panic

[senthilkumaar2021]

Hi

Thanks for the reply

I was using kernel 2.6.28.5 and 2.6.30.5 but i got kernel panic in both 
versions

my iptable and ebtable are as follows

iptables -t mangle -N DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT

iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT

iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129

 ebtables -t broute -A BROUTING -i $CLIENT_IFACE -p ipv4 --ip-proto tcp --ip-dport 80 -j redirect --redirect-target DROP

 ebtables -t broute -A BROUTING -i $INET_IFACE -p ipv4 --ip-proto tcp --ip-sport 80 -j redirect --redirect-target DROP

Regards
senthil

> Hi,
>
> >I am using iptables version 1.43 and getting the following error in
> >machine while using squid in tproxy.we are using Linux machine as
> >bridge too
> >i was able to capture part of the error only please help me in solving
> >the issue
>
> What kernel version are you using? I don't know if it's the same
> problem but I was experiencing kernel panic with tproxy in recent
> kernels if there were any traffic to port 80 to the box itself. I was
> able to workaround the problem using RETURN target on the local traffic
> before "-m socket" match and DIVERT target.
>
> XX.XX.XX.XX in the example is the local box address.
>
> My mangle starts this way:
>
>        IPTM="${IPT} -t mangle"
>        echo $" * Launching mangle RULEZ..."
>
>
>        # Setup a chain DIVERT to mark packets
>        $IPTM -N DIVERT
>        $IPTM -A DIVERT -j MARK --set-mark 1
>        $IPTM -A DIVERT -j ACCEPT
>
>        # Bez przekierowywania na proxy w przypadku polaczen do 
> lokalnego serwera www (videocache)
>        $IPTM -A PREROUTING -p tcp -d XX.XX.XX.XX --dport 80 -j ACCEPT
>
>        # Use DIVERT to prevent existing connections going through 
> TPROXY twice
>        $IPTM -A PREROUTING -p tcp -m socket -j DIVERT --dport 80       
> # regula dla ruchu klienci->swiat
>        $IPTM -A PREROUTING -p tcp -m socket -j DIVERT --sport 80
>
>
> Hope this helps.
>
> Best regards,
> Marek