* kernel 5.5 kernel and tg3 driver
@ 2011-10-10 6:34 sri
2011-10-10 15:50 ` Mulyadi Santosa
0 siblings, 1 reply; 4+ messages in thread
From: sri @ 2011-10-10 6:34 UTC (permalink / raw)
To: kernelnewbies
Hello Experts,
I was using click-1.6.0 on centos 5.3 kernel and recently* ported that to
centos5.5 kernel.*.
After porting, am getting the (*low stack detected by irq handler) and the
*following errors on the console frequently and causing the system
unresponsive totally.
Even after reboot, this issue is re-occurring some time after every 3 days.
System info:
kernel: centos 5.5 (2.6.194-el5)
kernel Modules using: click router
NIC driver: tg3 (broadcom)
Note: The above setup was working successfully for years earlier.
I suspect tg3 driver is doing something here but not sure of how to confirm
this.
Am looking for pointers to start with so that I can investigate. I tried to
do web search on this but not useful.
Also, would like to know any ways/tools to debug using the addresses given
in the stack trace.
Thanks in advance.
######################## *START OF LOGS* ############################
Oct 1 14:16:56 server03 kernel: low stack detected by irq handler
Oct 1 14:16:56 server03 kernel: [<c0405e9b>] do_IRQ+0x86/0xc3
Oct 1 14:16:56 server03 kernel: [<c04046e6>] common_interrupt+0x1a/0x20
Oct 1 14:16:56 server03 kernel: [<c05a5c06>] neigh_lookup+0x23/0x7f
Oct 1 14:16:56 server03 kernel: [<c05d6800>] arp_bind_neighbour+0x50/0x71
Oct 1 14:16:56 server03 kernel: [<c05b7b44>] rt_intern_hash+0x372/0x480
Oct 1 14:16:56 server03 kernel: [<c05b8352>] __ip_route_output_key+0x700/
0x79e
Oct 1 14:16:56 server03 kernel: [<c05b83d4>]
__ip_route_output_key+0x782/0x79e
Oct 1 14:16:57 server03 kernel: [<c05b8403>]
ip_route_output_flow+0x13/0x1d4
Oct 1 14:16:57 server03 kernel: [<c05b85cd>] ip_route_output_key+0x9/0xb
Oct 1 14:16:57 server03 kernel: [<c05bdd5d>] ip_send_reply+0xf8/0x209
Oct 1 14:16:57 server03 kernel: [<c045f1a6>]
cache_alloc_refill+0x122/0x3e4
Oct 1 14:16:57 server03 kernel: [<c05b7c2f>] rt_intern_hash+0x45d/0x480
Oct 1 14:16:57 server03 kernel: [<c05cfc56>] tcp_v4_send_reset+0x104/0x11c
Oct 1 14:16:57 server03 kernel: [<c05d0879>] tcp_v4_do_rcv+0x240/0x269
Oct 1 14:16:57 server03 kernel: [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct 1 14:16:57 server03 kernel: [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct 1 14:16:57 server03 kernel: [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct 1 14:16:57 server03 kernel: [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct 1 14:16:57 server03 kernel: [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct 1 14:16:57 server03 kernel: [<fb1ce85d>]
_ZN12EnsureEther24pushEiP6Packet+0x1d/0x20 [click]
Oct 1 14:16:57 server03 kernel: [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct 1 14:16:57 server03 kernel: [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct 1 14:16:57 server03 kernel: [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct 1 14:16:57 server03 kernel: [<fb19e42d>]
_ZN4IPRw7Mapping9make_pairEiRK8IPFlowIDS3_iiPS0_S4_+0xad/0xd0 [click]
Oct 1 14:16:57 server03 kernel: [<fb19fcfb>]
_ZN4IPRw7Pattern14create_mappingEiRK8IPFlowIDiiPNS_7MappingES5_RK7HashMapIS1_S5_E+0x1cb/0x1f0
[click]
Oct 1 14:16:57 server03 kernel: [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct 1 14:16:57 server03 kernel: [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct 1 14:16:57 server03 kernel: [<fb1ea535>]
_ZN11MacThrottle10l2PushProcEiP6Packet+0x55/0x3b0 [click]
Oct 1 14:16:57 server03 kernel: [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct 1 14:16:57 server03 kernel: [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct 1 14:16:57 server03 kernel: [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct 1 14:16:58 server03 kernel: [<fb1eab96>]
_ZN11MacThrottle4pushEiP6Packet+0x306/0x1890 [click]
Oct 1 14:16:58 server03 kernel: [<c0417f22>] enqueue_task+0x29/0x39
Oct 1 14:16:58 server03 kernel: [<c041805a>] __activate_task+0x4a/0x59
Oct 1 14:16:58 server03 kernel: [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct 1 14:16:58 server03 kernel: [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct 1 14:16:58 server03 kernel: [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct 1 14:16:58 server03 kernel: [<fb1ee075>]
_ZN17NtlmLoginDetector4pushEiP6Packet+0x65/0x15d0 [click]
Oct 1 14:16:58 server03 kernel: [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct 1 14:16:58 server03 kernel: [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct 1 14:16:58 server03 kernel: [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct 1 14:16:58 server03 kernel: [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct 1 14:16:58 server03 kernel: [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct 1 14:16:58 server03 kernel: [<fb1d8e86>]
_ZN11HashIPTable4pushEiP6Packet+0xf6/0x2d0 [click]
Oct 1 14:16:58 server03 kernel: [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct 1 14:16:58 server03 kernel: [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct 1 14:16:58 server03 kernel: [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct 1 14:16:58 server03 kernel: [<fb1e4517>]
_ZN12IPPortLookup4pushEiP6Packet+0x47/0x9e0 [click]
Oct 1 14:16:58 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f
Oct 1 14:16:58 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct 1 14:16:58 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f
Oct 1 14:16:58 server03 kernel: [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct 1 14:16:58 server03 kernel: [<fb1cd5fc>]
_ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click]
Oct 1 14:16:59 server03 kernel: [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct 1 14:16:59 server03 kernel: [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct 1 14:16:59 server03 kernel: [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct 1 14:16:59 server03 kernel: [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct 1 14:16:59 server03 kernel: [<fb1b1853>]
_ZN11ARPQuerier214send_query_forERK9IPAddressitit+0x133/0x3f0 [click]
Oct 1 14:16:59 server03 kernel: [<fb1b1dba>]
_ZN11ARPQuerier29handle_ipEP6Packet+0x2aa/0x5f0 [click]
Oct 1 14:16:59 server03 kernel: [<fb1b31bd>]
_ZN11ARPQuerier24pushEiP6Packet+0xcd/0x110 [click]
Oct 1 14:16:59 server03 kernel: [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct 1 14:16:59 server03 kernel: [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct 1 14:16:59 server03 kernel: [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct 1 14:16:59 server03 kernel: [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct 1 14:16:59 server03 kernel: [<fb1e234f>]
_ZN13IPFilterGroup4pushEiP6Packet+0x19f/0xa80 [click]
Oct 1 14:16:59 server03 kernel: [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct 1 14:16:59 server03 kernel: [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct 1 14:16:59 server03 kernel: [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct 1 14:16:59 server03 kernel: [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct 1 14:16:59 server03 kernel: [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct 1 14:16:59 server03 kernel: [<c05a5c59>] neigh_lookup+0x76/0x7f
Oct 1 14:16:59 server03 kernel: [<fb1d3492>]
_ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click]
Oct 1 14:16:59 server03 kernel: [<fb1e66cf>]
_ZN15LinearIPFilter24pushEiP6Packet+0x4f/0x3f0 [click]
Oct 1 14:16:59 server03 kernel: [<fb1b80e8>]
_ZN11ARPQuerier420handle_arp_retaggingEP6Packett+0x58/0x940 [click]
Oct 1 14:16:59 server03 kernel: [<c05bf82d>] ip_output+0x1f0/0x227
Oct 1 14:16:59 server03 kernel: [<fb1dd289>]
_ZN14HashMACLookup54pushEiP6Packet+0x249/0x890 [click]
Oct 1 14:16:59 server03 kernel: [<fb1d45b7>]
_ZN13HashIPLookup54pushEiP6Packet+0xc7/0x110 [click]
Oct 1 14:17:00 server03 kernel: [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct 1 14:17:00 server03 kernel: [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct 1 14:17:00 server03 kernel: [<fb1dcd23>]
_ZN14HashMACLookup514push_extern_ipEP6Packet+0x1b3/0x4d0 [click]
Oct 1 14:17:00 server03 kernel: [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct 1 14:17:00 server03 kernel: [<c05d065b>] tcp_v4_do_rcv+0x22/0x269
Oct 1 14:17:00 server03 kernel: [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct 1 14:17:00 server03 kernel: [<fb1bfb12>]
_ZN11ARPQuerier415handle_responseEP6Packet+0x32/0x8f0 [click]
Oct 1 14:17:00 server03 kernel: [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct 1 14:17:00 server03 kernel: [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct 1 14:17:00 server03 kernel: [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct 1 14:17:00 server03 kernel: [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct 1 14:17:00 server03 kernel: [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct 1 14:17:00 server03 kernel: [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct 1 14:17:00 server03 kernel: [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct 1 14:17:00 server03 kernel: [<c05d7777>] arp_rcv+0xd0/0xe1
Oct 1 14:17:00 server03 kernel: [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct 1 14:17:00 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f
Oct 1 14:17:00 server03 kernel: [<c043f334>] __do_IRQ+0x0/0x9f
Oct 1 14:17:00 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct 1 14:17:00 server03 kernel: [<c0411f41>]
smp_apic_timer_interrupt+0x32/0x45
Oct 1 14:17:00 server03 kernel: [<c040470b>]
apic_timer_interrupt+0x1f/0x24
Oct 1 14:17:00 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct 1 14:17:00 server03 kernel: [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct 1 14:17:01 server03 kernel: [<fb1fe516>]
_ZN10Classifier4pushEiP6Packet+0x76/0xe0 [click]
Oct 1 14:17:01 server03 kernel: [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct 1 14:17:01 server03 kernel: [<fb1ab2a5>]
_ZN10FromDevice8run_taskEP4Task+0xa5/0x120 [click]
Oct 1 14:17:01 server03 kernel: [<fb16c127>]
_ZN12RouterThread6driverEv+0x177/0x290 [click]
Oct 1 14:17:01 server03 kernel: [<fb21c822>] _Z11click_schedPv+0x82/0x130
[click]
Oct 1 14:17:01 server03 kernel: [<fb21c7a0>] _Z11click_schedPv+0x0/0x130
[click]
Oct 1 14:17:01 server03 kernel: [<c0404987>] kernel_thread_helper+0x7/0x10
Oct 1 14:17:01 server03 kernel: =======================
#######################* END OF LOGS #*
#####################################################
--
--
Krishna Mohan B
--
--
Krishna Mohan B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20111010/b002ab4e/attachment-0001.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* kernel 5.5 kernel and tg3 driver
2011-10-10 6:34 kernel 5.5 kernel and tg3 driver sri
@ 2011-10-10 15:50 ` Mulyadi Santosa
2011-10-10 18:24 ` sri
0 siblings, 1 reply; 4+ messages in thread
From: Mulyadi Santosa @ 2011-10-10 15:50 UTC (permalink / raw)
To: kernelnewbies
Hi...
On Mon, Oct 10, 2011 at 13:34, sri <bskmohan@gmail.com> wrote:
>
> ######################## START OF LOGS ############################
> Oct? 1 14:16:56 server03 kernel: low stack detected by irq handler
> Oct? 1 14:16:56 server03 kernel:? [<c0405e9b>] do_IRQ+0x86/0xc3
> Oct? 1 14:16:56 server03 kernel:? [<c04046e6>] common_interrupt+0x1a/0x20
> Oct? 1 14:16:56 server03 kernel:? [<c05a5c06>] neigh_lookup+0x23/0x7f
> Oct? 1 14:16:56 server03 kernel:? [<c05d6800>] arp_bind_neighbour+0x50/0x71
> Oct? 1 14:16:56 server03 kernel:? [<c05b7b44>] rt_intern_hash+0x372/0x480
> Oct? 1 14:16:56 server03 kernel:? [<c05b8352>] __ip_route_output_key+0x700/
Looks like something is exhausting kernel stack way too fast. IIRC in
4K kernel stack scenario, 4K is assigned for normal usage and another
4K for irq.
So, IMHO, either something is leaking, or code paths are mixing too
much and too fast so they all together exhaust the irq stack.
For debugging, I am not sure...maybe ftrace stack usage could help
you. Grep your kernel .config and find out everything that contains
"stack". Or go to "kernel hacking" section and dig into ftrace
section.
Good luck....
--
regards,
Mulyadi Santosa
Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com
^ permalink raw reply [flat|nested] 4+ messages in thread
* kernel 5.5 kernel and tg3 driver
2011-10-10 15:50 ` Mulyadi Santosa
@ 2011-10-10 18:24 ` sri
2011-10-11 7:50 ` amit mehta
0 siblings, 1 reply; 4+ messages in thread
From: sri @ 2011-10-10 18:24 UTC (permalink / raw)
To: kernelnewbies
Thanks for the Inputs.
As there is not kernel crash, crash files are not create here to debug more.
Is there any way to force the kernel to create kernel crash files under
/var/crash.
This helps more If am not wrong.
Regards,
Sri
On Mon, Oct 10, 2011 at 9:20 PM, Mulyadi Santosa
<mulyadi.santosa@gmail.com>wrote:
> Hi...
>
> On Mon, Oct 10, 2011 at 13:34, sri <bskmohan@gmail.com> wrote:
> >
> > ######################## START OF LOGS ############################
> > Oct 1 14:16:56 server03 kernel: low stack detected by irq handler
> > Oct 1 14:16:56 server03 kernel: [<c0405e9b>] do_IRQ+0x86/0xc3
> > Oct 1 14:16:56 server03 kernel: [<c04046e6>] common_interrupt+0x1a/0x20
> > Oct 1 14:16:56 server03 kernel: [<c05a5c06>] neigh_lookup+0x23/0x7f
> > Oct 1 14:16:56 server03 kernel: [<c05d6800>]
> arp_bind_neighbour+0x50/0x71
> > Oct 1 14:16:56 server03 kernel: [<c05b7b44>] rt_intern_hash+0x372/0x480
> > Oct 1 14:16:56 server03 kernel: [<c05b8352>]
> __ip_route_output_key+0x700/
>
> Looks like something is exhausting kernel stack way too fast. IIRC in
> 4K kernel stack scenario, 4K is assigned for normal usage and another
> 4K for irq.
>
> So, IMHO, either something is leaking, or code paths are mixing too
> much and too fast so they all together exhaust the irq stack.
>
> For debugging, I am not sure...maybe ftrace stack usage could help
> you. Grep your kernel .config and find out everything that contains
> "stack". Or go to "kernel hacking" section and dig into ftrace
> section.
>
> Good luck....
>
> --
> regards,
>
> Mulyadi Santosa
> Freelance Linux trainer and consultant
>
> blog: the-hydra.blogspot.com
> training: mulyaditraining.blogspot.com
>
--
--
Krishna Mohan B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20111010/4e88d410/attachment.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* kernel 5.5 kernel and tg3 driver
2011-10-10 18:24 ` sri
@ 2011-10-11 7:50 ` amit mehta
0 siblings, 0 replies; 4+ messages in thread
From: amit mehta @ 2011-10-11 7:50 UTC (permalink / raw)
To: kernelnewbies
On Mon, Oct 10, 2011 at 11:54 PM, sri <bskmohan@gmail.com> wrote:
> Thanks for the Inputs.
> As there is not kernel crash, crash files are not create here to debug more.
> Is there any way to force the kernel to create kernel crash files under
> /var/crash.
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
but before doing this, you need to enable kdump to dump
the crash image under /var/crash
More on kdump and crash utility to debug kernel crash:
http://people.redhat.com/anderson/crash_whitepaper/
Thanks,
Amit
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2011-10-11 7:50 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-10-10 6:34 kernel 5.5 kernel and tg3 driver sri
2011-10-10 15:50 ` Mulyadi Santosa
2011-10-10 18:24 ` sri
2011-10-11 7:50 ` amit mehta
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).