From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: [PATCH] ip_gre: fix kernel panic with icmp_dest_unreach Date: Tue, 21 May 2013 14:01:03 -0700 Message-ID: <1369170063.3301.251.camel@edumazet-glaptop> References: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: netdev To: Daniel Petre Return-path: Received: from mail-oa0-f42.google.com ([209.85.219.42]:47899 "EHLO mail-oa0-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750766Ab3EUVBH (ORCPT ); Tue, 21 May 2013 17:01:07 -0400 Received: by mail-oa0-f42.google.com with SMTP id i10so1514369oag.1 for ; Tue, 21 May 2013 14:01:06 -0700 (PDT) In-Reply-To: Sender: netdev-owner@vger.kernel.org List-ID: On Tue, 2013-05-21 at 20:53 +0300, Daniel Petre wrote: > Hello, > This mini patch mitigates the kernel panic with "stack-protector: Kernel stack > is corrupted" when using gre tunnels and network interface flaps and ip_gre > sends a icmp dest unreachable which gets in the IP stack and most probably messes things up. > > Current ip_gre panics both 3.7.10 and 3.8.13 few seconds after ether interface is down, > i managed to take a look at the vmcore with crash utility to find each time icmp_send : > > crash> bt > > PID: 0 TASK: ffffffff81813420 CPU: 0 COMMAND: "swapper/0" > #0 [ffff88003fc03798] machine_kexec at ffffffff81027430 > #1 [ffff88003fc037e8] crash_kexec at ffffffff8107da80 > #2 [ffff88003fc038b8] panic at ffffffff81540026 > #3 [ffff88003fc03938] __stack_chk_fail at ffffffff81037f77 > #4 [ffff88003fc03948] icmp_send at ffffffff814d5fec > #5 [ffff88003fc03ae8] ipv4_link_failure at ffffffff814a1795 > #6 [ffff88003fc03af8] ipgre_tunnel_xmit at ffffffff814e7965 > #7 [ffff88003fc03b78] dev_hard_start_xmit at ffffffff8146e032 > #8 [ffff88003fc03bc8] sch_direct_xmit at ffffffff81487d66 > #9 [ffff88003fc03c08] __qdisc_run at ffffffff81487efd > #10 [ffff88003fc03c48] dev_queue_xmit at ffffffff8146e5a7 > #11 [ffff88003fc03c88] ip_finish_output at ffffffff814ab596 > #12 [ffff88003fc03ce8] __netif_receive_skb at ffffffff8146ed13 > #13 [ffff88003fc03d88] napi_gro_receive at ffffffff8146fc50 > #14 [ffff88003fc03da8] e1000_clean_rx_irq at ffffffff813bc67b > #15 [ffff88003fc03e48] e1000e_poll at ffffffff813c3a20 > #16 [ffff88003fc03e98] net_rx_action at ffffffff8146f796 > #17 [ffff88003fc03ee8] __do_softirq at ffffffff8103ebb9 > #18 [ffff88003fc03f38] call_softirq at ffffffff8154444c > #19 [ffff88003fc03f50] do_softirq at ffffffff810047dd > #20 [ffff88003fc03f80] do_IRQ at ffffffff81003f6c > --- --- > #21 [ffffffff81801ea8] ret_from_intr at ffffffff81542c2a > [exception RIP: mwait_idle+95] > RIP: ffffffff8100ad8f RSP: ffffffff81801f50 RFLAGS: 00000246 > RAX: 0000000000000000 RBX: ffffffff8154194e RCX: 0000000000000000 > RDX: 0000000000000000 RSI: ffffffff81801fd8 RDI: ffff88003fc0d840 > RBP: ffffffff8185be80 R8: 0000000000000000 R9: 0000000000000001 > R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 > R13: ffffffff81813420 R14: ffff88003fc11000 R15: ffffffff81813420 > ORIG_RAX: ffffffffffffff1e CS: 0010 SS: 0018 > #22 [ffffffff81801f50] cpu_idle at ffffffff8100b126 > > crash> log > > [..] > > [ 6772.560124] e1000e: eth3 NIC Link is Down > [ 6786.680876] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow > Control: Rx > [ 6928.050119] e1000e: eth3 NIC Link is Down > [ 6945.240875] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow > Control: Rx > [ 6945.738103] Kernel panic - not syncing: stack-protector: Kernel stack > is corrupted in: ffffffff814d5fec > > [ 6945.738165] Pid: 0, comm: swapper/0 Not tainted 3.8.13 #3 > [ 6945.738189] Call Trace: > [ 6945.738212] [] ? panic+0xbf/0x1c9 > [ 6945.738245] [] ? ip_finish_output+0x255/0x3e0 > [ 6945.738271] [] ? icmp_send+0x6bc/0x730 > [ 6945.738296] [] ? __stack_chk_fail+0x17/0x30 > [ 6945.738320] [] ? icmp_send+0x6bc/0x730 > [ 6945.738344] [] ? nommu_map_sg+0x110/0x110 > [ 6945.738369] [] ? _raw_spin_lock_bh+0x9/0x30 > [ 6945.738393] [] ? ipv4_link_failure+0x15/0x60 > [ 6945.738418] [] ? ipgre_tunnel_xmit+0x7f5/0x9f0 > [ 6945.738443] [] ? dev_hard_start_xmit+0x102/0x490 > [ 6945.738470] [] ? sch_direct_xmit+0x106/0x1e0 > [ 6945.738494] [] ? __qdisc_run+0xbd/0x150 > [ 6945.738518] [] ? dev_queue_xmit+0x1e7/0x3a0 > [ 6945.738542] [] ? ip_finish_output+0x2e6/0x3e0 > [ 6945.738567] [] ? __netif_receive_skb+0x5b3/0x7c0 > [ 6945.738591] [] ? netif_receive_skb+0x24/0x80 > [ 6945.738616] [] ? napi_gro_receive+0x110/0x140 > [ 6945.738642] [] ? e1000_clean_rx_irq+0x29b/0x490 > [ 6945.738667] [] ? e1000e_poll+0x90/0x3a0 > [ 6945.738690] [] ? net_rx_action+0xc6/0x1e0 > [ 6945.738715] [] ? __do_softirq+0xa9/0x170 > [ 6945.738739] [] ? call_softirq+0x1c/0x30 > [ 6945.738762] [] ? do_softirq+0x4d/0x80 > [ 6945.738786] [] ? irq_exit+0x7d/0x90 > [ 6945.738808] [] ? do_IRQ+0x5c/0xd0 > [ 6945.738831] [] ? common_interrupt+0x6a/0x6a > [ 6945.738854] [] ? __schedule+0x26e/0x5b0 > [ 6945.738884] [] ? mwait_idle+0x5f/0x70 > [ 6945.738907] [] ? cpu_idle+0xf6/0x110 > [ 6945.738930] [] ? start_kernel+0x33d/0x348 > [ 6945.738954] [] ? repair_env_string+0x5b/0x5b > [ 6945.738978] [] ? x86_64_start_kernel+0xee/0xf2 > > > Signed-off-by: Daniel Petre > --- > > --- linux-3.8.13/net/ipv4/ip_gre.c.orig 2013-05-21 20:28:37.340537935 +0300 > +++ linux-3.8.13/net/ipv4/ip_gre.c 2013-05-21 20:32:47.248722835 +0300 > @@ -728,7 +728,9 @@ static int ipgre_rcv(struct sk_buff *skb > gro_cells_receive(&tunnel->gro_cells, skb); > return 0; > } > - icmp_send(skb, ICMP_DEST_UNREACH, ICMP_PORT_UNREACH, 0); > + /* don't send icmp destination unreachable if tunnel is down > + the IP stack gets corrupted and machine panics! > + icmp_send(skb, ICMP_DEST_UNREACH, ICMP_PORT_UNREACH, 0); */ > > drop: > kfree_skb(skb); Hmm... can you reproduce this bug on latest kernel ? (preferably David Miller net tree : http://git.kernel.org/cgit/linux/kernel/git/davem/net.git ) Thanks