From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: NULL pointer dereference panic in stable (2.6.33.2), amd64 Date: Mon, 12 Apr 2010 22:09:24 +0200 Message-ID: <1271102964.16881.164.camel@edumazet-laptop> References: <201004112338.47019.nuclearcat@nuclearcat.com> <1271064703.16881.16.camel@edumazet-laptop> <201004121911.27955.nuclearcat@nuclearcat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Krishna Kumar2 , David Miller , netdev@vger.kernel.org To: Denys Fedorysychenko Return-path: Received: from mail-bw0-f219.google.com ([209.85.218.219]:61370 "EHLO mail-bw0-f219.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753863Ab0DLVPZ (ORCPT ); Mon, 12 Apr 2010 17:15:25 -0400 Received: by bwz19 with SMTP id 19so4458bwz.21 for ; Mon, 12 Apr 2010 14:15:23 -0700 (PDT) In-Reply-To: <201004121911.27955.nuclearcat@nuclearcat.com> Sender: netdev-owner@vger.kernel.org List-ID: Le lundi 12 avril 2010 =C3=A0 19:11 +0300, Denys Fedorysychenko a =C3=A9= crit : > On Monday 12 April 2010 12:31:43 Eric Dumazet wrote: > . > Seems problem still remain. Patched kernel, but paniced now. > Btw, i dont have any multiqueue card, i think. > pi=C3=A8ce jointe document texte brut (x.txt) > Apr 12 18:46:58 80.83.17.1 dropbear[4843]: exit before auth: Disconne= ct received > Apr 12 18:46:59 80.83.17.1 dropbear[4845]: Child connection from 82.1= 13.44.186:48692 > Apr 12 18:46:59 80.83.17.1 dropbear[4844]: exit before auth: Disconne= ct received > Apr 12 18:46:59 80.83.17.1 kernel: [12598.956375] BUG: unable to hand= le kernel NULL pointer dereference at (null) > Apr 12 18:46:59 80.83.17.1 kernel: [12598.956571] IP: [] dev_queue_xmit+0x28c/0x46d > Apr 12 18:46:59 80.83.17.1 kernel: [12598.956762] PGD 21debc067 PUD 2= 1c881067 PMD 0=20 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.956947] Oops: 0000 [#1] SMP= =20 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957126] last sysfs file: /s= ys/devices/virtual/vc/vcs3/dev > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957311] CPU 0=20 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] Pid: 0, comm: swapp= er Not tainted 2.6.33.2-build-0052test-64 #2 / =20 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] RIP: 0010:[] [] dev_queue_xmit+0x28c/0x46d > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] RSP: 0000:ffff88002= 8203a30 EFLAGS: 00010202 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] RAX: 00000000000020= 00 RBX: 0000000000000000 RCX: ffff880209d8a900 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] RDX: ffff88021d8700= 00 RSI: 0000000000000000 RDI: ffff88020a7b48e8 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] RBP: ffff880028203a= 60 R08: ffff88021c8be89c R09: ffff88021c8bec00 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] R10: dead0000002002= 00 R11: dead000000100100 R12: ffff88021f98a880 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] R13: ffff88021d5c09= 00 R14: ffff88020a7b48e8 R15: ffff88021cbad000 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] FS: 00000000000000= 00(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] CS: 0010 DS: 0000 = ES: 0000 CR0: 000000008005003b > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] CR2: 00000000000000= 00 CR3: 000000021c9d8000 CR4: 00000000000006f0 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] DR0: 00000000000000= 00 DR1: 0000000000000000 DR2: 0000000000000000 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] DR3: 00000000000000= 00 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] Process swapper (pi= d: 0, threadinfo ffffffff81392000, task ffffffff813a1020) > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] Stack: > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] ffff88021d870000 f= fff88021d5c0900 0000000000000042 ffff88021d5c0900 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] <0> ffff88021cbad00= 0 ffff88021cbad000 ffff880028203a80 ffffffffa01c12a9 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] <0> 000000000000000= 0 ffff88020a7b48e8 ffff880028203ad0 ffffffff811e540e > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] Call Trace: > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] =20 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] vlan_dev_hwaccel_hard_start_xmit+0x68/0x86 [8021q] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] dev_hard_start_xmit+0x232/0x304 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] sch_direct_xmit+0x5d/0x16b > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] __qdisc_run+0xbc/0xdc > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] dev_queue_xmit+0x346/0x46d > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ip_finish_output2+0x1c2/0x206 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ip_finish_output+0x68/0x6a > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ip_output+0xa0/0xa5 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ip_forward_finish+0x2e/0x32 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ip_forward+0x2c2/0x322 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ip_rcv_finish+0x2f0/0x30a > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ip_rcv+0x27d/0x2a4 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ? vlan_hwaccel_do_receive+0x2b/0xda > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] netif_receive_skb+0x450/0x475 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] napi_skb_finish+0x24/0x3b > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] vlan_gro_receive+0x7c/0x81 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] e1000_receive_skb+0x4a/0x65 [e1000e] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] e1000_clean_rx_irq+0x1eb/0x29c [e1000e] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] e1000_clean+0x75/0x22e [e1000e] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ? hfsc_dequeue+0x171/0x2a6 [sch_hfsc] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] net_rx_action+0xa7/0x17a > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] __do_softirq+0x96/0x11a > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] call_softirq+0x1c/0x28 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] do_softirq+0x33/0x68 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] irq_exit+0x36/0x75 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] do_IRQ+0xaa/0xc1 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ret_from_intr+0x0/0xa > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] =20 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ? mwait_idle+0x66/0x6b > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] ? enter_idle+0x20/0x22 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] cpu_idle+0x55/0x8d > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] rest_init+0x79/0x7b > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] start_kernel+0x362/0x36d > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] x86_64_start_reservations+0xa5/0xa9 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] [] x86_64_start_kernel+0xdd/0xe4 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] Code: e2 48 8b 55 d= 0 49 c1 e4 07 66 41 8b 86 a6 00 00 00 4c 03 a2 00 03 00 00 80 e4 cf 80 = cc 20 49 8b 5c 24 08 66 41 89 86 a6 00 00 00 <48> 83 3b 00 0f 84 bb 00 = 00 00 4c 8d ab 9c 00 00 00 4c 89 ef e8=20 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] RIP [] dev_queue_xmit+0x28c/0x46d > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] RSP > Apr 12 18:46:59 80.83.17.1 kernel: [12598.957342] CR2: 00000000000000= 00 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.974856] ---[ end trace 739e= 5480c8ab784f ]--- > Apr 12 18:46:59 80.83.17.1 kernel: [12598.975082] Kernel panic - not = syncing: Fatal exception in interrupt > Apr 12 18:46:59 80.83.17.1 kernel: [12598.975311] Pid: 0, comm: swapp= er Tainted: G D 2.6.33.2-build-0052test-64 #2 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.975706] Call Trace: > Apr 12 18:46:59 80.83.17.1 kernel: [12598.975920] [] panic+0xa0/0x161 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.976200] [] ? apic_timer_interrupt+0x13/0x20 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.976431] [] ? kmsg_dump+0x112/0x12c > Apr 12 18:46:59 80.83.17.1 kernel: [12598.976657] [] oops_end+0xaa/0xba > Apr 12 18:46:59 80.83.17.1 kernel: [12598.976882] [] no_context+0x1f3/0x202 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.977113] [] __bad_area_nosemaphore+0x1ba/0x1e0 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.977347] [] ? swiotlb_map_page+0x0/0xd5 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.977577] [] ? pci_map_single+0x8a/0x99 [e1000e] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.977806] [] ? swiotlb_dma_mapping_error+0x18/0x25 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.978045] [] ? pci_dma_mapping_error+0x31/0x3d [e1000e] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.978282] [] ? e1000_xmit_frame+0x6ce/0xa43 [e1000e] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.978513] [] bad_area_nosemaphore+0xe/0x10 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.978741] [] do_page_fault+0x114/0x24a > Apr 12 18:46:59 80.83.17.1 kernel: [12598.978967] [] page_fault+0x1f/0x30 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.979196] [] ? dev_queue_xmit+0x28c/0x46d > Apr 12 18:46:59 80.83.17.1 kernel: [12598.979426] [] vlan_dev_hwaccel_hard_start_xmit+0x68/0x86 [8021q] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.979821] [] dev_hard_start_xmit+0x232/0x304 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.980055] [] sch_direct_xmit+0x5d/0x16b > Apr 12 18:46:59 80.83.17.1 kernel: [12598.980284] [] __qdisc_run+0xbc/0xdc > Apr 12 18:46:59 80.83.17.1 kernel: [12598.980514] [] dev_queue_xmit+0x346/0x46d > Apr 12 18:46:59 80.83.17.1 kernel: [12598.980740] [] ip_finish_output2+0x1c2/0x206 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.980966] [] ip_finish_output+0x68/0x6a > Apr 12 18:46:59 80.83.17.1 kernel: [12598.981197] [] ip_output+0xa0/0xa5 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.981427] [] ip_forward_finish+0x2e/0x32 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.981654] [] ip_forward+0x2c2/0x322 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.981880] [] ip_rcv_finish+0x2f0/0x30a > Apr 12 18:46:59 80.83.17.1 kernel: [12598.982111] [] ip_rcv+0x27d/0x2a4 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.982337] [] ? vlan_hwaccel_do_receive+0x2b/0xda > Apr 12 18:46:59 80.83.17.1 kernel: [12598.982566] [] netif_receive_skb+0x450/0x475 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.982793] [] napi_skb_finish+0x24/0x3b > Apr 12 18:46:59 80.83.17.1 kernel: [12598.983025] [] vlan_gro_receive+0x7c/0x81 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.983260] [] e1000_receive_skb+0x4a/0x65 [e1000e] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.983492] [] e1000_clean_rx_irq+0x1eb/0x29c [e1000e] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.983727] [] e1000_clean+0x75/0x22e [e1000e] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.983955] [] ? hfsc_dequeue+0x171/0x2a6 [sch_hfsc] > Apr 12 18:46:59 80.83.17.1 kernel: [12598.984190] [] net_rx_action+0xa7/0x17a > Apr 12 18:46:59 80.83.17.1 kernel: [12598.984416] [] __do_softirq+0x96/0x11a > Apr 12 18:46:59 80.83.17.1 kernel: [12598.984642] [] call_softirq+0x1c/0x28 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.984866] [] do_softirq+0x33/0x68 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.985097] [] irq_exit+0x36/0x75 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.985323] [] do_IRQ+0xaa/0xc1 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.985546] [] ret_from_intr+0x0/0xa > Apr 12 18:46:59 80.83.17.1 kernel: [12598.985770] [] ? mwait_idle+0x66/0x6b > Apr 12 18:46:59 80.83.17.1 kernel: [12598.986048] [] ? enter_idle+0x20/0x22 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.986284] [] cpu_idle+0x55/0x8d > Apr 12 18:46:59 80.83.17.1 kernel: [12598.986507] [] rest_init+0x79/0x7b > Apr 12 18:46:59 80.83.17.1 kernel: [12598.986730] [] start_kernel+0x362/0x36d > Apr 12 18:46:59 80.83.17.1 kernel: [12598.986955] [] x86_64_start_reservations+0xa5/0xa9 > Apr 12 18:46:59 80.83.17.1 kernel: [12598.987189] [] x86_64_start_kernel+0xdd/0xe4 This is becoming tricky :( This is forwarding case, no socket involved in this case. If no multiqueue is involved, I dont see how it can happen. We should take a look at requeues (qdisc congestion), there might be a problem with them.