* BUG in skb_pull with e1000e, PPTP, and L2TP @ 2011-10-17 22:19 Bradley Peterson 2011-10-17 22:25 ` Stephen Hemminger 2011-10-18 2:24 ` Eric Dumazet 0 siblings, 2 replies; 10+ messages in thread From: Bradley Peterson @ 2011-10-17 22:19 UTC (permalink / raw) To: netdev; +Cc: e1000-devel, Bruce Allan, Jesse Brandeburg, John Ronciak I have servers running as PPTP and L2TP/IPSec endpoints. They run other services, but the VPN endpoints seem to be the problem (the problem goes away when VPN is disabled). The servers that are using the e1000e driver crash with "kernel BUG at include/linux/skbuff.h:1186!" using linux 2.6.38. I saw a similar BUG in the same function on 2.6.22, with both e1000e and igb, using 3rd party pptp and l2tp modules. I have other servers, running tg3 and forcedeth drivers, which don't have this crash. I can't reproduce the BUG in my development, and it happens randomly in production. So, testing is difficult. I'm working on testing with 3.0 next. Here are 3 separate instances of the crash. The traces are different, but the BUG is always the same. Thanks for any pointers or help, Bradley Peterson [32173.294224] ------------[ cut here ]------------ [32173.298873] kernel BUG at include/linux/skbuff.h:1186! [32173.304029] invalid opcode: 0000 [#1] SMP [32173.308184] last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map [32173.316039] CPU 1 [32173.317891] Modules linked in: authenc esp4 xfrm4_mode_transport arc4 ppp_mppe tcp_diag inet_diag xt_NOTRACK iptable_raw pptp gre l2tp_ppp pppox ppp_generic slhc l2tp_netlink l 2tp_core tun deflate zlib_deflate twofish_generic twofish_x86_64 twofish_common camellia serpent blowfish cast5 des_generic xcbc rmd160 sha512_generic sha256_generic crypto_null a f_key iptable_nat nf_nat xt_mark iptable_mangle bonding 8021q garp stp llc ipv6 sp5100_tco i2c_piix4 i2c_core e1000e amd64_edac_mod serio_raw ghes microcode k10temp edac_core hed edac_mce_amd raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 pata_acpi firewire_ohci ata_generic firewire_core crc_itu_t pata_atiixp 3w_9xxx [last unloaded: scsi_wait_scan] [32173.385465] [32173.386965] Pid: 0, comm: kworker/0:0 Not tainted 2.6.38.8-32.1.fix.fc14.x86_64 #1 SGI.COM System Product Name/KGP(M)E-D16 [32173.398135] RIP: 0010:[<ffffffff813d2f0d>] [<ffffffff813d2f0d>] __skb_pull258] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 [32173.588842] [<ffffffff81410bb5>] ip_rcv+0x21b/0x246 [32173.593816] [<ffffffff813dd584>] __netif_receive_skb+0x426/0x45c [32173.599925] [<ffffffff81053443>] ? select_task_rq_fair+0x57a/0x57f [32173.606225] [<ffffffff813da220>] ? arch_local_irq_save+0x16/0x1c [32173.612337] [<ffffffff813dd495>] __netif_receive_skb+0x337/0x45c [32173.618450] [<ffffffff810482c7>] ? check_preempt_curr+0x45/0x70 [32173.624478] [<ffffffff8104baa0>] ? ttwu_post_activation+0x60/0xf9 [32173.630669] [<ffffffff813dd641>] process_backlog+0x87/0x15d [32173.636351] [<ffffffff8148982f>] ? _raw_spin_unlock_irqrestore+0x17/0x19 [32173.643165] [<ffffffff813de528>] net_rx_action+0xac/0x1b1 [32173.648675] [<ffffffff8105efaa>] __do_softirq+0xd2/0x19e [32173.654082] [<ffffffff81010fad>] ? paravirt_read_tsc+0x9/0xd [32173.659850] [<ffffffff810114d6>] ? sched_clock+0x9/0xd [32173.665082] [<ffffffff8100bb5c>] call_softirq+0x1c/0x30 [32173.670417] [<ffffffff8100d287>] do_softirq+0x46/0x83 [32173.675565] [<ffffffff8105f132>] irq_exit+0x49/0x8b [32173.680547] [<ffffffff81022b66>] smp_call_function_single_interrupt+0x25/0x27 [32173.687786] [<ffffffff8100b7b3>] call_function_single_interrupt+0x13/0x20 [32173.694662] <EOI> [32173.696798] [<ffffffff8102c61d>] ? native_safe_halt+0xb/0xd [32173.702508] [<ffffffff81011fac>] ? need_resched+0x23/0x2d [32173.708005] [<ffffffff810120fa>] default_idle+0x4e/0x86 [32173.713345] [<ffffffff8100932a>] cpu_idle+0xaa/0xcc [32173.718339] [<ffffffff81482062>] start_secondary+0x20d/0x20f [32173.724092] Code: 68 2b b7 d8 00 00 00 03 b7 e0 00 00 00 89 b7 cc 00 00 00 c9 c3 55 48 89 e5 66 66 66 66 90 8b 57 68 29 f2 3b 57 6c 89 57 68 73 02 <0f> 0b 89 f0 48 03 87 e0 00 00 00 48 89 87 e0 00 00 00 c9 c3 55 [32173.744370] RIP [<ffffffff813d2f0d>] __skb_pull+0x16/0x2a [32173.749920] RSP <ffff8800dfa23b80> [32173.753820] ---[ end trace 83b8ebd5dde8ff41 ]--- [16165.077006] ------------[ cut here ]------------ [16165.077936] kernel BUG at include/linux/skbuff.h:1186! [16165.082856] invalid opcode: 0000 [#1] SMP [16165.082856] last sysfs file: /sys/devices/virtual/net/ppp29/queues/rx-0/rps_flow_cnt [16165.095731] CPU 1 [16165.095731] Modules linked in: arc4 ppp_mppe tcp_diag inet_diag xt_NOTRACK iptable_raw pptp gre l2tp_ppp pppox ppp_generic slhc l2tp_netlink l2tp_core tun deflate zlib_deflate twofish_generic twofish_x86_64 twofish_common camellia serpent blowfish cast5 des_generic xcbc rmd160 sha512_generic sha256_generic crypto_null af_key iptable_nat nf_nat xt_mark i ptable_mangle bonding 8021q garp stp llc ipv6 sp5100_tco e1000e k10temp i2c_piix4 amd64_edac_mod i2c_core edac_core ghes hed edac_mce_amd microcode serio_raw raid456 async_raid6_r ecov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 pata_acpi firewire_ohci ata_generic firewire_core crc_itu_t pata_atiixp 3w_9xxx [last unloaded: scsi_wait_scan] [16165.163315] [16165.163315] Pid: 0, comm: kworker/0:0 Not tainted 2.6.38.8-32.1.fix.fc14.x86_64 #1 SGI.COM System Product Name/KGP(M)E-D16 [16165.163315] RIP: 0010:[<ffffffff813d2f0d>] [<ffffffff813d2f0d>] __skb_pull+0x16/0x2a [16165.163315] RSP: 0018:ffff8800dfa23b80 EFLAGS: 00010287 [16165.163315] RAX: 0000000000000000 RBX: ffff880141cec000 RCX: 000000000000005c [16165.196875] RDX: 000000000000057f RSI: 0000000000000010 RDI: ffff880141cec000 [16165.203325] RBP: ffff8800dfa23b80 R08: 00000000ff34033f R09: 0000000000000000 [1616165.384622] [<ffffffff8104a480>] ? update_shares+0xb7/0xf4 [16165.394969] [<ffffffff813dd641>] process_backlog+0x87/0x15d [16165.394969] [<ffffffff81489816>] ? _raw_spin_lock_irq+0x1f/0x21 [16165.405933] [<ffffffff813de528>] net_rx_action+0xac/0x1b1 [16165.410153] [<ffffffff8105efaa>] __do_softirq+0xd2/0x19e [16165.410153] [<ffffffff81010fad>] ? paravirt_read_tsc+0x9/0xd [16165.410153] [<ffffffff810114d6>] ? sched_clock+0x9/0xd [16165.410153] [<ffffffff8100bb5c>] call_softirq+0x1c/0x30 [16165.410153] [<ffffffff8100d287>] do_softirq+0x46/0x83 [16165.410153] [<ffffffff8105f132>] irq_exit+0x49/0x8b [16165.410153] [<ffffffff81022b66>] smp_call_function_single_interrupt+0x25/0x27 [16165.447293] [<ffffffff8100b7b3>] call_function_single_interrupt+0x13/0x20 [16165.447293] <EOI> [16165.459948] [<ffffffff810b8394>] ? rcu_needs_cpu+0x10e/0x1bf [16165.465027] [<ffffffff8102c61d>] ? native_safe_halt+0xb/0xd [16165.470461] [<ffffffff81011fac>] ? need_resched+0x23/0x2d [16165.477519] [<ffffffff810120fa>] default_idle+0x4e/0x86 [16165.477974] [<ffffffff8100932a>] cpu_idle+0xaa/0xcc [16165.477974] [<ffffffff81482062>] start_secondary+0x20d/0x20f [16165.477974] Code: 68 2b b7 d8 00 00 00 03 b7 e0 00 00 00 89 b7 cc 00 00 00 c9 c3 55 48 89 e5 66 66 66 66 90 8b 57 68 29 f2 3b 57 6c 89 57 68 73 02 <0f> 0b 89 f0 48 03 87 e0 00 00 00 48 89 87 e0 00 00 00 c9 c3 55 [16165.477974] RIP [<ffffffff813d2f0d>] __skb_pull+0x16/0x2a [16165.477974] RSP <ffff8800dfa23b80> [16165.523203] ---[ end trace f793f200ecc5d20f ]--- [17950.922006] ------------[ cut here ]------------ [17950.922941] kernel BUG at include/linux/skbuff.h:1186! [17950.928042] invalid opcode: 0000 [#1] SMP [17950.928042] last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map [17950.943036] CPU 7 [17950.943036] Modules linked in: authenc esp4 xfrm4_mode_transport tcp_diag inet_diag xt_NOTRACK iptable_raw arc4 ppp_mppe pptp gre l2tp_ppp pppox ppp_generic slhc l2tp_netlink l 2tp_core tun deflate zlib_deflate twofish_generic twofish_x86_64 twofish_common camellia serpent blowfish cast5 des_generic xcbc rmd160 sha512_generic sha256_generic crypto_null a f_key iptable_nat nf_nat xt_mark iptable_mangle bonding 8021q garp stp llc ipv6 e1000e sp5100_tco i2c_piix4 k10temp i2c_core amd64_edac_mod ghes edac_core hed serio_raw edac_mce_a md microcode raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 pata_acpi ata_generic firewire_ohci firewire_core crc_itu_t pata_atiixp 3w_9xxx [last unloaded: scsi_wait_scan] [17950.969223] [17950.969223] Pid: 0, comm: kworker/0:1 Not tainted 2.6.38.8-32.1.fix.fc14.x86_64 #1 SGI.COM System Product Name/KGP(M)E-D16 [17950.969223] RIP: 0010:[<ffffffff813d2f0d>] [<ffffffff813d2f0d>] __skb_pull+0x16/0x2a [17950.969223] RSP: 0018:ffff8800dfae3b80 EFLAGS: 00010287 [17950.969223] RAX: 0000000000000000 RBX: ffff88017089f600 RCX: 0000000000000221 [17951.040852] RDX: 000000000000057f RSI: 0000000000000010 RDI: ffff88017089f600 [17951.050257] RBP: ffff8800dfae3b80 R08: 0000000000000000 R09: ffff8800dfae39c0 [17951.050257] R10: ffff88020e362758 R11: ffff880200000001 R12: ffff8800b31eac00 [17951.050257] R13: ffff88013ba2cc72 R14: ffffffffa0280230 R15: ffff880208362000 [17951.050257] FS: 00007fb9a3fee7e0(0000) GS:ffff8800dfae0000(0000) knlGS:0000000000000000 [17951.080066] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [17951.087033] CR2: 00007ffb65c2e000 CR3: 000000014ab0a000 CR4: 00000000000006e0 [17951.087033] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17951.100032] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17951.108481] Process kworker/0:1 (pid: 0, threadinfo ffff88020f60e000, task ffff88020f611730) [17951.117822] Stack: [17951.119564] ffff8800dfae3b90 ffffffff813d2f36 ffff8800dfae3bc0 ffffffffa0286824 [17951.121222] ffff8800dfae3bf0 ffff8800b31eac00 ffff88017089f600 0000000000000000 [17951.121222] ffff8800dfae3c00 ffffffff813d17c4 0000000000000000 0000000000000000 [17951.121222] Call Trace: [17951.142737] <IRQ> [17951.142737] [<ffffffff813d2f36>] skb_pull+0x15/0x17 [17951.142737] [<ffffffffa0286824>] pptp_rcv_core+0x126/0x19a [pptp] [17951.152725] [<ffffffff813d17c4>] sk_receive_skb+0x69/0x105 [17951.163558] [<ffffffffa0286993>] pptp_rcv+0xc8/0xdc [pptp] [17951.165092] [<ffffffffa02800a3>] gre_rcv+0x62/0x75 [gre] [17951.165092] [<ffffffff81410784>] ip_local_deliver_finish+0x150/0x1c1 [17951.177599] [<ffffffff81410634>] ? ip_local_deliver_finish+0x0/0x1c1 [17951.177599] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 [17951.177599] [<ffffffff81410996>] ip_local_deliver+0x51/0x55 [17951.177599] [<ffffffff814105b9>] ip_rcv_finish+0x31a/0x33e [17951.177599] [<ffffffff8141029f>] ? ip_rcv_finish+0x0/0x33e [17951.204898] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 [17951.214651] [<ffffffff81410bb5>] ip_rcv+0x21b/0x246 [17951.219683] [<ffffffff813dd584>] __netif_receive_skb+0x426/0x45c [17951.219683] [<ffffffff813da220>] ? arch_local_irq_save+0x16/0x1c [17951.219683] [<ffffffff813dd495>] __netif_receive_skb+0x337/0x45c [17951.234702] [<ffffffff81022954>] ? native_send_call_func_single_ipi+0x23/0x25 [17951.245864] [<ffffffff813dd641>] process_backlog+0x87/0x15d [17951.247180] [<ffffffff8123f315>] ? timerqueue_add+0x89/0xa8 [17951.257133] [<ffffffff813de528>] net_rx_action+0xac/0x1b1 [17951.262265] [<ffffffff8105efaa>] __do_softirq+0xd2/0x19e [17951.265220] [<ffffffff81010fad>] ? paravirt_read_tsc+0x9/0xd [17951.273703] [<ffffffff810114d6>] ? sched_clock+0x9/0xd [17951.274966] [<ffffffff8100bb5c>] call_softirq+0x1c/0x30 [17951.274966] [<ffffffff8100d287>] do_softirq+0x46/0x83 [17951.274966] [<ffffffff8105f132>] irq_exit+0x49/0x8b [17951.274966] [<ffffffff81022b66>] smp_call_function_single_interrupt+0x25/0x27 [17951.274966] [<ffffffff8100b7b3>] call_function_single_interrupt+0x13/0x20 [17951.274966] <EOI> [17951.274966] [<ffffffff8102c61d>] ? native_safe_halt+0xb/0xd [17951.274966] [<ffffffff81011fac>] ? need_resched+0x23/0x2d [17951.320741] [<ffffffff810120fa>] default_idle+0x4e/0x86 [17951.320741] [<ffffffff8100932a>] cpu_idle+0xaa/0xcc [17951.320741] [<ffffffff81482062>] start_secondary+0x20d/0x20f [17951.320741] Code: 68 2b b7 d8 00 00 00 03 b7 e0 00 00 00 89 b7 cc 00 00 00 c9 c3 55 48 89 e5 66 66 66 66 90 8b 57 68 29 f2 3b 57 6c 89 57 68 73 02 <0f> 0b 89 f0 48 03 87 e0 00 00 00 48 89 87 e0 00 00 00 c9 c3 55 [17951.352436] RIP [<ffffffff813d2f0d>] __skb_pull+0x16/0x2a [17951.352436] RSP <ffff8800dfae3b80> [17951.367951] ---[ end trace af7b2da986dde7ca ]--- ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2d-oct _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: BUG in skb_pull with e1000e, PPTP, and L2TP 2011-10-17 22:19 BUG in skb_pull with e1000e, PPTP, and L2TP Bradley Peterson @ 2011-10-17 22:25 ` Stephen Hemminger 2011-10-17 22:30 ` Bradley Peterson 2011-10-18 2:24 ` Eric Dumazet 1 sibling, 1 reply; 10+ messages in thread From: Stephen Hemminger @ 2011-10-17 22:25 UTC (permalink / raw) To: Bradley Peterson Cc: netdev, Jeff Kirsher, Jesse Brandeburg, Bruce Allan, Carolyn Wyborny, Don Skidmore, Greg Rose, PJ Waskiewicz, Alex Duyck, John Ronciak, e1000-devel On Mon, 17 Oct 2011 17:19:53 -0500 Bradley Peterson <despite@gmail.com> wrote: > using 3rd > party pptp and l2tp modules. More than likely your 3rd party modules are buggy and can't handle all the possible types of skb layout. I have seen out of tree code that can't handle non-linear skb's. You need to take it up with the those developer. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: BUG in skb_pull with e1000e, PPTP, and L2TP 2011-10-17 22:25 ` Stephen Hemminger @ 2011-10-17 22:30 ` Bradley Peterson 0 siblings, 0 replies; 10+ messages in thread From: Bradley Peterson @ 2011-10-17 22:30 UTC (permalink / raw) To: Stephen Hemminger Cc: netdev, Jeff Kirsher, Jesse Brandeburg, Bruce Allan, Carolyn Wyborny, Don Skidmore, Greg Rose, PJ Waskiewicz, Alex Duyck, John Ronciak, e1000-devel On Mon, Oct 17, 2011 at 5:25 PM, Stephen Hemminger <shemminger@vyatta.com> wrote: > On Mon, 17 Oct 2011 17:19:53 -0500 > Bradley Peterson <despite@gmail.com> wrote: > >> using 3rd >> party pptp and l2tp modules. > > More than likely your 3rd party modules are buggy and > can't handle all the possible types of skb layout. > I have seen out of tree code that can't handle non-linear > skb's. You need to take it up with the those developer. > Oh, I should've been more clear -- I am no longer using any 3rd party modules. That referred to my previous setup with 2.6.22 (I knew better than to report that). This bug is with 2.6.38 using in-tree modules. Bradley Peterson ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: BUG in skb_pull with e1000e, PPTP, and L2TP 2011-10-17 22:19 BUG in skb_pull with e1000e, PPTP, and L2TP Bradley Peterson 2011-10-17 22:25 ` Stephen Hemminger @ 2011-10-18 2:24 ` Eric Dumazet 2011-10-18 3:51 ` Eric Dumazet 1 sibling, 1 reply; 10+ messages in thread From: Eric Dumazet @ 2011-10-18 2:24 UTC (permalink / raw) To: Bradley Peterson Cc: netdev, Jeff Kirsher, Jesse Brandeburg, Bruce Allan, Carolyn Wyborny, Don Skidmore, Greg Rose, PJ Waskiewicz, Alex Duyck, John Ronciak, e1000-devel Le lundi 17 octobre 2011 à 17:19 -0500, Bradley Peterson a écrit : > I have servers running as PPTP and L2TP/IPSec endpoints. They run > other services, but the VPN endpoints seem to be the problem (the > problem goes away when VPN is disabled). The servers that are using > the e1000e driver crash with "kernel BUG at > include/linux/skbuff.h:1186!" using linux 2.6.38. I saw a similar BUG > in the same function on 2.6.22, with both e1000e and igb, using 3rd > party pptp and l2tp modules. I have other servers, running tg3 and > forcedeth drivers, which don't have this crash. > > I can't reproduce the BUG in my development, and it happens randomly > in production. So, testing is difficult. I'm working on testing with > 3.0 next. > > Here are 3 separate instances of the crash. The traces are different, > but the BUG is always the same. > > Thanks for any pointers or help, > Bradley Peterson > > [32173.294224] ------------[ cut here ]------------ > [32173.298873] kernel BUG at include/linux/skbuff.h:1186! > [32173.304029] invalid opcode: 0000 [#1] SMP > [32173.308184] last sysfs file: > /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map > [32173.316039] CPU 1 > [32173.317891] Modules linked in: authenc esp4 xfrm4_mode_transport > arc4 ppp_mppe tcp_diag inet_diag xt_NOTRACK iptable_raw pptp gre > l2tp_ppp pppox ppp_generic slhc l2tp_netlink l > 2tp_core tun deflate zlib_deflate twofish_generic twofish_x86_64 > twofish_common camellia serpent blowfish cast5 des_generic xcbc rmd160 > sha512_generic sha256_generic crypto_null a > f_key iptable_nat nf_nat xt_mark iptable_mangle bonding 8021q garp stp > llc ipv6 sp5100_tco i2c_piix4 i2c_core e1000e amd64_edac_mod serio_raw > ghes microcode k10temp edac_core hed > edac_mce_amd raid456 async_raid6_recov async_pq raid6_pq async_xor xor > async_memcpy async_tx raid1 pata_acpi firewire_ohci ata_generic > firewire_core crc_itu_t pata_atiixp 3w_9xxx > [last unloaded: scsi_wait_scan] > [32173.385465] > [32173.386965] Pid: 0, comm: kworker/0:0 Not tainted > 2.6.38.8-32.1.fix.fc14.x86_64 #1 SGI.COM System Product > Name/KGP(M)E-D16 > [32173.398135] RIP: 0010:[<ffffffff813d2f0d>] [<ffffffff813d2f0d>] > __skb_pull258] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 > [32173.588842] [<ffffffff81410bb5>] ip_rcv+0x21b/0x246 > [32173.593816] [<ffffffff813dd584>] __netif_receive_skb+0x426/0x45c > [32173.599925] [<ffffffff81053443>] ? select_task_rq_fair+0x57a/0x57f > [32173.606225] [<ffffffff813da220>] ? arch_local_irq_save+0x16/0x1c > [32173.612337] [<ffffffff813dd495>] __netif_receive_skb+0x337/0x45c > [32173.618450] [<ffffffff810482c7>] ? check_preempt_curr+0x45/0x70 > [32173.624478] [<ffffffff8104baa0>] ? ttwu_post_activation+0x60/0xf9 > [32173.630669] [<ffffffff813dd641>] process_backlog+0x87/0x15d > [32173.636351] [<ffffffff8148982f>] ? _raw_spin_unlock_irqrestore+0x17/0x19 > [32173.643165] [<ffffffff813de528>] net_rx_action+0xac/0x1b1 > [32173.648675] [<ffffffff8105efaa>] __do_softirq+0xd2/0x19e > [32173.654082] [<ffffffff81010fad>] ? paravirt_read_tsc+0x9/0xd > [32173.659850] [<ffffffff810114d6>] ? sched_clock+0x9/0xd > [32173.665082] [<ffffffff8100bb5c>] call_softirq+0x1c/0x30 > [32173.670417] [<ffffffff8100d287>] do_softirq+0x46/0x83 > [32173.675565] [<ffffffff8105f132>] irq_exit+0x49/0x8b > [32173.680547] [<ffffffff81022b66>] > smp_call_function_single_interrupt+0x25/0x27 > [32173.687786] [<ffffffff8100b7b3>] call_function_single_interrupt+0x13/0x20 > [32173.694662] <EOI> > [32173.696798] [<ffffffff8102c61d>] ? native_safe_halt+0xb/0xd > [32173.702508] [<ffffffff81011fac>] ? need_resched+0x23/0x2d > [32173.708005] [<ffffffff810120fa>] default_idle+0x4e/0x86 > [32173.713345] [<ffffffff8100932a>] cpu_idle+0xaa/0xcc > [32173.718339] [<ffffffff81482062>] start_secondary+0x20d/0x20f > [32173.724092] Code: 68 2b b7 d8 00 00 00 03 b7 e0 00 00 00 89 b7 cc > 00 00 00 c9 c3 55 48 89 e5 66 66 66 66 90 8b 57 68 29 f2 3b 57 6c 89 > 57 68 73 02 <0f> 0b 89 f0 48 03 87 e0 00 > 00 00 48 89 87 e0 00 00 00 c9 c3 55 > [32173.744370] RIP [<ffffffff813d2f0d>] __skb_pull+0x16/0x2a > [32173.749920] RSP <ffff8800dfa23b80> > [32173.753820] ---[ end trace 83b8ebd5dde8ff41 ]--- > > > > > > [16165.077006] ------------[ cut here ]------------ > [16165.077936] kernel BUG at include/linux/skbuff.h:1186! > [16165.082856] invalid opcode: 0000 [#1] SMP > [16165.082856] last sysfs file: > /sys/devices/virtual/net/ppp29/queues/rx-0/rps_flow_cnt > [16165.095731] CPU 1 > [16165.095731] Modules linked in: arc4 ppp_mppe tcp_diag inet_diag > xt_NOTRACK iptable_raw pptp gre l2tp_ppp pppox ppp_generic slhc > l2tp_netlink l2tp_core tun deflate zlib_deflate > twofish_generic twofish_x86_64 twofish_common camellia serpent > blowfish cast5 des_generic xcbc rmd160 sha512_generic sha256_generic > crypto_null af_key iptable_nat nf_nat xt_mark i > ptable_mangle bonding 8021q garp stp llc ipv6 sp5100_tco e1000e > k10temp i2c_piix4 amd64_edac_mod i2c_core edac_core ghes hed > edac_mce_amd microcode serio_raw raid456 async_raid6_r > ecov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 > pata_acpi firewire_ohci ata_generic firewire_core crc_itu_t > pata_atiixp 3w_9xxx [last unloaded: scsi_wait_scan] > [16165.163315] > [16165.163315] Pid: 0, comm: kworker/0:0 Not tainted > 2.6.38.8-32.1.fix.fc14.x86_64 #1 SGI.COM System Product > Name/KGP(M)E-D16 > [16165.163315] RIP: 0010:[<ffffffff813d2f0d>] [<ffffffff813d2f0d>] > __skb_pull+0x16/0x2a > [16165.163315] RSP: 0018:ffff8800dfa23b80 EFLAGS: 00010287 > [16165.163315] RAX: 0000000000000000 RBX: ffff880141cec000 RCX: 000000000000005c > [16165.196875] RDX: 000000000000057f RSI: 0000000000000010 RDI: ffff880141cec000 > [16165.203325] RBP: ffff8800dfa23b80 R08: 00000000ff34033f R09: 0000000000000000 > [1616165.384622] [<ffffffff8104a480>] ? update_shares+0xb7/0xf4 > [16165.394969] [<ffffffff813dd641>] process_backlog+0x87/0x15d > [16165.394969] [<ffffffff81489816>] ? _raw_spin_lock_irq+0x1f/0x21 > [16165.405933] [<ffffffff813de528>] net_rx_action+0xac/0x1b1 > [16165.410153] [<ffffffff8105efaa>] __do_softirq+0xd2/0x19e > [16165.410153] [<ffffffff81010fad>] ? paravirt_read_tsc+0x9/0xd > [16165.410153] [<ffffffff810114d6>] ? sched_clock+0x9/0xd > [16165.410153] [<ffffffff8100bb5c>] call_softirq+0x1c/0x30 > [16165.410153] [<ffffffff8100d287>] do_softirq+0x46/0x83 > [16165.410153] [<ffffffff8105f132>] irq_exit+0x49/0x8b > [16165.410153] [<ffffffff81022b66>] > smp_call_function_single_interrupt+0x25/0x27 > [16165.447293] [<ffffffff8100b7b3>] call_function_single_interrupt+0x13/0x20 > [16165.447293] <EOI> > [16165.459948] [<ffffffff810b8394>] ? rcu_needs_cpu+0x10e/0x1bf > [16165.465027] [<ffffffff8102c61d>] ? native_safe_halt+0xb/0xd > [16165.470461] [<ffffffff81011fac>] ? need_resched+0x23/0x2d > [16165.477519] [<ffffffff810120fa>] default_idle+0x4e/0x86 > [16165.477974] [<ffffffff8100932a>] cpu_idle+0xaa/0xcc > [16165.477974] [<ffffffff81482062>] start_secondary+0x20d/0x20f > [16165.477974] Code: 68 2b b7 d8 00 00 00 03 b7 e0 00 00 00 89 b7 cc > 00 00 00 c9 c3 55 48 89 e5 66 66 66 66 90 8b 57 68 29 f2 3b 57 6c 89 > 57 68 73 02 <0f> 0b 89 f0 48 03 87 e0 00 > 00 00 48 89 87 e0 00 00 00 c9 c3 55 > [16165.477974] RIP [<ffffffff813d2f0d>] __skb_pull+0x16/0x2a > [16165.477974] RSP <ffff8800dfa23b80> > [16165.523203] ---[ end trace f793f200ecc5d20f ]--- > > > > > > [17950.922006] ------------[ cut here ]------------ > [17950.922941] kernel BUG at include/linux/skbuff.h:1186! > [17950.928042] invalid opcode: 0000 [#1] SMP > [17950.928042] last sysfs file: > /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map > [17950.943036] CPU 7 > [17950.943036] Modules linked in: authenc esp4 xfrm4_mode_transport > tcp_diag inet_diag xt_NOTRACK iptable_raw arc4 ppp_mppe pptp gre > l2tp_ppp pppox ppp_generic slhc l2tp_netlink l > 2tp_core tun deflate zlib_deflate twofish_generic twofish_x86_64 > twofish_common camellia serpent blowfish cast5 des_generic xcbc rmd160 > sha512_generic sha256_generic crypto_null a > f_key iptable_nat nf_nat xt_mark iptable_mangle bonding 8021q garp stp > llc ipv6 e1000e sp5100_tco i2c_piix4 k10temp i2c_core amd64_edac_mod > ghes edac_core hed serio_raw edac_mce_a > md microcode raid456 async_raid6_recov async_pq raid6_pq async_xor xor > async_memcpy async_tx raid1 pata_acpi ata_generic firewire_ohci > firewire_core crc_itu_t pata_atiixp 3w_9xxx > [last unloaded: scsi_wait_scan] > [17950.969223] > [17950.969223] Pid: 0, comm: kworker/0:1 Not tainted > 2.6.38.8-32.1.fix.fc14.x86_64 #1 SGI.COM System Product > Name/KGP(M)E-D16 > [17950.969223] RIP: 0010:[<ffffffff813d2f0d>] [<ffffffff813d2f0d>] > __skb_pull+0x16/0x2a > [17950.969223] RSP: 0018:ffff8800dfae3b80 EFLAGS: 00010287 > [17950.969223] RAX: 0000000000000000 RBX: ffff88017089f600 RCX: 0000000000000221 > [17951.040852] RDX: 000000000000057f RSI: 0000000000000010 RDI: ffff88017089f600 > [17951.050257] RBP: ffff8800dfae3b80 R08: 0000000000000000 R09: ffff8800dfae39c0 > [17951.050257] R10: ffff88020e362758 R11: ffff880200000001 R12: ffff8800b31eac00 > [17951.050257] R13: ffff88013ba2cc72 R14: ffffffffa0280230 R15: ffff880208362000 > [17951.050257] FS: 00007fb9a3fee7e0(0000) GS:ffff8800dfae0000(0000) > knlGS:0000000000000000 > [17951.080066] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [17951.087033] CR2: 00007ffb65c2e000 CR3: 000000014ab0a000 CR4: 00000000000006e0 > [17951.087033] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [17951.100032] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [17951.108481] Process kworker/0:1 (pid: 0, threadinfo > ffff88020f60e000, task ffff88020f611730) > [17951.117822] Stack: > [17951.119564] ffff8800dfae3b90 ffffffff813d2f36 ffff8800dfae3bc0 > ffffffffa0286824 > [17951.121222] ffff8800dfae3bf0 ffff8800b31eac00 ffff88017089f600 > 0000000000000000 > [17951.121222] ffff8800dfae3c00 ffffffff813d17c4 0000000000000000 > 0000000000000000 > [17951.121222] Call Trace: > [17951.142737] <IRQ> > [17951.142737] [<ffffffff813d2f36>] skb_pull+0x15/0x17 > [17951.142737] [<ffffffffa0286824>] pptp_rcv_core+0x126/0x19a [pptp] > [17951.152725] [<ffffffff813d17c4>] sk_receive_skb+0x69/0x105 > [17951.163558] [<ffffffffa0286993>] pptp_rcv+0xc8/0xdc [pptp] > [17951.165092] [<ffffffffa02800a3>] gre_rcv+0x62/0x75 [gre] > [17951.165092] [<ffffffff81410784>] ip_local_deliver_finish+0x150/0x1c1 > [17951.177599] [<ffffffff81410634>] ? ip_local_deliver_finish+0x0/0x1c1 > [17951.177599] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 > [17951.177599] [<ffffffff81410996>] ip_local_deliver+0x51/0x55 > [17951.177599] [<ffffffff814105b9>] ip_rcv_finish+0x31a/0x33e > [17951.177599] [<ffffffff8141029f>] ? ip_rcv_finish+0x0/0x33e > [17951.204898] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 > [17951.214651] [<ffffffff81410bb5>] ip_rcv+0x21b/0x246 > [17951.219683] [<ffffffff813dd584>] __netif_receive_skb+0x426/0x45c > [17951.219683] [<ffffffff813da220>] ? arch_local_irq_save+0x16/0x1c > [17951.219683] [<ffffffff813dd495>] __netif_receive_skb+0x337/0x45c > [17951.234702] [<ffffffff81022954>] ? > native_send_call_func_single_ipi+0x23/0x25 > [17951.245864] [<ffffffff813dd641>] process_backlog+0x87/0x15d > [17951.247180] [<ffffffff8123f315>] ? timerqueue_add+0x89/0xa8 > [17951.257133] [<ffffffff813de528>] net_rx_action+0xac/0x1b1 > [17951.262265] [<ffffffff8105efaa>] __do_softirq+0xd2/0x19e > [17951.265220] [<ffffffff81010fad>] ? paravirt_read_tsc+0x9/0xd > [17951.273703] [<ffffffff810114d6>] ? sched_clock+0x9/0xd > [17951.274966] [<ffffffff8100bb5c>] call_softirq+0x1c/0x30 > [17951.274966] [<ffffffff8100d287>] do_softirq+0x46/0x83 > [17951.274966] [<ffffffff8105f132>] irq_exit+0x49/0x8b > [17951.274966] [<ffffffff81022b66>] > smp_call_function_single_interrupt+0x25/0x27 > [17951.274966] [<ffffffff8100b7b3>] call_function_single_interrupt+0x13/0x20 > [17951.274966] <EOI> > [17951.274966] [<ffffffff8102c61d>] ? native_safe_halt+0xb/0xd > [17951.274966] [<ffffffff81011fac>] ? need_resched+0x23/0x2d > [17951.320741] [<ffffffff810120fa>] default_idle+0x4e/0x86 > [17951.320741] [<ffffffff8100932a>] cpu_idle+0xaa/0xcc > [17951.320741] [<ffffffff81482062>] start_secondary+0x20d/0x20f > [17951.320741] Code: 68 2b b7 d8 00 00 00 03 b7 e0 00 00 00 89 b7 cc > 00 00 00 c9 c3 55 48 89 e5 66 66 66 66 90 8b 57 68 29 f2 3b 57 6c 89 > 57 68 73 02 <0f> 0b 89 f0 48 03 87 e0 00 > 00 00 48 89 87 e0 00 00 00 c9 c3 55 > [17951.352436] RIP [<ffffffff813d2f0d>] __skb_pull+0x16/0x2a > [17951.352436] RSP <ffff8800dfae3b80> > [17951.367951] ---[ end trace af7b2da986dde7ca ]--- > -- Could you please try following patch ? [PATCH] pptp: pptp_rcv_core() misses pskb_may_pull() call e1000e uses paged frags, so any layer incorrectly pulling bytes from skb can trigger a BUG in skb_pull() [951.142737] [<ffffffff813d2f36>] skb_pull+0x15/0x17 [951.142737] [<ffffffffa0286824>] pptp_rcv_core+0x126/0x19a [pptp] [951.152725] [<ffffffff813d17c4>] sk_receive_skb+0x69/0x105 [951.163558] [<ffffffffa0286993>] pptp_rcv+0xc8/0xdc [pptp] [951.165092] [<ffffffffa02800a3>] gre_rcv+0x62/0x75 [gre] [951.165092] [<ffffffff81410784>] ip_local_deliver_finish+0x150/0x1c1 [951.177599] [<ffffffff81410634>] ? ip_local_deliver_finish+0x0/0x1c1 [951.177599] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 [951.177599] [<ffffffff81410996>] ip_local_deliver+0x51/0x55 [951.177599] [<ffffffff814105b9>] ip_rcv_finish+0x31a/0x33e [951.177599] [<ffffffff8141029f>] ? ip_rcv_finish+0x0/0x33e [951.204898] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 [951.214651] [<ffffffff81410bb5>] ip_rcv+0x21b/0x246 pptp_rcv_core() is a nice example of a function assuming everything it needs is available in skb head. Reported-by: Bradley Peterson <despite@gmail.com> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> --- drivers/net/ppp/pptp.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/drivers/net/ppp/pptp.c b/drivers/net/ppp/pptp.c index eae542a..d0197e3 100644 --- a/drivers/net/ppp/pptp.c +++ b/drivers/net/ppp/pptp.c @@ -305,11 +305,16 @@ static int pptp_rcv_core(struct sock *sk, struct sk_buff *skb) } header = (struct pptp_gre_header *)(skb->data); + headersize = sizeof(*header); /* test if acknowledgement present */ if (PPTP_GRE_IS_A(header->ver)) { - __u32 ack = (PPTP_GRE_IS_S(header->flags)) ? - header->ack : header->seq; /* ack in different place if S = 0 */ + __u32 ack; + + if (!pskb_may_pull(skb, headersize)) + goto drop; + ack = (PPTP_GRE_IS_S(header->flags)) ? + header->ack : header->seq; /* ack in different place if S = 0 */ ack = ntohl(ack); @@ -318,21 +323,18 @@ static int pptp_rcv_core(struct sock *sk, struct sk_buff *skb) /* also handle sequence number wrap-around */ if (WRAPPED(ack, opt->ack_recv)) opt->ack_recv = ack; + } else { + headersize -= sizeof(header->ack); } - /* test if payload present */ if (!PPTP_GRE_IS_S(header->flags)) goto drop; - headersize = sizeof(*header); payload_len = ntohs(header->payload_len); seq = ntohl(header->seq); - /* no ack present? */ - if (!PPTP_GRE_IS_A(header->ver)) - headersize -= sizeof(header->ack); /* check for incomplete packet (length smaller than expected) */ - if (skb->len - headersize < payload_len) + if (!pskb_may_pull(skb, headersize + payload_len)) goto drop; payload = skb->data + headersize; ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: BUG in skb_pull with e1000e, PPTP, and L2TP 2011-10-18 2:24 ` Eric Dumazet @ 2011-10-18 3:51 ` Eric Dumazet 2011-10-18 3:59 ` Eric Dumazet 0 siblings, 1 reply; 10+ messages in thread From: Eric Dumazet @ 2011-10-18 3:51 UTC (permalink / raw) To: Bradley Peterson Cc: netdev, Jeff Kirsher, Jesse Brandeburg, Bruce Allan, Carolyn Wyborny, Don Skidmore, Greg Rose, PJ Waskiewicz, Alex Duyck, John Ronciak, e1000-devel Le mardi 18 octobre 2011 à 04:24 +0200, Eric Dumazet a écrit : > diff --git a/drivers/net/ppp/pptp.c b/drivers/net/ppp/pptp.c > index eae542a..d0197e3 100644 > --- a/drivers/net/ppp/pptp.c > +++ b/drivers/net/ppp/pptp.c > @@ -305,11 +305,16 @@ static int pptp_rcv_core(struct sock *sk, struct sk_buff *skb) > } > > header = (struct pptp_gre_header *)(skb->data); > + headersize = sizeof(*header); > > /* test if acknowledgement present */ > if (PPTP_GRE_IS_A(header->ver)) { > - __u32 ack = (PPTP_GRE_IS_S(header->flags)) ? > - header->ack : header->seq; /* ack in different place if S = 0 */ > + __u32 ack; > + > + if (!pskb_may_pull(skb, headersize)) > + goto drop; Oh well, this is buggy, I need to set header again, I'll send an updated patch header = (struct pptp_gre_header *)(skb->data); > + ack = (PPTP_GRE_IS_S(header->flags)) ? > + header->ack : header->seq; /* ack in different place if S = 0 */ > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: BUG in skb_pull with e1000e, PPTP, and L2TP 2011-10-18 3:51 ` Eric Dumazet @ 2011-10-18 3:59 ` Eric Dumazet 2011-10-19 7:31 ` David Miller 2011-10-24 21:59 ` Bradley Peterson 0 siblings, 2 replies; 10+ messages in thread From: Eric Dumazet @ 2011-10-18 3:59 UTC (permalink / raw) To: Bradley Peterson Cc: Don, e1000-devel, netdev, Bruce Allan, Jesse Brandeburg, John Ronciak Le mardi 18 octobre 2011 à 05:51 +0200, Eric Dumazet a écrit : > Le mardi 18 octobre 2011 à 04:24 +0200, Eric Dumazet a écrit : > > > diff --git a/drivers/net/ppp/pptp.c b/drivers/net/ppp/pptp.c > > index eae542a..d0197e3 100644 > > --- a/drivers/net/ppp/pptp.c > > +++ b/drivers/net/ppp/pptp.c > > @@ -305,11 +305,16 @@ static int pptp_rcv_core(struct sock *sk, struct sk_buff *skb) > > } > > > > header = (struct pptp_gre_header *)(skb->data); > > + headersize = sizeof(*header); > > > > /* test if acknowledgement present */ > > if (PPTP_GRE_IS_A(header->ver)) { > > - __u32 ack = (PPTP_GRE_IS_S(header->flags)) ? > > - header->ack : header->seq; /* ack in different place if S = 0 */ > > + __u32 ack; > > + > > + if (!pskb_may_pull(skb, headersize)) > > + goto drop; > > Oh well, this is buggy, I need to set header again, I'll send an updated > patch > [PATCH v2] pptp: pptp_rcv_core() misses pskb_may_pull() call e1000e uses paged frags, so any layer incorrectly pulling bytes from skb can trigger a BUG in skb_pull() [951.142737] [<ffffffff813d2f36>] skb_pull+0x15/0x17 [951.142737] [<ffffffffa0286824>] pptp_rcv_core+0x126/0x19a [pptp] [951.152725] [<ffffffff813d17c4>] sk_receive_skb+0x69/0x105 [951.163558] [<ffffffffa0286993>] pptp_rcv+0xc8/0xdc [pptp] [951.165092] [<ffffffffa02800a3>] gre_rcv+0x62/0x75 [gre] [951.165092] [<ffffffff81410784>] ip_local_deliver_finish+0x150/0x1c1 [951.177599] [<ffffffff81410634>] ? ip_local_deliver_finish+0x0/0x1c1 [951.177599] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 [951.177599] [<ffffffff81410996>] ip_local_deliver+0x51/0x55 [951.177599] [<ffffffff814105b9>] ip_rcv_finish+0x31a/0x33e [951.177599] [<ffffffff8141029f>] ? ip_rcv_finish+0x0/0x33e [951.204898] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 [951.214651] [<ffffffff81410bb5>] ip_rcv+0x21b/0x246 pptp_rcv_core() is a nice example of a function assuming everything it needs is available in skb head. Reported-by: Bradley Peterson <despite@gmail.com> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> --- drivers/net/ppp/pptp.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/drivers/net/ppp/pptp.c b/drivers/net/ppp/pptp.c index eae542a..29730fd 100644 --- a/drivers/net/ppp/pptp.c +++ b/drivers/net/ppp/pptp.c @@ -305,11 +305,18 @@ static int pptp_rcv_core(struct sock *sk, struct sk_buff *skb) } header = (struct pptp_gre_header *)(skb->data); + headersize = sizeof(*header); /* test if acknowledgement present */ if (PPTP_GRE_IS_A(header->ver)) { - __u32 ack = (PPTP_GRE_IS_S(header->flags)) ? - header->ack : header->seq; /* ack in different place if S = 0 */ + __u32 ack; + + if (!pskb_may_pull(skb, headersize)) + goto drop; + header = (struct pptp_gre_header *)(skb->data); + + /* ack in different place if S = 0 */ + ack = PPTP_GRE_IS_S(header->flags) ? header->ack : header->seq; ack = ntohl(ack); @@ -318,21 +325,18 @@ static int pptp_rcv_core(struct sock *sk, struct sk_buff *skb) /* also handle sequence number wrap-around */ if (WRAPPED(ack, opt->ack_recv)) opt->ack_recv = ack; + } else { + headersize -= sizeof(header->ack); } - /* test if payload present */ if (!PPTP_GRE_IS_S(header->flags)) goto drop; - headersize = sizeof(*header); payload_len = ntohs(header->payload_len); seq = ntohl(header->seq); - /* no ack present? */ - if (!PPTP_GRE_IS_A(header->ver)) - headersize -= sizeof(header->ack); /* check for incomplete packet (length smaller than expected) */ - if (skb->len - headersize < payload_len) + if (!pskb_may_pull(skb, headersize + payload_len)) goto drop; payload = skb->data + headersize; ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2d-oct _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: BUG in skb_pull with e1000e, PPTP, and L2TP 2011-10-18 3:59 ` Eric Dumazet @ 2011-10-19 7:31 ` David Miller 2011-10-19 7:47 ` Eric Dumazet 2011-10-24 21:59 ` Bradley Peterson 1 sibling, 1 reply; 10+ messages in thread From: David Miller @ 2011-10-19 7:31 UTC (permalink / raw) To: eric.dumazet Cc: e1000-devel, netdev, bruce.w.allan, jesse.brandeburg, john.ronciak, despite From: Eric Dumazet <eric.dumazet@gmail.com> Date: Tue, 18 Oct 2011 05:59:53 +0200 > [PATCH v2] pptp: pptp_rcv_core() misses pskb_may_pull() call > > e1000e uses paged frags, so any layer incorrectly pulling bytes from skb > can trigger a BUG in skb_pull() > > [951.142737] [<ffffffff813d2f36>] skb_pull+0x15/0x17 > [951.142737] [<ffffffffa0286824>] pptp_rcv_core+0x126/0x19a [pptp] > [951.152725] [<ffffffff813d17c4>] sk_receive_skb+0x69/0x105 > [951.163558] [<ffffffffa0286993>] pptp_rcv+0xc8/0xdc [pptp] > [951.165092] [<ffffffffa02800a3>] gre_rcv+0x62/0x75 [gre] > [951.165092] [<ffffffff81410784>] ip_local_deliver_finish+0x150/0x1c1 > [951.177599] [<ffffffff81410634>] ? ip_local_deliver_finish+0x0/0x1c1 > [951.177599] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 > [951.177599] [<ffffffff81410996>] ip_local_deliver+0x51/0x55 > [951.177599] [<ffffffff814105b9>] ip_rcv_finish+0x31a/0x33e > [951.177599] [<ffffffff8141029f>] ? ip_rcv_finish+0x0/0x33e > [951.204898] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 > [951.214651] [<ffffffff81410bb5>] ip_rcv+0x21b/0x246 > > pptp_rcv_core() is a nice example of a function assuming everything it > needs is available in skb head. > > Reported-by: Bradley Peterson <despite@gmail.com> > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> I assume by the driver paths in the patch that you think this is 'net-next' material and not suitable for plain 'net', right? ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2d-oct _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: BUG in skb_pull with e1000e, PPTP, and L2TP 2011-10-19 7:31 ` David Miller @ 2011-10-19 7:47 ` Eric Dumazet 2011-10-19 7:51 ` David Miller 0 siblings, 1 reply; 10+ messages in thread From: Eric Dumazet @ 2011-10-19 7:47 UTC (permalink / raw) To: David Miller Cc: e1000-devel, netdev, bruce.w.allan, jesse.brandeburg, john.ronciak, despite Le mercredi 19 octobre 2011 à 03:31 -0400, David Miller a écrit : > From: Eric Dumazet <eric.dumazet@gmail.com> > Date: Tue, 18 Oct 2011 05:59:53 +0200 > > > [PATCH v2] pptp: pptp_rcv_core() misses pskb_may_pull() call > > > > e1000e uses paged frags, so any layer incorrectly pulling bytes from skb > > can trigger a BUG in skb_pull() > > > > [951.142737] [<ffffffff813d2f36>] skb_pull+0x15/0x17 > > [951.142737] [<ffffffffa0286824>] pptp_rcv_core+0x126/0x19a [pptp] > > [951.152725] [<ffffffff813d17c4>] sk_receive_skb+0x69/0x105 > > [951.163558] [<ffffffffa0286993>] pptp_rcv+0xc8/0xdc [pptp] > > [951.165092] [<ffffffffa02800a3>] gre_rcv+0x62/0x75 [gre] > > [951.165092] [<ffffffff81410784>] ip_local_deliver_finish+0x150/0x1c1 > > [951.177599] [<ffffffff81410634>] ? ip_local_deliver_finish+0x0/0x1c1 > > [951.177599] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 > > [951.177599] [<ffffffff81410996>] ip_local_deliver+0x51/0x55 > > [951.177599] [<ffffffff814105b9>] ip_rcv_finish+0x31a/0x33e > > [951.177599] [<ffffffff8141029f>] ? ip_rcv_finish+0x0/0x33e > > [951.204898] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 > > [951.214651] [<ffffffff81410bb5>] ip_rcv+0x21b/0x246 > > > > pptp_rcv_core() is a nice example of a function assuming everything it > > needs is available in skb head. > > > > Reported-by: Bradley Peterson <despite@gmail.com> > > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> > > I assume by the driver paths in the patch that you think this > is 'net-next' material and not suitable for plain 'net', right? I incorrectly thought this driver was at the same location in net & net-next, and I my net-next tree was more convenient to compile this. I can respin patch on net tree if you prefer. Thanks ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2d-oct _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: BUG in skb_pull with e1000e, PPTP, and L2TP 2011-10-19 7:47 ` Eric Dumazet @ 2011-10-19 7:51 ` David Miller 0 siblings, 0 replies; 10+ messages in thread From: David Miller @ 2011-10-19 7:51 UTC (permalink / raw) To: eric.dumazet Cc: e1000-devel, netdev, bruce.w.allan, jesse.brandeburg, john.ronciak, despite From: Eric Dumazet <eric.dumazet@gmail.com> Date: Wed, 19 Oct 2011 09:47:15 +0200 > I incorrectly thought this driver was at the same location in net & > net-next, and I my net-next tree was more convenient to compile this. > > I can respin patch on net tree if you prefer. No need, I took care of adjusting the paths. Thanks! ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2d-oct _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: BUG in skb_pull with e1000e, PPTP, and L2TP 2011-10-18 3:59 ` Eric Dumazet 2011-10-19 7:31 ` David Miller @ 2011-10-24 21:59 ` Bradley Peterson 1 sibling, 0 replies; 10+ messages in thread From: Bradley Peterson @ 2011-10-24 21:59 UTC (permalink / raw) To: Eric Dumazet Cc: e1000-devel, netdev, Bruce Allan, Jesse Brandeburg, John Ronciak On Mon, Oct 17, 2011 at 10:59 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > Le mardi 18 octobre 2011 à 05:51 +0200, Eric Dumazet a écrit : >> Le mardi 18 octobre 2011 à 04:24 +0200, Eric Dumazet a écrit : >> >> > diff --git a/drivers/net/ppp/pptp.c b/drivers/net/ppp/pptp.c >> > index eae542a..d0197e3 100644 >> > --- a/drivers/net/ppp/pptp.c >> > +++ b/drivers/net/ppp/pptp.c >> > @@ -305,11 +305,16 @@ static int pptp_rcv_core(struct sock *sk, struct sk_buff *skb) >> > } >> > >> > header = (struct pptp_gre_header *)(skb->data); >> > + headersize = sizeof(*header); >> > >> > /* test if acknowledgement present */ >> > if (PPTP_GRE_IS_A(header->ver)) { >> > - __u32 ack = (PPTP_GRE_IS_S(header->flags)) ? >> > - header->ack : header->seq; /* ack in different place if S = 0 */ >> > + __u32 ack; >> > + >> > + if (!pskb_may_pull(skb, headersize)) >> > + goto drop; >> >> Oh well, this is buggy, I need to set header again, I'll send an updated >> patch >> > > [PATCH v2] pptp: pptp_rcv_core() misses pskb_may_pull() call > > e1000e uses paged frags, so any layer incorrectly pulling bytes from skb > can trigger a BUG in skb_pull() > > [951.142737] [<ffffffff813d2f36>] skb_pull+0x15/0x17 > [951.142737] [<ffffffffa0286824>] pptp_rcv_core+0x126/0x19a [pptp] > [951.152725] [<ffffffff813d17c4>] sk_receive_skb+0x69/0x105 > [951.163558] [<ffffffffa0286993>] pptp_rcv+0xc8/0xdc [pptp] > [951.165092] [<ffffffffa02800a3>] gre_rcv+0x62/0x75 [gre] > [951.165092] [<ffffffff81410784>] ip_local_deliver_finish+0x150/0x1c1 > [951.177599] [<ffffffff81410634>] ? ip_local_deliver_finish+0x0/0x1c1 > [951.177599] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 > [951.177599] [<ffffffff81410996>] ip_local_deliver+0x51/0x55 > [951.177599] [<ffffffff814105b9>] ip_rcv_finish+0x31a/0x33e > [951.177599] [<ffffffff8141029f>] ? ip_rcv_finish+0x0/0x33e > [951.204898] [<ffffffff81410846>] NF_HOOK.clone.7+0x51/0x58 > [951.214651] [<ffffffff81410bb5>] ip_rcv+0x21b/0x246 > > pptp_rcv_core() is a nice example of a function assuming everything it > needs is available in skb head. > > Reported-by: Bradley Peterson <despite@gmail.com> > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> > --- > drivers/net/ppp/pptp.c | 20 ++++++++++++-------- > 1 file changed, 12 insertions(+), 8 deletions(-) > > diff --git a/drivers/net/ppp/pptp.c b/drivers/net/ppp/pptp.c > index eae542a..29730fd 100644 > --- a/drivers/net/ppp/pptp.c > +++ b/drivers/net/ppp/pptp.c > @@ -305,11 +305,18 @@ static int pptp_rcv_core(struct sock *sk, struct sk_buff *skb) > } > > header = (struct pptp_gre_header *)(skb->data); > + headersize = sizeof(*header); > > /* test if acknowledgement present */ > if (PPTP_GRE_IS_A(header->ver)) { > - __u32 ack = (PPTP_GRE_IS_S(header->flags)) ? > - header->ack : header->seq; /* ack in different place if S = 0 */ > + __u32 ack; > + > + if (!pskb_may_pull(skb, headersize)) > + goto drop; > + header = (struct pptp_gre_header *)(skb->data); > + > + /* ack in different place if S = 0 */ > + ack = PPTP_GRE_IS_S(header->flags) ? header->ack : header->seq; > > ack = ntohl(ack); > > @@ -318,21 +325,18 @@ static int pptp_rcv_core(struct sock *sk, struct sk_buff *skb) > /* also handle sequence number wrap-around */ > if (WRAPPED(ack, opt->ack_recv)) > opt->ack_recv = ack; > + } else { > + headersize -= sizeof(header->ack); > } > - > /* test if payload present */ > if (!PPTP_GRE_IS_S(header->flags)) > goto drop; > > - headersize = sizeof(*header); > payload_len = ntohs(header->payload_len); > seq = ntohl(header->seq); > > - /* no ack present? */ > - if (!PPTP_GRE_IS_A(header->ver)) > - headersize -= sizeof(header->ack); > /* check for incomplete packet (length smaller than expected) */ > - if (skb->len - headersize < payload_len) > + if (!pskb_may_pull(skb, headersize + payload_len)) > goto drop; > > payload = skb->data + headersize; > > > This patch has been working for me. 5 days uptime, with no crashes. Thanks for your help! Bradley Peterson ------------------------------------------------------------------------------ The demand for IT networking professionals continues to grow, and the demand for specialized networking skills is growing even more rapidly. Take a complimentary Learning@Cisco Self-Assessment and learn about Cisco certifications, training, and career opportunities. http://p.sf.net/sfu/cisco-dev2dev _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2011-10-24 21:59 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-10-17 22:19 BUG in skb_pull with e1000e, PPTP, and L2TP Bradley Peterson 2011-10-17 22:25 ` Stephen Hemminger 2011-10-17 22:30 ` Bradley Peterson 2011-10-18 2:24 ` Eric Dumazet 2011-10-18 3:51 ` Eric Dumazet 2011-10-18 3:59 ` Eric Dumazet 2011-10-19 7:31 ` David Miller 2011-10-19 7:47 ` Eric Dumazet 2011-10-19 7:51 ` David Miller 2011-10-24 21:59 ` Bradley Peterson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).