* [BUG] xfrm: unable to handle kernel NULL pointer dereference
@ 2018-11-10 19:34 Jean-Philippe Menil
2018-11-16 18:48 ` Lennert Buytenhek
0 siblings, 1 reply; 4+ messages in thread
From: Jean-Philippe Menil @ 2018-11-10 19:34 UTC (permalink / raw)
To: steffen.klassert; +Cc: herbert, davem, netdev, kuznet, yoshfuji
[-- Attachment #1: Type: text/plain, Size: 3005 bytes --]
Hi guys,
we're seeing unexpected crashes from kernel 4.15 to 4.18.17, using IPsec
VTI interfaces, on several vpn hosts, since upgrade from 4.4.
Attached, the offended oops against 4.18.
Output of decodedecode:
[ 37.134864] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00
00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe
<f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
All code
========
0: 8b 44 24 70 mov 0x70(%rsp),%eax
4: 0f c8 bswap %eax
6: 89 87 b4 00 00 00 mov %eax,0xb4(%rdi)
c: 48 8b 86 20 05 00 00 mov 0x520(%rsi),%rax
13: 8b 80 f8 14 00 00 mov 0x14f8(%rax),%eax
19: 85 c0 test %eax,%eax
1b: 75 05 jne 0x22
1d: 48 85 d2 test %rdx,%rdx
20: 74 0e je 0x30
22: 48 8b 43 58 mov 0x58(%rbx),%rax
26: 48 83 e0 fe and $0xfffffffffffffffe,%rax
2a:* f6 40 38 04 testb $0x4,0x38(%rax) <--
trapping instruction
2e: 74 7d je 0xad
30: 44 89 b3 b4 00 00 00 mov %r14d,0xb4(%rbx)
37: 49 8b 44 24 20 mov 0x20(%r12),%rax
3c: 48 rex.W
3d: 39 .byte 0x39
3e: 86 20 xchg %ah,(%rax)
Code starting with the faulting instruction
===========================================
0: f6 40 38 04 testb $0x4,0x38(%rax)
4: 74 7d je 0x83
6: 44 89 b3 b4 00 00 00 mov %r14d,0xb4(%rbx)
d: 49 8b 44 24 20 mov 0x20(%r12),%rax
12: 48 rex.W
13: 39 .byte 0x39
14: 86 20 xchg %ah,(%rax)
if my understanding is correct, we fail here:
/build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/include/net/xfrm.h:
1169 return (!net->xfrm.policy_count[dir] && !skb->sp) ||
0x0000000000000b19 <+185>: testb $0x4,0x38(%rax)
0x0000000000000b1d <+189>: je 0xb9c <vti_rcv_cb+316>
(gdb) list *0x0000000000000b19
0xb19 is in vti_rcv_cb
(/build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/include/net/xfrm.h:1169).
1164 int ndir = dir | (reverse ? XFRM_POLICY_MASK + 1 : 0);
1165
1166 if (sk && sk->sk_policy[XFRM_POLICY_IN])
1167 return __xfrm_policy_check(sk, ndir, skb, family);
1168
1169 return (!net->xfrm.policy_count[dir] && !skb->sp) ||
1170 (skb_dst(skb)->flags & DST_NOPOLICY) ||
1171 __xfrm_policy_check(sk, ndir, skb, family);
1172 }
1173
I really have hard time to understand why skb seem to be freed twice.
I'm not able to repeat the bug in lab, but it happened regulary in prod,
seem to depend of the workload.
Any help will be appreciated.
Let me know if you need further informations.
Regards,
Jean-Philippe
[-- Attachment #2: oops.txt --]
[-- Type: text/plain, Size: 14750 bytes --]
[ 31.154360] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
[ 31.162233] PGD 0 P4D 0
[ 31.164786] Oops: 0000 [#1] SMP PTI
[ 31.168291] CPU: 5 PID: 42 Comm: ksoftirqd/5 Not tainted 4.18.0-11-generic #12~18.04.1-Ubuntu
[ 31.176854] Hardware name: Supermicro Super Server/X10SDV-4C-7TP4F, BIOS 1.0b 11/21/2016
[ 31.184980] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 31.189962] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 31.208916] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 31.214160] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 31.221328] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 31.228485] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 31.235643] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 31.242803] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 31.249964] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000
[ 31.258077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 31.263848] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 31.271004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 31.278163] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 31.285320] Call Trace:
[ 31.287789] xfrm4_rcv_cb+0x4a/0x70
[ 31.291297] xfrm_input+0x58f/0x8f0
[ 31.294807] vti_input+0xaa/0x110 [ip_vti]
[ 31.298926] vti_rcv+0x33/0x3c [ip_vti]
[ 31.302783] xfrm4_esp_rcv+0x39/0x50
[ 31.306375] ip_local_deliver_finish+0x62/0x200
[ 31.310923] ip_local_deliver+0xdf/0xf0
[ 31.314775] ? ip_rcv_finish+0x420/0x420
[ 31.318718] ip_rcv_finish+0x126/0x420
[ 31.322486] ip_rcv+0x28f/0x360
[ 31.325655] ? inet_del_offload+0x40/0x40
[ 31.329686] __netif_receive_skb_core+0x48c/0xb70
[ 31.334413] ? kmem_cache_alloc+0xb4/0x1d0
[ 31.338532] ? __build_skb+0x2b/0xf0
[ 31.342128] __netif_receive_skb+0x18/0x60
[ 31.346244] ? __netif_receive_skb+0x18/0x60
[ 31.350536] netif_receive_skb_internal+0x45/0xe0
[ 31.355263] napi_gro_receive+0xc5/0xf0
[ 31.359141] mlx5e_handle_rx_cqe+0x1b2/0x5d0 [mlx5_core]
[ 31.364476] ? skb_release_all+0x24/0x30
[ 31.368430] mlx5e_poll_rx_cq+0xd3/0x990 [mlx5_core]
[ 31.373432] mlx5e_napi_poll+0x9b/0xc60 [mlx5_core]
[ 31.378333] ? __switch_to_asm+0x34/0x70
[ 31.382270] ? __switch_to_asm+0x40/0x70
[ 31.386214] ? __switch_to_asm+0x34/0x70
[ 31.391056] ? __switch_to_asm+0x40/0x70
[ 31.395905] ? __switch_to_asm+0x34/0x70
[ 31.400743] net_rx_action+0x140/0x3a0
[ 31.405379] ? __switch_to+0xad/0x500
[ 31.409887] __do_softirq+0xe4/0x2bb
[ 31.414448] run_ksoftirqd+0x2b/0x40
[ 31.418862] smpboot_thread_fn+0xfc/0x170
[ 31.423700] kthread+0x121/0x140
[ 31.427701] ? sort_range+0x30/0x30
[ 31.432040] ? kthread_create_worker_on_cpu+0x70/0x70
[ 31.437816] ret_from_fork+0x35/0x40
[ 31.442219] Modules linked in: esp6 authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo ip_vti ip_tunnel ip6_vti ip6_tunnel tunnel6 8021q garp mrp stp llc bonding ipt_REJECT nf_reject_ipv4 nfnetlink_log n
fnetlink xt_NFLOG xt_hl xt_limit xt_nat xt_TCPMSS xt_HL xt_comment xt_tcpudp xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_connmark xt_mark iptable_mangle xt_CT nf_conntrack xt_addrtype iptable_raw bpfilter ipmi_ssif gpio_
ich intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate intel_rapl_perf input_leds joydev mei_me intel_pch_thermal ioatdma mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid sch_fq_codel
[ 31.519488] ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear mlx5_ib ib_uverbs ib
_core raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast pcbc ttm drm_kms_helper aesni_intel syscopyarea aes_x86_64 sysfillrect mxm_wmi crypto_simd sysimgblt cryptd glue_helper fb_sys_fops mlx5_core ixgbe igb mpt3sas drm ahci tls libahci i2c_algo_bit m
lxfw raid_class dca devlink mdio scsi_transport_sas wmi
[ 31.578877] CR2: 0000000000000038
[ 31.583249] ---[ end trace c4bada38847a0075 ]---
[ 31.737166] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 31.737167] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 31.737209] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 31.737212] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 31.737213] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 31.737216] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 31.737219] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 31.737220] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 31.737222] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000
[ 31.737224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 31.737225] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 31.737227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 31.737228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 31.737230] Kernel panic - not syncing: Fatal exception in interrupt
[ 31.737264] Kernel Offset: 0x3c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 36.558279] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[ 36.566769] ------------[ cut here ]------------
[ 36.572306] sched: Unexpected reschedule of offline CPU#3!
[ 36.578714] WARNING: CPU: 5 PID: 42 at /build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/arch/x86/kernel/smp.c:128 native_smp_send_resched
ule+0x3a/0x40
[ 36.594262] Modules linked in: esp6 authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
esp4 ah4 af_key xfrm_algo ip_vti ip_tunnel ip6_vti ip6_tunnel tunnel6 8021q garp mrp stp llc bonding ipt_REJECT nf_reject_ipv4 nfnetlink_log n
fnetlink xt_NFLOG xt_hl xt_limit xt_nat xt_TCPMSS xt_HL xt_comment xt_tcpudp xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack
_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_connmark xt_mark iptable_mangle xt_CT nf_conntrack xt_addrtype iptable_raw bpfilter ipmi_ssif gpio_
ich intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate intel_rapl_perf input_leds joydev m
ei_me intel_pch_thermal ioatdma mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid sch_fq_codel
[ 36.673307] ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compres
s raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear mlx5_ib ib_uverbs ib
_core raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast pcbc ttm drm_kms_helper aesni_intel syscopyarea aes_x
86_64 sysfillrect mxm_wmi crypto_simd sysimgblt cryptd glue_helper fb_sys_fops mlx5_core ixgbe igb mpt3sas drm ahci tls libahci i2c_algo_bit m
lxfw raid_class dca devlink mdio scsi_transport_sas wmi
[ 36.733827] CPU: 5 PID: 42 Comm: ksoftirqd/5 Tainted: G D 4.18.0-11-generic #12~18.04.1-Ubuntu
[ 36.745027] Hardware name: Supermicro Super Server/X10SDV-4C-7TP4F, BIOS 1.0b 11/21/2016
[ 36.754392] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 36.761143] Code: c6 62 01 73 17 48 8b 05 24 d2 17 01 be fd 00 00 00 48 8b 40 30 e8 96 96 ba 00 5d c3 89 fe 48 c7 c7 90 2a cd 85 e8 26 4a 0
3 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec 20 65 48
[ 36.782546] RSP: 0018:ffff9a35bfd43b68 EFLAGS: 00010082
[ 36.789019] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000006
[ 36.797379] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9a35bfd564b0
[ 36.805721] RBP: ffff9a35bfd43b68 R08: 00000000000004b7 R09: 0000000000cdcdcd
[ 36.814046] R10: 0000000000000324 R11: 00000000ffffffff R12: ffff9a35bfce2c40
[ 36.822350] R13: ffff9a3598e35c00 R14: 0000000000000008 R15: 0000000000000003
[ 36.830637] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000
[ 36.839869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 36.846742] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 36.855009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 36.863275] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 36.871510] Call Trace:
[ 36.875048] <IRQ>
[ 36.878136] resched_curr+0x5d/0xc0
[ 36.882689] check_preempt_wakeup+0x130/0x240
[ 36.888116] check_preempt_curr+0x2d/0x90
[ 36.893184] ttwu_do_wakeup+0x1e/0x140
[ 36.897984] ttwu_do_activate+0x77/0x80
[ 36.902857] try_to_wake_up+0x1d6/0x450
[ 36.907726] ? __netif_receive_skb_core+0x48c/0xb70
[ 36.913625] default_wake_function+0x12/0x20
[ 36.918914] __wake_up_common+0x73/0x130
[ 36.923851] __wake_up_locked+0x16/0x20
[ 36.928706] ep_poll_callback+0xcb/0x2b0
[ 36.933634] __wake_up_common+0x73/0x130
[ 36.938549] __wake_up_common_lock+0x80/0xc0
[ 36.943787] ? tick_sched_do_timer+0x60/0x60
[ 36.949011] __wake_up+0x13/0x20
[ 36.953190] wake_up_klogd_work_func+0x40/0x60
[ 36.958560] irq_work_run_list+0x52/0x80
[ 36.963388] irq_work_tick+0x3b/0x50
[ 36.967841] update_process_times+0x42/0x60
[ 36.972883] tick_sched_handle+0x25/0x70
[ 36.977644] tick_sched_timer+0x3c/0x80
[ 36.982285] __hrtimer_run_queues+0x10f/0x280
[ 36.987429] hrtimer_interrupt+0xe7/0x240
[ 36.992202] ? rcu_irq_exit+0x1d/0x20
[ 36.996602] smp_apic_timer_interrupt+0x6f/0x130
[ 37.001929] apic_timer_interrupt+0xf/0x20
[ 37.006712] </IRQ>
[ 37.009476] RIP: 0010:panic+0x1fe/0x244
[ 37.013951] Code: eb a6 83 3d 37 05 8f 01 00 74 05 e8 d0 73 02 00 48 c7 c6 20 f1 57 86 48 c7 c7 c8 cd cd 85 e8 f0 78 06 00 fb 66 0f 1f 44 0
0 00 <31> db e8 2f c6 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 df 04 8f
[ 37.034119] RSP: 0018:ffffbc61832e3668 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 37.042324] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 37.050086] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9a35bfd564b0
[ 37.057835] RBP: ffffbc61832e36e0 R08: 00000000000004b5 R09: 0000000000cdcdcd
[ 37.065570] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000000
[ 37.006712] </IRQ>
[ 37.009476] RIP: 0010:panic+0x1fe/0x244
[ 37.013951] Code: eb a6 83 3d 37 05 8f 01 00 74 05 e8 d0 73 02 00 48 c7 c6 20 f1 57 86 48 c7 c7 c8 cd cd 85 e8 f0 78 06 00 fb 66 0f 1f 44 00 00 <31> db e8 2f c6 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 df 04 8f
[ 37.034119] RSP: 0018:ffffbc61832e3668 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 37.042324] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 37.050086] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9a35bfd564b0
[ 37.057835] RBP: ffffbc61832e36e0 R08: 00000000000004b5 R09: 0000000000cdcdcd
[ 37.065570] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000000
[ 37.073287] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000000
[ 37.080999] ? panic+0x1f7/0x244
[ 37.084812] oops_end+0xce/0xe0
[ 37.088531] no_context+0x17c/0x400
[ 37.092585] ? skcipher_walk_first+0x4c/0x110
[ 37.097500] __bad_area_nosemaphore+0x115/0x1d0
[ 37.102591] ? cbc_decrypt+0xb1/0xe0 [aesni_intel]
[ 37.107926] bad_area_nosemaphore+0x14/0x20
[ 37.112654] __do_page_fault+0xd4/0x4d0
[ 37.117030] do_page_fault+0x2d/0xf0
[ 37.121143] ? skb_copy_bits+0x61/0x260
[ 37.125511] page_fault+0x1e/0x30
[ 37.129362] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 37.134864] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 37.154892] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 37.160711] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 37.168451] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 37.176196] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 37.183950] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 37.191716] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 37.199481] ? esp_input_done2+0x6b/0x320 [esp4]
[ 37.204740] xfrm4_rcv_cb+0x4a/0x70
[ 37.208873] xfrm_input+0x58f/0x8f0
[ 37.212996] vti_input+0xaa/0x110 [ip_vti]
[ 37.217735] vti_rcv+0x33/0x3c [ip_vti]
[ 37.222209] xfrm4_esp_rcv+0x39/0x50
[ 37.226424] ip_local_deliver_finish+0x62/0x200
[ 37.231599] ip_local_deliver+0xdf/0xf0
[ 37.236072] ? ip_rcv_finish+0x420/0x420
[ 37.240635] ip_rcv_finish+0x126/0x420
[ 37.245025] ip_rcv+0x28f/0x360
[ 37.248801] ? inet_del_offload+0x40/0x40
[ 37.253457] __netif_receive_skb_core+0x48c/0xb70
[ 37.258807] ? kmem_cache_alloc+0xb4/0x1d0
[ 37.263541] ? __build_skb+0x2b/0xf0
[ 37.267749] __netif_receive_skb+0x18/0x60
[ 37.272483] ? __netif_receive_skb+0x18/0x60
[ 37.277391] netif_receive_skb_internal+0x45/0xe0
[ 37.282732] napi_gro_receive+0xc5/0xf0
[ 37.287232] mlx5e_handle_rx_cqe+0x1b2/0x5d0 [mlx5_core]
[ 37.293170] ? skb_release_all+0x24/0x30
[ 37.297708] mlx5e_poll_rx_cq+0xd3/0x990 [mlx5_core]
[ 37.303281] mlx5e_napi_poll+0x9b/0xc60 [mlx5_core]
[ 37.308752] ? __switch_to_asm+0x34/0x70
[ 37.313266] ? __switch_to_asm+0x40/0x70
[ 37.317768] ? __switch_to_asm+0x34/0x70
[ 37.322259] ? __switch_to_asm+0x40/0x70
[ 37.326742] ? __switch_to_asm+0x34/0x70
[ 37.331214] net_rx_action+0x140/0x3a0
[ 37.335519] ? __switch_to+0xad/0x500
[ 37.339728] __do_softirq+0xe4/0x2bb
[ 37.343852] run_ksoftirqd+0x2b/0x40
[ 37.347971] smpboot_thread_fn+0xfc/0x170
[ 37.352522] kthread+0x121/0x140
[ 37.356282] ? sort_range+0x30/0x30
[ 37.360303] ? kthread_create_worker_on_cpu+0x70/0x70
[ 37.365895] ret_from_fork+0x35/0x40
[ 37.370010] ---[ end trace c4bada38847a0076 ]---
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [BUG] xfrm: unable to handle kernel NULL pointer dereference 2018-11-10 19:34 [BUG] xfrm: unable to handle kernel NULL pointer dereference Jean-Philippe Menil @ 2018-11-16 18:48 ` Lennert Buytenhek 2018-11-16 19:12 ` Steffen Klassert 0 siblings, 1 reply; 4+ messages in thread From: Lennert Buytenhek @ 2018-11-16 18:48 UTC (permalink / raw) To: steffen.klassert, herbert Cc: Jean-Philippe Menil, davem, netdev, kuznet, yoshfuji On Sat, Nov 10, 2018 at 08:34:34PM +0100, Jean-Philippe Menil wrote: > we're seeing unexpected crashes from kernel 4.15 to 4.18.17, using > IPsec VTI interfaces, on several vpn hosts, since upgrade from 4.4. I looked into this with Jean-Philippe, and it appears to be crashing on a NULL pointer dereference in the inlined xfrm_policy_check() call in vti_rcv_cb(), and specifically on the skb_dst(skb) dereference in __xfrm_policy_check2(): return (!net->xfrm.policy_count[dir] && !skb->sp) || (skb_dst(skb)->flags & DST_NOPOLICY) || <===== __xfrm_policy_check(sk, ndir, skb, family); Commit 9e1437937807 ("xfrm: Fix NULL pointer dereference when skb_dst_force clears the dst_entry.") fixes a very similar problem on the output and forward paths, but our issue seems to be triggering on the input path. This hack patch seems to make the crashes go away, and the printk added triggers with approximately the same regularity as the crashes used to occur, so the fix from 9e1437937807 probably needs to be extended to the input path somewhat like this. Thanks! diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index 352abca2605f..c666e29441b4 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -381,6 +381,12 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) XFRM_SKB_CB(skb)->seq.input.hi = seq_hi; skb_dst_force(skb); + if (!skb_dst(skb)) { + if (net_ratelimit()) + printk(KERN_CRIT "OH CRAP\n"); + goto drop; + } + dev_hold(skb->dev); if (crypto_done) > Attached, the offended oops against 4.18. > > Output of decodedecode: > > [ 37.134864] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 > 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 > 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20 > All code > ======== > 0: 8b 44 24 70 mov 0x70(%rsp),%eax > 4: 0f c8 bswap %eax > 6: 89 87 b4 00 00 00 mov %eax,0xb4(%rdi) > c: 48 8b 86 20 05 00 00 mov 0x520(%rsi),%rax > 13: 8b 80 f8 14 00 00 mov 0x14f8(%rax),%eax > 19: 85 c0 test %eax,%eax > 1b: 75 05 jne 0x22 > 1d: 48 85 d2 test %rdx,%rdx > 20: 74 0e je 0x30 > 22: 48 8b 43 58 mov 0x58(%rbx),%rax > 26: 48 83 e0 fe and $0xfffffffffffffffe,%rax > 2a:* f6 40 38 04 testb $0x4,0x38(%rax) <-- trapping > instruction > 2e: 74 7d je 0xad > 30: 44 89 b3 b4 00 00 00 mov %r14d,0xb4(%rbx) > 37: 49 8b 44 24 20 mov 0x20(%r12),%rax > 3c: 48 rex.W > 3d: 39 .byte 0x39 > 3e: 86 20 xchg %ah,(%rax) > > Code starting with the faulting instruction > =========================================== > 0: f6 40 38 04 testb $0x4,0x38(%rax) > 4: 74 7d je 0x83 > 6: 44 89 b3 b4 00 00 00 mov %r14d,0xb4(%rbx) > d: 49 8b 44 24 20 mov 0x20(%r12),%rax > 12: 48 rex.W > 13: 39 .byte 0x39 > 14: 86 20 xchg %ah,(%rax) > > > if my understanding is correct, we fail here: > > /build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/include/net/xfrm.h: > 1169 return (!net->xfrm.policy_count[dir] && !skb->sp) || > 0x0000000000000b19 <+185>: testb $0x4,0x38(%rax) > 0x0000000000000b1d <+189>: je 0xb9c <vti_rcv_cb+316> > > (gdb) list *0x0000000000000b19 > 0xb19 is in vti_rcv_cb > (/build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/include/net/xfrm.h:1169). > 1164 int ndir = dir | (reverse ? XFRM_POLICY_MASK + 1 : 0); > 1165 > 1166 if (sk && sk->sk_policy[XFRM_POLICY_IN]) > 1167 return __xfrm_policy_check(sk, ndir, skb, family); > 1168 > 1169 return (!net->xfrm.policy_count[dir] && !skb->sp) || > 1170 (skb_dst(skb)->flags & DST_NOPOLICY) || > 1171 __xfrm_policy_check(sk, ndir, skb, family); > 1172 } > 1173 > > I really have hard time to understand why skb seem to be freed twice. > > I'm not able to repeat the bug in lab, but it happened regulary in prod, > seem to depend of the workload. > > Any help will be appreciated. > > Let me know if you need further informations. > > Regards, > > Jean-Philippe > [ 31.154360] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 > [ 31.162233] PGD 0 P4D 0 > [ 31.164786] Oops: 0000 [#1] SMP PTI > [ 31.168291] CPU: 5 PID: 42 Comm: ksoftirqd/5 Not tainted 4.18.0-11-generic #12~18.04.1-Ubuntu > [ 31.176854] Hardware name: Supermicro Super Server/X10SDV-4C-7TP4F, BIOS 1.0b 11/21/2016 > [ 31.184980] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti] > [ 31.189962] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20 > [ 31.208916] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246 > [ 31.214160] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002 > [ 31.221328] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00 > [ 31.228485] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b > [ 31.235643] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900 > [ 31.242803] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00 > [ 31.249964] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000 > [ 31.258077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 31.263848] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0 > [ 31.271004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 31.278163] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 31.285320] Call Trace: > [ 31.287789] xfrm4_rcv_cb+0x4a/0x70 > [ 31.291297] xfrm_input+0x58f/0x8f0 > [ 31.294807] vti_input+0xaa/0x110 [ip_vti] > [ 31.298926] vti_rcv+0x33/0x3c [ip_vti] > [ 31.302783] xfrm4_esp_rcv+0x39/0x50 > [ 31.306375] ip_local_deliver_finish+0x62/0x200 > [ 31.310923] ip_local_deliver+0xdf/0xf0 > [ 31.314775] ? ip_rcv_finish+0x420/0x420 > [ 31.318718] ip_rcv_finish+0x126/0x420 > [ 31.322486] ip_rcv+0x28f/0x360 > [ 31.325655] ? inet_del_offload+0x40/0x40 > [ 31.329686] __netif_receive_skb_core+0x48c/0xb70 > [ 31.334413] ? kmem_cache_alloc+0xb4/0x1d0 > [ 31.338532] ? __build_skb+0x2b/0xf0 > [ 31.342128] __netif_receive_skb+0x18/0x60 > [ 31.346244] ? __netif_receive_skb+0x18/0x60 > [ 31.350536] netif_receive_skb_internal+0x45/0xe0 > [ 31.355263] napi_gro_receive+0xc5/0xf0 > [ 31.359141] mlx5e_handle_rx_cqe+0x1b2/0x5d0 [mlx5_core] > [ 31.364476] ? skb_release_all+0x24/0x30 > [ 31.368430] mlx5e_poll_rx_cq+0xd3/0x990 [mlx5_core] > [ 31.373432] mlx5e_napi_poll+0x9b/0xc60 [mlx5_core] > [ 31.378333] ? __switch_to_asm+0x34/0x70 > [ 31.382270] ? __switch_to_asm+0x40/0x70 > [ 31.386214] ? __switch_to_asm+0x34/0x70 > [ 31.391056] ? __switch_to_asm+0x40/0x70 > [ 31.395905] ? __switch_to_asm+0x34/0x70 > [ 31.400743] net_rx_action+0x140/0x3a0 > [ 31.405379] ? __switch_to+0xad/0x500 > [ 31.409887] __do_softirq+0xe4/0x2bb > [ 31.414448] run_ksoftirqd+0x2b/0x40 > [ 31.418862] smpboot_thread_fn+0xfc/0x170 > [ 31.423700] kthread+0x121/0x140 > [ 31.427701] ? sort_range+0x30/0x30 > [ 31.432040] ? kthread_create_worker_on_cpu+0x70/0x70 > [ 31.437816] ret_from_fork+0x35/0x40 > [ 31.442219] Modules linked in: esp6 authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo ip_vti ip_tunnel ip6_vti ip6_tunnel tunnel6 8021q garp mrp stp llc bonding ipt_REJECT nf_reject_ipv4 nfnetlink_log n > fnetlink xt_NFLOG xt_hl xt_limit xt_nat xt_TCPMSS xt_HL xt_comment xt_tcpudp xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_connmark xt_mark iptable_mangle xt_CT nf_conntrack xt_addrtype iptable_raw bpfilter ipmi_ssif gpio_ > ich intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate intel_rapl_perf input_leds joydev mei_me intel_pch_thermal ioatdma mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid sch_fq_codel > [ 31.519488] ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear mlx5_ib ib_uverbs ib > _core raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast pcbc ttm drm_kms_helper aesni_intel syscopyarea aes_x86_64 sysfillrect mxm_wmi crypto_simd sysimgblt cryptd glue_helper fb_sys_fops mlx5_core ixgbe igb mpt3sas drm ahci tls libahci i2c_algo_bit m > lxfw raid_class dca devlink mdio scsi_transport_sas wmi > [ 31.578877] CR2: 0000000000000038 > [ 31.583249] ---[ end trace c4bada38847a0075 ]--- > [ 31.737166] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti] > [ 31.737167] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20 > [ 31.737209] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246 > [ 31.737212] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002 > [ 31.737213] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00 > [ 31.737216] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b > [ 31.737219] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900 > [ 31.737220] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00 > [ 31.737222] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000 > [ 31.737224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 31.737225] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0 > [ 31.737227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 31.737228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 31.737230] Kernel panic - not syncing: Fatal exception in interrupt > [ 31.737264] Kernel Offset: 0x3c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > [ 36.558279] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- > [ 36.566769] ------------[ cut here ]------------ > [ 36.572306] sched: Unexpected reschedule of offline CPU#3! > [ 36.578714] WARNING: CPU: 5 PID: 42 at /build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/arch/x86/kernel/smp.c:128 native_smp_send_resched > ule+0x3a/0x40 > [ 36.594262] Modules linked in: esp6 authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp > esp4 ah4 af_key xfrm_algo ip_vti ip_tunnel ip6_vti ip6_tunnel tunnel6 8021q garp mrp stp llc bonding ipt_REJECT nf_reject_ipv4 nfnetlink_log n > fnetlink xt_NFLOG xt_hl xt_limit xt_nat xt_TCPMSS xt_HL xt_comment xt_tcpudp xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack > _ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_connmark xt_mark iptable_mangle xt_CT nf_conntrack xt_addrtype iptable_raw bpfilter ipmi_ssif gpio_ > ich intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate intel_rapl_perf input_leds joydev m > ei_me intel_pch_thermal ioatdma mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid sch_fq_codel > [ 36.673307] ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compres > s raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear mlx5_ib ib_uverbs ib > _core raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast pcbc ttm drm_kms_helper aesni_intel syscopyarea aes_x > 86_64 sysfillrect mxm_wmi crypto_simd sysimgblt cryptd glue_helper fb_sys_fops mlx5_core ixgbe igb mpt3sas drm ahci tls libahci i2c_algo_bit m > lxfw raid_class dca devlink mdio scsi_transport_sas wmi > [ 36.733827] CPU: 5 PID: 42 Comm: ksoftirqd/5 Tainted: G D 4.18.0-11-generic #12~18.04.1-Ubuntu > [ 36.745027] Hardware name: Supermicro Super Server/X10SDV-4C-7TP4F, BIOS 1.0b 11/21/2016 > [ 36.754392] RIP: 0010:native_smp_send_reschedule+0x3a/0x40 > [ 36.761143] Code: c6 62 01 73 17 48 8b 05 24 d2 17 01 be fd 00 00 00 48 8b 40 30 e8 96 96 ba 00 5d c3 89 fe 48 c7 c7 90 2a cd 85 e8 26 4a 0 > 3 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec 20 65 48 > [ 36.782546] RSP: 0018:ffff9a35bfd43b68 EFLAGS: 00010082 > [ 36.789019] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000006 > [ 36.797379] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9a35bfd564b0 > [ 36.805721] RBP: ffff9a35bfd43b68 R08: 00000000000004b7 R09: 0000000000cdcdcd > [ 36.814046] R10: 0000000000000324 R11: 00000000ffffffff R12: ffff9a35bfce2c40 > [ 36.822350] R13: ffff9a3598e35c00 R14: 0000000000000008 R15: 0000000000000003 > [ 36.830637] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000 > [ 36.839869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 36.846742] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0 > [ 36.855009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 36.863275] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 36.871510] Call Trace: > [ 36.875048] <IRQ> > [ 36.878136] resched_curr+0x5d/0xc0 > [ 36.882689] check_preempt_wakeup+0x130/0x240 > [ 36.888116] check_preempt_curr+0x2d/0x90 > [ 36.893184] ttwu_do_wakeup+0x1e/0x140 > [ 36.897984] ttwu_do_activate+0x77/0x80 > [ 36.902857] try_to_wake_up+0x1d6/0x450 > [ 36.907726] ? __netif_receive_skb_core+0x48c/0xb70 > [ 36.913625] default_wake_function+0x12/0x20 > [ 36.918914] __wake_up_common+0x73/0x130 > [ 36.923851] __wake_up_locked+0x16/0x20 > [ 36.928706] ep_poll_callback+0xcb/0x2b0 > [ 36.933634] __wake_up_common+0x73/0x130 > [ 36.938549] __wake_up_common_lock+0x80/0xc0 > [ 36.943787] ? tick_sched_do_timer+0x60/0x60 > [ 36.949011] __wake_up+0x13/0x20 > [ 36.953190] wake_up_klogd_work_func+0x40/0x60 > [ 36.958560] irq_work_run_list+0x52/0x80 > [ 36.963388] irq_work_tick+0x3b/0x50 > [ 36.967841] update_process_times+0x42/0x60 > [ 36.972883] tick_sched_handle+0x25/0x70 > [ 36.977644] tick_sched_timer+0x3c/0x80 > [ 36.982285] __hrtimer_run_queues+0x10f/0x280 > [ 36.987429] hrtimer_interrupt+0xe7/0x240 > [ 36.992202] ? rcu_irq_exit+0x1d/0x20 > [ 36.996602] smp_apic_timer_interrupt+0x6f/0x130 > [ 37.001929] apic_timer_interrupt+0xf/0x20 > [ 37.006712] </IRQ> > [ 37.009476] RIP: 0010:panic+0x1fe/0x244 > [ 37.013951] Code: eb a6 83 3d 37 05 8f 01 00 74 05 e8 d0 73 02 00 48 c7 c6 20 f1 57 86 48 c7 c7 c8 cd cd 85 e8 f0 78 06 00 fb 66 0f 1f 44 0 > 0 00 <31> db e8 2f c6 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 df 04 8f > [ 37.034119] RSP: 0018:ffffbc61832e3668 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13 > [ 37.042324] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006 > [ 37.050086] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9a35bfd564b0 > [ 37.057835] RBP: ffffbc61832e36e0 R08: 00000000000004b5 R09: 0000000000cdcdcd > [ 37.065570] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000000 > [ 37.006712] </IRQ> > [ 37.009476] RIP: 0010:panic+0x1fe/0x244 > [ 37.013951] Code: eb a6 83 3d 37 05 8f 01 00 74 05 e8 d0 73 02 00 48 c7 c6 20 f1 57 86 48 c7 c7 c8 cd cd 85 e8 f0 78 06 00 fb 66 0f 1f 44 00 00 <31> db e8 2f c6 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 df 04 8f > [ 37.034119] RSP: 0018:ffffbc61832e3668 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13 > [ 37.042324] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006 > [ 37.050086] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9a35bfd564b0 > [ 37.057835] RBP: ffffbc61832e36e0 R08: 00000000000004b5 R09: 0000000000cdcdcd > [ 37.065570] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000000 > [ 37.073287] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000000 > [ 37.080999] ? panic+0x1f7/0x244 > [ 37.084812] oops_end+0xce/0xe0 > [ 37.088531] no_context+0x17c/0x400 > [ 37.092585] ? skcipher_walk_first+0x4c/0x110 > [ 37.097500] __bad_area_nosemaphore+0x115/0x1d0 > [ 37.102591] ? cbc_decrypt+0xb1/0xe0 [aesni_intel] > [ 37.107926] bad_area_nosemaphore+0x14/0x20 > [ 37.112654] __do_page_fault+0xd4/0x4d0 > [ 37.117030] do_page_fault+0x2d/0xf0 > [ 37.121143] ? skb_copy_bits+0x61/0x260 > [ 37.125511] page_fault+0x1e/0x30 > [ 37.129362] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti] > [ 37.134864] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20 > [ 37.154892] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246 > [ 37.160711] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002 > [ 37.168451] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00 > [ 37.176196] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b > [ 37.183950] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900 > [ 37.191716] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00 > [ 37.199481] ? esp_input_done2+0x6b/0x320 [esp4] > [ 37.204740] xfrm4_rcv_cb+0x4a/0x70 > [ 37.208873] xfrm_input+0x58f/0x8f0 > [ 37.212996] vti_input+0xaa/0x110 [ip_vti] > [ 37.217735] vti_rcv+0x33/0x3c [ip_vti] > [ 37.222209] xfrm4_esp_rcv+0x39/0x50 > [ 37.226424] ip_local_deliver_finish+0x62/0x200 > [ 37.231599] ip_local_deliver+0xdf/0xf0 > [ 37.236072] ? ip_rcv_finish+0x420/0x420 > [ 37.240635] ip_rcv_finish+0x126/0x420 > [ 37.245025] ip_rcv+0x28f/0x360 > [ 37.248801] ? inet_del_offload+0x40/0x40 > [ 37.253457] __netif_receive_skb_core+0x48c/0xb70 > [ 37.258807] ? kmem_cache_alloc+0xb4/0x1d0 > [ 37.263541] ? __build_skb+0x2b/0xf0 > [ 37.267749] __netif_receive_skb+0x18/0x60 > [ 37.272483] ? __netif_receive_skb+0x18/0x60 > [ 37.277391] netif_receive_skb_internal+0x45/0xe0 > [ 37.282732] napi_gro_receive+0xc5/0xf0 > [ 37.287232] mlx5e_handle_rx_cqe+0x1b2/0x5d0 [mlx5_core] > [ 37.293170] ? skb_release_all+0x24/0x30 > [ 37.297708] mlx5e_poll_rx_cq+0xd3/0x990 [mlx5_core] > [ 37.303281] mlx5e_napi_poll+0x9b/0xc60 [mlx5_core] > [ 37.308752] ? __switch_to_asm+0x34/0x70 > [ 37.313266] ? __switch_to_asm+0x40/0x70 > [ 37.317768] ? __switch_to_asm+0x34/0x70 > [ 37.322259] ? __switch_to_asm+0x40/0x70 > [ 37.326742] ? __switch_to_asm+0x34/0x70 > [ 37.331214] net_rx_action+0x140/0x3a0 > [ 37.335519] ? __switch_to+0xad/0x500 > [ 37.339728] __do_softirq+0xe4/0x2bb > [ 37.343852] run_ksoftirqd+0x2b/0x40 > [ 37.347971] smpboot_thread_fn+0xfc/0x170 > [ 37.352522] kthread+0x121/0x140 > [ 37.356282] ? sort_range+0x30/0x30 > [ 37.360303] ? kthread_create_worker_on_cpu+0x70/0x70 > [ 37.365895] ret_from_fork+0x35/0x40 > [ 37.370010] ---[ end trace c4bada38847a0076 ]--- ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [BUG] xfrm: unable to handle kernel NULL pointer dereference 2018-11-16 18:48 ` Lennert Buytenhek @ 2018-11-16 19:12 ` Steffen Klassert 2018-11-22 6:44 ` Steffen Klassert 0 siblings, 1 reply; 4+ messages in thread From: Steffen Klassert @ 2018-11-16 19:12 UTC (permalink / raw) To: Lennert Buytenhek Cc: herbert, Jean-Philippe Menil, davem, netdev, kuznet, yoshfuji On Fri, Nov 16, 2018 at 08:48:00PM +0200, Lennert Buytenhek wrote: > On Sat, Nov 10, 2018 at 08:34:34PM +0100, Jean-Philippe Menil wrote: > > > we're seeing unexpected crashes from kernel 4.15 to 4.18.17, using > > IPsec VTI interfaces, on several vpn hosts, since upgrade from 4.4. > > I looked into this with Jean-Philippe, and it appears to be crashing > on a NULL pointer dereference in the inlined xfrm_policy_check() call > in vti_rcv_cb(), and specifically on the skb_dst(skb) dereference in > __xfrm_policy_check2(): > > return (!net->xfrm.policy_count[dir] && !skb->sp) || > (skb_dst(skb)->flags & DST_NOPOLICY) || <===== > __xfrm_policy_check(sk, ndir, skb, family); > > Commit 9e1437937807 ("xfrm: Fix NULL pointer dereference when > skb_dst_force clears the dst_entry.") fixes a very similar problem on > the output and forward paths, but our issue seems to be triggering on > the input path. Yes, this is the same problem. skb_dst_force() does not really force a refcount anymore, it might clear the dst pointer instead (maybe this function should be renamed). Want to submit a fix? If not I'll go to fix that. Thanks! ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [BUG] xfrm: unable to handle kernel NULL pointer dereference 2018-11-16 19:12 ` Steffen Klassert @ 2018-11-22 6:44 ` Steffen Klassert 0 siblings, 0 replies; 4+ messages in thread From: Steffen Klassert @ 2018-11-22 6:44 UTC (permalink / raw) To: Lennert Buytenhek Cc: herbert, Jean-Philippe Menil, davem, netdev, kuznet, yoshfuji On Fri, Nov 16, 2018 at 08:12:46PM +0100, Steffen Klassert wrote: > On Fri, Nov 16, 2018 at 08:48:00PM +0200, Lennert Buytenhek wrote: > > On Sat, Nov 10, 2018 at 08:34:34PM +0100, Jean-Philippe Menil wrote: > > > > > we're seeing unexpected crashes from kernel 4.15 to 4.18.17, using > > > IPsec VTI interfaces, on several vpn hosts, since upgrade from 4.4. > > > > I looked into this with Jean-Philippe, and it appears to be crashing > > on a NULL pointer dereference in the inlined xfrm_policy_check() call > > in vti_rcv_cb(), and specifically on the skb_dst(skb) dereference in > > __xfrm_policy_check2(): > > > > return (!net->xfrm.policy_count[dir] && !skb->sp) || > > (skb_dst(skb)->flags & DST_NOPOLICY) || <===== > > __xfrm_policy_check(sk, ndir, skb, family); > > > > Commit 9e1437937807 ("xfrm: Fix NULL pointer dereference when > > skb_dst_force clears the dst_entry.") fixes a very similar problem on > > the output and forward paths, but our issue seems to be triggering on > > the input path. > > Yes, this is the same problem. skb_dst_force() does not > really force a refcount anymore, it might clear the dst > pointer instead (maybe this function should be renamed). I plan to apply this patch to the ipsec tree: [PATCH RFC] xfrm: Fix NULL pointer dereference in xfrm_input when skb_dst_force clears the dst_entry. Since commit 222d7dbd258d ("net: prevent dst uses after free") skb_dst_force() might clear the dst_entry attached to the skb. The xfrm code don't expect this to happen, so we crash with a NULL pointer dereference in this case. Fix it by checking skb_dst(skb) for NULL after skb_dst_force() and drop the packet in cast the dst_entry was cleared. We also move the skb_dst_force() to a codepath that is not used when the transformation was offloaded, because in this case we don't have a dst_entry attached to the skb. The output and forwarding path was already fixed by commit 9e1437937807 ("xfrm: Fix NULL pointer dereference when skb_dst_force clears the dst_entry.") Fixes: 222d7dbd258d ("net: prevent dst uses after free") Reported-by: Jean-Philippe Menil <jpmenil@gmail.com> Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com> --- net/xfrm/xfrm_input.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index 684c0bc01e2c..d5635908587f 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -346,6 +346,12 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) skb->sp->xvec[skb->sp->len++] = x; + skb_dst_force(skb); + if (!skb_dst(skb)) { + XFRM_INC_STATS(net, LINUX_MIB_XFRMINERROR); + goto drop; + } + lock: spin_lock(&x->lock); @@ -385,7 +391,6 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) XFRM_SKB_CB(skb)->seq.input.low = seq; XFRM_SKB_CB(skb)->seq.input.hi = seq_hi; - skb_dst_force(skb); dev_hold(skb->dev); if (crypto_done) -- 2.17.1 ^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2018-11-22 17:22 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-11-10 19:34 [BUG] xfrm: unable to handle kernel NULL pointer dereference Jean-Philippe Menil 2018-11-16 18:48 ` Lennert Buytenhek 2018-11-16 19:12 ` Steffen Klassert 2018-11-22 6:44 ` Steffen Klassert
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox