* [4.9.10] ip_route_me_harder() reading off-slab
@ 2017-02-17 5:34 Daniel J Blueman
2017-02-17 6:23 ` Willy Tarreau
2017-02-17 7:39 ` Florian Westphal
0 siblings, 2 replies; 6+ messages in thread
From: Daniel J Blueman @ 2017-02-17 5:34 UTC (permalink / raw)
To: Netdev
Cc: David S. Miller, Eric W. Biederman, David Ahern, Jan Beulich,
Eric Dumazet
When booting a VM in libvirt/KVM attached to a local bridge and KASAN
enabled on 4.9.10, we see a stream of KASAN warnings about off-slab
access [1].
Let me know if you'd like more debug.
Thanks,
Daniel
-- [1]
[ 473.579567] BUG: KASAN: slab-out-of-bounds in
ip_route_me_harder+0xbd5/0xf20 at addr ffff8801e1eb28a8
[ 473.579577] Read of size 1 by task vcselab/10339
[ 473.579590] CPU: 1 PID: 10339 Comm: vcselab Tainted: G B 4.9.10-debug+ #2
[ 473.579596] Hardware name: Dell Inc. XPS 13 9360/0T3FTF, BIOS 1.3.2
01/18/2017
[ 473.579602] ffff880236086ed0 ffffffff8aed83a1 ffff8802324fe6c0
ffff8801e1eb26f8
[ 473.579626] ffff880236086ef8 ffffffff8a849521 ffff880236086f90
ffff8801e1eb26f0
[ 473.579645] ffff8802324fe6c0 ffff880236086f80 ffffffff8a8497ba
ffffffff8a848b2d
[ 473.579662] Call Trace:
[ 473.579667] <IRQ>
[ 473.579685] [<ffffffff8aed83a1>] dump_stack+0x85/0xc4
[ 473.579698] [<ffffffff8a849521>] kasan_object_err+0x21/0x70
[ 473.579709] [<ffffffff8a8497ba>] kasan_report_error+0x1fa/0x500
[ 473.579720] [<ffffffff8a848b2d>] ? kasan_kmalloc+0xad/0xe0
[ 473.579737] [<ffffffff8a849b21>] __asan_report_load1_noabort+0x61/0x70
[ 473.579749] [<ffffffff8bbe42f5>] ? ip_route_me_harder+0xbd5/0xf20
[ 473.579759] [<ffffffff8bbe42f5>] ip_route_me_harder+0xbd5/0xf20
[ 473.579772] [<ffffffff8bbe3720>] ? nf_ip_saveroute+0x320/0x320
[ 473.579785] [<ffffffff8bdd0c05>] ? entry_SYSCALL_64_fastpath+0x23/0xc6
[ 473.579801] [<ffffffffc047846a>] iptable_mangle_hook+0x3da/0x5f0
[iptable_mangle]
[ 473.579814] [<ffffffff8ba4f080>] nf_iterate+0x110/0x2d0
[ 473.579826] [<ffffffff8ba4f336>] nf_hook_slow+0xf6/0x1b0
[ 473.579839] [<ffffffff8ba4f240>] ? nf_iterate+0x2d0/0x2d0
[ 473.579850] [<ffffffff8ba885fb>] ? __ip_local_out+0x28b/0x720
[ 473.579860] [<ffffffff8ba886d6>] __ip_local_out+0x366/0x720
[ 473.579869] [<ffffffff8ba885fb>] ? __ip_local_out+0x28b/0x720
[ 473.579879] [<ffffffff8ba88370>] ? ip_finish_output+0x9b0/0x9b0
[ 473.579894] [<ffffffff8ba83b90>] ?
__ip_flush_pending_frames.isra.43+0x2e0/0x2e0
[ 473.579905] [<ffffffff8ba88aae>] ip_local_out+0x1e/0x130
[ 473.579915] [<ffffffff8ba8910d>] ip_build_and_send_pkt+0x54d/0xad0
[ 473.579927] [<ffffffff8bb1fdb4>] tcp_v4_send_synack+0x184/0x290
[ 473.579937] [<ffffffff8bb1fc30>] ? tcp_v4_send_check+0x90/0x90
[ 473.579950] [<ffffffff8ba9c047>] ? inet_ehash_insert+0x407/0x910
[ 473.579965] [<ffffffff8bad365b>] tcp_conn_request+0x1f5b/0x2a20
[ 473.579977] [<ffffffff8bad1700>] ? tcp_check_space+0x580/0x580
[ 473.579991] [<ffffffff8a40e235>] ? default_wake_function+0x35/0x50
[ 473.580007] [<ffffffff8a48aac0>] ? debug_check_no_locks_freed+0x290/0x290
[ 473.580018] [<ffffffff8a48aac0>] ? debug_check_no_locks_freed+0x290/0x290
[ 473.580031] [<ffffffffc032afc4>] ? ipt_do_table+0xb14/0x1ac0 [ip_tables]
[ 473.580041] [<ffffffff8a48a4cd>] ? trace_hardirqs_on+0xd/0x10
[ 473.580055] [<ffffffff8a38cc50>] ? __local_bh_enable_ip+0x70/0xc0
[ 473.580067] [<ffffffff8bb19b94>] tcp_v4_conn_request+0x134/0x1e0
[ 473.580079] [<ffffffff8bcf39e8>] tcp_v6_conn_request+0x1b8/0x230
[ 473.580089] [<ffffffff8baead5d>] tcp_rcv_state_process+0x61d/0x41a0
[ 473.580101] [<ffffffff8b9d9987>] ? sk_filter_trim_cap+0x2a7/0x6a0
[ 473.580114] [<ffffffff8baea740>] ? tcp_finish_connect+0x600/0x600
[ 473.580125] [<ffffffff8b9d99a6>] ? sk_filter_trim_cap+0x2c6/0x6a0
[ 473.580135] [<ffffffff8b9d97d8>] ? sk_filter_trim_cap+0xf8/0x6a0
[ 473.580145] [<ffffffff8bb16a4a>] ? tcp_md5_do_lookup+0x4a/0x190
[ 473.580157] [<ffffffff8b9d96e0>] ? sk_filter_is_valid_access+0x60/0x60
[ 473.580170] [<ffffffff8bdb3cfa>] ? tcp_v4_inbound_md5_hash+0x139/0x3bb
[ 473.580180] [<ffffffff8bdb3bc1>] ? ncsi_start_dev+0x111/0x111
[ 473.580190] [<ffffffff8bb1dba8>] tcp_v4_do_rcv+0x2c8/0x8c0
[ 473.580201] [<ffffffff8bb22988>] tcp_v4_rcv+0x23a8/0x2fc0
[ 473.580214] [<ffffffffc04005c7>] ? ipv4_confirm+0x117/0x3d0
[nf_conntrack_ipv4]
[ 473.580228] [<ffffffff8ba6f829>] ip_local_deliver_finish+0x2b9/0x970
[ 473.580241] [<ffffffff8ba6f69a>] ? ip_local_deliver_finish+0x12a/0x970
[ 473.580251] [<ffffffff8ba705a4>] ip_local_deliver+0x1b4/0x460
[ 473.580259] [<ffffffff8ba705f2>] ? ip_local_deliver+0x202/0x460
[ 473.580267] [<ffffffff8ba703f0>] ? ip_call_ra_chain+0x510/0x510
[ 473.580280] [<ffffffffc16d8195>] ? iptable_nat_ipv4_in+0x15/0x20
[iptable_nat]
[ 473.580290] [<ffffffff8ba4f002>] ? nf_iterate+0x92/0x2d0
[ 473.580302] [<ffffffff8ba6f570>] ? ip_rcv_finish+0x18e0/0x18e0
[ 473.580313] [<ffffffff8ba4f336>] ? nf_hook_slow+0xf6/0x1b0
[ 473.580323] [<ffffffff8ba6e2e5>] ip_rcv_finish+0x655/0x18e0
[ 473.580331] [<ffffffff8ba7122b>] ? ip_rcv+0x9db/0x1280
[ 473.580341] [<ffffffff8ba71093>] ip_rcv+0x843/0x1280
[ 473.580352] [<ffffffff8ba71123>] ? ip_rcv+0x8d3/0x1280
[ 473.580363] [<ffffffff8a423619>] ? __enqueue_entity+0x139/0x230
[ 473.580373] [<ffffffff8ba70850>] ? ip_local_deliver+0x460/0x460
[ 473.580382] [<ffffffff8ba6dc90>] ? inet_del_offload+0x40/0x40
[ 473.580393] [<ffffffff8ba70850>] ? ip_local_deliver+0x460/0x460
[ 473.580407] [<ffffffff8b981319>] __netif_receive_skb_core+0x15d9/0x2c90
[ 473.580419] [<ffffffff8a446e31>] ? enqueue_task_fair+0x261/0x2980
[ 473.580428] [<ffffffff8a48aac0>] ? debug_check_no_locks_freed+0x290/0x290
[ 473.580439] [<ffffffff8b97fd40>] ? netif_wake_subqueue+0x1c0/0x1c0
[ 473.580453] [<ffffffff8b9829f4>] __netif_receive_skb+0x24/0x150
[ 473.580468] [<ffffffff8b982bf7>] process_backlog+0xd7/0x610
[ 473.580477] [<ffffffff8b982d24>] ? process_backlog+0x204/0x610
[ 473.580487] [<ffffffff8a46ad5b>] ? swake_up+0x3b/0x60
[ 473.580498] [<ffffffff8b987c81>] net_rx_action+0x731/0xe60
[ 473.580510] [<ffffffff8b987550>] ? sk_busy_loop+0xae0/0xae0
[ 473.580527] [<ffffffff8a52623f>] ? clockevents_program_event+0x1cf/0x300
[ 473.580537] [<ffffffff8a38c42c>] ? __local_bh_enable+0x3c/0x70
[ 473.580548] [<ffffffff8bdd3fbe>] __do_softirq+0x21e/0x889
[ 473.580560] [<ffffffff8bdd1c0c>] do_softirq_own_stack+0x1c/0x30
[ 473.580564] <EOI>
[ 473.580578] [<ffffffff8a38cba5>] do_softirq.part.17+0x65/0xa0
[ 473.580588] [<ffffffff8ba84ad7>] ? ip_finish_output2+0x657/0x1040
[ 473.580597] [<ffffffff8a38cc99>] __local_bh_enable_ip+0xb9/0xc0
[ 473.580606] [<ffffffff8ba84b00>] ip_finish_output2+0x680/0x1040
[ 473.580615] [<ffffffff8ba87f68>] ? ip_finish_output+0x5a8/0x9b0
[ 473.580626] [<ffffffff8ba84480>] ? ip_copy_metadata+0x7a0/0x7a0
[ 473.580640] [<ffffffff8ba4f336>] ? nf_hook_slow+0xf6/0x1b0
[ 473.580651] [<ffffffff8ba4f240>] ? nf_iterate+0x2d0/0x2d0
[ 473.580660] [<ffffffff8ba87f68>] ip_finish_output+0x5a8/0x9b0
[ 473.580670] [<ffffffff8ba8c036>] ip_output+0x1d6/0x520
[ 473.580679] [<ffffffff8ba8c07d>] ? ip_output+0x21d/0x520
[ 473.580692] [<ffffffff8ba8be60>] ? ip_mc_output+0xc10/0xc10
[ 473.580704] [<ffffffff8ba879c0>] ? ip_fragment.constprop.54+0x220/0x220
[ 473.580714] [<ffffffff8ba88b0d>] ip_local_out+0x7d/0x130
[ 473.580724] [<ffffffff8ba89e87>] ip_queue_xmit+0x7f7/0x1bc0
[ 473.580733] [<ffffffff8ba896ce>] ? ip_queue_xmit+0x3e/0x1bc0
[ 473.580749] [<ffffffff8b935147>] ? __skb_clone+0x97/0x7d0
[ 473.580760] [<ffffffff8baf907c>] tcp_transmit_skb+0x172c/0x3430
[ 473.580771] [<ffffffff8a8489b6>] ? kasan_unpoison_shadow+0x36/0x50
[ 473.580782] [<ffffffff8baf7950>] ? __tcp_select_window+0x6b0/0x6b0
[ 473.580795] [<ffffffff8bbae2c2>] ? fib_table_lookup+0xde2/0x1580
[ 473.580808] [<ffffffff8bab769a>] ? sk_stream_alloc_skb+0x2da/0x770
[ 473.580816] [<ffffffff8baf369f>] ? tcp_mtup_init+0x1af/0x330
[ 473.580827] [<ffffffff8bb025fd>] tcp_connect+0x1ffd/0x2e30
[ 473.580836] [<ffffffff8a48a4cd>] ? trace_hardirqs_on+0xd/0x10
[ 473.580850] [<ffffffff8bb00600>] ? tcp_push_one+0xf0/0xf0
[ 473.580862] [<ffffffff8b960d81>] ? secure_tcp_sequence_number+0x101/0x190
[ 473.580873] [<ffffffff8b960c80>] ? secure_dccpv6_sequence_number+0x440/0x440
[ 473.580885] [<ffffffff8ba5eec0>] ? ip_rt_update_pmtu+0xd10/0xd10
[ 473.580896] [<ffffffff8bc08ab1>] ? xfrm_lookup_route+0x21/0x160
[ 473.580910] [<ffffffff8bb18938>] tcp_v4_connect+0xe08/0x1cd0
[ 473.580923] [<ffffffff8bb7a56b>] __inet_stream_connect+0x64b/0xd70
[ 473.580934] [<ffffffff8bb79f20>] ? inet_bind+0x880/0x880
[ 473.580946] [<ffffffff8b922900>] ? lock_sock_nested+0x90/0x110
[ 473.580955] [<ffffffff8a48a4cd>] ? trace_hardirqs_on+0xd/0x10
[ 473.580965] [<ffffffff8a38cc50>] ? __local_bh_enable_ip+0x70/0xc0
[ 473.580980] [<ffffffff8bb7ace5>] inet_stream_connect+0x55/0xa0
[ 473.580991] [<ffffffff8b91739c>] SYSC_connect+0x22c/0x2d0
[ 473.581000] [<ffffffff8b917170>] ? SYSC_bind+0x240/0x240
[ 473.581011] [<ffffffff8a90ec62>] ? set_close_on_exec+0xc2/0x170
[ 473.581021] [<ffffffff8bdd03e7>] ? _raw_spin_unlock+0x27/0x40
[ 473.581035] [<ffffffff8a90ec62>] ? set_close_on_exec+0xc2/0x170
[ 473.581046] [<ffffffff8a8e8386>] ? SyS_fcntl+0x666/0xde0
[ 473.581056] [<ffffffff8a8e7d20>] ? f_getown+0xb0/0xb0
[ 473.581067] [<ffffffff8a20401a>] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 473.581078] [<ffffffff8b91999e>] SyS_connect+0xe/0x10
[ 473.581091] [<ffffffff8bdd0c05>] entry_SYSCALL_64_fastpath+0x23/0xc6
[ 473.581102] Object at ffff8801e1eb26f8, in cache request_sock_TCP size: 352
[ 473.581105] Allocated:
[ 473.581109] PID = 0
[ 473.581112] (stack is not available)
[ 473.581115] Freed:
[ 473.581119] PID = 0
[ 473.581122] (stack is not available)
[ 473.581125] Memory state around the buggy address:
[ 473.581134] ffff8801e1eb2780: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 473.581140] ffff8801e1eb2800: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 473.581147] >ffff8801e1eb2880: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 473.581151] ^
[ 473.581157] ffff8801e1eb2900: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 473.581164] ffff8801e1eb2980: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
--
Daniel J Blueman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [4.9.10] ip_route_me_harder() reading off-slab
2017-02-17 5:34 [4.9.10] ip_route_me_harder() reading off-slab Daniel J Blueman
@ 2017-02-17 6:23 ` Willy Tarreau
2017-02-17 7:39 ` Florian Westphal
1 sibling, 0 replies; 6+ messages in thread
From: Willy Tarreau @ 2017-02-17 6:23 UTC (permalink / raw)
To: Daniel J Blueman
Cc: Netdev, David S. Miller, Eric W. Biederman, David Ahern,
Jan Beulich, Eric Dumazet
On Fri, Feb 17, 2017 at 01:34:11PM +0800, Daniel J Blueman wrote:
> When booting a VM in libvirt/KVM attached to a local bridge and KASAN
> enabled on 4.9.10, we see a stream of KASAN warnings about off-slab
> access [1].
Did it start to appear with 4.9.10 or is 4.9.10 the first 4.9 kernel
you tried (ie is it a regression between earlier kernels and 4.9 or
a recent faulty stable backport into 4.9) ?
Willy
> Let me know if you'd like more debug.
>
> Thanks,
> Daniel
>
> -- [1]
>
> [ 473.579567] BUG: KASAN: slab-out-of-bounds in
> ip_route_me_harder+0xbd5/0xf20 at addr ffff8801e1eb28a8
> [ 473.579577] Read of size 1 by task vcselab/10339
> [ 473.579590] CPU: 1 PID: 10339 Comm: vcselab Tainted: G B 4.9.10-debug+ #2
> [ 473.579596] Hardware name: Dell Inc. XPS 13 9360/0T3FTF, BIOS 1.3.2
> 01/18/2017
> [ 473.579602] ffff880236086ed0 ffffffff8aed83a1 ffff8802324fe6c0
> ffff8801e1eb26f8
> [ 473.579626] ffff880236086ef8 ffffffff8a849521 ffff880236086f90
> ffff8801e1eb26f0
> [ 473.579645] ffff8802324fe6c0 ffff880236086f80 ffffffff8a8497ba
> ffffffff8a848b2d
> [ 473.579662] Call Trace:
> [ 473.579667] <IRQ>
> [ 473.579685] [<ffffffff8aed83a1>] dump_stack+0x85/0xc4
> [ 473.579698] [<ffffffff8a849521>] kasan_object_err+0x21/0x70
> [ 473.579709] [<ffffffff8a8497ba>] kasan_report_error+0x1fa/0x500
> [ 473.579720] [<ffffffff8a848b2d>] ? kasan_kmalloc+0xad/0xe0
> [ 473.579737] [<ffffffff8a849b21>] __asan_report_load1_noabort+0x61/0x70
> [ 473.579749] [<ffffffff8bbe42f5>] ? ip_route_me_harder+0xbd5/0xf20
> [ 473.579759] [<ffffffff8bbe42f5>] ip_route_me_harder+0xbd5/0xf20
> [ 473.579772] [<ffffffff8bbe3720>] ? nf_ip_saveroute+0x320/0x320
> [ 473.579785] [<ffffffff8bdd0c05>] ? entry_SYSCALL_64_fastpath+0x23/0xc6
> [ 473.579801] [<ffffffffc047846a>] iptable_mangle_hook+0x3da/0x5f0
> [iptable_mangle]
> [ 473.579814] [<ffffffff8ba4f080>] nf_iterate+0x110/0x2d0
> [ 473.579826] [<ffffffff8ba4f336>] nf_hook_slow+0xf6/0x1b0
> [ 473.579839] [<ffffffff8ba4f240>] ? nf_iterate+0x2d0/0x2d0
> [ 473.579850] [<ffffffff8ba885fb>] ? __ip_local_out+0x28b/0x720
> [ 473.579860] [<ffffffff8ba886d6>] __ip_local_out+0x366/0x720
> [ 473.579869] [<ffffffff8ba885fb>] ? __ip_local_out+0x28b/0x720
> [ 473.579879] [<ffffffff8ba88370>] ? ip_finish_output+0x9b0/0x9b0
> [ 473.579894] [<ffffffff8ba83b90>] ?
> __ip_flush_pending_frames.isra.43+0x2e0/0x2e0
> [ 473.579905] [<ffffffff8ba88aae>] ip_local_out+0x1e/0x130
> [ 473.579915] [<ffffffff8ba8910d>] ip_build_and_send_pkt+0x54d/0xad0
> [ 473.579927] [<ffffffff8bb1fdb4>] tcp_v4_send_synack+0x184/0x290
> [ 473.579937] [<ffffffff8bb1fc30>] ? tcp_v4_send_check+0x90/0x90
> [ 473.579950] [<ffffffff8ba9c047>] ? inet_ehash_insert+0x407/0x910
> [ 473.579965] [<ffffffff8bad365b>] tcp_conn_request+0x1f5b/0x2a20
> [ 473.579977] [<ffffffff8bad1700>] ? tcp_check_space+0x580/0x580
> [ 473.579991] [<ffffffff8a40e235>] ? default_wake_function+0x35/0x50
> [ 473.580007] [<ffffffff8a48aac0>] ? debug_check_no_locks_freed+0x290/0x290
> [ 473.580018] [<ffffffff8a48aac0>] ? debug_check_no_locks_freed+0x290/0x290
> [ 473.580031] [<ffffffffc032afc4>] ? ipt_do_table+0xb14/0x1ac0 [ip_tables]
> [ 473.580041] [<ffffffff8a48a4cd>] ? trace_hardirqs_on+0xd/0x10
> [ 473.580055] [<ffffffff8a38cc50>] ? __local_bh_enable_ip+0x70/0xc0
> [ 473.580067] [<ffffffff8bb19b94>] tcp_v4_conn_request+0x134/0x1e0
> [ 473.580079] [<ffffffff8bcf39e8>] tcp_v6_conn_request+0x1b8/0x230
> [ 473.580089] [<ffffffff8baead5d>] tcp_rcv_state_process+0x61d/0x41a0
> [ 473.580101] [<ffffffff8b9d9987>] ? sk_filter_trim_cap+0x2a7/0x6a0
> [ 473.580114] [<ffffffff8baea740>] ? tcp_finish_connect+0x600/0x600
> [ 473.580125] [<ffffffff8b9d99a6>] ? sk_filter_trim_cap+0x2c6/0x6a0
> [ 473.580135] [<ffffffff8b9d97d8>] ? sk_filter_trim_cap+0xf8/0x6a0
> [ 473.580145] [<ffffffff8bb16a4a>] ? tcp_md5_do_lookup+0x4a/0x190
> [ 473.580157] [<ffffffff8b9d96e0>] ? sk_filter_is_valid_access+0x60/0x60
> [ 473.580170] [<ffffffff8bdb3cfa>] ? tcp_v4_inbound_md5_hash+0x139/0x3bb
> [ 473.580180] [<ffffffff8bdb3bc1>] ? ncsi_start_dev+0x111/0x111
> [ 473.580190] [<ffffffff8bb1dba8>] tcp_v4_do_rcv+0x2c8/0x8c0
> [ 473.580201] [<ffffffff8bb22988>] tcp_v4_rcv+0x23a8/0x2fc0
> [ 473.580214] [<ffffffffc04005c7>] ? ipv4_confirm+0x117/0x3d0
> [nf_conntrack_ipv4]
> [ 473.580228] [<ffffffff8ba6f829>] ip_local_deliver_finish+0x2b9/0x970
> [ 473.580241] [<ffffffff8ba6f69a>] ? ip_local_deliver_finish+0x12a/0x970
> [ 473.580251] [<ffffffff8ba705a4>] ip_local_deliver+0x1b4/0x460
> [ 473.580259] [<ffffffff8ba705f2>] ? ip_local_deliver+0x202/0x460
> [ 473.580267] [<ffffffff8ba703f0>] ? ip_call_ra_chain+0x510/0x510
> [ 473.580280] [<ffffffffc16d8195>] ? iptable_nat_ipv4_in+0x15/0x20
> [iptable_nat]
> [ 473.580290] [<ffffffff8ba4f002>] ? nf_iterate+0x92/0x2d0
> [ 473.580302] [<ffffffff8ba6f570>] ? ip_rcv_finish+0x18e0/0x18e0
> [ 473.580313] [<ffffffff8ba4f336>] ? nf_hook_slow+0xf6/0x1b0
> [ 473.580323] [<ffffffff8ba6e2e5>] ip_rcv_finish+0x655/0x18e0
> [ 473.580331] [<ffffffff8ba7122b>] ? ip_rcv+0x9db/0x1280
> [ 473.580341] [<ffffffff8ba71093>] ip_rcv+0x843/0x1280
> [ 473.580352] [<ffffffff8ba71123>] ? ip_rcv+0x8d3/0x1280
> [ 473.580363] [<ffffffff8a423619>] ? __enqueue_entity+0x139/0x230
> [ 473.580373] [<ffffffff8ba70850>] ? ip_local_deliver+0x460/0x460
> [ 473.580382] [<ffffffff8ba6dc90>] ? inet_del_offload+0x40/0x40
> [ 473.580393] [<ffffffff8ba70850>] ? ip_local_deliver+0x460/0x460
> [ 473.580407] [<ffffffff8b981319>] __netif_receive_skb_core+0x15d9/0x2c90
> [ 473.580419] [<ffffffff8a446e31>] ? enqueue_task_fair+0x261/0x2980
> [ 473.580428] [<ffffffff8a48aac0>] ? debug_check_no_locks_freed+0x290/0x290
> [ 473.580439] [<ffffffff8b97fd40>] ? netif_wake_subqueue+0x1c0/0x1c0
> [ 473.580453] [<ffffffff8b9829f4>] __netif_receive_skb+0x24/0x150
> [ 473.580468] [<ffffffff8b982bf7>] process_backlog+0xd7/0x610
> [ 473.580477] [<ffffffff8b982d24>] ? process_backlog+0x204/0x610
> [ 473.580487] [<ffffffff8a46ad5b>] ? swake_up+0x3b/0x60
> [ 473.580498] [<ffffffff8b987c81>] net_rx_action+0x731/0xe60
> [ 473.580510] [<ffffffff8b987550>] ? sk_busy_loop+0xae0/0xae0
> [ 473.580527] [<ffffffff8a52623f>] ? clockevents_program_event+0x1cf/0x300
> [ 473.580537] [<ffffffff8a38c42c>] ? __local_bh_enable+0x3c/0x70
> [ 473.580548] [<ffffffff8bdd3fbe>] __do_softirq+0x21e/0x889
> [ 473.580560] [<ffffffff8bdd1c0c>] do_softirq_own_stack+0x1c/0x30
> [ 473.580564] <EOI>
> [ 473.580578] [<ffffffff8a38cba5>] do_softirq.part.17+0x65/0xa0
> [ 473.580588] [<ffffffff8ba84ad7>] ? ip_finish_output2+0x657/0x1040
> [ 473.580597] [<ffffffff8a38cc99>] __local_bh_enable_ip+0xb9/0xc0
> [ 473.580606] [<ffffffff8ba84b00>] ip_finish_output2+0x680/0x1040
> [ 473.580615] [<ffffffff8ba87f68>] ? ip_finish_output+0x5a8/0x9b0
> [ 473.580626] [<ffffffff8ba84480>] ? ip_copy_metadata+0x7a0/0x7a0
> [ 473.580640] [<ffffffff8ba4f336>] ? nf_hook_slow+0xf6/0x1b0
> [ 473.580651] [<ffffffff8ba4f240>] ? nf_iterate+0x2d0/0x2d0
> [ 473.580660] [<ffffffff8ba87f68>] ip_finish_output+0x5a8/0x9b0
> [ 473.580670] [<ffffffff8ba8c036>] ip_output+0x1d6/0x520
> [ 473.580679] [<ffffffff8ba8c07d>] ? ip_output+0x21d/0x520
> [ 473.580692] [<ffffffff8ba8be60>] ? ip_mc_output+0xc10/0xc10
> [ 473.580704] [<ffffffff8ba879c0>] ? ip_fragment.constprop.54+0x220/0x220
> [ 473.580714] [<ffffffff8ba88b0d>] ip_local_out+0x7d/0x130
> [ 473.580724] [<ffffffff8ba89e87>] ip_queue_xmit+0x7f7/0x1bc0
> [ 473.580733] [<ffffffff8ba896ce>] ? ip_queue_xmit+0x3e/0x1bc0
> [ 473.580749] [<ffffffff8b935147>] ? __skb_clone+0x97/0x7d0
> [ 473.580760] [<ffffffff8baf907c>] tcp_transmit_skb+0x172c/0x3430
> [ 473.580771] [<ffffffff8a8489b6>] ? kasan_unpoison_shadow+0x36/0x50
> [ 473.580782] [<ffffffff8baf7950>] ? __tcp_select_window+0x6b0/0x6b0
> [ 473.580795] [<ffffffff8bbae2c2>] ? fib_table_lookup+0xde2/0x1580
> [ 473.580808] [<ffffffff8bab769a>] ? sk_stream_alloc_skb+0x2da/0x770
> [ 473.580816] [<ffffffff8baf369f>] ? tcp_mtup_init+0x1af/0x330
> [ 473.580827] [<ffffffff8bb025fd>] tcp_connect+0x1ffd/0x2e30
> [ 473.580836] [<ffffffff8a48a4cd>] ? trace_hardirqs_on+0xd/0x10
> [ 473.580850] [<ffffffff8bb00600>] ? tcp_push_one+0xf0/0xf0
> [ 473.580862] [<ffffffff8b960d81>] ? secure_tcp_sequence_number+0x101/0x190
> [ 473.580873] [<ffffffff8b960c80>] ? secure_dccpv6_sequence_number+0x440/0x440
> [ 473.580885] [<ffffffff8ba5eec0>] ? ip_rt_update_pmtu+0xd10/0xd10
> [ 473.580896] [<ffffffff8bc08ab1>] ? xfrm_lookup_route+0x21/0x160
> [ 473.580910] [<ffffffff8bb18938>] tcp_v4_connect+0xe08/0x1cd0
> [ 473.580923] [<ffffffff8bb7a56b>] __inet_stream_connect+0x64b/0xd70
> [ 473.580934] [<ffffffff8bb79f20>] ? inet_bind+0x880/0x880
> [ 473.580946] [<ffffffff8b922900>] ? lock_sock_nested+0x90/0x110
> [ 473.580955] [<ffffffff8a48a4cd>] ? trace_hardirqs_on+0xd/0x10
> [ 473.580965] [<ffffffff8a38cc50>] ? __local_bh_enable_ip+0x70/0xc0
> [ 473.580980] [<ffffffff8bb7ace5>] inet_stream_connect+0x55/0xa0
> [ 473.580991] [<ffffffff8b91739c>] SYSC_connect+0x22c/0x2d0
> [ 473.581000] [<ffffffff8b917170>] ? SYSC_bind+0x240/0x240
> [ 473.581011] [<ffffffff8a90ec62>] ? set_close_on_exec+0xc2/0x170
> [ 473.581021] [<ffffffff8bdd03e7>] ? _raw_spin_unlock+0x27/0x40
> [ 473.581035] [<ffffffff8a90ec62>] ? set_close_on_exec+0xc2/0x170
> [ 473.581046] [<ffffffff8a8e8386>] ? SyS_fcntl+0x666/0xde0
> [ 473.581056] [<ffffffff8a8e7d20>] ? f_getown+0xb0/0xb0
> [ 473.581067] [<ffffffff8a20401a>] ? trace_hardirqs_on_thunk+0x1a/0x1c
> [ 473.581078] [<ffffffff8b91999e>] SyS_connect+0xe/0x10
> [ 473.581091] [<ffffffff8bdd0c05>] entry_SYSCALL_64_fastpath+0x23/0xc6
> [ 473.581102] Object at ffff8801e1eb26f8, in cache request_sock_TCP size: 352
> [ 473.581105] Allocated:
> [ 473.581109] PID = 0
> [ 473.581112] (stack is not available)
> [ 473.581115] Freed:
> [ 473.581119] PID = 0
> [ 473.581122] (stack is not available)
> [ 473.581125] Memory state around the buggy address:
> [ 473.581134] ffff8801e1eb2780: fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc fc fc
> [ 473.581140] ffff8801e1eb2800: fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc fc fc
> [ 473.581147] >ffff8801e1eb2880: fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc fc fc
> [ 473.581151] ^
> [ 473.581157] ffff8801e1eb2900: fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc fc fc
> [ 473.581164] ffff8801e1eb2980: fc fc fc fc fc fc fc fc fc fc fc fc
> fc fc fc fc
> --
> Daniel J Blueman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [4.9.10] ip_route_me_harder() reading off-slab
2017-02-17 5:34 [4.9.10] ip_route_me_harder() reading off-slab Daniel J Blueman
2017-02-17 6:23 ` Willy Tarreau
@ 2017-02-17 7:39 ` Florian Westphal
2017-02-27 14:41 ` Daniel J Blueman
1 sibling, 1 reply; 6+ messages in thread
From: Florian Westphal @ 2017-02-17 7:39 UTC (permalink / raw)
To: Daniel J Blueman
Cc: Netdev, David S. Miller, Eric W. Biederman, David Ahern,
Jan Beulich, Eric Dumazet, netfilter-devel, pablo
Daniel J Blueman <daniel@quora.org> wrote:
[ CC nf-devel, pablo ]
> When booting a VM in libvirt/KVM attached to a local bridge and KASAN
> enabled on 4.9.10, we see a stream of KASAN warnings about off-slab
> access [1].
>
> Let me know if you'd like more debug.
Does this patch help?
Subject: [PATCH nf] netfilter: use skb_to_full_sk in ip_route_me_harder
inet_sk(skb->sk) is illegal in case skb is attached to request socket.
Fixes: ca6fb0651883 ("tcp: attach SYNACK messages to request sockets instead of listener")
Reported by: Daniel J Blueman <daniel@quora.org>
Signed-off-by: Florian Westphal <fw@strlen.de>
---
net/ipv4/netfilter.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/net/ipv4/netfilter.c b/net/ipv4/netfilter.c
index b3cc1335adbc..c0cc6aa8cfaa 100644
--- a/net/ipv4/netfilter.c
+++ b/net/ipv4/netfilter.c
@@ -23,7 +23,8 @@ int ip_route_me_harder(struct net *net, struct sk_buff *skb, unsigned int addr_t
struct rtable *rt;
struct flowi4 fl4 = {};
__be32 saddr = iph->saddr;
- __u8 flags = skb->sk ? inet_sk_flowi_flags(skb->sk) : 0;
+ const struct sock *sk = skb_to_full_sk(skb);
+ __u8 flags = sk ? inet_sk_flowi_flags(sk) : 0;
struct net_device *dev = skb_dst(skb)->dev;
unsigned int hh_len;
@@ -40,7 +41,7 @@ int ip_route_me_harder(struct net *net, struct sk_buff *skb, unsigned int addr_t
fl4.daddr = iph->daddr;
fl4.saddr = saddr;
fl4.flowi4_tos = RT_TOS(iph->tos);
- fl4.flowi4_oif = skb->sk ? skb->sk->sk_bound_dev_if : 0;
+ fl4.flowi4_oif = sk ? sk->sk_bound_dev_if : 0;
if (!fl4.flowi4_oif)
fl4.flowi4_oif = l3mdev_master_ifindex(dev);
fl4.flowi4_mark = skb->mark;
@@ -61,7 +62,7 @@ int ip_route_me_harder(struct net *net, struct sk_buff *skb, unsigned int addr_t
xfrm_decode_session(skb, flowi4_to_flowi(&fl4), AF_INET) == 0) {
struct dst_entry *dst = skb_dst(skb);
skb_dst_set(skb, NULL);
- dst = xfrm_lookup(net, dst, flowi4_to_flowi(&fl4), skb->sk, 0);
+ dst = xfrm_lookup(net, dst, flowi4_to_flowi(&fl4), sk, 0);
if (IS_ERR(dst))
return PTR_ERR(dst);
skb_dst_set(skb, dst);
--
2.10.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [4.9.10] ip_route_me_harder() reading off-slab
[not found] ` <1487309781.1311.71.camel@edumazet-glaptop3.roam.corp.google.com>
@ 2017-02-18 6:56 ` Daniel J Blueman
0 siblings, 0 replies; 6+ messages in thread
From: Daniel J Blueman @ 2017-02-18 6:56 UTC (permalink / raw)
To: Eric Dumazet
Cc: Linux Kernel, David S. Miller, Eric W. Biederman, David Ahern,
Jan Beulich, Netdev, Florian Westphal
On 17 February 2017 at 13:36, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Fri, 2017-02-17 at 12:36 +0800, Daniel J Blueman wrote:
>> When booting a VM in libvirt/KVM attached to a local bridge and KASAN
>> enabled on 4.9.10, we see a stream of KASAN warnings about off-slab
>> access [1].
>>
>> Let me know if you'd like more debug.
>
> Could you try the following patch ?
>
> Thanks !
>
> diff --git a/net/ipv4/netfilter.c b/net/ipv4/netfilter.c
> index b3cc1335adbc1a20dcd225d0501b0a286d27e3c8..18839e59da849f0988924bcbc9873965a3681eb0 100644
> --- a/net/ipv4/netfilter.c
> +++ b/net/ipv4/netfilter.c
> @@ -23,7 +23,8 @@ int ip_route_me_harder(struct net *net, struct sk_buff *skb, unsigned int addr_t
> struct rtable *rt;
> struct flowi4 fl4 = {};
> __be32 saddr = iph->saddr;
> - __u8 flags = skb->sk ? inet_sk_flowi_flags(skb->sk) : 0;
> + struct sock *sk = skb->sk;
> + __u8 flags = sk && sk_fullsock(sk) ? inet_sk_flowi_flags(sk) : 0;
> struct net_device *dev = skb_dst(skb)->dev;
> unsigned int hh_len;
>
> @@ -40,7 +41,7 @@ int ip_route_me_harder(struct net *net, struct sk_buff *skb, unsigned int addr_t
> fl4.daddr = iph->daddr;
> fl4.saddr = saddr;
> fl4.flowi4_tos = RT_TOS(iph->tos);
> - fl4.flowi4_oif = skb->sk ? skb->sk->sk_bound_dev_if : 0;
> + fl4.flowi4_oif = sk ? sk->sk_bound_dev_if : 0;
> if (!fl4.flowi4_oif)
> fl4.flowi4_oif = l3mdev_master_ifindex(dev);
> fl4.flowi4_mark = skb->mark;
> @@ -61,7 +62,7 @@ int ip_route_me_harder(struct net *net, struct sk_buff *skb, unsigned int addr_t
> xfrm_decode_session(skb, flowi4_to_flowi(&fl4), AF_INET) == 0) {
> struct dst_entry *dst = skb_dst(skb);
> skb_dst_set(skb, NULL);
> - dst = xfrm_lookup(net, dst, flowi4_to_flowi(&fl4), skb->sk, 0);
> + dst = xfrm_lookup(net, dst, flowi4_to_flowi(&fl4), sk, 0);
> if (IS_ERR(dst))
> return PTR_ERR(dst);
> skb_dst_set(skb, dst);
Fine work! This nicely resolves the issue. I'll test Florian's
proposed fix also.
Tested-by: Daniel J Blueman <daniel@quora.org>
Thanks,
Dan
--
Daniel J Blueman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [4.9.10] ip_route_me_harder() reading off-slab
2017-02-17 7:39 ` Florian Westphal
@ 2017-02-27 14:41 ` Daniel J Blueman
2017-02-28 11:50 ` Pablo Neira Ayuso
0 siblings, 1 reply; 6+ messages in thread
From: Daniel J Blueman @ 2017-02-27 14:41 UTC (permalink / raw)
To: Florian Westphal
Cc: Netdev, David S. Miller, Eric W. Biederman, David Ahern,
Jan Beulich, Eric Dumazet, netfilter-devel, pablo
On 17 February 2017 at 15:39, Florian Westphal <fw@strlen.de> wrote:
> Daniel J Blueman <daniel@quora.org> wrote:
>
> [ CC nf-devel, pablo ]
>
>> When booting a VM in libvirt/KVM attached to a local bridge and KASAN
>> enabled on 4.9.10, we see a stream of KASAN warnings about off-slab
>> access [1].
>>
>> Let me know if you'd like more debug.
>
> Does this patch help?
>
> Subject: [PATCH nf] netfilter: use skb_to_full_sk in ip_route_me_harder
>
> inet_sk(skb->sk) is illegal in case skb is attached to request socket.
>
> Fixes: ca6fb0651883 ("tcp: attach SYNACK messages to request sockets instead of listener")
> Reported by: Daniel J Blueman <daniel@quora.org>
> Signed-off-by: Florian Westphal <fw@strlen.de>
> ---
> net/ipv4/netfilter.c | 7 ++++---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/net/ipv4/netfilter.c b/net/ipv4/netfilter.c
> index b3cc1335adbc..c0cc6aa8cfaa 100644
> --- a/net/ipv4/netfilter.c
> +++ b/net/ipv4/netfilter.c
> @@ -23,7 +23,8 @@ int ip_route_me_harder(struct net *net, struct sk_buff *skb, unsigned int addr_t
> struct rtable *rt;
> struct flowi4 fl4 = {};
> __be32 saddr = iph->saddr;
> - __u8 flags = skb->sk ? inet_sk_flowi_flags(skb->sk) : 0;
> + const struct sock *sk = skb_to_full_sk(skb);
> + __u8 flags = sk ? inet_sk_flowi_flags(sk) : 0;
> struct net_device *dev = skb_dst(skb)->dev;
> unsigned int hh_len;
>
> @@ -40,7 +41,7 @@ int ip_route_me_harder(struct net *net, struct sk_buff *skb, unsigned int addr_t
> fl4.daddr = iph->daddr;
> fl4.saddr = saddr;
> fl4.flowi4_tos = RT_TOS(iph->tos);
> - fl4.flowi4_oif = skb->sk ? skb->sk->sk_bound_dev_if : 0;
> + fl4.flowi4_oif = sk ? sk->sk_bound_dev_if : 0;
> if (!fl4.flowi4_oif)
> fl4.flowi4_oif = l3mdev_master_ifindex(dev);
> fl4.flowi4_mark = skb->mark;
> @@ -61,7 +62,7 @@ int ip_route_me_harder(struct net *net, struct sk_buff *skb, unsigned int addr_t
> xfrm_decode_session(skb, flowi4_to_flowi(&fl4), AF_INET) == 0) {
> struct dst_entry *dst = skb_dst(skb);
> skb_dst_set(skb, NULL);
> - dst = xfrm_lookup(net, dst, flowi4_to_flowi(&fl4), skb->sk, 0);
> + dst = xfrm_lookup(net, dst, flowi4_to_flowi(&fl4), sk, 0);
> if (IS_ERR(dst))
> return PTR_ERR(dst);
> skb_dst_set(skb, dst);
Apologies for the delays; this also addresses the issue just fine.
Tested-by: Daniel J Blueman <daniel@quora.org>
Dan
--
Daniel J Blueman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [4.9.10] ip_route_me_harder() reading off-slab
2017-02-27 14:41 ` Daniel J Blueman
@ 2017-02-28 11:50 ` Pablo Neira Ayuso
0 siblings, 0 replies; 6+ messages in thread
From: Pablo Neira Ayuso @ 2017-02-28 11:50 UTC (permalink / raw)
To: Daniel J Blueman
Cc: Florian Westphal, Netdev, David S. Miller, Eric W. Biederman,
David Ahern, Jan Beulich, Eric Dumazet, netfilter-devel
On Mon, Feb 27, 2017 at 10:41:48PM +0800, Daniel J Blueman wrote:
> On 17 February 2017 at 15:39, Florian Westphal <fw@strlen.de> wrote:
> > Daniel J Blueman <daniel@quora.org> wrote:
> >
> > [ CC nf-devel, pablo ]
> >
> >> When booting a VM in libvirt/KVM attached to a local bridge and KASAN
> >> enabled on 4.9.10, we see a stream of KASAN warnings about off-slab
> >> access [1].
> >>
> >> Let me know if you'd like more debug.
> >
> > Does this patch help?
> >
> > Subject: [PATCH nf] netfilter: use skb_to_full_sk in ip_route_me_harder
> >
> > inet_sk(skb->sk) is illegal in case skb is attached to request socket.
> >
> > Fixes: ca6fb0651883 ("tcp: attach SYNACK messages to request sockets instead of listener")
> > Reported by: Daniel J Blueman <daniel@quora.org>
> > Signed-off-by: Florian Westphal <fw@strlen.de>
[...]
> Apologies for the delays; this also addresses the issue just fine.
>
> Tested-by: Daniel J Blueman <daniel@quora.org>
Applied, thanks for testing.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-02-28 11:50 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-02-17 5:34 [4.9.10] ip_route_me_harder() reading off-slab Daniel J Blueman
2017-02-17 6:23 ` Willy Tarreau
2017-02-17 7:39 ` Florian Westphal
2017-02-27 14:41 ` Daniel J Blueman
2017-02-28 11:50 ` Pablo Neira Ayuso
[not found] <CAMVG2sszgMW0C3LB-u4bqyMYFgnarXspn9iX0GF9+KMW0A0bJg@mail.gmail.com>
[not found] ` <1487309781.1311.71.camel@edumazet-glaptop3.roam.corp.google.com>
2017-02-18 6:56 ` Daniel J Blueman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).