* Re: WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 [not found] <CAASgV=s63Mj5Ki-ep0pUrOpGonaaOZm3TEeENk+rjxA=koxPfg@mail.gmail.com> @ 2017-09-08 17:21 ` Cong Wang 2017-09-08 18:46 ` Eric Dumazet 0 siblings, 1 reply; 7+ messages in thread From: Cong Wang @ 2017-09-08 17:21 UTC (permalink / raw) To: Shankara Pailoor Cc: LKML, syzkaller, David Miller, Alexey Kuznetsov, Hideaki YOSHIFUJI, Linux Kernel Network Developers (Cc'ing netdev) On Fri, Sep 8, 2017 at 5:59 AM, Shankara Pailoor <sp3485@columbia.edu> wrote: > Hi, > > I found a warning while fuzzing with Syzkaller on linux 4.13-rc7 on > x86_64. The full stack trace is below: > > WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 > refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:186 > Kernel panic - not syncing: panic_on_warn set ... > > CPU: 2 PID: 4277 Comm: syz-executor0 Not tainted 4.13.0-rc7 #3 > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > Ubuntu-1.8.2-1ubuntu1 04/01/2014 > Call Trace: > <IRQ> > __dump_stack lib/dump_stack.c:16 [inline] > dump_stack+0xf7/0x1aa lib/dump_stack.c:52 > panic+0x1ae/0x3a7 kernel/panic.c:180 > __warn+0x1c4/0x1d9 kernel/panic.c:541 > report_bug+0x211/0x2d0 lib/bug.c:183 > fixup_bug+0x40/0x90 arch/x86/kernel/traps.c:190 > do_trap_no_signal arch/x86/kernel/traps.c:224 [inline] > do_trap+0x260/0x390 arch/x86/kernel/traps.c:273 > do_error_trap+0x118/0x340 arch/x86/kernel/traps.c:310 > do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:323 > invalid_op+0x18/0x20 arch/x86/entry/entry_64.S:846 > RIP: 0010:refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:186 > RSP: 0018:ffff88006e006b60 EFLAGS: 00010286 > RAX: 0000000000000026 RBX: 0000000000000000 RCX: 0000000000000000 > RDX: 0000000000000026 RSI: 1ffff1000dc00d2c RDI: ffffed000dc00d60 > RBP: ffff88006e006bf0 R08: 0000000000000001 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff1000dc00d6d > R13: 00000000ffffffff R14: 0000000000000001 R15: ffff88006ce9d340 > refcount_dec_and_test+0x1a/0x20 lib/refcount.c:211 > reqsk_put+0x71/0x2b0 include/net/request_sock.h:123 > tcp_v4_rcv+0x259e/0x2e20 net/ipv4/tcp_ipv4.c:1729 > ip_local_deliver_finish+0x2e2/0xba0 net/ipv4/ip_input.c:216 > NF_HOOK include/linux/netfilter.h:248 [inline] > ip_local_deliver+0x1ce/0x6d0 net/ipv4/ip_input.c:257 > dst_input include/net/dst.h:477 [inline] > ip_rcv_finish+0x8db/0x19c0 net/ipv4/ip_input.c:397 > NF_HOOK include/linux/netfilter.h:248 [inline] > ip_rcv+0xc3f/0x17d0 net/ipv4/ip_input.c:488 > __netif_receive_skb_core+0x1fb7/0x31f0 net/core/dev.c:4298 > __netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4336 > process_backlog+0x1c5/0x6d0 net/core/dev.c:5102 > napi_poll net/core/dev.c:5499 [inline] > net_rx_action+0x6d3/0x14a0 net/core/dev.c:5565 > __do_softirq+0x2cb/0xb2d kernel/softirq.c:284 > do_softirq_own_stack+0x1c/0x30 arch/x86/entry/entry_64.S:898 > </IRQ> > do_softirq.part.16+0x63/0x80 kernel/softirq.c:328 > do_softirq kernel/softirq.c:176 [inline] > __local_bh_enable_ip+0x84/0x90 kernel/softirq.c:181 > local_bh_enable include/linux/bottom_half.h:31 [inline] > rcu_read_unlock_bh include/linux/rcupdate.h:705 [inline] > ip_finish_output2+0x8ad/0x1360 net/ipv4/ip_output.c:231 > ip_finish_output+0x74e/0xb80 net/ipv4/ip_output.c:317 > NF_HOOK_COND include/linux/netfilter.h:237 [inline] > ip_output+0x1cc/0x850 net/ipv4/ip_output.c:405 > dst_output include/net/dst.h:471 [inline] > ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124 > ip_queue_xmit+0x8c6/0x1810 net/ipv4/ip_output.c:504 > tcp_transmit_skb+0x1963/0x3320 net/ipv4/tcp_output.c:1123 > tcp_send_ack.part.35+0x38c/0x620 net/ipv4/tcp_output.c:3575 > tcp_send_ack+0x49/0x60 net/ipv4/tcp_output.c:3545 > tcp_rcv_synsent_state_process net/ipv4/tcp_input.c:5795 [inline] > tcp_rcv_state_process+0x4876/0x4b60 net/ipv4/tcp_input.c:5930 > tcp_v4_do_rcv+0x58a/0x820 net/ipv4/tcp_ipv4.c:1483 > sk_backlog_rcv include/net/sock.h:907 [inline] > __release_sock+0x124/0x360 net/core/sock.c:2223 > release_sock+0xa4/0x2a0 net/core/sock.c:2715 > inet_wait_for_connect net/ipv4/af_inet.c:557 [inline] > __inet_stream_connect+0x671/0xf00 net/ipv4/af_inet.c:643 > inet_stream_connect+0x58/0xa0 net/ipv4/af_inet.c:682 > SYSC_connect+0x204/0x470 net/socket.c:1628 > SyS_connect+0x24/0x30 net/socket.c:1609 > entry_SYSCALL_64_fastpath+0x18/0xad > RIP: 0033:0x451e59 > RSP: 002b:00007f474843fc08 EFLAGS: 00000216 ORIG_RAX: 000000000000002a > RAX: ffffffffffffffda RBX: 0000000000718000 RCX: 0000000000451e59 > RDX: 0000000000000010 RSI: 0000000020002000 RDI: 0000000000000007 > RBP: 0000000000000046 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000216 R12: 0000000000000000 > R13: 00007ffc040a0f8f R14: 00007f47484409c0 R15: 0000000000000000 > > > > > I found that the following program is able to reproduce the warning: > > > Pastebin: https://pastebin.com/B75BdYKz > > Here are my configs: https://pastebin.com/zRYCXbak > > Regards, > Shankara > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 2017-09-08 17:21 ` WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 Cong Wang @ 2017-09-08 18:46 ` Eric Dumazet 2017-09-11 3:34 ` Eric Dumazet 0 siblings, 1 reply; 7+ messages in thread From: Eric Dumazet @ 2017-09-08 18:46 UTC (permalink / raw) To: Cong Wang Cc: Shankara Pailoor, LKML, syzkaller, David Miller, Alexey Kuznetsov, Hideaki YOSHIFUJI, Linux Kernel Network Developers On Fri, 2017-09-08 at 10:21 -0700, Cong Wang wrote: > (Cc'ing netdev) > > On Fri, Sep 8, 2017 at 5:59 AM, Shankara Pailoor <sp3485@columbia.edu> wrote: > > Hi, > > > > I found a warning while fuzzing with Syzkaller on linux 4.13-rc7 on > > x86_64. The full stack trace is below: > > > > WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 > > refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:186 > > Kernel panic - not syncing: panic_on_warn set ... > > > > CPU: 2 PID: 4277 Comm: syz-executor0 Not tainted 4.13.0-rc7 #3 > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > > Ubuntu-1.8.2-1ubuntu1 04/01/2014 > > Call Trace: > > <IRQ> > > __dump_stack lib/dump_stack.c:16 [inline] > > dump_stack+0xf7/0x1aa lib/dump_stack.c:52 > > panic+0x1ae/0x3a7 kernel/panic.c:180 > > __warn+0x1c4/0x1d9 kernel/panic.c:541 > > report_bug+0x211/0x2d0 lib/bug.c:183 > > fixup_bug+0x40/0x90 arch/x86/kernel/traps.c:190 > > do_trap_no_signal arch/x86/kernel/traps.c:224 [inline] > > do_trap+0x260/0x390 arch/x86/kernel/traps.c:273 > > do_error_trap+0x118/0x340 arch/x86/kernel/traps.c:310 > > do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:323 > > invalid_op+0x18/0x20 arch/x86/entry/entry_64.S:846 > > RIP: 0010:refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:186 > > RSP: 0018:ffff88006e006b60 EFLAGS: 00010286 > > RAX: 0000000000000026 RBX: 0000000000000000 RCX: 0000000000000000 > > RDX: 0000000000000026 RSI: 1ffff1000dc00d2c RDI: ffffed000dc00d60 > > RBP: ffff88006e006bf0 R08: 0000000000000001 R09: 0000000000000000 > > R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff1000dc00d6d > > R13: 00000000ffffffff R14: 0000000000000001 R15: ffff88006ce9d340 > > refcount_dec_and_test+0x1a/0x20 lib/refcount.c:211 > > reqsk_put+0x71/0x2b0 include/net/request_sock.h:123 > > tcp_v4_rcv+0x259e/0x2e20 net/ipv4/tcp_ipv4.c:1729 > > ip_local_deliver_finish+0x2e2/0xba0 net/ipv4/ip_input.c:216 > > NF_HOOK include/linux/netfilter.h:248 [inline] > > ip_local_deliver+0x1ce/0x6d0 net/ipv4/ip_input.c:257 > > dst_input include/net/dst.h:477 [inline] > > ip_rcv_finish+0x8db/0x19c0 net/ipv4/ip_input.c:397 > > NF_HOOK include/linux/netfilter.h:248 [inline] > > ip_rcv+0xc3f/0x17d0 net/ipv4/ip_input.c:488 > > __netif_receive_skb_core+0x1fb7/0x31f0 net/core/dev.c:4298 > > __netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4336 > > process_backlog+0x1c5/0x6d0 net/core/dev.c:5102 > > napi_poll net/core/dev.c:5499 [inline] > > net_rx_action+0x6d3/0x14a0 net/core/dev.c:5565 > > __do_softirq+0x2cb/0xb2d kernel/softirq.c:284 > > do_softirq_own_stack+0x1c/0x30 arch/x86/entry/entry_64.S:898 > > </IRQ> > > do_softirq.part.16+0x63/0x80 kernel/softirq.c:328 > > do_softirq kernel/softirq.c:176 [inline] > > __local_bh_enable_ip+0x84/0x90 kernel/softirq.c:181 > > local_bh_enable include/linux/bottom_half.h:31 [inline] > > rcu_read_unlock_bh include/linux/rcupdate.h:705 [inline] > > ip_finish_output2+0x8ad/0x1360 net/ipv4/ip_output.c:231 > > ip_finish_output+0x74e/0xb80 net/ipv4/ip_output.c:317 > > NF_HOOK_COND include/linux/netfilter.h:237 [inline] > > ip_output+0x1cc/0x850 net/ipv4/ip_output.c:405 > > dst_output include/net/dst.h:471 [inline] > > ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124 > > ip_queue_xmit+0x8c6/0x1810 net/ipv4/ip_output.c:504 > > tcp_transmit_skb+0x1963/0x3320 net/ipv4/tcp_output.c:1123 > > tcp_send_ack.part.35+0x38c/0x620 net/ipv4/tcp_output.c:3575 > > tcp_send_ack+0x49/0x60 net/ipv4/tcp_output.c:3545 > > tcp_rcv_synsent_state_process net/ipv4/tcp_input.c:5795 [inline] > > tcp_rcv_state_process+0x4876/0x4b60 net/ipv4/tcp_input.c:5930 > > tcp_v4_do_rcv+0x58a/0x820 net/ipv4/tcp_ipv4.c:1483 > > sk_backlog_rcv include/net/sock.h:907 [inline] > > __release_sock+0x124/0x360 net/core/sock.c:2223 > > release_sock+0xa4/0x2a0 net/core/sock.c:2715 > > inet_wait_for_connect net/ipv4/af_inet.c:557 [inline] > > __inet_stream_connect+0x671/0xf00 net/ipv4/af_inet.c:643 > > inet_stream_connect+0x58/0xa0 net/ipv4/af_inet.c:682 > > SYSC_connect+0x204/0x470 net/socket.c:1628 > > SyS_connect+0x24/0x30 net/socket.c:1609 > > entry_SYSCALL_64_fastpath+0x18/0xad > > RIP: 0033:0x451e59 > > RSP: 002b:00007f474843fc08 EFLAGS: 00000216 ORIG_RAX: 000000000000002a > > RAX: ffffffffffffffda RBX: 0000000000718000 RCX: 0000000000451e59 > > RDX: 0000000000000010 RSI: 0000000020002000 RDI: 0000000000000007 > > RBP: 0000000000000046 R08: 0000000000000000 R09: 0000000000000000 > > R10: 0000000000000000 R11: 0000000000000216 R12: 0000000000000000 > > R13: 00007ffc040a0f8f R14: 00007f47484409c0 R15: 0000000000000000 > > > > > > > > > > I found that the following program is able to reproduce the warning: > > > > > > Pastebin: https://pastebin.com/B75BdYKz > > > > Here are my configs: https://pastebin.com/zRYCXbak > > > > Regards, > > Shankara > > Sweet, thanks for the report, I will have a look. It seems one reqsk_put(req); is missing, but that would lead to a memory leak, not a double reqsk_put(req) :/ ( Same fix is needed for IPv6 ) diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index a63486afa7a7e7b4dce88b65bc27cfa872a3ba2f..d9416b5162bc1bdd1acd34fcb4da21cb6b62d0ae 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -1669,9 +1669,9 @@ int tcp_v4_rcv(struct sk_buff *skb) */ sock_hold(sk); refcounted = true; - if (tcp_filter(sk, skb)) - goto discard_and_relse; - nsk = tcp_check_req(sk, skb, req, false); + nsk = NULL; + if (!tcp_filter(sk, skb)) + nsk = tcp_check_req(sk, skb, req, false); if (!nsk) { reqsk_put(req); goto discard_and_relse; ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 2017-09-08 18:46 ` Eric Dumazet @ 2017-09-11 3:34 ` Eric Dumazet 2017-09-11 21:43 ` Shankara Pailoor 0 siblings, 1 reply; 7+ messages in thread From: Eric Dumazet @ 2017-09-11 3:34 UTC (permalink / raw) To: Shankara Pailoor Cc: LKML, syzkaller, David Miller, Linux Kernel Network Developers, Cong Wang On Fri, 2017-09-08 at 11:46 -0700, Eric Dumazet wrote: > On Fri, 2017-09-08 at 10:21 -0700, Cong Wang wrote: > > (Cc'ing netdev) > > > > On Fri, Sep 8, 2017 at 5:59 AM, Shankara Pailoor <sp3485@columbia.edu> wrote: > > > Hi, > > > > > > I found a warning while fuzzing with Syzkaller on linux 4.13-rc7 on > > > x86_64. The full stack trace is below: > > > > > > WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 > > > refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:186 > > > Kernel panic - not syncing: panic_on_warn set ... > > > > > > CPU: 2 PID: 4277 Comm: syz-executor0 Not tainted 4.13.0-rc7 #3 > > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > > > Ubuntu-1.8.2-1ubuntu1 04/01/2014 > > > Call Trace: > > > <IRQ> > > > __dump_stack lib/dump_stack.c:16 [inline] > > > dump_stack+0xf7/0x1aa lib/dump_stack.c:52 > > > panic+0x1ae/0x3a7 kernel/panic.c:180 > > > __warn+0x1c4/0x1d9 kernel/panic.c:541 > > > report_bug+0x211/0x2d0 lib/bug.c:183 > > > fixup_bug+0x40/0x90 arch/x86/kernel/traps.c:190 > > > do_trap_no_signal arch/x86/kernel/traps.c:224 [inline] > > > do_trap+0x260/0x390 arch/x86/kernel/traps.c:273 > > > do_error_trap+0x118/0x340 arch/x86/kernel/traps.c:310 > > > do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:323 > > > invalid_op+0x18/0x20 arch/x86/entry/entry_64.S:846 > > > RIP: 0010:refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:186 > > > RSP: 0018:ffff88006e006b60 EFLAGS: 00010286 > > > RAX: 0000000000000026 RBX: 0000000000000000 RCX: 0000000000000000 > > > RDX: 0000000000000026 RSI: 1ffff1000dc00d2c RDI: ffffed000dc00d60 > > > RBP: ffff88006e006bf0 R08: 0000000000000001 R09: 0000000000000000 > > > R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff1000dc00d6d > > > R13: 00000000ffffffff R14: 0000000000000001 R15: ffff88006ce9d340 > > > refcount_dec_and_test+0x1a/0x20 lib/refcount.c:211 > > > reqsk_put+0x71/0x2b0 include/net/request_sock.h:123 > > > tcp_v4_rcv+0x259e/0x2e20 net/ipv4/tcp_ipv4.c:1729 > > > ip_local_deliver_finish+0x2e2/0xba0 net/ipv4/ip_input.c:216 > > > NF_HOOK include/linux/netfilter.h:248 [inline] > > > ip_local_deliver+0x1ce/0x6d0 net/ipv4/ip_input.c:257 > > > dst_input include/net/dst.h:477 [inline] > > > ip_rcv_finish+0x8db/0x19c0 net/ipv4/ip_input.c:397 > > > NF_HOOK include/linux/netfilter.h:248 [inline] > > > ip_rcv+0xc3f/0x17d0 net/ipv4/ip_input.c:488 > > > __netif_receive_skb_core+0x1fb7/0x31f0 net/core/dev.c:4298 > > > __netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4336 > > > process_backlog+0x1c5/0x6d0 net/core/dev.c:5102 > > > napi_poll net/core/dev.c:5499 [inline] > > > net_rx_action+0x6d3/0x14a0 net/core/dev.c:5565 > > > __do_softirq+0x2cb/0xb2d kernel/softirq.c:284 > > > do_softirq_own_stack+0x1c/0x30 arch/x86/entry/entry_64.S:898 > > > </IRQ> > > > do_softirq.part.16+0x63/0x80 kernel/softirq.c:328 > > > do_softirq kernel/softirq.c:176 [inline] > > > __local_bh_enable_ip+0x84/0x90 kernel/softirq.c:181 > > > local_bh_enable include/linux/bottom_half.h:31 [inline] > > > rcu_read_unlock_bh include/linux/rcupdate.h:705 [inline] > > > ip_finish_output2+0x8ad/0x1360 net/ipv4/ip_output.c:231 > > > ip_finish_output+0x74e/0xb80 net/ipv4/ip_output.c:317 > > > NF_HOOK_COND include/linux/netfilter.h:237 [inline] > > > ip_output+0x1cc/0x850 net/ipv4/ip_output.c:405 > > > dst_output include/net/dst.h:471 [inline] > > > ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124 > > > ip_queue_xmit+0x8c6/0x1810 net/ipv4/ip_output.c:504 > > > tcp_transmit_skb+0x1963/0x3320 net/ipv4/tcp_output.c:1123 > > > tcp_send_ack.part.35+0x38c/0x620 net/ipv4/tcp_output.c:3575 > > > tcp_send_ack+0x49/0x60 net/ipv4/tcp_output.c:3545 > > > tcp_rcv_synsent_state_process net/ipv4/tcp_input.c:5795 [inline] > > > tcp_rcv_state_process+0x4876/0x4b60 net/ipv4/tcp_input.c:5930 > > > tcp_v4_do_rcv+0x58a/0x820 net/ipv4/tcp_ipv4.c:1483 > > > sk_backlog_rcv include/net/sock.h:907 [inline] > > > __release_sock+0x124/0x360 net/core/sock.c:2223 > > > release_sock+0xa4/0x2a0 net/core/sock.c:2715 > > > inet_wait_for_connect net/ipv4/af_inet.c:557 [inline] > > > __inet_stream_connect+0x671/0xf00 net/ipv4/af_inet.c:643 > > > inet_stream_connect+0x58/0xa0 net/ipv4/af_inet.c:682 > > > SYSC_connect+0x204/0x470 net/socket.c:1628 > > > SyS_connect+0x24/0x30 net/socket.c:1609 > > > entry_SYSCALL_64_fastpath+0x18/0xad > > > RIP: 0033:0x451e59 > > > RSP: 002b:00007f474843fc08 EFLAGS: 00000216 ORIG_RAX: 000000000000002a > > > RAX: ffffffffffffffda RBX: 0000000000718000 RCX: 0000000000451e59 > > > RDX: 0000000000000010 RSI: 0000000020002000 RDI: 0000000000000007 > > > RBP: 0000000000000046 R08: 0000000000000000 R09: 0000000000000000 > > > R10: 0000000000000000 R11: 0000000000000216 R12: 0000000000000000 > > > R13: 00007ffc040a0f8f R14: 00007f47484409c0 R15: 0000000000000000 > > > > > > > > > > > > > > > I found that the following program is able to reproduce the warning: > > > > > > > > > Pastebin: https://pastebin.com/B75BdYKz > > > > > > Here are my configs: https://pastebin.com/zRYCXbak > > > > > > Regards, > > > Shankara > > > > > Sweet, thanks for the report, I will have a look. > > It seems one reqsk_put(req); is missing, but that would lead to a memory > leak, not a double reqsk_put(req) :/ OK, it seems the following patch should fix the issue, can you try it ? (not compiled yet, I will test this tomorrow) I will have to double check this wont bring other problems :/ Thanks. diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index 4089c013cb03b12e31ddffbb7ae903542c012ae0..b9c64b40a83af1e151f553ba0e624fae5060ffd0 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -916,7 +916,6 @@ static void inet_child_forget(struct sock *sk, struct request_sock *req, tcp_sk(child)->fastopen_rsk = NULL; } inet_csk_destroy_sock(child); - reqsk_put(req); } struct sock *inet_csk_reqsk_queue_add(struct sock *sk, @@ -987,6 +986,7 @@ void inet_csk_listen_stop(struct sock *sk) sock_hold(child); inet_child_forget(sk, req, child); + reqsk_put(req); bh_unlock_sock(child); local_bh_enable(); sock_put(child); ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 2017-09-11 3:34 ` Eric Dumazet @ 2017-09-11 21:43 ` Shankara Pailoor 2017-09-11 21:58 ` Eric Dumazet 2017-09-11 22:58 ` [PATCH net] tcp/dccp: remove reqsk_put() from inet_child_forget() Eric Dumazet 0 siblings, 2 replies; 7+ messages in thread From: Shankara Pailoor @ 2017-09-11 21:43 UTC (permalink / raw) To: Eric Dumazet Cc: LKML, syzkaller, David Miller, Linux Kernel Network Developers, Cong Wang, Andrew Zhu Aday Hi Eric, I applied your patch and I no longer observed the warning. Thanks! I will let you know if any further problems show up during fuzzing. Regards, Shankara On Sun, Sep 10, 2017 at 11:34 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Fri, 2017-09-08 at 11:46 -0700, Eric Dumazet wrote: >> On Fri, 2017-09-08 at 10:21 -0700, Cong Wang wrote: >> > (Cc'ing netdev) >> > >> > On Fri, Sep 8, 2017 at 5:59 AM, Shankara Pailoor <sp3485@columbia.edu> wrote: >> > > Hi, >> > > >> > > I found a warning while fuzzing with Syzkaller on linux 4.13-rc7 on >> > > x86_64. The full stack trace is below: >> > > >> > > WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 >> > > refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:186 >> > > Kernel panic - not syncing: panic_on_warn set ... >> > > >> > > CPU: 2 PID: 4277 Comm: syz-executor0 Not tainted 4.13.0-rc7 #3 >> > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS >> > > Ubuntu-1.8.2-1ubuntu1 04/01/2014 >> > > Call Trace: >> > > <IRQ> >> > > __dump_stack lib/dump_stack.c:16 [inline] >> > > dump_stack+0xf7/0x1aa lib/dump_stack.c:52 >> > > panic+0x1ae/0x3a7 kernel/panic.c:180 >> > > __warn+0x1c4/0x1d9 kernel/panic.c:541 >> > > report_bug+0x211/0x2d0 lib/bug.c:183 >> > > fixup_bug+0x40/0x90 arch/x86/kernel/traps.c:190 >> > > do_trap_no_signal arch/x86/kernel/traps.c:224 [inline] >> > > do_trap+0x260/0x390 arch/x86/kernel/traps.c:273 >> > > do_error_trap+0x118/0x340 arch/x86/kernel/traps.c:310 >> > > do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:323 >> > > invalid_op+0x18/0x20 arch/x86/entry/entry_64.S:846 >> > > RIP: 0010:refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:186 >> > > RSP: 0018:ffff88006e006b60 EFLAGS: 00010286 >> > > RAX: 0000000000000026 RBX: 0000000000000000 RCX: 0000000000000000 >> > > RDX: 0000000000000026 RSI: 1ffff1000dc00d2c RDI: ffffed000dc00d60 >> > > RBP: ffff88006e006bf0 R08: 0000000000000001 R09: 0000000000000000 >> > > R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff1000dc00d6d >> > > R13: 00000000ffffffff R14: 0000000000000001 R15: ffff88006ce9d340 >> > > refcount_dec_and_test+0x1a/0x20 lib/refcount.c:211 >> > > reqsk_put+0x71/0x2b0 include/net/request_sock.h:123 >> > > tcp_v4_rcv+0x259e/0x2e20 net/ipv4/tcp_ipv4.c:1729 >> > > ip_local_deliver_finish+0x2e2/0xba0 net/ipv4/ip_input.c:216 >> > > NF_HOOK include/linux/netfilter.h:248 [inline] >> > > ip_local_deliver+0x1ce/0x6d0 net/ipv4/ip_input.c:257 >> > > dst_input include/net/dst.h:477 [inline] >> > > ip_rcv_finish+0x8db/0x19c0 net/ipv4/ip_input.c:397 >> > > NF_HOOK include/linux/netfilter.h:248 [inline] >> > > ip_rcv+0xc3f/0x17d0 net/ipv4/ip_input.c:488 >> > > __netif_receive_skb_core+0x1fb7/0x31f0 net/core/dev.c:4298 >> > > __netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4336 >> > > process_backlog+0x1c5/0x6d0 net/core/dev.c:5102 >> > > napi_poll net/core/dev.c:5499 [inline] >> > > net_rx_action+0x6d3/0x14a0 net/core/dev.c:5565 >> > > __do_softirq+0x2cb/0xb2d kernel/softirq.c:284 >> > > do_softirq_own_stack+0x1c/0x30 arch/x86/entry/entry_64.S:898 >> > > </IRQ> >> > > do_softirq.part.16+0x63/0x80 kernel/softirq.c:328 >> > > do_softirq kernel/softirq.c:176 [inline] >> > > __local_bh_enable_ip+0x84/0x90 kernel/softirq.c:181 >> > > local_bh_enable include/linux/bottom_half.h:31 [inline] >> > > rcu_read_unlock_bh include/linux/rcupdate.h:705 [inline] >> > > ip_finish_output2+0x8ad/0x1360 net/ipv4/ip_output.c:231 >> > > ip_finish_output+0x74e/0xb80 net/ipv4/ip_output.c:317 >> > > NF_HOOK_COND include/linux/netfilter.h:237 [inline] >> > > ip_output+0x1cc/0x850 net/ipv4/ip_output.c:405 >> > > dst_output include/net/dst.h:471 [inline] >> > > ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124 >> > > ip_queue_xmit+0x8c6/0x1810 net/ipv4/ip_output.c:504 >> > > tcp_transmit_skb+0x1963/0x3320 net/ipv4/tcp_output.c:1123 >> > > tcp_send_ack.part.35+0x38c/0x620 net/ipv4/tcp_output.c:3575 >> > > tcp_send_ack+0x49/0x60 net/ipv4/tcp_output.c:3545 >> > > tcp_rcv_synsent_state_process net/ipv4/tcp_input.c:5795 [inline] >> > > tcp_rcv_state_process+0x4876/0x4b60 net/ipv4/tcp_input.c:5930 >> > > tcp_v4_do_rcv+0x58a/0x820 net/ipv4/tcp_ipv4.c:1483 >> > > sk_backlog_rcv include/net/sock.h:907 [inline] >> > > __release_sock+0x124/0x360 net/core/sock.c:2223 >> > > release_sock+0xa4/0x2a0 net/core/sock.c:2715 >> > > inet_wait_for_connect net/ipv4/af_inet.c:557 [inline] >> > > __inet_stream_connect+0x671/0xf00 net/ipv4/af_inet.c:643 >> > > inet_stream_connect+0x58/0xa0 net/ipv4/af_inet.c:682 >> > > SYSC_connect+0x204/0x470 net/socket.c:1628 >> > > SyS_connect+0x24/0x30 net/socket.c:1609 >> > > entry_SYSCALL_64_fastpath+0x18/0xad >> > > RIP: 0033:0x451e59 >> > > RSP: 002b:00007f474843fc08 EFLAGS: 00000216 ORIG_RAX: 000000000000002a >> > > RAX: ffffffffffffffda RBX: 0000000000718000 RCX: 0000000000451e59 >> > > RDX: 0000000000000010 RSI: 0000000020002000 RDI: 0000000000000007 >> > > RBP: 0000000000000046 R08: 0000000000000000 R09: 0000000000000000 >> > > R10: 0000000000000000 R11: 0000000000000216 R12: 0000000000000000 >> > > R13: 00007ffc040a0f8f R14: 00007f47484409c0 R15: 0000000000000000 >> > > >> > > >> > > >> > > >> > > I found that the following program is able to reproduce the warning: >> > > >> > > >> > > Pastebin: https://pastebin.com/B75BdYKz >> > > >> > > Here are my configs: https://pastebin.com/zRYCXbak >> > > >> > > Regards, >> > > Shankara >> > > >> >> Sweet, thanks for the report, I will have a look. >> >> It seems one reqsk_put(req); is missing, but that would lead to a memory >> leak, not a double reqsk_put(req) :/ > > OK, it seems the following patch should fix the issue, can you try it ? > > (not compiled yet, I will test this tomorrow) > > I will have to double check this wont bring other problems :/ > > Thanks. > > diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c > index 4089c013cb03b12e31ddffbb7ae903542c012ae0..b9c64b40a83af1e151f553ba0e624fae5060ffd0 100644 > --- a/net/ipv4/inet_connection_sock.c > +++ b/net/ipv4/inet_connection_sock.c > @@ -916,7 +916,6 @@ static void inet_child_forget(struct sock *sk, struct request_sock *req, > tcp_sk(child)->fastopen_rsk = NULL; > } > inet_csk_destroy_sock(child); > - reqsk_put(req); > } > > struct sock *inet_csk_reqsk_queue_add(struct sock *sk, > @@ -987,6 +986,7 @@ void inet_csk_listen_stop(struct sock *sk) > sock_hold(child); > > inet_child_forget(sk, req, child); > + reqsk_put(req); > bh_unlock_sock(child); > local_bh_enable(); > sock_put(child); > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 2017-09-11 21:43 ` Shankara Pailoor @ 2017-09-11 21:58 ` Eric Dumazet 2017-09-11 22:58 ` [PATCH net] tcp/dccp: remove reqsk_put() from inet_child_forget() Eric Dumazet 1 sibling, 0 replies; 7+ messages in thread From: Eric Dumazet @ 2017-09-11 21:58 UTC (permalink / raw) To: Shankara Pailoor Cc: LKML, syzkaller, David Miller, Linux Kernel Network Developers, Cong Wang, Andrew Zhu Aday On Mon, 2017-09-11 at 17:43 -0400, Shankara Pailoor wrote: > Hi Eric, > > I applied your patch and I no longer observed the warning. Thanks! I > will let you know if any further problems show up during fuzzing. Great, I also verified this fixed the problem, I will send the official patch shortly. Thanks ! ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH net] tcp/dccp: remove reqsk_put() from inet_child_forget() 2017-09-11 21:43 ` Shankara Pailoor 2017-09-11 21:58 ` Eric Dumazet @ 2017-09-11 22:58 ` Eric Dumazet 2017-09-13 3:39 ` David Miller 1 sibling, 1 reply; 7+ messages in thread From: Eric Dumazet @ 2017-09-11 22:58 UTC (permalink / raw) To: Shankara Pailoor, David Miller Cc: Linux Kernel Network Developers, Cong Wang, Andrew Zhu Aday From: Eric Dumazet <edumazet@google.com> Back in linux-4.4, I inadvertently put a call to reqsk_put() in inet_child_forget(), forgetting it could be called from two different points. In the case it is called from inet_csk_reqsk_queue_add(), we want to keep the reference on the request socket, since it is released later by the caller (tcp_v{4|6}_rcv()) This bug never showed up because atomic_dec_and_test() was not signaling the underflow, and SLAB_DESTROY_BY RCU semantic for request sockets prevented the request to be put in quarantine. Recent conversion of socket refcount from atomic_t to refcount_t finally exposed the bug. So move the reqsk_put() to inet_csk_listen_stop() to fix this. Thanks to Shankara Pailoor for using syzkaller and providing a nice set of .config and C repro. WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:186 Kernel panic - not syncing: panic_on_warn set ... CPU: 2 PID: 4277 Comm: syz-executor0 Not tainted 4.13.0-rc7 #3 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 Call Trace: <IRQ> __dump_stack lib/dump_stack.c:16 [inline] dump_stack+0xf7/0x1aa lib/dump_stack.c:52 panic+0x1ae/0x3a7 kernel/panic.c:180 __warn+0x1c4/0x1d9 kernel/panic.c:541 report_bug+0x211/0x2d0 lib/bug.c:183 fixup_bug+0x40/0x90 arch/x86/kernel/traps.c:190 do_trap_no_signal arch/x86/kernel/traps.c:224 [inline] do_trap+0x260/0x390 arch/x86/kernel/traps.c:273 do_error_trap+0x118/0x340 arch/x86/kernel/traps.c:310 do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:323 invalid_op+0x18/0x20 arch/x86/entry/entry_64.S:846 RIP: 0010:refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:186 RSP: 0018:ffff88006e006b60 EFLAGS: 00010286 RAX: 0000000000000026 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000026 RSI: 1ffff1000dc00d2c RDI: ffffed000dc00d60 RBP: ffff88006e006bf0 R08: 0000000000000001 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff1000dc00d6d R13: 00000000ffffffff R14: 0000000000000001 R15: ffff88006ce9d340 refcount_dec_and_test+0x1a/0x20 lib/refcount.c:211 reqsk_put+0x71/0x2b0 include/net/request_sock.h:123 tcp_v4_rcv+0x259e/0x2e20 net/ipv4/tcp_ipv4.c:1729 ip_local_deliver_finish+0x2e2/0xba0 net/ipv4/ip_input.c:216 NF_HOOK include/linux/netfilter.h:248 [inline] ip_local_deliver+0x1ce/0x6d0 net/ipv4/ip_input.c:257 dst_input include/net/dst.h:477 [inline] ip_rcv_finish+0x8db/0x19c0 net/ipv4/ip_input.c:397 NF_HOOK include/linux/netfilter.h:248 [inline] ip_rcv+0xc3f/0x17d0 net/ipv4/ip_input.c:488 __netif_receive_skb_core+0x1fb7/0x31f0 net/core/dev.c:4298 __netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4336 process_backlog+0x1c5/0x6d0 net/core/dev.c:5102 napi_poll net/core/dev.c:5499 [inline] net_rx_action+0x6d3/0x14a0 net/core/dev.c:5565 __do_softirq+0x2cb/0xb2d kernel/softirq.c:284 do_softirq_own_stack+0x1c/0x30 arch/x86/entry/entry_64.S:898 </IRQ> do_softirq.part.16+0x63/0x80 kernel/softirq.c:328 do_softirq kernel/softirq.c:176 [inline] __local_bh_enable_ip+0x84/0x90 kernel/softirq.c:181 local_bh_enable include/linux/bottom_half.h:31 [inline] rcu_read_unlock_bh include/linux/rcupdate.h:705 [inline] ip_finish_output2+0x8ad/0x1360 net/ipv4/ip_output.c:231 ip_finish_output+0x74e/0xb80 net/ipv4/ip_output.c:317 NF_HOOK_COND include/linux/netfilter.h:237 [inline] ip_output+0x1cc/0x850 net/ipv4/ip_output.c:405 dst_output include/net/dst.h:471 [inline] ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124 ip_queue_xmit+0x8c6/0x1810 net/ipv4/ip_output.c:504 tcp_transmit_skb+0x1963/0x3320 net/ipv4/tcp_output.c:1123 tcp_send_ack.part.35+0x38c/0x620 net/ipv4/tcp_output.c:3575 tcp_send_ack+0x49/0x60 net/ipv4/tcp_output.c:3545 tcp_rcv_synsent_state_process net/ipv4/tcp_input.c:5795 [inline] tcp_rcv_state_process+0x4876/0x4b60 net/ipv4/tcp_input.c:5930 tcp_v4_do_rcv+0x58a/0x820 net/ipv4/tcp_ipv4.c:1483 sk_backlog_rcv include/net/sock.h:907 [inline] __release_sock+0x124/0x360 net/core/sock.c:2223 release_sock+0xa4/0x2a0 net/core/sock.c:2715 inet_wait_for_connect net/ipv4/af_inet.c:557 [inline] __inet_stream_connect+0x671/0xf00 net/ipv4/af_inet.c:643 inet_stream_connect+0x58/0xa0 net/ipv4/af_inet.c:682 SYSC_connect+0x204/0x470 net/socket.c:1628 SyS_connect+0x24/0x30 net/socket.c:1609 entry_SYSCALL_64_fastpath+0x18/0xad RIP: 0033:0x451e59 RSP: 002b:00007f474843fc08 EFLAGS: 00000216 ORIG_RAX: 000000000000002a RAX: ffffffffffffffda RBX: 0000000000718000 RCX: 0000000000451e59 RDX: 0000000000000010 RSI: 0000000020002000 RDI: 0000000000000007 RBP: 0000000000000046 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000216 R12: 0000000000000000 R13: 00007ffc040a0f8f R14: 00007f47484409c0 R15: 0000000000000000 Fixes: ebb516af60e1 ("tcp/dccp: fix race at listener dismantle phase") Signed-off-by: Eric Dumazet <edumazet@google.com> Reported-by: Shankara Pailoor <sp3485@columbia.edu> Tested-by: Shankara Pailoor <sp3485@columbia.edu> --- net/ipv4/inet_connection_sock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index 4089c013cb03b12e31ddffbb7ae903542c012ae0..b9c64b40a83af1e151f553ba0e624fae5060ffd0 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -916,7 +916,6 @@ static void inet_child_forget(struct sock *sk, struct request_sock *req, tcp_sk(child)->fastopen_rsk = NULL; } inet_csk_destroy_sock(child); - reqsk_put(req); } struct sock *inet_csk_reqsk_queue_add(struct sock *sk, @@ -987,6 +986,7 @@ void inet_csk_listen_stop(struct sock *sk) sock_hold(child); inet_child_forget(sk, req, child); + reqsk_put(req); bh_unlock_sock(child); local_bh_enable(); sock_put(child); ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH net] tcp/dccp: remove reqsk_put() from inet_child_forget() 2017-09-11 22:58 ` [PATCH net] tcp/dccp: remove reqsk_put() from inet_child_forget() Eric Dumazet @ 2017-09-13 3:39 ` David Miller 0 siblings, 0 replies; 7+ messages in thread From: David Miller @ 2017-09-13 3:39 UTC (permalink / raw) To: eric.dumazet; +Cc: sp3485, netdev, xiyou.wangcong, andrew.aday From: Eric Dumazet <eric.dumazet@gmail.com> Date: Mon, 11 Sep 2017 15:58:38 -0700 > From: Eric Dumazet <edumazet@google.com> > > Back in linux-4.4, I inadvertently put a call to reqsk_put() in > inet_child_forget(), forgetting it could be called from two different > points. > > In the case it is called from inet_csk_reqsk_queue_add(), we want to > keep the reference on the request socket, since it is released later by > the caller (tcp_v{4|6}_rcv()) > > This bug never showed up because atomic_dec_and_test() was not signaling > the underflow, and SLAB_DESTROY_BY RCU semantic for request sockets > prevented the request to be put in quarantine. > > Recent conversion of socket refcount from atomic_t to refcount_t finally > exposed the bug. > > So move the reqsk_put() to inet_csk_listen_stop() to fix this. > > Thanks to Shankara Pailoor for using syzkaller and providing > a nice set of .config and C repro. ... > Fixes: ebb516af60e1 ("tcp/dccp: fix race at listener dismantle phase") > Signed-off-by: Eric Dumazet <edumazet@google.com> > Reported-by: Shankara Pailoor <sp3485@columbia.edu> > Tested-by: Shankara Pailoor <sp3485@columbia.edu> Applied and queued up for -stable. Thanks. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-09-13 3:39 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAASgV=s63Mj5Ki-ep0pUrOpGonaaOZm3TEeENk+rjxA=koxPfg@mail.gmail.com>
2017-09-08 17:21 ` WARNING: CPU: 2 PID: 4277 at lib/refcount.c:186 Cong Wang
2017-09-08 18:46 ` Eric Dumazet
2017-09-11 3:34 ` Eric Dumazet
2017-09-11 21:43 ` Shankara Pailoor
2017-09-11 21:58 ` Eric Dumazet
2017-09-11 22:58 ` [PATCH net] tcp/dccp: remove reqsk_put() from inet_child_forget() Eric Dumazet
2017-09-13 3:39 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).