* soft lockup at __skb_recv_datagram() when fuzzing with trinity as root in VM
@ 2013-02-11 19:25 Tommi Rantala
2013-02-12 0:19 ` Eric Dumazet
0 siblings, 1 reply; 11+ messages in thread
From: Tommi Rantala @ 2013-02-11 19:25 UTC (permalink / raw)
To: netdev; +Cc: Dave Jones
Hello,
I am quite easily reproducing this lockup when fuzzing with Trinity as
the root user in a virtual machine. It seems to be busy-looping in the
do-while loop in __skb_recv_datagram().
[ 83.541011] INFO: rcu_sched detected stalls on CPUs/tasks: {}
(detected by 0, t=26002 jiffies, g=27673, c=27672, q=75)
[ 83.541011] INFO: Stall ended before state dump start
[ 108.067010] BUG: soft lockup - CPU#0 stuck for 22s! [trinity-child31:2847]
[ 108.067010] irq event stamp: 244034822
[ 108.067010] hardirqs last enabled at (244034821):
[<ffffffff81ca2da5>] _raw_spin_unlock_irqrestore+0x55/0x70
[ 108.067010] hardirqs last disabled at (244034822):
[<ffffffff81ca4fad>] apic_timer_interrupt+0x6d/0x80
[ 108.067010] softirqs last enabled at (244030010):
[<ffffffff810a086a>] __do_softirq+0x1ca/0x240
[ 108.067010] softirqs last disabled at (244030005):
[<ffffffff81ca56fc>] call_softirq+0x1c/0x30
[ 108.067010] CPU 0
[ 108.067010] Pid: 2847, comm: trinity-child31 Tainted: G W
3.8.0-rc7+ #73 Bochs Bochs
[ 108.067010] RIP: 0010:[<ffffffff81ca2daa>] [<ffffffff81ca2daa>]
_raw_spin_unlock_irqrestore+0x5a/0x70
[ 108.067010] RSP: 0018:ffff88002fb5db38 EFLAGS: 00000286
[ 108.067010] RAX: ffff8800201ec520 RBX: ffffffff810d54fa RCX: 0000000000005220
[ 108.067010] RDX: ffff8800201ec520 RSI: 0000000000000001 RDI: 0000000000000286
[ 108.067010] RBP: ffff88002fb5db48 R08: 0000000000000068 R09: 0000000000000001
[ 108.067010] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff810f5b9d
[ 108.067010] R13: ffff88002fb5daa8 R14: 00000019294ba499 R15: 0000000000000086
[ 108.067010] FS: 00007f6aabc57700(0000) GS:ffff88003e000000(0000)
knlGS:0000000000000000
[ 108.067010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 108.067010] CR2: 0000000000000009 CR3: 000000002fb08000 CR4: 00000000000006f0
[ 108.067010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 108.067010] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 108.067010] Process trinity-child31 (pid: 2847, threadinfo
ffff88002fb5c000, task ffff8800201ec520)
[ 108.067010] Stack:
[ 108.067010] ffff88002fb5dc10 ffff88002fb5dc14 ffff88002fb5dbf8
ffffffff818cc103
[ 108.067010] ffff8800391a7d80 ffff8800201ec520 ffff88002fb5dbb8
7fffffffffffffff
[ 108.067010] ffff88002fb5dc54 40001202810d54fa ffff8800201ec520
ffff8800277f87e8
[ 108.067010] Call Trace:
[ 108.067010] [<ffffffff818cc103>] __skb_recv_datagram+0x1a3/0x3b0
[ 108.067010] [<ffffffff818cbbe0>] ?
csum_partial_copy_fromiovecend+0x220/0x220
[ 108.067010] [<ffffffff818cc33d>] skb_recv_datagram+0x2d/0x30
[ 108.067010] [<ffffffff813029a0>] ? selinux_syslog+0x70/0x70
[ 108.067010] [<ffffffff819ed43d>] rawv6_recvmsg+0xad/0x240
[ 108.067010] [<ffffffff818c4b04>] sock_common_recvmsg+0x34/0x50
[ 108.067010] [<ffffffff818bc8ec>] sock_recvmsg+0xbc/0xf0
[ 108.067010] [<ffffffff81084adf>] ? kvm_clock_read+0x1f/0x30
[ 108.067010] [<ffffffff810612d9>] ? sched_clock+0x9/0x10
[ 108.067010] [<ffffffff818bf31e>] sys_recvfrom+0xde/0x150
[ 108.067010] [<ffffffff810f5abd>] ? trace_hardirqs_on+0xd/0x10
[ 108.067010] [<ffffffff81ca2deb>] ? _raw_spin_unlock_irq+0x2b/0x40
[ 108.067010] [<ffffffff81ca4355>] ? sysret_check+0x22/0x5d
[ 108.067010] [<ffffffff810f5a15>] ? trace_hardirqs_on_caller+0x155/0x1f0
[ 108.067010] [<ffffffff8135718e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 108.067010] [<ffffffff81ca4329>] system_call_fastpath+0x16/0x1b
[ 108.067010] Code: ff f6 c7 02 75 1b 48 89 df 57 9d 0f 1f 44 00 00
e8 fc 2d 45 ff eb 19 66 2e 0f 1f 84 00 00 00 00 00 e8 0b 2d 45 ff 48
89 df 57 9d <0f> 1f 44 00 00 48 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 80 00
00 00
Tommi
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: soft lockup at __skb_recv_datagram() when fuzzing with trinity as root in VM 2013-02-11 19:25 soft lockup at __skb_recv_datagram() when fuzzing with trinity as root in VM Tommi Rantala @ 2013-02-12 0:19 ` Eric Dumazet 2013-02-12 3:15 ` Eric Dumazet 0 siblings, 1 reply; 11+ messages in thread From: Eric Dumazet @ 2013-02-12 0:19 UTC (permalink / raw) To: Tommi Rantala; +Cc: netdev, Dave Jones, Pavel Emelyanov On Mon, 2013-02-11 at 21:25 +0200, Tommi Rantala wrote: > Hello, > > I am quite easily reproducing this lockup when fuzzing with Trinity as > the root user in a virtual machine. It seems to be busy-looping in the > do-while loop in __skb_recv_datagram(). > > [ 83.541011] INFO: rcu_sched detected stalls on CPUs/tasks: {} > (detected by 0, t=26002 jiffies, g=27673, c=27672, q=75) > [ 83.541011] INFO: Stall ended before state dump start > [ 108.067010] BUG: soft lockup - CPU#0 stuck for 22s! [trinity-child31:2847] > [ 108.067010] irq event stamp: 244034822 > [ 108.067010] hardirqs last enabled at (244034821): > [<ffffffff81ca2da5>] _raw_spin_unlock_irqrestore+0x55/0x70 > [ 108.067010] hardirqs last disabled at (244034822): > [<ffffffff81ca4fad>] apic_timer_interrupt+0x6d/0x80 > [ 108.067010] softirqs last enabled at (244030010): > [<ffffffff810a086a>] __do_softirq+0x1ca/0x240 > [ 108.067010] softirqs last disabled at (244030005): > [<ffffffff81ca56fc>] call_softirq+0x1c/0x30 > [ 108.067010] CPU 0 > [ 108.067010] Pid: 2847, comm: trinity-child31 Tainted: G W > 3.8.0-rc7+ #73 Bochs Bochs > [ 108.067010] RIP: 0010:[<ffffffff81ca2daa>] [<ffffffff81ca2daa>] > _raw_spin_unlock_irqrestore+0x5a/0x70 > [ 108.067010] RSP: 0018:ffff88002fb5db38 EFLAGS: 00000286 > [ 108.067010] RAX: ffff8800201ec520 RBX: ffffffff810d54fa RCX: 0000000000005220 > [ 108.067010] RDX: ffff8800201ec520 RSI: 0000000000000001 RDI: 0000000000000286 > [ 108.067010] RBP: ffff88002fb5db48 R08: 0000000000000068 R09: 0000000000000001 > [ 108.067010] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff810f5b9d > [ 108.067010] R13: ffff88002fb5daa8 R14: 00000019294ba499 R15: 0000000000000086 > [ 108.067010] FS: 00007f6aabc57700(0000) GS:ffff88003e000000(0000) > knlGS:0000000000000000 > [ 108.067010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 108.067010] CR2: 0000000000000009 CR3: 000000002fb08000 CR4: 00000000000006f0 > [ 108.067010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 108.067010] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 108.067010] Process trinity-child31 (pid: 2847, threadinfo > ffff88002fb5c000, task ffff8800201ec520) > [ 108.067010] Stack: > [ 108.067010] ffff88002fb5dc10 ffff88002fb5dc14 ffff88002fb5dbf8 > ffffffff818cc103 > [ 108.067010] ffff8800391a7d80 ffff8800201ec520 ffff88002fb5dbb8 > 7fffffffffffffff > [ 108.067010] ffff88002fb5dc54 40001202810d54fa ffff8800201ec520 > ffff8800277f87e8 > [ 108.067010] Call Trace: > [ 108.067010] [<ffffffff818cc103>] __skb_recv_datagram+0x1a3/0x3b0 > [ 108.067010] [<ffffffff818cbbe0>] ? > csum_partial_copy_fromiovecend+0x220/0x220 > [ 108.067010] [<ffffffff818cc33d>] skb_recv_datagram+0x2d/0x30 > [ 108.067010] [<ffffffff813029a0>] ? selinux_syslog+0x70/0x70 > [ 108.067010] [<ffffffff819ed43d>] rawv6_recvmsg+0xad/0x240 > [ 108.067010] [<ffffffff818c4b04>] sock_common_recvmsg+0x34/0x50 > [ 108.067010] [<ffffffff818bc8ec>] sock_recvmsg+0xbc/0xf0 > [ 108.067010] [<ffffffff81084adf>] ? kvm_clock_read+0x1f/0x30 > [ 108.067010] [<ffffffff810612d9>] ? sched_clock+0x9/0x10 > [ 108.067010] [<ffffffff818bf31e>] sys_recvfrom+0xde/0x150 > [ 108.067010] [<ffffffff810f5abd>] ? trace_hardirqs_on+0xd/0x10 > [ 108.067010] [<ffffffff81ca2deb>] ? _raw_spin_unlock_irq+0x2b/0x40 > [ 108.067010] [<ffffffff81ca4355>] ? sysret_check+0x22/0x5d > [ 108.067010] [<ffffffff810f5a15>] ? trace_hardirqs_on_caller+0x155/0x1f0 > [ 108.067010] [<ffffffff8135718e>] ? trace_hardirqs_on_thunk+0x3a/0x3f > [ 108.067010] [<ffffffff81ca4329>] system_call_fastpath+0x16/0x1b > [ 108.067010] Code: ff f6 c7 02 75 1b 48 89 df 57 9d 0f 1f 44 00 00 > e8 fc 2d 45 ff eb 19 66 2e 0f 1f 84 00 00 00 00 00 e8 0b 2d 45 ff 48 > 89 df 57 9d <0f> 1f 44 00 00 48 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 80 00 > 00 00 > > Tommi Seems MSG_PEEK issue wait_for_packet() is unable to wait if one packet is in receive_queue. So yes, we basically loop forever. Bug added in commit 3f518bf745cbd6007d8069100fb9cb09e960c872 (datagram: Add offset argument to __skb_recv_datagram) CC Pavel Emelyanov ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: soft lockup at __skb_recv_datagram() when fuzzing with trinity as root in VM 2013-02-12 0:19 ` Eric Dumazet @ 2013-02-12 3:15 ` Eric Dumazet 2013-02-12 7:42 ` Tommi Rantala 0 siblings, 1 reply; 11+ messages in thread From: Eric Dumazet @ 2013-02-12 3:15 UTC (permalink / raw) To: Tommi Rantala; +Cc: netdev, Dave Jones, Pavel Emelyanov On Mon, 2013-02-11 at 16:19 -0800, Eric Dumazet wrote: > On Mon, 2013-02-11 at 21:25 +0200, Tommi Rantala wrote: > > Hello, > > > > I am quite easily reproducing this lockup when fuzzing with Trinity as > > the root user in a virtual machine. It seems to be busy-looping in the > > do-while loop in __skb_recv_datagram(). > > > > [ 83.541011] INFO: rcu_sched detected stalls on CPUs/tasks: {} > > (detected by 0, t=26002 jiffies, g=27673, c=27672, q=75) > > [ 83.541011] INFO: Stall ended before state dump start > > [ 108.067010] BUG: soft lockup - CPU#0 stuck for 22s! [trinity-child31:2847] > > [ 108.067010] irq event stamp: 244034822 > > [ 108.067010] hardirqs last enabled at (244034821): > > [<ffffffff81ca2da5>] _raw_spin_unlock_irqrestore+0x55/0x70 > > [ 108.067010] hardirqs last disabled at (244034822): > > [<ffffffff81ca4fad>] apic_timer_interrupt+0x6d/0x80 > > [ 108.067010] softirqs last enabled at (244030010): > > [<ffffffff810a086a>] __do_softirq+0x1ca/0x240 > > [ 108.067010] softirqs last disabled at (244030005): > > [<ffffffff81ca56fc>] call_softirq+0x1c/0x30 > > [ 108.067010] CPU 0 > > [ 108.067010] Pid: 2847, comm: trinity-child31 Tainted: G W > > 3.8.0-rc7+ #73 Bochs Bochs > > [ 108.067010] RIP: 0010:[<ffffffff81ca2daa>] [<ffffffff81ca2daa>] > > _raw_spin_unlock_irqrestore+0x5a/0x70 > > [ 108.067010] RSP: 0018:ffff88002fb5db38 EFLAGS: 00000286 > > [ 108.067010] RAX: ffff8800201ec520 RBX: ffffffff810d54fa RCX: 0000000000005220 > > [ 108.067010] RDX: ffff8800201ec520 RSI: 0000000000000001 RDI: 0000000000000286 > > [ 108.067010] RBP: ffff88002fb5db48 R08: 0000000000000068 R09: 0000000000000001 > > [ 108.067010] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff810f5b9d > > [ 108.067010] R13: ffff88002fb5daa8 R14: 00000019294ba499 R15: 0000000000000086 > > [ 108.067010] FS: 00007f6aabc57700(0000) GS:ffff88003e000000(0000) > > knlGS:0000000000000000 > > [ 108.067010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 108.067010] CR2: 0000000000000009 CR3: 000000002fb08000 CR4: 00000000000006f0 > > [ 108.067010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > [ 108.067010] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > [ 108.067010] Process trinity-child31 (pid: 2847, threadinfo > > ffff88002fb5c000, task ffff8800201ec520) > > [ 108.067010] Stack: > > [ 108.067010] ffff88002fb5dc10 ffff88002fb5dc14 ffff88002fb5dbf8 > > ffffffff818cc103 > > [ 108.067010] ffff8800391a7d80 ffff8800201ec520 ffff88002fb5dbb8 > > 7fffffffffffffff > > [ 108.067010] ffff88002fb5dc54 40001202810d54fa ffff8800201ec520 > > ffff8800277f87e8 > > [ 108.067010] Call Trace: > > [ 108.067010] [<ffffffff818cc103>] __skb_recv_datagram+0x1a3/0x3b0 > > [ 108.067010] [<ffffffff818cbbe0>] ? > > csum_partial_copy_fromiovecend+0x220/0x220 > > [ 108.067010] [<ffffffff818cc33d>] skb_recv_datagram+0x2d/0x30 > > [ 108.067010] [<ffffffff813029a0>] ? selinux_syslog+0x70/0x70 > > [ 108.067010] [<ffffffff819ed43d>] rawv6_recvmsg+0xad/0x240 > > [ 108.067010] [<ffffffff818c4b04>] sock_common_recvmsg+0x34/0x50 > > [ 108.067010] [<ffffffff818bc8ec>] sock_recvmsg+0xbc/0xf0 > > [ 108.067010] [<ffffffff81084adf>] ? kvm_clock_read+0x1f/0x30 > > [ 108.067010] [<ffffffff810612d9>] ? sched_clock+0x9/0x10 > > [ 108.067010] [<ffffffff818bf31e>] sys_recvfrom+0xde/0x150 > > [ 108.067010] [<ffffffff810f5abd>] ? trace_hardirqs_on+0xd/0x10 > > [ 108.067010] [<ffffffff81ca2deb>] ? _raw_spin_unlock_irq+0x2b/0x40 > > [ 108.067010] [<ffffffff81ca4355>] ? sysret_check+0x22/0x5d > > [ 108.067010] [<ffffffff810f5a15>] ? trace_hardirqs_on_caller+0x155/0x1f0 > > [ 108.067010] [<ffffffff8135718e>] ? trace_hardirqs_on_thunk+0x3a/0x3f > > [ 108.067010] [<ffffffff81ca4329>] system_call_fastpath+0x16/0x1b > > [ 108.067010] Code: ff f6 c7 02 75 1b 48 89 df 57 9d 0f 1f 44 00 00 > > e8 fc 2d 45 ff eb 19 66 2e 0f 1f 84 00 00 00 00 00 e8 0b 2d 45 ff 48 > > 89 df 57 9d <0f> 1f 44 00 00 48 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 80 00 > > 00 00 > > > > Tommi > > Seems MSG_PEEK issue > > wait_for_packet() is unable to wait if one packet is in receive_queue. > > So yes, we basically loop forever. > > Bug added in commit 3f518bf745cbd6007d8069100fb9cb09e960c872 > (datagram: Add offset argument to __skb_recv_datagram) > > CC Pavel Emelyanov > If I am not mistaken, we can have skb with 0 bytes in them diff --git a/net/core/datagram.c b/net/core/datagram.c index 0337e2b..368f9c3 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -187,7 +187,7 @@ struct sk_buff *__skb_recv_datagram(struct sock *sk, unsigned int flags, skb_queue_walk(queue, skb) { *peeked = skb->peeked; if (flags & MSG_PEEK) { - if (*off >= skb->len) { + if (*off >= skb->len && skb->len) { *off -= skb->len; continue; } ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: soft lockup at __skb_recv_datagram() when fuzzing with trinity as root in VM 2013-02-12 3:15 ` Eric Dumazet @ 2013-02-12 7:42 ` Tommi Rantala 2013-02-12 16:16 ` [PATCH] net: fix infinite loop in __skb_recv_datagram() Eric Dumazet 0 siblings, 1 reply; 11+ messages in thread From: Tommi Rantala @ 2013-02-12 7:42 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev, Dave Jones, Pavel Emelyanov 2013/2/12 Eric Dumazet <eric.dumazet@gmail.com>: > On Mon, 2013-02-11 at 16:19 -0800, Eric Dumazet wrote: >> On Mon, 2013-02-11 at 21:25 +0200, Tommi Rantala wrote: >> > Hello, >> > >> > I am quite easily reproducing this lockup when fuzzing with Trinity as >> > the root user in a virtual machine. It seems to be busy-looping in the >> > do-while loop in __skb_recv_datagram(). >> > >> > [ 83.541011] INFO: rcu_sched detected stalls on CPUs/tasks: {} >> > (detected by 0, t=26002 jiffies, g=27673, c=27672, q=75) >> > [ 83.541011] INFO: Stall ended before state dump start >> > [ 108.067010] BUG: soft lockup - CPU#0 stuck for 22s! [trinity-child31:2847] >> > [ 108.067010] irq event stamp: 244034822 >> > [ 108.067010] hardirqs last enabled at (244034821): >> > [<ffffffff81ca2da5>] _raw_spin_unlock_irqrestore+0x55/0x70 >> > [ 108.067010] hardirqs last disabled at (244034822): >> > [<ffffffff81ca4fad>] apic_timer_interrupt+0x6d/0x80 >> > [ 108.067010] softirqs last enabled at (244030010): >> > [<ffffffff810a086a>] __do_softirq+0x1ca/0x240 >> > [ 108.067010] softirqs last disabled at (244030005): >> > [<ffffffff81ca56fc>] call_softirq+0x1c/0x30 >> > [ 108.067010] CPU 0 >> > [ 108.067010] Pid: 2847, comm: trinity-child31 Tainted: G W >> > 3.8.0-rc7+ #73 Bochs Bochs >> > [ 108.067010] RIP: 0010:[<ffffffff81ca2daa>] [<ffffffff81ca2daa>] >> > _raw_spin_unlock_irqrestore+0x5a/0x70 >> > [ 108.067010] RSP: 0018:ffff88002fb5db38 EFLAGS: 00000286 >> > [ 108.067010] RAX: ffff8800201ec520 RBX: ffffffff810d54fa RCX: 0000000000005220 >> > [ 108.067010] RDX: ffff8800201ec520 RSI: 0000000000000001 RDI: 0000000000000286 >> > [ 108.067010] RBP: ffff88002fb5db48 R08: 0000000000000068 R09: 0000000000000001 >> > [ 108.067010] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff810f5b9d >> > [ 108.067010] R13: ffff88002fb5daa8 R14: 00000019294ba499 R15: 0000000000000086 >> > [ 108.067010] FS: 00007f6aabc57700(0000) GS:ffff88003e000000(0000) >> > knlGS:0000000000000000 >> > [ 108.067010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> > [ 108.067010] CR2: 0000000000000009 CR3: 000000002fb08000 CR4: 00000000000006f0 >> > [ 108.067010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> > [ 108.067010] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >> > [ 108.067010] Process trinity-child31 (pid: 2847, threadinfo >> > ffff88002fb5c000, task ffff8800201ec520) >> > [ 108.067010] Stack: >> > [ 108.067010] ffff88002fb5dc10 ffff88002fb5dc14 ffff88002fb5dbf8 >> > ffffffff818cc103 >> > [ 108.067010] ffff8800391a7d80 ffff8800201ec520 ffff88002fb5dbb8 >> > 7fffffffffffffff >> > [ 108.067010] ffff88002fb5dc54 40001202810d54fa ffff8800201ec520 >> > ffff8800277f87e8 >> > [ 108.067010] Call Trace: >> > [ 108.067010] [<ffffffff818cc103>] __skb_recv_datagram+0x1a3/0x3b0 >> > [ 108.067010] [<ffffffff818cbbe0>] ? >> > csum_partial_copy_fromiovecend+0x220/0x220 >> > [ 108.067010] [<ffffffff818cc33d>] skb_recv_datagram+0x2d/0x30 >> > [ 108.067010] [<ffffffff813029a0>] ? selinux_syslog+0x70/0x70 >> > [ 108.067010] [<ffffffff819ed43d>] rawv6_recvmsg+0xad/0x240 >> > [ 108.067010] [<ffffffff818c4b04>] sock_common_recvmsg+0x34/0x50 >> > [ 108.067010] [<ffffffff818bc8ec>] sock_recvmsg+0xbc/0xf0 >> > [ 108.067010] [<ffffffff81084adf>] ? kvm_clock_read+0x1f/0x30 >> > [ 108.067010] [<ffffffff810612d9>] ? sched_clock+0x9/0x10 >> > [ 108.067010] [<ffffffff818bf31e>] sys_recvfrom+0xde/0x150 >> > [ 108.067010] [<ffffffff810f5abd>] ? trace_hardirqs_on+0xd/0x10 >> > [ 108.067010] [<ffffffff81ca2deb>] ? _raw_spin_unlock_irq+0x2b/0x40 >> > [ 108.067010] [<ffffffff81ca4355>] ? sysret_check+0x22/0x5d >> > [ 108.067010] [<ffffffff810f5a15>] ? trace_hardirqs_on_caller+0x155/0x1f0 >> > [ 108.067010] [<ffffffff8135718e>] ? trace_hardirqs_on_thunk+0x3a/0x3f >> > [ 108.067010] [<ffffffff81ca4329>] system_call_fastpath+0x16/0x1b >> > [ 108.067010] Code: ff f6 c7 02 75 1b 48 89 df 57 9d 0f 1f 44 00 00 >> > e8 fc 2d 45 ff eb 19 66 2e 0f 1f 84 00 00 00 00 00 e8 0b 2d 45 ff 48 >> > 89 df 57 9d <0f> 1f 44 00 00 48 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 80 00 >> > 00 00 >> > >> > Tommi >> >> Seems MSG_PEEK issue >> >> wait_for_packet() is unable to wait if one packet is in receive_queue. >> >> So yes, we basically loop forever. >> >> Bug added in commit 3f518bf745cbd6007d8069100fb9cb09e960c872 >> (datagram: Add offset argument to __skb_recv_datagram) >> >> CC Pavel Emelyanov >> > > If I am not mistaken, we can have skb with 0 bytes in them Thanks Eric, with the patch applied, I am no longer able to reproduce the bug with Trinity. Tommi > diff --git a/net/core/datagram.c b/net/core/datagram.c > index 0337e2b..368f9c3 100644 > --- a/net/core/datagram.c > +++ b/net/core/datagram.c > @@ -187,7 +187,7 @@ struct sk_buff *__skb_recv_datagram(struct sock *sk, unsigned int flags, > skb_queue_walk(queue, skb) { > *peeked = skb->peeked; > if (flags & MSG_PEEK) { > - if (*off >= skb->len) { > + if (*off >= skb->len && skb->len) { > *off -= skb->len; > continue; > } > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH] net: fix infinite loop in __skb_recv_datagram() 2013-02-12 7:42 ` Tommi Rantala @ 2013-02-12 16:16 ` Eric Dumazet 2013-02-12 16:18 ` Pavel Emelyanov 2013-02-12 21:07 ` David Miller 0 siblings, 2 replies; 11+ messages in thread From: Eric Dumazet @ 2013-02-12 16:16 UTC (permalink / raw) To: Tommi Rantala, David Miller; +Cc: netdev, Dave Jones, Pavel Emelyanov From: Eric Dumazet <edumazet@google.com> Tommi was fuzzing with trinity and reported the following problem : commit 3f518bf745 (datagram: Add offset argument to __skb_recv_datagram) missed that a raw socket receive queue can contain skbs with no payload. We can loop in __skb_recv_datagram() with MSG_PEEK mode, because wait_for_packet() is not prepared to skip these skbs. [ 83.541011] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0, t=26002 jiffies, g=27673, c=27672, q=75) [ 83.541011] INFO: Stall ended before state dump start [ 108.067010] BUG: soft lockup - CPU#0 stuck for 22s! [trinity-child31:2847] ... [ 108.067010] Call Trace: [ 108.067010] [<ffffffff818cc103>] __skb_recv_datagram+0x1a3/0x3b0 [ 108.067010] [<ffffffff818cc33d>] skb_recv_datagram+0x2d/0x30 [ 108.067010] [<ffffffff819ed43d>] rawv6_recvmsg+0xad/0x240 [ 108.067010] [<ffffffff818c4b04>] sock_common_recvmsg+0x34/0x50 [ 108.067010] [<ffffffff818bc8ec>] sock_recvmsg+0xbc/0xf0 [ 108.067010] [<ffffffff818bf31e>] sys_recvfrom+0xde/0x150 [ 108.067010] [<ffffffff81ca4329>] system_call_fastpath+0x16/0x1b Reported-by: Tommi Rantala <tt.rantala@gmail.com> Tested-by: Tommi Rantala <tt.rantala@gmail.com> Signed-off-by: Eric Dumazet <edumazet@google.com> Cc: Pavel Emelyanov <xemul@parallels.com> --- net/core/datagram.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/datagram.c b/net/core/datagram.c index 0337e2b..368f9c3 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -187,7 +187,7 @@ struct sk_buff *__skb_recv_datagram(struct sock *sk, unsigned int flags, skb_queue_walk(queue, skb) { *peeked = skb->peeked; if (flags & MSG_PEEK) { - if (*off >= skb->len) { + if (*off >= skb->len && skb->len) { *off -= skb->len; continue; } ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH] net: fix infinite loop in __skb_recv_datagram() 2013-02-12 16:16 ` [PATCH] net: fix infinite loop in __skb_recv_datagram() Eric Dumazet @ 2013-02-12 16:18 ` Pavel Emelyanov 2013-02-12 21:07 ` David Miller 1 sibling, 0 replies; 11+ messages in thread From: Pavel Emelyanov @ 2013-02-12 16:18 UTC (permalink / raw) To: Eric Dumazet, David Miller; +Cc: Tommi Rantala, netdev, Dave Jones On 02/12/2013 08:16 PM, Eric Dumazet wrote: > From: Eric Dumazet <edumazet@google.com> > > Tommi was fuzzing with trinity and reported the following problem : > > commit 3f518bf745 (datagram: Add offset argument to __skb_recv_datagram) > missed that a raw socket receive queue can contain skbs with no payload. > > We can loop in __skb_recv_datagram() with MSG_PEEK mode, because > wait_for_packet() is not prepared to skip these skbs. > > [ 83.541011] INFO: rcu_sched detected stalls on CPUs/tasks: {} > (detected by 0, t=26002 jiffies, g=27673, c=27672, q=75) > [ 83.541011] INFO: Stall ended before state dump start > [ 108.067010] BUG: soft lockup - CPU#0 stuck for 22s! [trinity-child31:2847] > ... > [ 108.067010] Call Trace: > [ 108.067010] [<ffffffff818cc103>] __skb_recv_datagram+0x1a3/0x3b0 > [ 108.067010] [<ffffffff818cc33d>] skb_recv_datagram+0x2d/0x30 > [ 108.067010] [<ffffffff819ed43d>] rawv6_recvmsg+0xad/0x240 > [ 108.067010] [<ffffffff818c4b04>] sock_common_recvmsg+0x34/0x50 > [ 108.067010] [<ffffffff818bc8ec>] sock_recvmsg+0xbc/0xf0 > [ 108.067010] [<ffffffff818bf31e>] sys_recvfrom+0xde/0x150 > [ 108.067010] [<ffffffff81ca4329>] system_call_fastpath+0x16/0x1b > > Reported-by: Tommi Rantala <tt.rantala@gmail.com> > Tested-by: Tommi Rantala <tt.rantala@gmail.com> > Signed-off-by: Eric Dumazet <edumazet@google.com> > Cc: Pavel Emelyanov <xemul@parallels.com> Acked-by: Pavel Emelyanov <xemul@parallels.com> Thanks! > --- > net/core/datagram.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/net/core/datagram.c b/net/core/datagram.c > index 0337e2b..368f9c3 100644 > --- a/net/core/datagram.c > +++ b/net/core/datagram.c > @@ -187,7 +187,7 @@ struct sk_buff *__skb_recv_datagram(struct sock *sk, unsigned int flags, > skb_queue_walk(queue, skb) { > *peeked = skb->peeked; > if (flags & MSG_PEEK) { > - if (*off >= skb->len) { > + if (*off >= skb->len && skb->len) { > *off -= skb->len; > continue; > } > > > . > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] net: fix infinite loop in __skb_recv_datagram() 2013-02-12 16:16 ` [PATCH] net: fix infinite loop in __skb_recv_datagram() Eric Dumazet 2013-02-12 16:18 ` Pavel Emelyanov @ 2013-02-12 21:07 ` David Miller 2013-02-15 12:41 ` Hannes Frederic Sowa 1 sibling, 1 reply; 11+ messages in thread From: David Miller @ 2013-02-12 21:07 UTC (permalink / raw) To: eric.dumazet; +Cc: tt.rantala, netdev, davej, xemul From: Eric Dumazet <eric.dumazet@gmail.com> Date: Tue, 12 Feb 2013 08:16:53 -0800 > From: Eric Dumazet <edumazet@google.com> > > Tommi was fuzzing with trinity and reported the following problem : > > commit 3f518bf745 (datagram: Add offset argument to __skb_recv_datagram) > missed that a raw socket receive queue can contain skbs with no payload. > > We can loop in __skb_recv_datagram() with MSG_PEEK mode, because > wait_for_packet() is not prepared to skip these skbs. ... > Reported-by: Tommi Rantala <tt.rantala@gmail.com> > Tested-by: Tommi Rantala <tt.rantala@gmail.com> > Signed-off-by: Eric Dumazet <edumazet@google.com> Applied, thanks. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] net: fix infinite loop in __skb_recv_datagram() 2013-02-12 21:07 ` David Miller @ 2013-02-15 12:41 ` Hannes Frederic Sowa 2013-02-15 17:43 ` Ben Hutchings 2013-02-15 18:56 ` David Miller 0 siblings, 2 replies; 11+ messages in thread From: Hannes Frederic Sowa @ 2013-02-15 12:41 UTC (permalink / raw) To: David Miller; +Cc: eric.dumazet, tt.rantala, netdev, davej, xemul On Tue, Feb 12, 2013 at 04:07:33PM -0500, David Miller wrote: > From: Eric Dumazet <eric.dumazet@gmail.com> > Date: Tue, 12 Feb 2013 08:16:53 -0800 > > > From: Eric Dumazet <edumazet@google.com> > > > > Tommi was fuzzing with trinity and reported the following problem : > > > > commit 3f518bf745 (datagram: Add offset argument to __skb_recv_datagram) > > missed that a raw socket receive queue can contain skbs with no payload. > > > > We can loop in __skb_recv_datagram() with MSG_PEEK mode, because > > wait_for_packet() is not prepared to skip these skbs. > ... > > Reported-by: Tommi Rantala <tt.rantala@gmail.com> > > Tested-by: Tommi Rantala <tt.rantala@gmail.com> > > Signed-off-by: Eric Dumazet <edumazet@google.com> > > Applied, thanks. This issue got a CVE: http://seclists.org/oss-sec/2013/q1/310 Perhaps it's something that should go to stable? Thanks, Hannes ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] net: fix infinite loop in __skb_recv_datagram() 2013-02-15 12:41 ` Hannes Frederic Sowa @ 2013-02-15 17:43 ` Ben Hutchings 2013-02-15 17:55 ` Hannes Frederic Sowa 2013-02-15 18:56 ` David Miller 1 sibling, 1 reply; 11+ messages in thread From: Ben Hutchings @ 2013-02-15 17:43 UTC (permalink / raw) To: Hannes Frederic Sowa Cc: David Miller, eric.dumazet, tt.rantala, netdev, davej, xemul On Fri, 2013-02-15 at 13:41 +0100, Hannes Frederic Sowa wrote: > On Tue, Feb 12, 2013 at 04:07:33PM -0500, David Miller wrote: > > From: Eric Dumazet <eric.dumazet@gmail.com> > > Date: Tue, 12 Feb 2013 08:16:53 -0800 > > > > > From: Eric Dumazet <edumazet@google.com> > > > > > > Tommi was fuzzing with trinity and reported the following problem : > > > > > > commit 3f518bf745 (datagram: Add offset argument to __skb_recv_datagram) > > > missed that a raw socket receive queue can contain skbs with no payload. > > > > > > We can loop in __skb_recv_datagram() with MSG_PEEK mode, because > > > wait_for_packet() is not prepared to skip these skbs. > > ... > > > Reported-by: Tommi Rantala <tt.rantala@gmail.com> > > > Tested-by: Tommi Rantala <tt.rantala@gmail.com> > > > Signed-off-by: Eric Dumazet <edumazet@google.com> > > > > Applied, thanks. > > This issue got a CVE: http://seclists.org/oss-sec/2013/q1/310 > Perhaps it's something that should go to stable? David has already worked that out for himself: http://patchwork.ozlabs.org/bundle/davem/stable/?state=* Ben. -- Ben Hutchings, Staff Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] net: fix infinite loop in __skb_recv_datagram() 2013-02-15 17:43 ` Ben Hutchings @ 2013-02-15 17:55 ` Hannes Frederic Sowa 0 siblings, 0 replies; 11+ messages in thread From: Hannes Frederic Sowa @ 2013-02-15 17:55 UTC (permalink / raw) To: Ben Hutchings Cc: David Miller, eric.dumazet, tt.rantala, netdev, davej, xemul On Fri, Feb 15, 2013 at 05:43:30PM +0000, Ben Hutchings wrote: > David has already worked that out for himself: > http://patchwork.ozlabs.org/bundle/davem/stable/?state=* Thanks, didn't know where to look. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] net: fix infinite loop in __skb_recv_datagram() 2013-02-15 12:41 ` Hannes Frederic Sowa 2013-02-15 17:43 ` Ben Hutchings @ 2013-02-15 18:56 ` David Miller 1 sibling, 0 replies; 11+ messages in thread From: David Miller @ 2013-02-15 18:56 UTC (permalink / raw) To: hannes; +Cc: eric.dumazet, tt.rantala, netdev, davej, xemul From: Hannes Frederic Sowa <hannes@stressinduktion.org> Date: Fri, 15 Feb 2013 13:41:41 +0100 > On Tue, Feb 12, 2013 at 04:07:33PM -0500, David Miller wrote: >> From: Eric Dumazet <eric.dumazet@gmail.com> >> Date: Tue, 12 Feb 2013 08:16:53 -0800 >> >> > From: Eric Dumazet <edumazet@google.com> >> > >> > Tommi was fuzzing with trinity and reported the following problem : >> > >> > commit 3f518bf745 (datagram: Add offset argument to __skb_recv_datagram) >> > missed that a raw socket receive queue can contain skbs with no payload. >> > >> > We can loop in __skb_recv_datagram() with MSG_PEEK mode, because >> > wait_for_packet() is not prepared to skip these skbs. >> ... >> > Reported-by: Tommi Rantala <tt.rantala@gmail.com> >> > Tested-by: Tommi Rantala <tt.rantala@gmail.com> >> > Signed-off-by: Eric Dumazet <edumazet@google.com> >> >> Applied, thanks. > > This issue got a CVE: http://seclists.org/oss-sec/2013/q1/310 > Perhaps it's something that should go to stable? It's already queued up in my -stable queue. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2013-02-15 18:56 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-02-11 19:25 soft lockup at __skb_recv_datagram() when fuzzing with trinity as root in VM Tommi Rantala 2013-02-12 0:19 ` Eric Dumazet 2013-02-12 3:15 ` Eric Dumazet 2013-02-12 7:42 ` Tommi Rantala 2013-02-12 16:16 ` [PATCH] net: fix infinite loop in __skb_recv_datagram() Eric Dumazet 2013-02-12 16:18 ` Pavel Emelyanov 2013-02-12 21:07 ` David Miller 2013-02-15 12:41 ` Hannes Frederic Sowa 2013-02-15 17:43 ` Ben Hutchings 2013-02-15 17:55 ` Hannes Frederic Sowa 2013-02-15 18:56 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).