From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Pim van Riezen <pi+lists@panelsix.com>
Cc: xen-devel@lists.xensource.com
Subject: Re: Xen hypervisor external denial of service vulnerability?
Date: Thu, 10 Feb 2011 12:08:29 -0500 [thread overview]
Message-ID: <20110210170828.GA3993@dumpdata.com> (raw)
In-Reply-To: <C02E5D73-4DAE-4D5A-BDEA-0B7486A14511@panelsix.com>
On Tue, Feb 08, 2011 at 06:21:25PM +0100, Pim van Riezen wrote:
>
> On Feb 8, 2011, at 18:08 , Pim van Riezen wrote:
>
> > On Feb 8, 2011, at 17:51 , Pasi Kärkkäinen wrote:
> >>
> >> Did you also make sure VMs don't use those 2 pcpus dedicated for dom0?
> >> You have to explicitly configure each VM not to use those pcpus.
> >
> > That seems to have done the trick.
>
> Alas, I was too soon in drawing a conclusion. After a new 10 minute run:
Did you try to run the 2.6.32 pvops type kernel? Asking b/c it looks like
the issue is due to the fact that mutex lock is held for a very very long time.
The spinlock implementation in 2.6.32 changed so it might provide a better
solution.
>
> Feb 8 18:12:30 telemann kernel: INFO: task bash:12225 blocked for more than 120 seconds.
> Feb 8 18:12:30 telemann kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:12:30 telemann kernel: bash D ffff88006ac7bd08 0 12225 1 8260 (L-TLB)
> Feb 8 18:12:30 telemann kernel: ffff88006ac7bb88 0000000000000246 0000000300000000 ffff88007ec3a6d8
> Feb 8 18:12:30 telemann kernel: 0000000000000009 ffff88006c16e820 ffff88007a5a9080 000000000008f03e
> Feb 8 18:12:30 telemann kernel: ffff88006c16ea08 ffffffff8022f10c
> Feb 8 18:12:30 telemann kernel: Call Trace:
> Feb 8 18:12:30 telemann kernel: [<ffffffff8022f10c>] __wake_up+0x38/0x4f
> Feb 8 18:12:30 telemann kernel: [<ffffffff880317ae>] :jbd:journal_stop+0x1f3/0x1ff
> Feb 8 18:12:30 telemann kernel: [<ffffffff802994d1>] flush_cpu_workqueue+0x83/0xb5
> Feb 8 18:12:30 telemann kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:12:30 telemann kernel: [<ffffffff80263914>] mutex_lock+0xd/0x1d
> Feb 8 18:12:30 telemann kernel: [<ffffffff80299563>] flush_workqueue+0x60/0x87
> Feb 8 18:12:41 telemann kernel: [<ffffffff80394af5>] release_dev+0x503/0x67b
> Feb 8 18:12:55 telemann kernel: [<ffffffff8020b860>] release_pages+0x158/0x165
> Feb 8 18:13:09 telemann kernel: [<ffffffff80255821>] tty_release+0x11/0x1a
> Feb 8 18:13:23 telemann kernel: [<ffffffff80213492>] __fput+0xd3/0x1bd
> Feb 8 18:13:38 telemann kernel: [<ffffffff802243cb>] filp_close+0x5c/0x64
> Feb 8 18:13:51 telemann kernel: [<ffffffff8023a392>] put_files_struct+0x63/0xae
> Feb 8 18:14:06 telemann kernel: [<ffffffff802160cd>] do_exit+0x31d/0x902
> Feb 8 18:14:19 telemann kernel: [<ffffffff8024ae4d>] cpuset_exit+0x0/0x88
> Feb 8 18:14:33 telemann kernel: [<ffffffff8022b920>] get_signal_to_deliver+0x477/0x4aa
> Feb 8 18:14:49 telemann kernel: [<ffffffff8025d19e>] do_notify_resume+0x9c/0x7ba
> Feb 8 18:15:01 telemann kernel: [<ffffffff80294ea1>] __group_send_sig_info+0xb9/0xc8
> Feb 8 18:15:08 telemann kernel: [<ffffffff8025cb0b>] group_send_sig_info+0x62/0x6f
> Feb 8 18:15:22 telemann kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:15:37 telemann kernel: [<ffffffff802afd73>] audit_syscall_entry+0x180/0x1b3
> Feb 8 18:15:49 telemann kernel: [<ffffffff80245a48>] sys_rt_sigreturn+0x327/0x35a
> Feb 8 18:16:03 telemann kernel: [<ffffffff802b0175>] audit_syscall_exit+0x336/0x362
> Feb 8 18:16:17 telemann kernel: [<ffffffff8026042c>] int_signal+0x12/0x17
> Feb 8 18:16:31 telemann kernel:
> Feb 8 18:16:44 telemann kernel: INFO: task bash:12225 blocked for more than 120 seconds.
> Feb 8 18:16:58 telemann kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:17:12 telemann kernel: bash D ffff88006ac7bd08 0 12225 1 8260 (L-TLB)
> Feb 8 18:17:26 telemann kernel: ffff88006ac7bb88 0000000000000246 0000000300000000 ffff88007ec3a6d8
> Feb 8 18:17:39 telemann kernel: 0000000000000009 ffff88006c16e820 ffff88007a5a9080 000000000008f03e
> Feb 8 18:17:54 telemann kernel: ffff88006c16ea08 ffffffff8022f10c
> Feb 8 18:18:08 telemann kernel: Call Trace:
> Feb 8 18:18:21 telemann kernel: [<ffffffff8022f10c>] __wake_up+0x38/0x4f
> Feb 8 18:18:34 telemann kernel: [<ffffffff880317ae>] :jbd:journal_stop+0x1f3/0x1ff
> Feb 8 18:18:47 telemann kernel: [<ffffffff802994d1>] flush_cpu_workqueue+0x83/0xb5
> Feb 8 18:18:58 telemann kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:18:58 telemann kernel: [<ffffffff80263914>] mutex_lock+0xd/0x1d
> Feb 8 18:18:58 telemann kernel: [<ffffffff80299563>] flush_workqueue+0x60/0x87
> Feb 8 18:18:58 telemann kernel: [<ffffffff80394af5>] release_dev+0x503/0x67b
> Feb 8 18:18:58 telemann kernel: [<ffffffff8020b860>] release_pages+0x158/0x165
> Feb 8 18:18:58 telemann kernel: [<ffffffff80255821>] tty_release+0x11/0x1a
> Feb 8 18:18:58 telemann kernel: [<ffffffff80213492>] __fput+0xd3/0x1bd
> Feb 8 18:18:58 telemann kernel: [<ffffffff802243cb>] filp_close+0x5c/0x64
> Feb 8 18:18:58 telemann kernel: [<ffffffff8023a392>] put_files_struct+0x63/0xae
> Feb 8 18:18:58 telemann kernel: [<ffffffff802160cd>] do_exit+0x31d/0x902
> Feb 8 18:18:58 telemann kernel: [<ffffffff8024ae4d>] cpuset_exit+0x0/0x88
> Feb 8 18:18:58 telemann kernel: [<ffffffff8022b920>] get_signal_to_deliver+0x477/0x4aa
> Feb 8 18:18:58 telemann kernel: [<ffffffff8025d19e>] do_notify_resume+0x9c/0x7ba
> Feb 8 18:18:58 telemann kernel: [<ffffffff80294ea1>] __group_send_sig_info+0xb9/0xc8
> Feb 8 18:18:58 telemann kernel: [<ffffffff8025cb0b>] group_send_sig_info+0x62/0x6f
> Feb 8 18:18:58 telemann kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:18:58 telemann kernel: [<ffffffff802afd73>] audit_syscall_entry+0x180/0x1b3
> Feb 8 18:18:58 telemann kernel: [<ffffffff80245a48>] sys_rt_sigreturn+0x327/0x35a
> Feb 8 18:18:58 telemann kernel: [<ffffffff802b0175>] audit_syscall_exit+0x336/0x362
> Feb 8 18:18:59 telemann kernel: [<ffffffff8026042c>] int_signal+0x12/0x17
> Feb 8 18:18:59 telemann kernel:
> Feb 8 18:18:59 telemann kernel: INFO: task bash:12225 blocked for more than 120 seconds.
> Feb 8 18:18:59 telemann kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:18:59 telemann kernel: bash D ffff88006ac7bd08 0 12225 1 8260 (L-TLB)
> Feb 8 18:18:59 telemann kernel: ffff88006ac7bb88 0000000000000246 0000000300000000 ffff88007ec3a6d8
> Feb 8 18:18:59 telemann kernel: 0000000000000009 ffff88006c16e820 ffff88007a5a9080 000000000008f03e
> Feb 8 18:18:59 telemann kernel: ffff88006c16ea08 ffffffff8022f10c
> Feb 8 18:18:59 telemann kernel: Call Trace:
> Feb 8 18:18:59 telemann kernel: [<ffffffff8022f10c>] __wake_up+0x38/0x4f
> Feb 8 18:18:59 telemann kernel: [<ffffffff880317ae>] :jbd:journal_stop+0x1f3/0x1ff
> Feb 8 18:18:59 telemann kernel: [<ffffffff802994d1>] flush_cpu_workqueue+0x83/0xb5
> Feb 8 18:18:59 telemann kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:18:59 telemann kernel: [<ffffffff80263914>] mutex_lock+0xd/0x1d
> Feb 8 18:18:59 telemann kernel: [<ffffffff80299563>] flush_workqueue+0x60/0x87
> Feb 8 18:18:59 telemann kernel: [<ffffffff80394af5>] release_dev+0x503/0x67b
> Feb 8 18:18:59 telemann kernel: [<ffffffff8020b860>] release_pages+0x158/0x165
> Feb 8 18:18:59 telemann kernel: [<ffffffff80255821>] tty_release+0x11/0x1a
> Feb 8 18:18:59 telemann kernel: [<ffffffff80213492>] __fput+0xd3/0x1bd
> Feb 8 18:18:59 telemann kernel: [<ffffffff802243cb>] filp_close+0x5c/0x64
> Feb 8 18:18:59 telemann kernel: [<ffffffff8023a392>] put_files_struct+0x63/0xae
> Feb 8 18:18:59 telemann kernel: [<ffffffff802160cd>] do_exit+0x31d/0x902
> Feb 8 18:18:59 telemann kernel: [<ffffffff8024ae4d>] cpuset_exit+0x0/0x88
> Feb 8 18:18:59 telemann kernel: [<ffffffff8022b920>] get_signal_to_deliver+0x477/0x4aa
> Feb 8 18:18:59 telemann kernel: [<ffffffff8025d19e>] do_notify_resume+0x9c/0x7ba
> Feb 8 18:18:59 telemann kernel: [<ffffffff80294ea1>] __group_send_sig_info+0xb9/0xc8
> Feb 8 18:18:59 telemann kernel: [<ffffffff8025cb0b>] group_send_sig_info+0x62/0x6f
> Feb 8 18:18:59 telemann kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:18:59 telemann kernel: [<ffffffff802afd73>] audit_syscall_entry+0x180/0x1b3
> Feb 8 18:18:59 telemann kernel: [<ffffffff80245a48>] sys_rt_sigreturn+0x327/0x35a
> Feb 8 18:18:59 telemann kernel: [<ffffffff802b0175>] audit_syscall_exit+0x336/0x362
> Feb 8 18:18:59 telemann kernel: [<ffffffff8026042c>] int_signal+0x12/0x17
> Feb 8 18:18:59 telemann kernel:
> Feb 8 18:18:59 telemann kernel: INFO: task bash:12225 blocked for more than 120 seconds.
> Feb 8 18:18:59 telemann kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:18:59 telemann kernel: bash D ffff88006ac7bd08 0 12225 1 8260 (L-TLB)
> Feb 8 18:18:59 telemann kernel: ffff88006ac7bb88 0000000000000246 0000000300000000 ffff88007ec3a6d8
> Feb 8 18:18:59 telemann kernel: 0000000000000009 ffff88006c16e820 ffff88007a5a9080 000000000008f03e
> Feb 8 18:18:59 telemann kernel: ffff88006c16ea08 ffffffff8022f10c
> Feb 8 18:18:59 telemann kernel: Call Trace:
> Feb 8 18:18:59 telemann kernel: [<ffffffff8022f10c>] __wake_up+0x38/0x4f
> Feb 8 18:18:59 telemann kernel: [<ffffffff880317ae>] :jbd:journal_stop+0x1f3/0x1ff
> Feb 8 18:18:59 telemann kernel: [<ffffffff802994d1>] flush_cpu_workqueue+0x83/0xb5
> Feb 8 18:18:59 telemann kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:18:59 telemann kernel: [<ffffffff80263914>] mutex_lock+0xd/0x1d
> Feb 8 18:18:59 telemann kernel: [<ffffffff80299563>] flush_workqueue+0x60/0x87
> Feb 8 18:18:59 telemann kernel: [<ffffffff80394af5>] release_dev+0x503/0x67b
> Feb 8 18:18:59 telemann kernel: [<ffffffff8020b860>] release_pages+0x158/0x165
> Feb 8 18:18:59 telemann kernel: [<ffffffff80255821>] tty_release+0x11/0x1a
> Feb 8 18:18:59 telemann kernel: [<ffffffff80213492>] __fput+0xd3/0x1bd
> Feb 8 18:18:59 telemann kernel: [<ffffffff802243cb>] filp_close+0x5c/0x64
> Feb 8 18:18:59 telemann kernel: [<ffffffff8023a392>] put_files_struct+0x63/0xae
> Feb 8 18:18:59 telemann kernel: [<ffffffff802160cd>] do_exit+0x31d/0x902
> Feb 8 18:18:59 telemann kernel: [<ffffffff8024ae4d>] cpuset_exit+0x0/0x88
> Feb 8 18:18:59 telemann kernel: [<ffffffff8022b920>] get_signal_to_deliver+0x477/0x4aa
> Feb 8 18:18:59 telemann kernel: [<ffffffff8025d19e>] do_notify_resume+0x9c/0x7ba
> Feb 8 18:19:00 telemann kernel: [<ffffffff80294ea1>] __group_send_sig_info+0xb9/0xc8
> Feb 8 18:19:00 telemann kernel: [<ffffffff8025cb0b>] group_send_sig_info+0x62/0x6f
> Feb 8 18:19:00 telemann kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:19:00 telemann kernel: [<ffffffff802afd73>] audit_syscall_entry+0x180/0x1b3
> Feb 8 18:19:00 telemann kernel: [<ffffffff80245a48>] sys_rt_sigreturn+0x327/0x35a
> Feb 8 18:19:00 telemann kernel: [<ffffffff802b0175>] audit_syscall_exit+0x336/0x362
> Feb 8 18:19:00 telemann kernel: [<ffffffff8026042c>] int_signal+0x12/0x17
> Feb 8 18:19:00 telemann kernel:
>
> Feb 8 18:11:23 handel kernel: xenbr0: received tcn bpdu on port 1(eth0)
> Feb 8 18:11:23 handel kernel: xenbr0: topology change detected, propagating
> Feb 8 18:14:54 handel kernel: INFO: task syslogd:11299 blocked for more than 120 seconds.
> Feb 8 18:14:54 handel kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:14:54 handel kernel: syslogd D 0000031e848fed46 0 11299 1 11302 11268 (NOTLB)
> Feb 8 18:14:54 handel kernel: ffff880079603d88 0000000000000282 0000000000000000 0000000000000001
> Feb 8 18:14:54 handel kernel: 000000000000000a ffff88007e5b9100 ffff88000002b040 0000000000026ea9
> Feb 8 18:14:54 handel kernel: ffff88007e5b92e8 0000000000000000
> Feb 8 18:14:54 handel kernel: Call Trace:
> Feb 8 18:14:54 handel kernel: [<ffffffff88036d5a>] :jbd:log_wait_commit+0xa3/0xf5
> Feb 8 18:14:54 handel kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:14:54 handel kernel: [<ffffffff8803178a>] :jbd:journal_stop+0x1cf/0x1ff
> Feb 8 18:14:54 handel kernel: [<ffffffff8023119d>] __writeback_single_inode+0x1e9/0x328
> Feb 8 18:19:15 handel kernel: [<ffffffff802d330d>] do_readv_writev+0x26e/0x291
> Feb 8 18:19:15 handel kernel: [<ffffffff802e5b8b>] sync_inode+0x24/0x33
> Feb 8 18:19:15 handel kernel: [<ffffffff8804c36d>] :ext3:ext3_sync_file+0xc9/0xdc
> Feb 8 18:19:15 handel kernel: [<ffffffff80251e07>] do_fsync+0x52/0xa4
> Feb 8 18:19:15 handel kernel: [<ffffffff802d3b11>] __do_fsync+0x23/0x36
> Feb 8 18:19:15 handel kernel: [<ffffffff802602f9>] tracesys+0xab/0xb6
> Feb 8 18:19:15 handel kernel:
> Feb 8 18:19:15 handel kernel: INFO: task syslogd:11299 blocked for more than 120 seconds.
> Feb 8 18:19:15 handel kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:19:15 handel kernel: syslogd D 0000031e848fed46 0 11299 1 11302 11268 (NOTLB)
> Feb 8 18:19:15 handel kernel: ffff880079603d88 0000000000000282 0000000000000000 0000000000000001
> Feb 8 18:19:15 handel kernel: 000000000000000a ffff88007e5b9100 ffff88000002b040 0000000000026ea9
> Feb 8 18:19:15 handel kernel: ffff88007e5b92e8 0000000000000000
> Feb 8 18:19:15 handel kernel: Call Trace:
> Feb 8 18:19:15 handel kernel: [<ffffffff88036d5a>] :jbd:log_wait_commit+0xa3/0xf5
> Feb 8 18:19:15 handel kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:19:15 handel kernel: [<ffffffff8803178a>] :jbd:journal_stop+0x1cf/0x1ff
> Feb 8 18:19:15 handel kernel: [<ffffffff8023119d>] __writeback_single_inode+0x1e9/0x328
> Feb 8 18:19:15 handel kernel: [<ffffffff802d330d>] do_readv_writev+0x26e/0x291
> Feb 8 18:19:15 handel kernel: [<ffffffff802e5b8b>] sync_inode+0x24/0x33
> Feb 8 18:19:15 handel kernel: [<ffffffff8804c36d>] :ext3:ext3_sync_file+0xc9/0xdc
> Feb 8 18:19:15 handel kernel: [<ffffffff80251e07>] do_fsync+0x52/0xa4
> Feb 8 18:19:15 handel kernel: [<ffffffff802d3b11>] __do_fsync+0x23/0x36
> Feb 8 18:19:15 handel kernel: [<ffffffff802602f9>] tracesys+0xab/0xb6
> Feb 8 18:19:15 handel kernel:
> Feb 8 18:19:15 handel kernel: INFO: task syslogd:11299 blocked for more than 120 seconds.
> Feb 8 18:19:15 handel kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:19:15 handel kernel: syslogd D 0000031e848fed46 0 11299 1 11302 11268 (NOTLB)
> Feb 8 18:19:15 handel kernel: ffff880079603d88 0000000000000282 0000000000000000 0000000000000001
> Feb 8 18:19:15 handel kernel: 000000000000000a ffff88007e5b9100 ffff88000002b040 0000000000026ea9
> Feb 8 18:19:15 handel kernel: ffff88007e5b92e8 0000000000000000
> Feb 8 18:19:15 handel kernel: Call Trace:
> Feb 8 18:19:15 handel kernel: [<ffffffff88036d5a>] :jbd:log_wait_commit+0xa3/0xf5
> Feb 8 18:19:15 handel kernel: [<ffffffff8029c48f>] autoremove_wake_function+0x0/0x2e
> Feb 8 18:19:16 handel kernel: [<ffffffff8803178a>] :jbd:journal_stop+0x1cf/0x1ff
> Feb 8 18:19:16 handel kernel: [<ffffffff8023119d>] __writeback_single_inode+0x1e9/0x328
> Feb 8 18:19:16 handel kernel: [<ffffffff802d330d>] do_readv_writev+0x26e/0x291
> Feb 8 18:19:16 handel kernel: [<ffffffff802e5b8b>] sync_inode+0x24/0x33
> Feb 8 18:19:16 handel kernel: [<ffffffff8804c36d>] :ext3:ext3_sync_file+0xc9/0xdc
> Feb 8 18:19:16 handel kernel: [<ffffffff80251e07>] do_fsync+0x52/0xa4
> Feb 8 18:19:16 handel kernel: [<ffffffff802d3b11>] __do_fsync+0x23/0x36
> Feb 8 18:19:16 handel kernel: [<ffffffff802602f9>] tracesys+0xab/0xb6
> Feb 8 18:19:16 handel kernel:
>
> Cheers,
> Pim
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
prev parent reply other threads:[~2011-02-10 17:08 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-02-08 12:22 Xen hypervisor external denial of service vulnerability? Pim van Riezen
2011-02-08 12:39 ` Pim van Riezen
2011-02-08 15:53 ` Pasi Kärkkäinen
2011-02-08 16:10 ` Pim van Riezen
2011-02-08 16:28 ` Pim van Riezen
2011-02-08 16:51 ` Pasi Kärkkäinen
2011-02-08 17:08 ` Pim van Riezen
2011-02-08 17:21 ` Pim van Riezen
2011-02-10 17:08 ` Konrad Rzeszutek Wilk [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110210170828.GA3993@dumpdata.com \
--to=konrad.wilk@oracle.com \
--cc=pi+lists@panelsix.com \
--cc=xen-devel@lists.xensource.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.