From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: Xen hypervisor external denial of service vulnerability? Date: Thu, 10 Feb 2011 12:08:29 -0500 Message-ID: <20110210170828.GA3993@dumpdata.com> References: <20110208155359.GW2754@reaktio.net> <4320A8B3-0E1A-40E7-8AF3-CABCDC822EAC@panelsix.com> <20110208165140.GX2754@reaktio.net> <613ECEE5-AA2C-486E-9438-18061A341644@panelsix.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Pim van Riezen Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org On Tue, Feb 08, 2011 at 06:21:25PM +0100, Pim van Riezen wrote: >=20 > On Feb 8, 2011, at 18:08 , Pim van Riezen wrote: >=20 > > On Feb 8, 2011, at 17:51 , Pasi K=E4rkk=E4inen wrote: > >>=20 > >> Did you also make sure VMs don't use those 2 pcpus dedicated for dom= 0?=20 > >> You have to explicitly configure each VM not to use those pcpus. > >=20 > > That seems to have done the trick. >=20 > Alas, I was too soon in drawing a conclusion. After a new 10 minute run= : Did you try to run the 2.6.32 pvops type kernel? Asking b/c it looks like the issue is due to the fact that mutex lock is held for a very very long= time. The spinlock implementation in 2.6.32 changed so it might provide a bette= r solution. >=20 > Feb 8 18:12:30 telemann kernel: INFO: task bash:12225 blocked for more= than 120 seconds. > Feb 8 18:12:30 telemann kernel: "echo 0 > /proc/sys/kernel/hung_task_t= imeout_secs" disables this message. > Feb 8 18:12:30 telemann kernel: bash D ffff88006ac7bd08 0= 12225 1 8260 (L-TLB) > Feb 8 18:12:30 telemann kernel: ffff88006ac7bb88 0000000000000246 0= 000000300000000 ffff88007ec3a6d8=20 > Feb 8 18:12:30 telemann kernel: 0000000000000009 ffff88006c16e820 f= fff88007a5a9080 000000000008f03e=20 > Feb 8 18:12:30 telemann kernel: ffff88006c16ea08 ffffffff8022f10c=20 > Feb 8 18:12:30 telemann kernel: Call Trace: > Feb 8 18:12:30 telemann kernel: [] __wake_up+0x38/0= x4f > Feb 8 18:12:30 telemann kernel: [] :jbd:journal_sto= p+0x1f3/0x1ff > Feb 8 18:12:30 telemann kernel: [] flush_cpu_workqu= eue+0x83/0xb5 > Feb 8 18:12:30 telemann kernel: [] autoremove_wake_= function+0x0/0x2e > Feb 8 18:12:30 telemann kernel: [] mutex_lock+0xd/0= x1d > Feb 8 18:12:30 telemann kernel: [] flush_workqueue+= 0x60/0x87 > Feb 8 18:12:41 telemann kernel: [] release_dev+0x50= 3/0x67b > Feb 8 18:12:55 telemann kernel: [] release_pages+0x= 158/0x165 > Feb 8 18:13:09 telemann kernel: [] tty_release+0x11= /0x1a > Feb 8 18:13:23 telemann kernel: [] __fput+0xd3/0x1b= d > Feb 8 18:13:38 telemann kernel: [] filp_close+0x5c/= 0x64 > Feb 8 18:13:51 telemann kernel: [] put_files_struct= +0x63/0xae > Feb 8 18:14:06 telemann kernel: [] do_exit+0x31d/0x= 902 > Feb 8 18:14:19 telemann kernel: [] cpuset_exit+0x0/= 0x88 > Feb 8 18:14:33 telemann kernel: [] get_signal_to_de= liver+0x477/0x4aa > Feb 8 18:14:49 telemann kernel: [] do_notify_resume= +0x9c/0x7ba > Feb 8 18:15:01 telemann kernel: [] __group_send_sig= _info+0xb9/0xc8 > Feb 8 18:15:08 telemann kernel: [] group_send_sig_i= nfo+0x62/0x6f > Feb 8 18:15:22 telemann kernel: [] autoremove_wake_= function+0x0/0x2e > Feb 8 18:15:37 telemann kernel: [] audit_syscall_en= try+0x180/0x1b3 > Feb 8 18:15:49 telemann kernel: [] sys_rt_sigreturn= +0x327/0x35a > Feb 8 18:16:03 telemann kernel: [] audit_syscall_ex= it+0x336/0x362 > Feb 8 18:16:17 telemann kernel: [] int_signal+0x12/= 0x17 > Feb 8 18:16:31 telemann kernel:=20 > Feb 8 18:16:44 telemann kernel: INFO: task bash:12225 blocked for more= than 120 seconds. > Feb 8 18:16:58 telemann kernel: "echo 0 > /proc/sys/kernel/hung_task_t= imeout_secs" disables this message. > Feb 8 18:17:12 telemann kernel: bash D ffff88006ac7bd08 0= 12225 1 8260 (L-TLB) > Feb 8 18:17:26 telemann kernel: ffff88006ac7bb88 0000000000000246 0= 000000300000000 ffff88007ec3a6d8=20 > Feb 8 18:17:39 telemann kernel: 0000000000000009 ffff88006c16e820 f= fff88007a5a9080 000000000008f03e=20 > Feb 8 18:17:54 telemann kernel: ffff88006c16ea08 ffffffff8022f10c=20 > Feb 8 18:18:08 telemann kernel: Call Trace: > Feb 8 18:18:21 telemann kernel: [] __wake_up+0x38/0= x4f > Feb 8 18:18:34 telemann kernel: [] :jbd:journal_sto= p+0x1f3/0x1ff > Feb 8 18:18:47 telemann kernel: [] flush_cpu_workqu= eue+0x83/0xb5 > Feb 8 18:18:58 telemann kernel: [] autoremove_wake_= function+0x0/0x2e > Feb 8 18:18:58 telemann kernel: [] mutex_lock+0xd/0= x1d > Feb 8 18:18:58 telemann kernel: [] flush_workqueue+= 0x60/0x87 > Feb 8 18:18:58 telemann kernel: [] release_dev+0x50= 3/0x67b > Feb 8 18:18:58 telemann kernel: [] release_pages+0x= 158/0x165 > Feb 8 18:18:58 telemann kernel: [] tty_release+0x11= /0x1a > Feb 8 18:18:58 telemann kernel: [] __fput+0xd3/0x1b= d > Feb 8 18:18:58 telemann kernel: [] filp_close+0x5c/= 0x64 > Feb 8 18:18:58 telemann kernel: [] put_files_struct= +0x63/0xae > Feb 8 18:18:58 telemann kernel: [] do_exit+0x31d/0x= 902 > Feb 8 18:18:58 telemann kernel: [] cpuset_exit+0x0/= 0x88 > Feb 8 18:18:58 telemann kernel: [] get_signal_to_de= liver+0x477/0x4aa > Feb 8 18:18:58 telemann kernel: [] do_notify_resume= +0x9c/0x7ba > Feb 8 18:18:58 telemann kernel: [] __group_send_sig= _info+0xb9/0xc8 > Feb 8 18:18:58 telemann kernel: [] group_send_sig_i= nfo+0x62/0x6f > Feb 8 18:18:58 telemann kernel: [] autoremove_wake_= function+0x0/0x2e > Feb 8 18:18:58 telemann kernel: [] audit_syscall_en= try+0x180/0x1b3 > Feb 8 18:18:58 telemann kernel: [] sys_rt_sigreturn= +0x327/0x35a > Feb 8 18:18:58 telemann kernel: [] audit_syscall_ex= it+0x336/0x362 > Feb 8 18:18:59 telemann kernel: [] int_signal+0x12/= 0x17 > Feb 8 18:18:59 telemann kernel:=20 > Feb 8 18:18:59 telemann kernel: INFO: task bash:12225 blocked for more= than 120 seconds. > Feb 8 18:18:59 telemann kernel: "echo 0 > /proc/sys/kernel/hung_task_t= imeout_secs" disables this message. > Feb 8 18:18:59 telemann kernel: bash D ffff88006ac7bd08 0= 12225 1 8260 (L-TLB) > Feb 8 18:18:59 telemann kernel: ffff88006ac7bb88 0000000000000246 0= 000000300000000 ffff88007ec3a6d8=20 > Feb 8 18:18:59 telemann kernel: 0000000000000009 ffff88006c16e820 f= fff88007a5a9080 000000000008f03e=20 > Feb 8 18:18:59 telemann kernel: ffff88006c16ea08 ffffffff8022f10c=20 > Feb 8 18:18:59 telemann kernel: Call Trace: > Feb 8 18:18:59 telemann kernel: [] __wake_up+0x38/0= x4f > Feb 8 18:18:59 telemann kernel: [] :jbd:journal_sto= p+0x1f3/0x1ff > Feb 8 18:18:59 telemann kernel: [] flush_cpu_workqu= eue+0x83/0xb5 > Feb 8 18:18:59 telemann kernel: [] autoremove_wake_= function+0x0/0x2e > Feb 8 18:18:59 telemann kernel: [] mutex_lock+0xd/0= x1d > Feb 8 18:18:59 telemann kernel: [] flush_workqueue+= 0x60/0x87 > Feb 8 18:18:59 telemann kernel: [] release_dev+0x50= 3/0x67b > Feb 8 18:18:59 telemann kernel: [] release_pages+0x= 158/0x165 > Feb 8 18:18:59 telemann kernel: [] tty_release+0x11= /0x1a > Feb 8 18:18:59 telemann kernel: [] __fput+0xd3/0x1b= d > Feb 8 18:18:59 telemann kernel: [] filp_close+0x5c/= 0x64 > Feb 8 18:18:59 telemann kernel: [] put_files_struct= +0x63/0xae > Feb 8 18:18:59 telemann kernel: [] do_exit+0x31d/0x= 902 > Feb 8 18:18:59 telemann kernel: [] cpuset_exit+0x0/= 0x88 > Feb 8 18:18:59 telemann kernel: [] get_signal_to_de= liver+0x477/0x4aa > Feb 8 18:18:59 telemann kernel: [] do_notify_resume= +0x9c/0x7ba > Feb 8 18:18:59 telemann kernel: [] __group_send_sig= _info+0xb9/0xc8 > Feb 8 18:18:59 telemann kernel: [] group_send_sig_i= nfo+0x62/0x6f > Feb 8 18:18:59 telemann kernel: [] autoremove_wake_= function+0x0/0x2e > Feb 8 18:18:59 telemann kernel: [] audit_syscall_en= try+0x180/0x1b3 > Feb 8 18:18:59 telemann kernel: [] sys_rt_sigreturn= +0x327/0x35a > Feb 8 18:18:59 telemann kernel: [] audit_syscall_ex= it+0x336/0x362 > Feb 8 18:18:59 telemann kernel: [] int_signal+0x12/= 0x17 > Feb 8 18:18:59 telemann kernel:=20 > Feb 8 18:18:59 telemann kernel: INFO: task bash:12225 blocked for more= than 120 seconds. > Feb 8 18:18:59 telemann kernel: "echo 0 > /proc/sys/kernel/hung_task_t= imeout_secs" disables this message. > Feb 8 18:18:59 telemann kernel: bash D ffff88006ac7bd08 0= 12225 1 8260 (L-TLB) > Feb 8 18:18:59 telemann kernel: ffff88006ac7bb88 0000000000000246 0= 000000300000000 ffff88007ec3a6d8=20 > Feb 8 18:18:59 telemann kernel: 0000000000000009 ffff88006c16e820 f= fff88007a5a9080 000000000008f03e=20 > Feb 8 18:18:59 telemann kernel: ffff88006c16ea08 ffffffff8022f10c=20 > Feb 8 18:18:59 telemann kernel: Call Trace: > Feb 8 18:18:59 telemann kernel: [] __wake_up+0x38/0= x4f > Feb 8 18:18:59 telemann kernel: [] :jbd:journal_sto= p+0x1f3/0x1ff > Feb 8 18:18:59 telemann kernel: [] flush_cpu_workqu= eue+0x83/0xb5 > Feb 8 18:18:59 telemann kernel: [] autoremove_wake_= function+0x0/0x2e > Feb 8 18:18:59 telemann kernel: [] mutex_lock+0xd/0= x1d > Feb 8 18:18:59 telemann kernel: [] flush_workqueue+= 0x60/0x87 > Feb 8 18:18:59 telemann kernel: [] release_dev+0x50= 3/0x67b > Feb 8 18:18:59 telemann kernel: [] release_pages+0x= 158/0x165 > Feb 8 18:18:59 telemann kernel: [] tty_release+0x11= /0x1a > Feb 8 18:18:59 telemann kernel: [] __fput+0xd3/0x1b= d > Feb 8 18:18:59 telemann kernel: [] filp_close+0x5c/= 0x64 > Feb 8 18:18:59 telemann kernel: [] put_files_struct= +0x63/0xae > Feb 8 18:18:59 telemann kernel: [] do_exit+0x31d/0x= 902 > Feb 8 18:18:59 telemann kernel: [] cpuset_exit+0x0/= 0x88 > Feb 8 18:18:59 telemann kernel: [] get_signal_to_de= liver+0x477/0x4aa > Feb 8 18:18:59 telemann kernel: [] do_notify_resume= +0x9c/0x7ba > Feb 8 18:19:00 telemann kernel: [] __group_send_sig= _info+0xb9/0xc8 > Feb 8 18:19:00 telemann kernel: [] group_send_sig_i= nfo+0x62/0x6f > Feb 8 18:19:00 telemann kernel: [] autoremove_wake_= function+0x0/0x2e > Feb 8 18:19:00 telemann kernel: [] audit_syscall_en= try+0x180/0x1b3 > Feb 8 18:19:00 telemann kernel: [] sys_rt_sigreturn= +0x327/0x35a > Feb 8 18:19:00 telemann kernel: [] audit_syscall_ex= it+0x336/0x362 > Feb 8 18:19:00 telemann kernel: [] int_signal+0x12/= 0x17 > Feb 8 18:19:00 telemann kernel:=20 >=20 > Feb 8 18:11:23 handel kernel: xenbr0: received tcn bpdu on port 1(eth0= ) > Feb 8 18:11:23 handel kernel: xenbr0: topology change detected, propag= ating > Feb 8 18:14:54 handel kernel: INFO: task syslogd:11299 blocked for mor= e than 120 seconds. > Feb 8 18:14:54 handel kernel: "echo 0 > /proc/sys/kernel/hung_task_tim= eout_secs" disables this message. > Feb 8 18:14:54 handel kernel: syslogd D 0000031e848fed46 0 1= 1299 1 11302 11268 (NOTLB) > Feb 8 18:14:54 handel kernel: ffff880079603d88 0000000000000282 000= 0000000000000 0000000000000001=20 > Feb 8 18:14:54 handel kernel: 000000000000000a ffff88007e5b9100 fff= f88000002b040 0000000000026ea9=20 > Feb 8 18:14:54 handel kernel: ffff88007e5b92e8 0000000000000000=20 > Feb 8 18:14:54 handel kernel: Call Trace: > Feb 8 18:14:54 handel kernel: [] :jbd:log_wait_comm= it+0xa3/0xf5 > Feb 8 18:14:54 handel kernel: [] autoremove_wake_fu= nction+0x0/0x2e > Feb 8 18:14:54 handel kernel: [] :jbd:journal_stop+= 0x1cf/0x1ff > Feb 8 18:14:54 handel kernel: [] __writeback_single= _inode+0x1e9/0x328 > Feb 8 18:19:15 handel kernel: [] do_readv_writev+0x= 26e/0x291 > Feb 8 18:19:15 handel kernel: [] sync_inode+0x24/0x= 33 > Feb 8 18:19:15 handel kernel: [] :ext3:ext3_sync_fi= le+0xc9/0xdc > Feb 8 18:19:15 handel kernel: [] do_fsync+0x52/0xa4 > Feb 8 18:19:15 handel kernel: [] __do_fsync+0x23/0x= 36 > Feb 8 18:19:15 handel kernel: [] tracesys+0xab/0xb6 > Feb 8 18:19:15 handel kernel:=20 > Feb 8 18:19:15 handel kernel: INFO: task syslogd:11299 blocked for mor= e than 120 seconds. > Feb 8 18:19:15 handel kernel: "echo 0 > /proc/sys/kernel/hung_task_tim= eout_secs" disables this message. > Feb 8 18:19:15 handel kernel: syslogd D 0000031e848fed46 0 1= 1299 1 11302 11268 (NOTLB) > Feb 8 18:19:15 handel kernel: ffff880079603d88 0000000000000282 000= 0000000000000 0000000000000001=20 > Feb 8 18:19:15 handel kernel: 000000000000000a ffff88007e5b9100 fff= f88000002b040 0000000000026ea9=20 > Feb 8 18:19:15 handel kernel: ffff88007e5b92e8 0000000000000000=20 > Feb 8 18:19:15 handel kernel: Call Trace: > Feb 8 18:19:15 handel kernel: [] :jbd:log_wait_comm= it+0xa3/0xf5 > Feb 8 18:19:15 handel kernel: [] autoremove_wake_fu= nction+0x0/0x2e > Feb 8 18:19:15 handel kernel: [] :jbd:journal_stop+= 0x1cf/0x1ff > Feb 8 18:19:15 handel kernel: [] __writeback_single= _inode+0x1e9/0x328 > Feb 8 18:19:15 handel kernel: [] do_readv_writev+0x= 26e/0x291 > Feb 8 18:19:15 handel kernel: [] sync_inode+0x24/0x= 33 > Feb 8 18:19:15 handel kernel: [] :ext3:ext3_sync_fi= le+0xc9/0xdc > Feb 8 18:19:15 handel kernel: [] do_fsync+0x52/0xa4 > Feb 8 18:19:15 handel kernel: [] __do_fsync+0x23/0x= 36 > Feb 8 18:19:15 handel kernel: [] tracesys+0xab/0xb6 > Feb 8 18:19:15 handel kernel:=20 > Feb 8 18:19:15 handel kernel: INFO: task syslogd:11299 blocked for mor= e than 120 seconds. > Feb 8 18:19:15 handel kernel: "echo 0 > /proc/sys/kernel/hung_task_tim= eout_secs" disables this message. > Feb 8 18:19:15 handel kernel: syslogd D 0000031e848fed46 0 1= 1299 1 11302 11268 (NOTLB) > Feb 8 18:19:15 handel kernel: ffff880079603d88 0000000000000282 000= 0000000000000 0000000000000001=20 > Feb 8 18:19:15 handel kernel: 000000000000000a ffff88007e5b9100 fff= f88000002b040 0000000000026ea9=20 > Feb 8 18:19:15 handel kernel: ffff88007e5b92e8 0000000000000000=20 > Feb 8 18:19:15 handel kernel: Call Trace: > Feb 8 18:19:15 handel kernel: [] :jbd:log_wait_comm= it+0xa3/0xf5 > Feb 8 18:19:15 handel kernel: [] autoremove_wake_fu= nction+0x0/0x2e > Feb 8 18:19:16 handel kernel: [] :jbd:journal_stop+= 0x1cf/0x1ff > Feb 8 18:19:16 handel kernel: [] __writeback_single= _inode+0x1e9/0x328 > Feb 8 18:19:16 handel kernel: [] do_readv_writev+0x= 26e/0x291 > Feb 8 18:19:16 handel kernel: [] sync_inode+0x24/0x= 33 > Feb 8 18:19:16 handel kernel: [] :ext3:ext3_sync_fi= le+0xc9/0xdc > Feb 8 18:19:16 handel kernel: [] do_fsync+0x52/0xa4 > Feb 8 18:19:16 handel kernel: [] __do_fsync+0x23/0x= 36 > Feb 8 18:19:16 handel kernel: [] tracesys+0xab/0xb6 > Feb 8 18:19:16 handel kernel:=20 >=20 > Cheers, > Pim >=20 >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel