From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sebastian Hetze Subject: kvm_queue_exception - what is going wrong here? Date: Wed, 16 Jun 2010 12:05:10 +0200 Message-ID: <20100616100511.1557B30303E4@mail.linux-ag.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii To: kvm@vger.kernel.org Return-path: Received: from ironport.linux-ag.com ([62.245.157.240]:16686 "EHLO ironport.linux-ag.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758745Ab0FPKOw (ORCPT ); Wed, 16 Jun 2010 06:14:52 -0400 Received: from localhost (mail.linux-ag.de [62.245.157.206]) by mail.linux-ag.de (Postfix) with ESMTP id 1557B30303E4 for ; Wed, 16 Jun 2010 12:05:11 +0200 (CEST) Content-Disposition: inline Sender: kvm-owner@vger.kernel.org List-ID: Hi *, we recently have encountered kvm_queue_exception events in dmesg. What is causing these exceptions? It appears that the guest is at least suffering performance loss. Is the qemu-system-x86 hang event linked to the exception? Is there anything we can do to prevent these exceptions to happen? Best regards, Sebastian --------------------------------- [1811816.496620] set_cr3: #GP, reserved bits [1811816.496624] ------------[ cut here ]------------ [1811816.496640] WARNING: at /usr/src/linux-2.6.31/arch/x86/kvm/x86.c:202 kvm_queue_exception_e+0x61/0x70 [kvm ]() [1811816.496643] Hardware name: S5520HC [1811816.496644] Modules linked in: drbd tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv 4 xt_state nf_conntrack ipt_REJECT xt_tcpudp ppdev iptable_filter ip_tables lp parport x_tables kvm_intel kvm bridge stp bonding e100 via_rhine 3c59x 8139too mii shpchp iTCO_wdt iTCO_vendor_support pcspkr joydev ses encl osure usbhid bnx2 aacraid e1000e igb dca fbcon tileblit font bitblit softcursor [1811816.496669] Pid: 26170, comm: qemu-system-x86 Not tainted 2.6.31-14-server #48 [1811816.496672] Call Trace: [1811816.496682] [] ? kvm_queue_exception_e+0x61/0x70 [kvm] [1811816.496689] [] warn_slowpath_common+0x78/0xd0 [1811816.496692] [] warn_slowpath_null+0xf/0x20 [1811816.496701] [] kvm_queue_exception_e+0x61/0x70 [kvm] [1811816.496711] [] load_guest_segment_descriptor+0x9f/0xb0 [kvm] [1811816.496721] [] kvm_load_segment_descriptor+0x43/0x110 [kvm] [1811816.496730] [] ? kvm_inject_gp+0x10/0x20 [kvm] [1811816.496740] [] ? kvm_set_cr3+0x15d/0x170 [kvm] [1811816.496749] [] load_state_from_tss32+0x167/0x240 [kvm] [1811816.496758] [] kvm_task_switch_32+0x119/0x130 [kvm] [1811816.496768] [] kvm_task_switch+0x153/0x300 [kvm] [1811816.496777] [] ? gfn_to_hva+0x11/0x90 [kvm] [1811816.496786] [] ? kvm_read_guest_page+0x62/0x70 [kvm] [1811816.496792] [] handle_task_switch+0x6b/0x160 [kvm_intel] [1811816.496796] [] vmx_handle_exit+0xf5/0x280 [kvm_intel] [1811816.496801] [] ? vmx_vcpu_run+0x292/0x856 [kvm_intel] [1811816.496811] [] vcpu_enter_guest+0x2cf/0x600 [kvm] [1811816.496814] [] ? finish_wait+0x5e/0x80 [1811816.496819] [] ? _spin_lock_irq+0x10/0x20 [1811816.496822] [] ? __down_read+0xc3/0xce [1811816.496825] [] ? autoremove_wake_function+0x0/0x40 [1811816.496834] [] __vcpu_run+0x63/0x330 [kvm] [1811816.496844] [] kvm_arch_vcpu_ioctl_run+0x8b/0x1f0 [kvm] [1811816.496853] [] kvm_vcpu_ioctl+0x2e2/0x630 [kvm] [1811816.496856] [] ? dequeue_signal+0x9f/0x180 [1811816.496859] [] vfs_ioctl+0x31/0xa0 [1811816.496862] [] do_vfs_ioctl+0x373/0x400 [1811816.496864] [] sys_ioctl+0x99/0xa0 [1811816.496862] [] do_vfs_ioctl+0x373/0x400 [1811816.496864] [] sys_ioctl+0x99/0xa0 [1811816.496867] [] ? do_sigpending+0xa9/0xc0 [1811816.496870] [] system_call_fastpath+0x16/0x1b [1811816.496872] ---[ end trace c6ea0da2c3c9d0e4 ]--- [1811816.496887] set_cr3: #GP, pdptrs reserved bits [1811816.496889] ------------[ cut here ]------------ [1811816.496898] WARNING: at /usr/src/linux-2.6.31/arch/x86/kvm/x86.c:202 kvm_queue_exception_e+0x61/0x70 [kvm]() [1811816.496900] Hardware name: S5520HC [1811816.496901] Modules linked in: drbd tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp ppdev iptable_filter ip_tables lp parport x_tables kvm_intel kvm bridge stp bonding e100 via_rhine 3c59x 8139too mii shpchp iTCO_wdt iTCO_vendor_support pcspkr joydev ses enclosure usbhid bnx2 aacraid e1000e igb dca fbcon tileblit font bitblit softcursor [1811816.496923] Pid: 26170, comm: qemu-system-x86 Tainted: G W 2.6.31-14-server #48 [1811816.496925] Call Trace: [1811816.496934] [] ? kvm_queue_exception_e+0x61/0x70 [kvm] [1811816.496937] [] warn_slowpath_common+0x78/0xd0 [1811816.496940] [] warn_slowpath_null+0xf/0x20 [1811816.496950] [] kvm_queue_exception_e+0x61/0x70 [kvm] [1811816.496959] [] load_guest_segment_descriptor+0x9f/0xb0 [kvm] [1811816.496969] [] kvm_load_segment_descriptor+0x43/0x110 [kvm] [1811816.496978] [] ? kvm_set_cr3+0xc2/0x170 [kvm] [1811816.496988] [] load_state_from_tss32+0x19c/0x240 [kvm] [1811816.496997] [] kvm_task_switch_32+0x119/0x130 [kvm] [1811816.497006] [] kvm_task_switch+0x153/0x300 [kvm] [1811816.497015] [] ? gfn_to_hva+0x11/0x90 [kvm] [1811816.497024] [] ? kvm_read_guest_page+0x62/0x70 [kvm] [1811816.497029] [] handle_task_switch+0x6b/0x160 [kvm_intel] [1811816.497034] [] vmx_handle_exit+0xf5/0x280 [kvm_intel] [1811816.497038] [] ? vmx_vcpu_run+0x292/0x856 [kvm_intel] [1811816.497048] [] vcpu_enter_guest+0x2cf/0x600 [kvm] [1811816.497052] [] ? __vmx_load_host_state+0xc3/0x1e0 [kvm_intel] [1811816.497055] [] ? finish_wait+0x5e/0x80 [1811816.497059] [] ? default_spin_lock_flags+0x9/0x10 [1811816.497062] [] ? _spin_lock_irq+0x10/0x20 [1811816.497065] [] ? __down_read+0xc3/0xce [1811816.497074] [] __vcpu_run+0x63/0x330 [kvm] [1811816.497083] [] kvm_arch_vcpu_ioctl_run+0x8b/0x1f0 [kvm] [1811816.497092] [] kvm_vcpu_ioctl+0x2e2/0x630 [kvm] [1811816.497095] [] ? dequeue_signal+0x9f/0x180 [1811816.497098] [] vfs_ioctl+0x31/0xa0 [1811816.497100] [] do_vfs_ioctl+0x373/0x400 [1811816.497103] [] sys_ioctl+0x99/0xa0 [1811816.497105] [] ? do_sigpending+0xa9/0xc0 [1811816.497109] [] system_call_fastpath+0x16/0x1b [1811816.497110] ---[ end trace c6ea0da2c3c9d0e5 ]--- ...... [3307007.208029] INFO: task qemu-system-x86:5327 blocked for more than 120 seconds. [3307007.208176] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [3307007.208350] qemu-system-x D 00000000ffffffff 0 5327 1 0x00000000 [3307007.208356] ffff880bc80f3af8 0000000000000086 0000000000015880 0000000000015880 [3307007.208364] 0000000000015880 0000000000015880 0000000000015880 0000000000015880 [3307007.208369] 0000000000015880 ffff880c4f8fad60 ffff880654d48000 ffff880c4f8fb110 [3307007.208374] Call Trace: [3307007.208390] [] io_schedule+0x37/0x50 [3307007.208403] [] dio_await_completion+0x71/0x100 [3307007.208415] [] direct_io_worker+0x2b6/0x390 [3307007.208425] [] __blockdev_direct_IO+0x287/0x3f0 [3307007.208436] [] ? blkdev_get_blocks+0x0/0xc0 [3307007.208445] [] blkdev_direct_IO+0x49/0x50 [3307007.208454] [] ? blkdev_get_blocks+0x0/0xc0 [3307007.208468] [] generic_file_direct_write+0xc8/0x230 [3307007.208479] [] __generic_file_aio_write_nolock+0x36e/0x470 [3307007.208494] [] ? hrtimer_try_to_cancel+0x3f/0x90 [3307007.208504] [] generic_file_aio_write_nolock+0x37/0xa0 [3307007.208516] [] do_sync_write+0xf1/0x140 [3307007.208527] [] ? autoremove_wake_function+0x0/0x40 [3307007.208539] [] ? futex_wake+0x113/0x130 [3307007.208552] [] ? security_file_permission+0x11/0x20 [3307007.208562] [] vfs_write+0xcb/0x130 [3307007.208571] [] sys_pwrite64+0x92/0xa0 [3307007.208582] [] system_call_fastpath+0x16/0x1b [3307007.208592] INFO: task qemu-system-x86:5320 blocked for more than 120 seconds. [3307007.208724] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [3307007.208858] qemu-system-x D 0000000000000000 0 5320 1 0x00400000 [3307007.208863] ffff880bb3d45af8 0000000000000082 0000000000015880 0000000000015880 [3307007.208868] 0000000000015880 0000000000015880 0000000000015880 0000000000015880 [3307007.208873] 0000000000015880 ffff880c4fd2c410 ffff880654c94410 ffff880c4fd2c7c0 [3307007.208877] Call Trace: [3307007.208882] [] io_schedule+0x37/0x50 [3307007.208886] [] dio_await_completion+0x71/0x100 [3307007.208890] [] direct_io_worker+0x2b6/0x390 [3307007.208895] [] __blockdev_direct_IO+0x287/0x3f0 [3307007.208904] [] ? blkdev_get_blocks+0x0/0xc0 [3307007.208916] [] blkdev_direct_IO+0x49/0x50 [3307007.208886] [] dio_await_completion+0x71/0x100 [3307007.208890] [] direct_io_worker+0x2b6/0x390 [3307007.208895] [] __blockdev_direct_IO+0x287/0x3f0 [3307007.208904] [] ? blkdev_get_blocks+0x0/0xc0 [3307007.208916] [] blkdev_direct_IO+0x49/0x50 [3307007.208926] [] ? blkdev_get_blocks+0x0/0xc0 [3307007.208936] [] generic_file_direct_write+0xc8/0x230 [3307007.208949] [] __generic_file_aio_write_nolock+0x36e/0x470 [3307007.208960] [] ? hrtimer_try_to_cancel+0x3f/0x90 [3307007.208971] [] generic_file_aio_write_nolock+0x37/0xa0 [3307007.208981] [] do_sync_write+0xf1/0x140 [3307007.208991] [] ? autoremove_wake_function+0x0/0x40 [3307007.209002] [] ? futex_wake+0x113/0x130 [3307007.209012] [] ? security_file_permission+0x11/0x20 [3307007.209023] [] vfs_write+0xcb/0x130 [3307007.209033] [] sys_pwrite64+0x92/0xa0 [3307007.209043] [] system_call_fastpath+0x16/0x1b ...... [4837287.970390] set_cr3: #GP, reserved bits [4837287.970394] ------------[ cut here ]------------ [4837287.970411] WARNING: at /usr/src/linux-2.6.31/arch/x86/kvm/x86.c:202 kvm_queue_exception_e+0x61/0x70 [kvm]() [4837287.970413] Hardware name: S5520HC [4837287.970414] Modules linked in: drbd tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp ppdev iptable_filter ip_tables lp parport x_tables kvm_intel kvm bridge stp bonding e100 via_rhine 3c59x 8139too mii shpchp iTCO_wdt iTCO_vendor_support pcspkr joydev ses enclosure usbhid bnx2 aacraid e1000e igb dca fbcon tileblit font bitblit softcursor [4837287.970440] Pid: 6524, comm: qemu-system-x86 Tainted: G W 2.6.31-14-server #48 [4837287.970442] Call Trace: [4837287.970453] [] ? kvm_queue_exception_e+0x61/0x70 [kvm] [4837287.970459] [] warn_slowpath_common+0x78/0xd0 [4837287.970462] [] warn_slowpath_null+0xf/0x20 [4837287.970472] [] kvm_queue_exception_e+0x61/0x70 [kvm] [4837287.970481] [] load_guest_segment_descriptor+0x9f/0xb0 [kvm] [4837287.970491] [] kvm_load_segment_descriptor+0x43/0x110 [kvm] [4837287.970501] [] ? kvm_set_cr3+0x15d/0x170 [kvm] [4837287.970510] [] load_state_from_tss32+0x19c/0x240 [kvm] [4837287.970520] [] kvm_task_switch_32+0x119/0x130 [kvm] [4837287.970529] [] kvm_task_switch+0x153/0x300 [kvm] [4837287.970535] [] handle_task_switch+0x6b/0x160 [kvm_intel] [4837287.970540] [] vmx_handle_exit+0xf5/0x280 [kvm_intel] [4837287.970545] [] ? vmx_vcpu_run+0x292/0x856 [kvm_intel] [4837287.970554] [] vcpu_enter_guest+0x2cf/0x600 [kvm] [4837287.970558] [] ? finish_wait+0x5e/0x80 [4837287.970563] [] ? _spin_lock_irq+0x10/0x20 [4837287.970566] [] ? __down_read+0xc3/0xce [4837287.970568] [] ? autoremove_wake_function+0x0/0x40 [4837287.970578] [] __vcpu_run+0x63/0x330 [kvm] [4837287.970587] [] kvm_arch_vcpu_ioctl_run+0x8b/0x1f0 [kvm] [4837287.970596] [] kvm_vcpu_ioctl+0x2e2/0x630 [kvm] [4837287.970599] [] ? dequeue_signal+0x9f/0x180 [4837287.970603] [] ? common_interrupt+0xe/0x13 [4837287.970607] [] vfs_ioctl+0x31/0xa0 [4837287.970609] [] do_vfs_ioctl+0x373/0x400 [4837287.970612] [] ? sys_ioctl+0x9b/0xa0 [4837287.970614] [] sys_ioctl+0x99/0xa0 [4837287.970617] [] ? do_sigpending+0xa9/0xc0 [4837287.970620] [] system_call_fastpath+0x16/0x1b [4837287.970622] ---[ end trace c6ea0da2c3c9d0e6 ]---