From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:60949) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1URzv6-0008Jl-LG for qemu-devel@nongnu.org; Tue, 16 Apr 2013 03:03:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1URzv5-0006g5-1d for qemu-devel@nongnu.org; Tue, 16 Apr 2013 03:03:12 -0400 Received: from mout.web.de ([212.227.15.3]:63521) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1URzv4-0006fy-OY for qemu-devel@nongnu.org; Tue, 16 Apr 2013 03:03:10 -0400 Message-ID: <516CF7AB.9090107@web.de> Date: Tue, 16 Apr 2013 09:03:07 +0200 From: Jan Kiszka MIME-Version: 1.0 References: <516BAF96.1010402@web.de> In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="----enig2ULSRPKMQXGNDRDFEFEIE" Subject: Re: [Qemu-devel] [qemu-devel] Bug Report: VM crashed for some kinds of vCPU in nested virtualization List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: =?UTF-8?B?IuadjuaYpeWlhyA8QXJ0aHVyIENodW5xaSBMaT4i?= Cc: qemu-devel@nongnu.org, kvm This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2ULSRPKMQXGNDRDFEFEIE Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 2013-04-16 05:49, =E6=9D=8E=E6=98=A5=E5=A5=87 wrote= : > I changed to the latest version of kvm kernel but the bug also occured.= >=20 > On the startup of L1 VM on the host, the host kern.log will output: > Apr 16 11:28:22 Blade1-02 kernel: [ 4908.458090] kvm [2808]: vcpu0 > unhandled rdmsr: 0x345 > Apr 16 11:28:22 Blade1-02 kernel: [ 4908.458166] kvm_set_msr_common: 22= > callbacks suppressed > Apr 16 11:28:22 Blade1-02 kernel: [ 4908.458169] kvm [2808]: vcpu0 > unhandled wrmsr: 0x40 data 0 > Apr 16 11:28:22 Blade1-02 kernel: [ 4908.458176] kvm [2808]: vcpu0 > unhandled wrmsr: 0x60 data 0 > Apr 16 11:28:22 Blade1-02 kernel: [ 4908.458182] kvm [2808]: vcpu0 > unhandled wrmsr: 0x41 data 0 > Apr 16 11:28:22 Blade1-02 kernel: [ 4908.458188] kvm [2808]: vcpu0 > unhandled wrmsr: 0x61 data 0 > Apr 16 11:28:22 Blade1-02 kernel: [ 4908.458194] kvm [2808]: vcpu0 > unhandled wrmsr: 0x42 data 0 > Apr 16 11:28:22 Blade1-02 kernel: [ 4908.458200] kvm [2808]: vcpu0 > unhandled wrmsr: 0x62 data 0 > Apr 16 11:28:22 Blade1-02 kernel: [ 4908.458206] kvm [2808]: vcpu0 > unhandled wrmsr: 0x43 data 0 > Apr 16 11:28:22 Blade1-02 kernel: [ 4908.458211] kvm [2808]: vcpu0 > unhandled wrmsr: 0x63 data 0 > Apr 16 11:28:23 Blade1-02 kernel: [ 4908.471014] kvm [2808]: vcpu1 > unhandled wrmsr: 0x40 data 0 > Apr 16 11:28:23 Blade1-02 kernel: [ 4908.471024] kvm [2808]: vcpu1 > unhandled wrmsr: 0x60 data 0 >=20 > When L1 VM starts and crashes, its kern.log will output: > Apr 16 11:28:55 kvm1 kernel: [ 33.590101] device tap0 entered promisc= uous > mode > Apr 16 11:28:55 kvm1 kernel: [ 33.590140] br0: port 2(tap0) entered > forwarding state > Apr 16 11:28:55 kvm1 kernel: [ 33.590146] br0: port 2(tap0) entered > forwarding state > Apr 16 11:29:04 kvm1 kernel: [ 42.592103] br0: port 2(tap0) entered > forwarding state > Apr 16 11:29:19 kvm1 kernel: [ 57.752731] kvm [1673]: vcpu0 unhandled= > rdmsr: 0x345 > Apr 16 11:29:19 kvm1 kernel: [ 57.797261] kvm [1673]: vcpu0 unhandled= > wrmsr: 0x40 data 0 > Apr 16 11:29:19 kvm1 kernel: [ 57.797315] kvm [1673]: vcpu0 unhandled= > wrmsr: 0x60 data 0 > Apr 16 11:29:19 kvm1 kernel: [ 57.797366] kvm [1673]: vcpu0 unhandled= > wrmsr: 0x41 data 0 > Apr 16 11:29:19 kvm1 kernel: [ 57.797416] kvm [1673]: vcpu0 unhandled= > wrmsr: 0x61 data 0 > Apr 16 11:29:19 kvm1 kernel: [ 57.797466] kvm [1673]: vcpu0 unhandled= > wrmsr: 0x42 data 0 > Apr 16 11:29:19 kvm1 kernel: [ 57.797516] kvm [1673]: vcpu0 unhandled= > wrmsr: 0x62 data 0 > Apr 16 11:29:19 kvm1 kernel: [ 57.797566] kvm [1673]: vcpu0 unhandled= > wrmsr: 0x43 data 0 > Apr 16 11:29:19 kvm1 kernel: [ 57.797616] kvm [1673]: vcpu0 unhandled= > wrmsr: 0x63 data 0 >=20 > The host will output simultaneously: > Apr 16 11:29:20 Blade1-02 kernel: [ 4966.314742] nested_vmx_run: VMCS > MSR_{LOAD,STORE} unsupported That's an important information. KVM is not yet implementing this feature, but L1 is using it - doomed to fail. This feature gap of nested VMX needs to be closed at some point. >=20 > And the callback trace displayed on the console is the same as the prev= ious > mail. >=20 > Besides, the L1 and L2 guest may sometimes crash and output nothing, wh= ile > sometimes it will output as above. >=20 >=20 > So this indicates that the msr controls may fail for core2duo CPU emula= tor. >=20 Maybe varying the CPU type (try e.g. -cpu kvm64,+vmx) reduces the likeliness of this scenario with KVM as guest. >=20 > For Jan, > I have traced the code of qemu and KVM and found the relevant code of e= rrno > "KVM: entry failed, hardware error 0x7". The relevant code is in kernel= > arch/x86/kvm/vmx.c, function vmx_handle_exit(): >=20 > if (exit_reason & VMX_EXIT_REASONS_FAILED_VMENTRY) { > vcpu->run->exit_reason =3D KVM_EXIT_FAIL_ENTRY; > vcpu->run->fail_entry.hardware_entry_failure_reason > =3D exit_reason; > return 0; > } >=20 > if (unlikely(vmx->fail)) { > vcpu->run->exit_reason =3D KVM_EXIT_FAIL_ENTRY; > vcpu->run->fail_entry.hardware_entry_failure_reason > =3D vmcs_read32(VM_INSTRUCTION_ERROR); > return 0; > } >=20 > The entry failed hardware error may be caused from these two points, bo= th > are caused by VMENTRY failed. Because macro VMX_EXIT_REASONS_FAILED_VME= NTRY > is 0x80000000 and the output errno is 0x7, so this error is caused by t= he > second branch. I'm not very clear what the result of > vmcs_read32(VM_INSTRUCTION_ERROR) refers to. Try to look this up in the Intel manual. It explains what instruction error 7 means. You will also find it when tracing down the error message of L0. Jan ------enig2ULSRPKMQXGNDRDFEFEIE Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlFs96wACgkQitSsb3rl5xQpfACeMYLAbc70wfoWxddYNUYRtN43 dcsAnA8BM/Y6wOJLLmHgN7xl4rQ7S1rV =TfWV -----END PGP SIGNATURE----- ------enig2ULSRPKMQXGNDRDFEFEIE--