From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: Failed vm entry with heavy use of emulator Date: Tue, 5 Jan 2016 13:45:33 +0000 Message-ID: <568BC8FD.6060406@citrix.com> References: <568BAF7E.802@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4450064015209657296==" Return-path: Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aGRvp-0004OO-9W for xen-devel@lists.xenproject.org; Tue, 05 Jan 2016 13:45:49 +0000 In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Tamas K Lengyel Cc: Xen-devel , Razvan Cojocaru List-Id: xen-devel@lists.xenproject.org --===============4450064015209657296== Content-Type: multipart/alternative; boundary="------------000401020002030801030401" --------------000401020002030801030401 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit On 05/01/16 12:05, Tamas K Lengyel wrote: > > > On Tue, Jan 5, 2016 at 12:56 PM, Andrew Cooper > > wrote: > > On 05/01/16 11:49, Tamas K Lengyel wrote: >> Hi all, >> I've been stress-testing the built-in emulator using the vm_event >> response VM_EVENT_FLAG_EMULATE feature. In the test I've turned >> all pages non-readable by default and all trapped instructions to >> be emulated. My test code can be found at >> https://github.com/tklengyel/xen/compare/read_emul?expand=1. >> >> The following crash is reproducible and has been verified by >> Razvan as well. >> >> (XEN) p2m.c:1726:d1v0 calling mem_access_emulate_one, kind 0 >> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid >> guest state (0). >> (XEN) ************* VMCS Area ************** >> (XEN) *** Guest State *** >> (XEN) CR0: actual=0x000000008001003b, shadow=0x000000008001003b, >> gh_mask=ffffffffffffffff >> (XEN) CR4: actual=0x00000000000426f9, shadow=0x00000000000406f9, >> gh_mask=ffffffffffffffff >> (XEN) CR3 = 0x0000000000185000 >> (XEN) PDPTE0 = 0x0000000000186001 PDPTE1 = 0x0000000000187001 >> (XEN) PDPTE2 = 0x0000000000188001 PDPTE3 = 0x0000000000189001 >> (XEN) RSP = 0x000000008276dc28 (0x000000008276dc28) RIP = >> 0x00000000826bce1c (0x00000000826bce1c) >> (XEN) RFLAGS=0x00000002 (0x00000002) DR7 = 0x0000000000000400 >> (XEN) Sysenter RSP=000000008078b000 CS:RIP=0008:00000000826830c0 >> (XEN) sel attr limit base >> (XEN) CS: 0008 0c09b ffffffff 0000000000000000 >> (XEN) DS: 0023 0c0f3 ffffffff 0000000000000000 >> (XEN) SS: 0010 0c093 ffffffff 0000000000000000 >> (XEN) ES: 0023 0c0f3 ffffffff 0000000000000000 >> (XEN) FS: 0030 04093 00003748 0000000082770c00 >> (XEN) GS: 0000 1c000 ffffffff 0000000000000000 >> (XEN) GDTR: 000003ff 0000000080b95000 >> (XEN) LDTR: 0000 1c000 ffffffff 0000000000000000 >> (XEN) IDTR: 000007ff 0000000080b95400 >> (XEN) TR: 0028 0008b 000020ab 00000000801da000 >> (XEN) EFER = 0x0000000000000000 PAT = 0x0007010600070106 >> (XEN) PreemptionTimer = 0x00000000 SM Base = 0x00000000 >> (XEN) DebugCtl = 0x0000000000000000 DebugExceptions = >> 0x0000000000000000 >> (XEN) Interruptibility = 00000000 ActivityState = 00000000 >> (XEN) *** Host State *** >> (XEN) RIP = 0xffff82d0802075c0 (vmx_asm_vmexit_handler) RSP = >> 0xffff830430d97f90 >> (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 >> (XEN) FSBase=0000000000000000 GSBase=0000000000000000 >> TRBase=ffff830430d9bc00 >> (XEN) GDTBase=ffff830430d8c000 IDTBase=ffff830430d98000 >> (XEN) CR0=000000008005003b CR3=00000004136d0000 CR4=00000000000426e0 >> (XEN) Sysenter RSP=ffff830430d97fc0 CS:RIP=e008:ffff82d08024db30 >> (XEN) EFER = 0x0000000000000000 PAT = 0x0000050100070406 >> (XEN) *** Control State *** >> (XEN) PinBased=0000003f CPUBased=b6a075fa SecondaryExec=000000eb >> (XEN) EntryControls=000051ff ExitControls=000fefff >> (XEN) ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000 >> (XEN) VMEntry: intr_info=800000d1 errcode=00000000 ilen=00000000 >> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 >> (XEN) reason=80000021 qualification=0000000000000000 >> (XEN) IDTVectoring: info=800000d1 errcode=00000000 >> (XEN) TSC Offset = 0x0000004ed9c86354 >> (XEN) TPR Threshold = 0x00 PostedIntrVec = 0x00 >> (XEN) EPT pointer = 0x000000041124e01e EPTP index = 0x0000 >> (XEN) Virtual processor ID = 0x0011 VMfunc controls = >> 0000000000000000 >> (XEN) ************************************** >> (XEN) domain_crash called from vmx.c:2761 >> >> Any tips on how to further debug this issue? > > Do you have a log of the instructions emulated? > > > I don't. Is there an easy way to get that beside manually sprinkling > debug messages around in the emulator? Not trivially, sadly. > > > > Has the emulator by any chance just emulated setting CR4.PAE? > > > Possibly but I don't think so as the guest has already been fully > booted so I would not expect it to touch that. At a guess, I think the fault is an emulated 'mov %reg, %cr3' while in 32bit PAE mode. The PDPTE{0..3} values look wonky. I encountered a similar crash with the xen test framework in HAP mode with a bad %cr3 update. The VMM is expected to emulate updates to PDPTE{0..3} if writes to %cr3 are trapped. See vmx_update_guest_cr() and vmx_load_pdptrs(). ~Andrew --------------000401020002030801030401 Content-Type: text/html; charset="utf-8" Content-Length: 9720 Content-Transfer-Encoding: quoted-printable
On 05/01/16 12:05, Tamas K Lengyel wrote:


On Tue, Jan 5, 2016 at 12:56 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
On 05/01/16 11:49, Tamas K Lengyel wrote:
Hi all,
I've been stress-testing the built-in emulator using the vm_event response VM_EVENT_FLAG_EMULATE feature. In the test I've turned all pages non-readable by default and all trapped instructions to be emulated. My test code can be found at https://github.com/tklengyel/xen/compare/read_emul=3Fexpand=3D1.

The following crash is reproducible and has been verified by Razvan as well.

(XEN) p2m.c:1726:d1v0 calling mem_access_emulate_one, kind 0
(XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0).
(XEN) ************* VMCS Area **************
(XEN) *** Guest State ***
(XEN) CR0: actual=3D0x000000008001003b, shadow=3D0x000000008001003b, gh_mask=3Dffffffffffffffff
(XEN) CR4: actual=3D0x00000000000426f9, shadow=3D0x00000000000406f9, gh_mask=3Dffffffffffffffff
(XEN) CR3 =3D 0x0000000000185000
(XEN) PDPTE0 =3D 0x0000000000186001=C2=A0 PDPTE1 =3D 0x0000000000187001
(XEN) PDPTE2 =3D 0x0000000000188001=C2=A0 PDPTE3 =3D 0x0000000000189001
(XEN) RSP =3D 0x000000008276dc28 (0x000000008276dc28)=C2=A0 RIP =3D 0x00000000826bce1c (0x00000000826bce1c)
(XEN) RFLAGS=3D0x00000002 (0x00000002)=C2=A0 DR7 =3D 0x0000000000000400
(XEN) Sysenter RSP=3D000000008078b000 CS:RIP=3D0008:00000000826830c0
(XEN)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sel=C2=A0 attr=C2=A0 limit=C2=A0=C2=A0 base
(XEN)=C2=A0=C2=A0 CS: 0008 0c09b ffffffff 0000000000000000
(XEN)=C2=A0=C2=A0 DS: 0023 0c0f3 ffffffff 0000000000000000
(XEN)=C2=A0=C2=A0 SS: 0010 0c093 ffffffff 0000000000000000
(XEN)=C2=A0=C2=A0 ES: 0023 0c0f3 ffffffff 0000000000000000
(XEN)=C2=A0=C2=A0 FS: 0030 04093 00003748 0000000082770c00
(XEN)=C2=A0=C2=A0 GS: 0000 1c000 ffffffff 0000000000000000
(XEN) GDTR:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 000003ff 0000000080b95000
(XEN) LDTR: 0000 1c000 ffffffff 0000000000000000
(XEN) IDTR:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 000007ff 0000000080b95400
(XEN)=C2=A0=C2=A0 TR: 0028 0008b 000020ab 00000000801da000
(XEN) EFER =3D 0x0000000000000000=C2=A0 PAT =3D 0x0007010600070106
(XEN) PreemptionTimer =3D 0x00000000=C2=A0 SM Base =3D 0x00000000
(XEN) DebugCtl =3D 0x0000000000000000=C2=A0 DebugExceptions =3D 0x0000000000000000
(XEN) Interruptibility =3D 00000000=C2=A0 ActivityState =3D 00000000
(XEN) *** Host State ***
(XEN) RIP =3D 0xffff82d0802075c0 (vmx_asm_vmexit_handler)=C2=A0 RSP =3D 0xffff830430d97f90
(XEN) CS=3De008 SS=3D0000 DS=3D0000 ES=3D0000 FS=3D0000 GS=3D0000 TR=3De040
(XEN) FSBase=3D0000000000000000 GSBase=3D0000000000000000 TRBase=3Dffff830430d9bc00
(XEN) GDTBase=3Dffff830430d8c000 IDTBase=3Dffff830430d98000
(XEN) CR0=3D000000008005003b CR3=3D00000004136d0000 CR4=3D00000000000426e0
(XEN) Sysenter RSP=3Dffff830430d97fc0 CS:RIP=3De008:ffff82d08024db30
(XEN) EFER =3D 0x0000000000000000=C2=A0 PAT =3D 0x0000050100070406
(XEN) *** Control State ***
(XEN) PinBased=3D0000003f CPUBased=3Db6a075fa SecondaryExec=3D000000eb
(XEN) EntryControls=3D000051ff ExitControls=3D000fefff
(XEN) ExceptionBitmap=3D00060042 PFECmask=3D00000000 PFECmatch=3D00000000
(XEN) VMEntry: intr_info=3D800000d1 errcode=3D00000000 ilen=3D00000000
(XEN) VMExit: intr_info=3D00000000 errcode=3D00000000 ilen=3D00000003
(XEN)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 reason=3D80000021 qualification=3D0000000000000000
(XEN) IDTVectoring: info=3D800000d1 errcode=3D00000000
(XEN) TSC Offset =3D 0x0000004ed9c86354
(XEN) TPR Threshold =3D 0x00=C2=A0 PostedIntrVec =3D 0x00
(XEN) EPT pointer =3D 0x000000041124e01e=C2=A0 EPTP index =3D 0x0000
(XEN) Virtual processor ID =3D 0x0011 VMfunc controls =3D 0000000000000000
(XEN) **************************************
(XEN) domain_crash called from vmx.c:2761

Any tips on how to further debug this issue=3F

Do you have a log of the instructions emulated=3F

I don't. Is there an easy way to get that beside manually sprinkling debug messages around in the emulator=3F

Not trivially, sadly.

=C2=A0

Has the emulator by any chance just emulated setting CR4.PAE=3F

Possibly but I don't think so as the guest has already been fully booted so I would not expect it to touch that.

At a guess, I think the fault is an emulated 'mov %reg, %cr3' while in 32bit PAE mode.=C2=A0 The PDPTE{0..3} values look wonky.

I encountered a similar crash with the xen test framework in HAP mode with a bad %cr3 update.=C2=A0 The VMM is expected to emulate updates to PDPTE{0..3} if writes to %cr3 are trapped.=C2=A0 See vmx_update_guest_cr() and vmx_load_pdptrs().

~Andrew
--------------000401020002030801030401-- --===============4450064015209657296== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============4450064015209657296==--