From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: Emulating in response of an int3 vm_event Date: Wed, 2 Dec 2015 18:41:00 +0000 Message-ID: <565F3B3C.7070604@citrix.com> References: <565CE349.4070207@bitdefender.com> <565D7923.5080806@citrix.com> <565F39B6.3020001@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2425221411938935409==" Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1a4CLZ-00016X-Tx for xen-devel@lists.xenproject.org; Wed, 02 Dec 2015 18:41:46 +0000 In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Tamas K Lengyel Cc: Xen-devel , Razvan Cojocaru List-Id: xen-devel@lists.xenproject.org --===============2425221411938935409== Content-Type: multipart/alternative; boundary="------------000401020009000100020305" --------------000401020009000100020305 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit On 02/12/15 18:38, Tamas K Lengyel wrote: > > > On Wed, Dec 2, 2015 at 1:34 PM, Andrew Cooper > > wrote: > > On 02/12/15 18:21, Tamas K Lengyel wrote: >> >> >> On Tue, Dec 1, 2015 at 5:40 AM, Andrew Cooper >> > wrote: >> >> On 01/12/15 01:21, Tamas K Lengyel wrote: >>> >>> >>> On Mon, Nov 30, 2015 at 7:01 PM, Razvan Cojocaru >>> >> > wrote: >>> >>> On 12/01/2015 01:32 AM, Tamas K Lengyel wrote: >>> > Hi all, >>> > I'm trying to extend the current vm_event system to be >>> able to emulate >>> > over an in-guest breakpoint using the >>> VM_EVENT_FLAG_SET_EMUL_READ_DATA >>> > feature. The idea is to have the vm_event listener >>> send back the >>> > contents of the memory that was overwritten by the >>> breakpoint >>> > instruction, have Xen emulate one instruction, and >>> resume execution >>> > normally afterwards. This would eliminate the need of >>> removing the >>> > breakpoint, singlestepping, and placing the breakpoint >>> back again. >>> > >>> > Unfortunately I encounter this crash when I call >>> > hvm_mem_access_emulate_one in the event response handler: >>> > >>> > (XEN) vm_event.c:72:d0v0 Checking flags on int3 >>> response 37 >>> > (XEN) Xen BUG at >>> /share/src/xen/xen/include/asm/hvm/vmx/vmx.h:372 >>> >> >> This BUG() is the cause of the crash. >> >> It is a bad parameter to VMREAD, by the looks of it. >> >> ~Andrew >> >> >> So this is quite confusing. The error seems to stem from >> (XEN) [] hvm_emulate_prepare+0x23/0x6c >> >> in the line >> hvmemul_ctxt->intr_shadow = >> hvm_funcs.get_interrupt_shadow(current); >> >> which is effectively a simple >> __vmread(GUEST_INTERRUPTIBILITY_INFO, &intr_shadow); >> >> My printk after this doesn't show up in the console so this must >> be where the bug triggers. >> >> (XEN) emulate.c:1828:d1v0 get interrupt shadow >> (XEN) emulate.c:1830:d1v0 get interrupt shadow done >> (XEN) emulate.c:1836:d1v0 get seg cs >> (XEN) emulate.c:1838:d1v0 get seg ss >> (XEN) vm_event.c:72:d0v0 Checking flags on int3 response 37 >> (XEN) emulate.c:1828:d0v0 get interrupt shadow >> (XEN) Xen BUG at /share/src/xen/xen/include/asm/hvm/vmx/vmx.h:372 >> (XEN) ----[ Xen-4.7-unstable x86_64 debug=y Tainted: C ]---- >> (XEN) CPU: 0 >> (XEN) RIP: e008:[] vmx_vmenter_helper+0x3d/0x1cd >> (XEN) RFLAGS: 0000000000010203 CONTEXT: hypervisor (d0v0) >> (XEN) rax: 0000000000004824 rbx: ffff8300ae30fb68 rcx: >> 0000000000000000 >> (XEN) rdx: ffff8300ae308000 rsi: 000000000000000a rdi: >> ffff8300ae550000 >> (XEN) rbp: ffff8300ae30fb28 rsp: ffff8300ae30fb28 r8: >> ffff830430de0000 >> (XEN) r9: 0000000000000004 r10: 0000000000000004 r11: >> 0000000000000001 >> (XEN) r12: ffff8300ae308000 r13: ffff8300ae30ff18 r14: >> ffff8300ae0f8000 >> (XEN) r15: ffff82d08028a480 cr0: 0000000080050033 cr4: >> 00000000000426e0 >> (XEN) cr3: 000000043df79000 cr2: 00007f761d656000 >> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 >> (XEN) Xen stack trace from rsp=ffff8300ae30fb28: >> (XEN) ffff8300ae30fb58 ffff82d0801d55ab 00007cff51cf0497 >> 0000000000000006 >> (XEN) 00000000ffffffff 0000000000000002 ffff8300ae30fc98 >> ffff82d0801d5776 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000048 >> ffff8300ae30fcd0 >> (XEN) ffff8300ae30fcd0 ffff830412da0c50 ffff8300ae30fcb8 >> ffff82d0801c02c1 >> (XEN) ffff8300ae30fcd0 ffff83040fbaf000 ffff8300ae30fe38 >> ffff82d08013a483 >> (XEN) 00007cff51cf0307 0000002500000001 0000000000000006 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) c214c48300000008 0000000064900010 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 >> 0000000000000000 >> (XEN) Xen call trace: >> (XEN) [] vmx_vmenter_helper+0x3d/0x1cd >> (XEN) [] hvm_emulate_prepare+0x50/0x10e >> (XEN) [] hvm_mem_access_emulate_one+0x49/0xd3 >> (XEN) [] >> vm_event_interrupt_emulate_check+0x5c/0x63 >> (XEN) [] vm_event_resume+0xa1/0x131 >> (XEN) [] >> vm_event.c#monitor_notification+0x25/0x28 >> (XEN) [] evtchn_send+0x126/0x17e >> (XEN) [] do_event_channel_op+0xe66/0x14be >> (XEN) [] lstar_enter+0xe2/0x13c >> (XEN) >> (XEN) >> (XEN) **************************************** >> (XEN) Panic on CPU 0: >> (XEN) Xen BUG at /share/src/xen/xen/include/asm/hvm/vmx/vmx.h:372 >> >> Why would that vmread fail there and why would the call trace >> tell me it's in vmx_vmenter_helper? > > The symbol information is incorrect because of the bugframe being > inside an UNLIKELY() block. > > I have a patch to fix it, > http://www.gossamer-threads.com/lists/xen/devel/404482?do=post_view_threaded > but this has been rejected. > > I don't see a way to make a global symbol for the current > translation units' unlikely section. I also don't believe it is a > sensible approach to take even if is a technical way to make it > happen, as it still leaves the stack trace not correctly > identifying the source of the issue, so the fix is in limbo. > > ~Andrew > > > I see, thanks for the explanation. > > In the meanwhile I noticed that the vmread fails as it happens in the > context of d0v0 instead of d1v1. Unfortunately the entire emulation > code is pretty much hard-coded to use "current" everywhere so I'm not > sure how I could make use of it here. You will need to vmx_vmcs_{enter,exit}() to get the correct vmcs in context. ~Andrew --------------000401020009000100020305 Content-Type: text/html; charset="utf-8" Content-Length: 18473 Content-Transfer-Encoding: quoted-printable
On 02/12/15 18:38, Tamas K Lengyel wrote:


On Wed, Dec 2, 2015 at 1:34 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
On 02/12/15 18:21, Tamas K Lengyel wrote:


On Tue, Dec 1, 2015 at 5:40 AM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
On 01/12/15 01:21, Tamas K Lengyel wrote:


On Mon, Nov 30, 2015 at 7:01 PM, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote:
On 12/01/2015 01:32 AM, Tamas K Lengyel wrote:
> Hi all,
> I'm trying to extend the current vm_event system to be able to emulate
> over an in-guest breakpoint using the VM_EVENT_FLAG_SET_EMUL_READ_DATA
> feature. The idea is to have the vm_event listener send back the
> contents of the memory that was overwritten by the breakpoint
> instruction, have Xen emulate one instruction, and resume execution
> normally afterwards. This would eliminate the need of removing the
> breakpoint, singlestepping, and placing the breakpoint back again.
>
> Unfortunately I encounter this crash when I call
> hvm_mem_access_emulate_one in the event response handler:
>
> (XEN) vm_event.c:72:d0v0 Checking flags on int3 response 37
> (XEN) Xen BUG at /share/src/xen/xen/include/asm/hvm/vmx/vmx.h:372

 This BUG() is the cause of the crash.

It is a bad parameter to VMREAD, by the looks of it.

~Andrew

So this is quite confusing. The error seems to stem from
(XEN)=C2=A0=C2=A0=C2=A0 [<ffff82d0801d557e>] hvm_emulate_prepare+0x23/0x6c

in the line
=C2=A0=C2=A0=C2=A0 hvmemul_ctxt->intr_shadow =3D hvm_funcs.get_interrupt_shadow(current);

which is effectively a simple
=C2=A0=C2=A0=C2=A0 __vmread(GUEST_INTERRUPTIBILITY_INFO, &intr_shadow);

My printk after this doesn't show up in the console so this must be where the bug triggers.

(XEN) emulate.c:1828:d1v0 get interrupt shadow
(XEN) emulate.c:1830:d1v0 get interrupt shadow done
(XEN) emulate.c:1836:d1v0 get seg cs
(XEN) emulate.c:1838:d1v0 get seg ss
(XEN) vm_event.c:72:d0v0 Checking flags on int3 response 37
(XEN) emulate.c:1828:d0v0 get interrupt shadow
(XEN) Xen BUG at /share/src/xen/xen/include/asm/hvm/vmx/vmx.h:372
(XEN) ----[ Xen-4.7-unstable=C2=A0 x86_64=C2=A0 debug=3Dy=C2=A0 Tainted:=C2=A0=C2=A0=C2=A0 C ]----
(XEN) CPU:=C2=A0=C2=A0=C2=A0 0
(XEN) RIP:=C2=A0=C2=A0=C2=A0 e008:[<ffff82d080202e00>] vmx_vmenter_helper+0x3d/0x1cd
(XEN) RFLAGS: 0000000000010203=C2=A0=C2=A0 CONTEXT: hypervisor (d0v0)
(XEN) rax: 0000000000004824=C2=A0=C2=A0 rbx: ffff8300ae30fb68=C2=A0=C2=A0 rcx: 0000000000000000
(XEN) rdx: ffff8300ae308000=C2=A0=C2=A0 rsi: 000000000000000a=C2=A0=C2=A0 rdi: ffff8300ae550000
(XEN) rbp: ffff8300ae30fb28=C2=A0=C2=A0 rsp: ffff8300ae30fb28=C2=A0=C2=A0 r8:=C2=A0 ffff830430de0000
(XEN) r9:=C2=A0 0000000000000004=C2=A0=C2=A0 r10: 0000000000000004=C2=A0=C2=A0 r11: 0000000000000001
(XEN) r12: ffff8300ae308000=C2=A0=C2=A0 r13: ffff8300ae30ff18=C2=A0=C2=A0 r14: ffff8300ae0f8000
(XEN) r15: ffff82d08028a480=C2=A0=C2=A0 cr0: 0000000080050033=C2=A0=C2=A0 cr4: 00000000000426e0
(XEN) cr3: 000000043df79000=C2=A0=C2=A0 cr2: 00007f761d656000
(XEN) ds: 0000=C2=A0=C2=A0 es: 0000=C2=A0=C2=A0 fs: 0000=C2=A0=C2=A0 gs: 0000=C2=A0=C2=A0 ss: e010=C2=A0=C2=A0 cs: e008
(XEN) Xen stack trace from rsp=3Dffff8300ae30fb28:
(XEN)=C2=A0=C2=A0=C2=A0 ffff8300ae30fb58 ffff82d0801d55ab 00007cff51cf0497 0000000000000006
(XEN)=C2=A0=C2=A0=C2=A0 00000000ffffffff 0000000000000002 ffff8300ae30fc98 ffff82d0801d5776
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000048 ffff8300ae30fcd0
(XEN)=C2=A0=C2=A0=C2=A0 ffff8300ae30fcd0 ffff830412da0c50 ffff8300ae30fcb8 ffff82d0801c02c1
(XEN)=C2=A0=C2=A0=C2=A0 ffff8300ae30fcd0 ffff83040fbaf000 ffff8300ae30fe38 ffff82d08013a483
(XEN)=C2=A0=C2=A0=C2=A0 00007cff51cf0307 0000002500000001 0000000000000006 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 c214c48300000008 0000000064900010 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)=C2=A0=C2=A0=C2=A0 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)=C2=A0=C2=A0=C2=A0 [<ffff82d080202e00>] vmx_vmenter_helper+0x3d/0x1cd
(XEN)=C2=A0=C2=A0=C2=A0 [<ffff82d0801d55ab>] hvm_emulate_prepare+0x50/0x10e
(XEN)=C2=A0=C2=A0=C2=A0 [<ffff82d0801d5776>] hvm_mem_access_emulate_one+0x49/0xd3
(XEN)=C2=A0=C2=A0=C2=A0 [<ffff82d0801c02c1>] vm_event_interrupt_emulate_check+0x5c/0x63
(XEN)=C2=A0=C2=A0=C2=A0 [<ffff82d08013a483>] vm_event_resume+0xa1/0x131
(XEN)=C2=A0=C2=A0=C2=A0 [<ffff82d08013a914>] vm_event.c#monitor_notification+0x25/0x28
(XEN)=C2=A0=C2=A0=C2=A0 [<ffff82d080108554>] evtchn_send+0x126/0x17e
(XEN)=C2=A0=C2=A0=C2=A0 [<ffff82d080109a74>] do_event_channel_op+0xe66/0x14be
(XEN)=C2=A0=C2=A0=C2=A0 [<ffff82d08024d992>] lstar_enter+0xe2/0x13c
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at /share/src/xen/xen/include/asm/hvm/vmx/vmx.h:372

Why would that vmread fail there and why would the call trace tell me it's in vmx_vmenter_helper=3F

The symbol information is incorrect because of the bugframe being inside an UNLIKELY() block.

I have a patch to fix it, http://www.gossamer-threads.com/lists/xen/devel/404482=3Fdo=3Dpost_view_threaded but this has been rejected.

I don't see a way to make a global symbol for the current translation units' unlikely section.=C2=A0 I also don't believe it is a sensible approach to take even if is a technical way to make it happen, as it still leaves the stack trace not correctly identifying the source of the issue, so the fix is in limbo.

~Andrew

I see, thanks for the explanation.

In the meanwhile I noticed that the vmread fails as it happens in the context of d0v0 instead of d1v1. Unfortunately the entire emulation code is pretty much hard-coded to use "current" everywhere so I'm not sure how I could make use of it here.

You will need to vmx_vmcs_{enter,exit}() to get the correct vmcs in context.

~Andrew
--------------000401020009000100020305-- --===============2425221411938935409== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2425221411938935409==--