RIP register value in p2m_mem_access_check()

RIP register value in p2m_mem_access_check()

[Razvan Cojocaru]

Hello again,

simple test case: added the following statement in 
xen/arch/x86/mm/p2m.c, in function p2m_mem_access_check():

printk("v->arch.user_regs.eip: 0x%016lx, __vmread(GUEST_RIP): 
0x%016lx\n", v->arch.user_regs.eip, __vmread(GUEST_RIP));

(where v is current) which dutifully prints out (at the first page fault):

(XEN) v->arch.user_regs.eip: 0xfffff80002c13a7b, __vmread(GUEST_RIP): 
0xfffff8000269ec0a

They're not equal, and furthermore __vmread(GUEST_RIP) is the correct 
one (the same value returned by xc_domain_hvm_getcontext_partial()).

The questions:

1. What's the Xen-friendliest way to retrieve the _correct_ values for 
RIP and all the other registers usually put in a struct hvm_hw_cpu 
instance, in p2m_mem_access_check()?

2. What's the explanation for why things run as they do now?


Thanks,
Razvan Cojocaru

Re: RIP register value in p2m_mem_access_check()

[Jan Beulich]

>>> On 06.03.13 at 11:55, Razvan Cojocaru <rzvncj@gmail.com> wrote:
> Hello again,
> 
> simple test case: added the following statement in 
> xen/arch/x86/mm/p2m.c, in function p2m_mem_access_check():
> 
> printk("v->arch.user_regs.eip: 0x%016lx, __vmread(GUEST_RIP): 
> 0x%016lx\n", v->arch.user_regs.eip, __vmread(GUEST_RIP));
> 
> (where v is current) which dutifully prints out (at the first page fault):
> 
> (XEN) v->arch.user_regs.eip: 0xfffff80002c13a7b, __vmread(GUEST_RIP): 
> 0xfffff8000269ec0a
> 
> They're not equal, and furthermore __vmread(GUEST_RIP) is the correct 
> one (the same value returned by xc_domain_hvm_getcontext_partial()).
> 
> The questions:
> 
> 1. What's the Xen-friendliest way to retrieve the _correct_ values for 
> RIP and all the other registers usually put in a struct hvm_hw_cpu 
> instance, in p2m_mem_access_check()?

v->arch.user_regs.eip is what you should look at. It's
expected to be in sync with the VMCS value as long as
nothing (e.g. emulation) modified the value. There's code in
vmx_asm_vmexit_handler to sync the fields, and in
vmx_asm_do_vmentry to sync them back in the opposite
direction.

> 2. What's the explanation for why things run as they do now?

If the above doesn't work, you're likely in bigger trouble. So
you will want to hunt down why the user_regs field isn't what
you expect it to be.

Jan

Re: RIP register value in p2m_mem_access_check()

[Razvan Cojocaru]

> v->arch.user_regs.eip is what you should look at. It's
> expected to be in sync with the VMCS value as long as
> nothing (e.g. emulation) modified the value. There's code in
> vmx_asm_vmexit_handler to sync the fields, and in
> vmx_asm_do_vmentry to sync them back in the opposite
> direction.

Thank you for your answer. Isn't it possible that emulation is indeed 
involved, since in p2m_mem_access_check() a page fault has occured, and 
RIP might point to _after_ the offending instruction?


Thanks,
Razvan Cojocaru

Re: RIP register value in p2m_mem_access_check()

[Jan Beulich]

>>> On 06.03.13 at 12:35, Razvan Cojocaru <rzvncj@gmail.com> wrote:
>>  v->arch.user_regs.eip is what you should look at. It's
>> expected to be in sync with the VMCS value as long as
>> nothing (e.g. emulation) modified the value. There's code in
>> vmx_asm_vmexit_handler to sync the fields, and in
>> vmx_asm_do_vmentry to sync them back in the opposite
>> direction.
> 
> Thank you for your answer. Isn't it possible that emulation is indeed 
> involved, since in p2m_mem_access_check() a page fault has occured, and 
> RIP might point to _after_ the offending instruction?

That wouldn't match your observation (the two values were
much farther apart), unless a call or jump got emulated.

Further, after emulation, the user_regs.eip value should be the
canonical one.

And finally, you said you took this right after a page fault
occurred, and - just like for any fault - the RIP the fault refers
to is the faulting instruction, not the one following it.

Jan

Re: RIP register value in p2m_mem_access_check()

[Razvan Cojocaru]

>> Thank you for your answer. Isn't it possible that emulation is indeed
>> involved, since in p2m_mem_access_check() a page fault has occured, and
>> RIP might point to _after_ the offending instruction?
>
> That wouldn't match your observation (the two values were
> much farther apart), unless a call or jump got emulated.

I've noticed that, and have indeed assumed that it was something like a 
call.

> Further, after emulation, the user_regs.eip value should be the
> canonical one.
>
> And finally, you said you took this right after a page fault
> occurred, and - just like for any fault - the RIP the fault refers
> to is the faulting instruction, not the one following it.

Fair enough. Will debug the issue.


Thanks,
Razvan Cojocaru

Re: RIP register value in p2m_mem_access_check()

[Razvan Cojocaru]

> That wouldn't match your observation (the two values were
> much farther apart), unless a call or jump got emulated.
>
> Further, after emulation, the user_regs.eip value should be the
> canonical one.

Well, I've downloaded and installed a fresh copy of Xen 4.2.0 from the 
official website, and added the printk() to p2m.c (the only modification 
I've done):

1311         req->vcpu_id = v->vcpu_id;
1312
1313         printk("v->arch.user_regs.eip: 0x%016lx, 
__vmread(GUEST_RIP):0x%016lx\n",
1314                v->arch.user_regs.eip, __vmread(GUEST_RIP));
1315     }

I then launched my guest VM, and tested it with the xen-access tool from 
tools/tests/xen-access, which I've run like this (1 is the machine ID, 
as printed out by 'sudo xm list'):

sudo ./xen-access 1 write

Here's what the hypervisor wrote (values that are different appear once 
per each VCPU):

(XEN) HVM1: Booting from 0000:7c00
(XEN) v->arch.user_regs.eip: 0x00000000fc002de9, 
__vmread(GUEST_RIP):0x0000000000007c17
(XEN) v->arch.user_regs.eip: 0x0000000000007c17, 
__vmread(GUEST_RIP):0x0000000000000667
(XEN) v->arch.user_regs.eip: 0x0000000000000667, 
__vmread(GUEST_RIP):0x0000000000000116
(XEN) v->arch.user_regs.eip: 0x0000000000000116, 
__vmread(GUEST_RIP):0x0000000000000116
(XEN) v->arch.user_regs.eip: 0x0000000000000116, 
__vmread(GUEST_RIP):0x00000000000005ab
(XEN) v->arch.user_regs.eip: 0x00000000000005ab, 
__vmread(GUEST_RIP):0x00000000000001e7
(XEN) v->arch.user_regs.eip: 0x00000000000001e7, 
__vmread(GUEST_RIP):0x00000000000001f3
(XEN) v->arch.user_regs.eip: 0x00000000000001f3, 
__vmread(GUEST_RIP):0x0000000000000b71
(XEN) v->arch.user_regs.eip: 0x0000000000000b71, 
__vmread(GUEST_RIP):0x0000000000000782
(XEN) v->arch.user_regs.eip: 0x0000000000000782, 
__vmread(GUEST_RIP):0x0000000000000782
(XEN) v->arch.user_regs.eip: 0x0000000000000782, 
__vmread(GUEST_RIP):0x0000000000000782

Not sure what's going on, but this time the only code I've added is the 
printk(). Any thoughts?


Thanks,
Razvan Cojocaru

Re: RIP register value in p2m_mem_access_check()

[Razvan Cojocaru]

Furthermore (with the fresh install of Xen 4.2.0), when modifying 
xen-access.c to fetch the registers and print RIP out:

624             case MEM_EVENT_REASON_VIOLATION:
625             {
626                 struct hvm_hw_cpu hwcpu;
627
628                 if (xc_domain_hvm_getcontext_partial(xch, domain_id, 
HVM_SAVE_CODE(CPU),
629                                              req.vcpu_id, &hwcpu, 
sizeof(hwcpu)) != 0)
630                 {
631                     ERROR("xc_domain_hvm_getcontext_partial() 
failed\n");
632                     interrupted = -1;
633                     continue;
634                 }
635
636                 printf("RIP: 0x%016lx\n", hwCpu.rip);
637                 fflush(stdout);

the RIP it prints out is always the __vmread(GUEST_RIP), not 
v->arch.user_regs.eip:

xen-access output:

Got event from Xen
RIP: 0x0000000000003272
Got event from Xen
PAGE ACCESS: -w- for GFN 25 (offset 0001ba) gla 00000000000251ba (vcpu 0)
RIP: 0x000000000000fed9
Got event from Xen
PAGE ACCESS: -w- for GFN 0 (offset 00046c) gla 000000000000046c (vcpu 0)
RIP: 0x0000000000000643
Got event from Xen
PAGE ACCESS: -w- for GFN 9f (offset 000e54) gla 000000000009fe54 (vcpu 0)
RIP: 0x0000000000000643
Got event from Xen
PAGE ACCESS: -w- for GFN 32 (offset 000042) gla 0000000000032042 (vcpu 0)
RIP: 0x000000000000009b
Got event from Xen
PAGE ACCESS: -w- for GFN 9a (offset 0003d9) gla 000000000009a3d9 (vcpu 0)
RIP: 0x0000000000000269

corresponding hypervisor log:

(XEN) v->arch.user_regs.eip: 0x0000000000003272, 
__vmread(GUEST_RIP):0x0000000000003272
(XEN) v->arch.user_regs.eip: 0x0000000000003272, 
__vmread(GUEST_RIP):0x000000000000fed9
(XEN) v->arch.user_regs.eip: 0x000000000000fed9, 
__vmread(GUEST_RIP):0x0000000000000643
(XEN) v->arch.user_regs.eip: 0x0000000000000643, 
__vmread(GUEST_RIP):0x0000000000000643
(XEN) v->arch.user_regs.eip: 0x0000000000000643, 
__vmread(GUEST_RIP):0x000000000000009b
(XEN) v->arch.user_regs.eip: 0x000000000000009b, 
__vmread(GUEST_RIP):0x0000000000000269


Hope this helps,
Razvan Cojocaru

Re: RIP register value in p2m_mem_access_check()

[Jan Beulich]

>>> On 06.03.13 at 14:49, Razvan Cojocaru <rzvncj@gmail.com> wrote:
>>  That wouldn't match your observation (the two values were
>> much farther apart), unless a call or jump got emulated.
>>
>> Further, after emulation, the user_regs.eip value should be the
>> canonical one.
> 
> Well, I've downloaded and installed a fresh copy of Xen 4.2.0 from the 
> official website, and added the printk() to p2m.c (the only modification 
> I've done):
> 
> 1311         req->vcpu_id = v->vcpu_id;
> 1312
> 1313         printk("v->arch.user_regs.eip: 0x%016lx, 
> __vmread(GUEST_RIP):0x%016lx\n",
> 1314                v->arch.user_regs.eip, __vmread(GUEST_RIP));
> 1315     }

I know close to nothing about the paths that can lead here, and
hence I can't assess whether expecting consistency at this point
is valid. All I can tell is that right after VM exit (i.e. at the top of
vmx_vmexit_handler()) the two values ought to be consistent.

Jan

Re: RIP register value in p2m_mem_access_check()

[Razvan Cojocaru]

> I know close to nothing about the paths that can lead here, and
> hence I can't assess whether expecting consistency at this point
> is valid. All I can tell is that right after VM exit (i.e. at the top of
> vmx_vmexit_handler()) the two values ought to be consistent.

Yes, that's true, however what I'm after is a way to have the 
information associated with the proper RIP _before_ the VM exit 
(basically all of the information that gets stuffed into the struct 
hvm_hw_cpu instance by xc_domain_hvm_getcontext_partial()). 
Specifically, I want that information inside p2m_mem_access_check().


Thanks for your help,
Razvan Cojocaru

Re: RIP register value in p2m_mem_access_check()

[Tim Deegan]

At 14:33 +0000 on 06 Mar (1362580383), Jan Beulich wrote:
> >>> On 06.03.13 at 14:49, Razvan Cojocaru <rzvncj@gmail.com> wrote:
> >>  That wouldn't match your observation (the two values were
> >> much farther apart), unless a call or jump got emulated.
> >>
> >> Further, after emulation, the user_regs.eip value should be the
> >> canonical one.
> > 
> > Well, I've downloaded and installed a fresh copy of Xen 4.2.0 from the 
> > official website, and added the printk() to p2m.c (the only modification 
> > I've done):
> > 
> > 1311         req->vcpu_id = v->vcpu_id;
> > 1312
> > 1313         printk("v->arch.user_regs.eip: 0x%016lx, 
> > __vmread(GUEST_RIP):0x%016lx\n",
> > 1314                v->arch.user_regs.eip, __vmread(GUEST_RIP));
> > 1315     }
> 
> I know close to nothing about the paths that can lead here, and
> hence I can't assess whether expecting consistency at this point
> is valid. All I can tell is that right after VM exit (i.e. at the top of
> vmx_vmexit_handler()) the two values ought to be consistent.

I don't think so -- v->arch.user_regs is only brought up to date during
context switches.  For the currently running vcpu you should be looking
at guest_cpu_user_regs().

Cheers,

Tim.

Re: RIP register value in p2m_mem_access_check()

[Jan Beulich]

>>> On 07.03.13 at 17:49, Tim Deegan <tim@xen.org> wrote:
> At 14:33 +0000 on 06 Mar (1362580383), Jan Beulich wrote:
>> >>> On 06.03.13 at 14:49, Razvan Cojocaru <rzvncj@gmail.com> wrote:
>> >>  That wouldn't match your observation (the two values were
>> >> much farther apart), unless a call or jump got emulated.
>> >>
>> >> Further, after emulation, the user_regs.eip value should be the
>> >> canonical one.
>> > 
>> > Well, I've downloaded and installed a fresh copy of Xen 4.2.0 from the 
>> > official website, and added the printk() to p2m.c (the only modification 
>> > I've done):
>> > 
>> > 1311         req->vcpu_id = v->vcpu_id;
>> > 1312
>> > 1313         printk("v->arch.user_regs.eip: 0x%016lx, 
>> > __vmread(GUEST_RIP):0x%016lx\n",
>> > 1314                v->arch.user_regs.eip, __vmread(GUEST_RIP));
>> > 1315     }
>> 
>> I know close to nothing about the paths that can lead here, and
>> hence I can't assess whether expecting consistency at this point
>> is valid. All I can tell is that right after VM exit (i.e. at the top of
>> vmx_vmexit_handler()) the two values ought to be consistent.
> 
> I don't think so -- v->arch.user_regs is only brought up to date during
> context switches.  For the currently running vcpu you should be looking
> at guest_cpu_user_regs().

Oh, of course, yes - I was mixing them up quite badly. Sorry
for that.

Jan

Re: RIP register value in p2m_mem_access_check()

[Razvan Cojocaru]

> I don't think so -- v->arch.user_regs is only brought up to date during
> context switches.  For the currently running vcpu you should be looking
> at guest_cpu_user_regs().

Yes, that's what did it - I discovered that a few hours earlier and
wanted to post the solution but you beat me to it :)


Thanks,
Razvan Cojocaru