hypercall execution path?

hypercall execution path?

[Bryan D. Payne]

I'm trying to trace the execution path taken by the system when I make a 
hypercall from a domU kernel.  What I'm seeing in the hypercall entry 
function in Xen is that the domU eip value points to a location in the 
domU hypercall page for the __HYPERCALL_sched_op hypercall, even though 
this is *not* the hypercall that I just called.

Could someone help clarify this for me?  Why would the vcpu for the 
guest domain have this seemingly incorrect eip value?

Thanks,
bryan

-
Bryan D. Payne
Graduate Student, Computer Science
Georgia Tech Information Security Center
http://www.bryanpayne.org

Re: hypercall execution path?

[Daniel Stodden]

[-- Attachment #1.1: Type: text/plain, Size: 1355 bytes --]

On Thu, 2007-02-22 at 19:16 +0100, Daniel Stodden wrote:
> On Thu, 2007-02-22 at 12:17 -0500, Bryan D. Payne wrote:
> > I'm trying to trace the execution path taken by the system when I make a 
> > hypercall from a domU kernel.  What I'm seeing in the hypercall entry 
> > function in Xen is that the domU eip value points to a location in the 
> > domU hypercall page for the __HYPERCALL_sched_op hypercall, even though 
> > this is *not* the hypercall that I just called.
> > 
> > Could someone help clarify this for me?  Why would the vcpu for the 
> > guest domain have this seemingly incorrect eip value?
> 
> hard to imagine :}
> 
> apart from emulating execution, the single place i'm aware of playing
> with eip are continuations.
> 
> so: what does it point at? what's the page address, which offset? 
> 
> after looking into traps.c: take good care with the layout: hypercall
> pages on x86_32 are 32 (!) bytes per entry (not 8 as the loop body might
> suggest). are your calculations correct?

(i keep forgetting cc's)

-- 
Daniel Stodden
LRR     -      Lehrstuhl für Rechnertechnik und Rechnerorganisation
Institut für Informatik der TU München             D-85748 Garching
http://www.lrr.in.tum.de/~stodden         mailto:stodden@cs.tum.edu
PGP Fingerprint: F5A4 1575 4C56 E26A 0B33  3D80 457E 82AE B0D8 735B

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: hypercall execution path?

[Bryan D. Payne]

> so: what does it point at? what's the page address, which offset? 

On my machine (xen 3.0.4_1, linux 2.6.16.33-xen), the EIP register in 
the guest's vcpu holds 0xc01013a7.  For reference, the hypercall page is 
0xc0101000, this is at offset 0x3a7 on that page.  Looking at the 
disassembly from that page at this location, I see the following:

mov 0x1d, %eax
int $0x82
ret

The EIP value is pointing at the 'ret' instruction.  Looking in the 
hypercall table, this (0x1d) is the sched_op hypercall (which is not the 
hypercall that I called).


> after looking into traps.c: take good care with the layout: hypercall
> pages on x86_32 are 32 (!) bytes per entry (not 8 as the loop body might
> suggest). are your calculations correct?

I believe so, but you can verify with the information that I provided 
above.  0x1d * 32 = 0x3a0... this matches the offset I'm seeing.

Hopefully someone can help me make sense of this :-)

Thanks,
bryan


-
Bryan D. Payne
Graduate Student, Computer Science
Georgia Tech Information Security Center
http://www.bryanpayne.org

Re: hypercall execution path?

[Ian Campbell]

On Thu, 2007-02-22 at 13:30 -0500, Bryan D. Payne wrote:
> > so: what does it point at? what's the page address, which offset? 
> 
> On my machine (xen 3.0.4_1, linux 2.6.16.33-xen), the EIP register in 
> the guest's vcpu holds 0xc01013a7.  For reference, the hypercall page is 
> 0xc0101000, this is at offset 0x3a7 on that page.  Looking at the 
> disassembly from that page at this location, I see the following:
> 
> mov 0x1d, %eax
> int $0x82
> ret
> 
> The EIP value is pointing at the 'ret' instruction.  Looking in the 
> hypercall table, this (0x1d) is the sched_op hypercall (which is not the 
> hypercall that I called).
> 
> 
> > after looking into traps.c: take good care with the layout: hypercall
> > pages on x86_32 are 32 (!) bytes per entry (not 8 as the loop body might
> > suggest). are your calculations correct?
> 
> I believe so, but you can verify with the information that I provided 
> above.  0x1d * 32 = 0x3a0... this matches the offset I'm seeing.
> 
> Hopefully someone can help me make sense of this :-)

The most common reason for sched_op being called is that the vcpu is
idle (SCHEDOP_block).

Are you sure you are looking at the correct vcpu?

How are you looking at the EIP register?

Ian.

Re: hypercall execution path?

[Bryan D. Payne]

> Are you sure you are looking at the correct vcpu?

There is only one vcpu assigned to this domain.  Based on this, I'm 
assuming that my code (shown below) is correct.


> How are you looking at the EIP register?

uint32_t eip = domain->vcpu[0]->arch.guest_context.user_regs.eip;


Thanks,
-bryan


-
Bryan D. Payne
Graduate Student, Computer Science
Georgia Tech Information Security Center
http://www.bryanpayne.org

Re: hypercall execution path?

[Ian Campbell]

On Thu, 2007-02-22 at 14:11 -0500, Bryan D. Payne wrote:
> > Are you sure you are looking at the correct vcpu?
> 
> There is only one vcpu assigned to this domain.  Based on this, I'm 
> assuming that my code (shown below) is correct.
> 
> 
> > How are you looking at the EIP register?
> 
> uint32_t eip = domain->vcpu[0]->arch.guest_context.user_regs.eip;

That seems reasonable.

Where have you put this tracing?

How you are arranging for the hypercall you are expecting to be called
and how you are matching that up with where your tracing is placed?

Perhaps you could post your code so we can see what you are actually
doing?

Ian.

Re: hypercall execution path?

[Bryan D. Payne]

> Where have you put this tracing?
> 
> How you are arranging for the hypercall you are expecting to be called
> and how you are matching that up with where your tracing is placed?
> 
> Perhaps you could post your code so we can see what you are actually
> doing?

This all started with me playing with grant tables.  I have a simple 
example setup where I am passing a string from a domU kernel module to a 
dom0 kernel module.  Dom0 allocates the page and sets permissions in the 
grant table using gnttab_grant_foreign_access.

In domU, the kernel module makes a GNTTABOP_copy hypercall to pass the 
string.  The domU code looks like this:

static int send_string (grant_ref_t gref)
{
     struct gnttab_copy op;
     char str[MAX_STR_LENGTH];

     /* hard code string for testing */
     memset(str, 0, MAX_STR_LENGTH);
     memcpy(str, "This is a test\n", 16);

     op.source.domid = DOMID_SELF;
     op.source.offset = (PAGE_SIZE-1) & (uint32_t)str;
     op.source.u.gmfn = virt_to_mfn(str);
     op.dest.domid = 0;
     op.dest.offset = 0;
     op.dest.u.ref = gref;
     op.len = strnlen(str, MAX_LOG_LENGTH);
     op.flags = GNTCOPY_dest_gref;

     HYPERVISOR_grant_table_op(GNTTABOP_priv_write, &op, 1);

     /* make sure that the hypercall succeeded */
     if (op.status){
         printk("Grant table operation failure\n");
         return 1;
     }

     return 0;
}

This code setup seems to work just fine.  I can pass the string to dom0 
without any problems.

Next, I wanted to study the execution path between the two kernel 
modules.  As part of this, I placed code in the hypervisor to print out 
the EIP value from domU while servicing the hypercall (the one shown in 
the code above).  I put this code in the __gnttab_copy function of 
common/grant_table.c, because this is the function that does most of the 
work for this hypercall.  Here I simply print out the EIP value, as 
indicated in my prior email:

uint32_t eip = sd->vcpu[0]->arch.guest_context.user_regs.eip;
gdprintk(XENLOG_WARNING, "eip=0x%x", eip);

Note that I'm using 'sd' for the domain since I'm interested in the 
domain that invoked the hypercall.  This is where I see the eip value 
that I indicated in my previous email.  Also note that I am doing this 
after 'sd' is initialized, of course :-)

My assumptions are as follows:

* When I make a hypercall from domU, the execution switches to the 
hypervisor immediately (as soon as the 'int' instruction is issued).

* The hypervisor services this hypercall and then returns to domU where 
it left off.

However, this is now what I'm seeing... and this is why I'm confused. 
Hopefully this helps explain the problem.  Let me know if you have any 
other questions about my setup.

Thanks,
bryan


-
Bryan D. Payne
Graduate Student, Computer Science
Georgia Tech Information Security Center
http://www.bryanpayne.org

Re: hypercall execution path?

[Ian Campbell]

On Thu, 2007-02-22 at 15:28 -0500, Bryan D. Payne wrote:
> uint32_t eip = sd->vcpu[0]->arch.guest_context.user_regs.eip;
> gdprintk(XENLOG_WARNING, "eip=0x%x", eip); 

sd is current->domain here so the guest context is saved on the stack
rather than in the domain structure. Therefore you need to use
guest_cpu_user_regs()->eip instead. I should have thought of that
before, sorry.

The context in the vcpu's arch.guest_context will be a stale left over
from the last time the domain was rescheduled, which is why it is doing
a sched_op since it was presumably just waking up from idle.

Ian.

Re: hypercall execution path?

[Bryan D. Payne]

> sd is current->domain here so the guest context is saved on the stack
> rather than in the domain structure. Therefore you need to use
> guest_cpu_user_regs()->eip instead. I should have thought of that
> before, sorry.

Thanks!  That gives a much more reasonable value :-)

-bryan


-
Bryan D. Payne
Graduate Student, Computer Science
Georgia Tech Information Security Center
http://www.bryanpayne.org