Switching to user mode from domU kernel

[Trammell Hudson <hudson@osresearch.net>]

I'm having trouble switching into user mode with my domU kernel --
everytime I call __HYPERVISOR_iret, the hypervisor calls into my
kernel via the registered callback handler.  None of the bits in
the event channel pending arrays are set.  If I return via the
iret call, the hypervisor immediately calls back into the handler.
The user RIP never changes, so it is clearly not making forward
progress.

The stackframe should be correctly setup with the flags, rip, cs,
rflags, rsp, and ss values and the currently installed LDT has
slots for the code and data segments that the user will use.

Am I not jumping into user space correctly?  Is there something
else that my code should do to make the transition?

On a documentation question, is __HYPERVISOR_iret documented
anywhere?  The Xen API* doesn't seem to include any details on it.
I deduced the stack arguments from looking at the code in the Linux
xen architecture and the xen-x86_64.h header.

This comment in the header, however, doesn't seem to match the
behaviour that my code seems to experience:

 * If flags contains VGCF_in_syscall:
 *   Restore RAX, RIP, RFLAGS, RSP.
 *   Discard R11, RCX, CS, SS.
 * Otherwise:
 *   Restore RAX, R11, RCX, CS:RIP, RFLAGS, SS:RSP.

I am passing flags==0 and only push the values for flags, CS:RIP,
RFLAGS, and SS:RSP.  If I push values for RAX, R11, and RCX on the
stack the kernel ends up in all sorts of weird places rather than
my intended RIP and RSP.


Thanks!
-- Trammell

*: http://www.cl.cam.ac.uk/research/srg/netos/xen/readmes/interface/interface.html