Re: Switching to user mode from domU kernel

[Trammell Hudson <hudson@osresearch.net>]

On Mon, Oct 01, 2007 at 06:24:07AM +0100, Keir Fraser wrote:
> On 30/9/07 21:11, "Trammell Hudson" <hudson@osresearch.net> wrote:
> > Am I not jumping into user space correctly?  Is there something
> > else that my code should do to make the transition?
> 
> It sounds like event delivery is masked before the iret, and then you have
> IF set in the RFLAGS value in the iret frame, which causes event delivery to
> be unmasked during iret.

That makes sense, although the event channel pending array is
all zero and the shared_info->vcpu_info[0].evtchn_upcall_pending
value is also zero.

If I do not set IF in RFLAGS, Xen crashes when it tries to handle
a GPF:

(XEN) traps.c:1587: GPF (001c): ffff830000165555 -> ffff83000016556b
(XEN) ----[ Xen-3.0.4-1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e010:[<ffff83000016364b>] restore_all_xen+0x1b/0x20
(XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
(XEN) rax: 000000400001c480   rbx: 000000600053dde8   rcx: 0000000000000017
(XEN) rdx: 00000000decafbad   rsi: 0000000000012345   rdi: 000000000badbabe
(XEN) rbp: 00000040005fffc0   rsp: ffff83000074ffa8   r8:  000000000000001f
(XEN) r9:  0000000000000017   r10: 00000040005fffc0   r11: 0000000000000000
(XEN) r12: 0000006000017000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: ffff830000163726   cr0: 000000008005003b   cr4: 00000000000006f0
(XEN) cr3: 000000002734d000   cr2: ffff820000001000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN) Xen stack trace from rsp=ffff83000074ffa8:
(XEN)    000001000001c480 0000006000101000 000000000000e033 000000400058bec0
(XEN)    0000006000102fa8 000000000000e02b 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 ffff830000744080
(XEN) Xen call trace:
(XEN)    [<ffff83000016364b>] restore_all_xen+0x1b/0x20
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) GENERAL PROTECTION FAULT
(XEN) [error_code=1000]
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...

> > I am passing flags==0 and only push the values for flags, CS:RIP,
> > RFLAGS, and SS:RSP.  If I push values for RAX, R11, and RCX on the
> > stack the kernel ends up in all sorts of weird places rather than
> > my intended RIP and RSP.
> 
> That doesn't make sense. The iret implementation (for an x86/64 guest)
> always expects RAX/R11/RCX on the stack.

That was my expectation as well, but if I push the three extra values
onto the stack then it jumps to the wrong place:

        context.rax     = 0x12345;
        context.r11     = 0xdecafbad;
        context.rcx     = 0xbadbabe;
        context.rflags  = 0x200;
        context.rip     = start_addr; // 01f:000000400001c480
        context.cs      = USER_CS;
        context.flags   = 0;
        context.rsp     = (uintptr_t) stack; // 017:00000040005fffc0
        context.ss      = USER_DS;

        __asm__ __volatile__ (
                "push   %0\n"
                "push   %1\n"
                "push   %2\n"
                "push   %3\n"
                "push   %4\n"
                "push   %5\n"
                "push   %6\n"
                "push   %7\n"
                "push   %8\n"
                "jmp hypercall_page + __HYPERVISOR_iret*32\n"
                :
                : "r" (context.ss),
                "r" (context.rsp),
                "r" (context.rflags),
                "r" (context.cs),
                "r" (context.rip),
                "r" (context.flags),
                "r" (context.rcx),
                "r" (context.r11),
                "r" (context.rax),
                "r" (USER_DS)
        );

Ends up with RIP e033:00000000decafbad and RSP e02b:000000400001c480:

(XEN) domain_crash_sync called from entry.S
(XEN) Domain 147 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.0.4-1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<00000000decafbad>]
(XEN) RFLAGS: 0000000000010202   CONTEXT: guest
(XEN) rax: 0000000000000017   rbx: 0000000000000200   rcx: 00000000decafbad
(XEN) rdx: 0000000000012345   rsi: 000000000badbabe   rdi: 0000000000000000
(XEN) rbp: 000000600053dde8   rsp: 000000400001c480   r8:  000000400001c480
(XEN) r9:  000000000000001f   r10: 0000000000000017   r11: 0000000000000200
(XEN) r12: 00000040005fffc0   r13: 0000006000017000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000006f0
(XEN) cr3: 0000000026ae5000   cr2: 00000000decafbad
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033

Looking in hypercall_page_initialise_ring3_kernel() it appears that
the hypervisor call does the rcx, r11 and rax pushing, not the
domU kernel calling the hypercall:

    /*
     * HYPERVISOR_iret is special because it doesn't return and expects a
     * special stack frame. Guests jump at this transfer point instead of
     * calling it.
     */
    p = (char *)(hypercall_page + (__HYPERVISOR_iret * 32));
    *(u8  *)(p+ 0) = 0x51;    /* push %rcx */
    *(u16 *)(p+ 1) = 0x5341;  /* push %r11 */
    *(u8  *)(p+ 3) = 0x50;    /* push %rax */
    *(u8  *)(p+ 4) = 0xb8;    /* mov  $__HYPERVISOR_iret,%eax */
    *(u32 *)(p+ 5) = __HYPERVISOR_iret;
    *(u16 *)(p+ 9) = 0x050f;  /* syscall */

-- Trammell