Re: Getting rid of invalid SYSCALL RSP under Xen?

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 23/07/2015 17:49, Andy Lutomirski wrote:
> Hi-

Hi.  Apologies for the delay.  I have been out of the office for a few days.

>
> In entry_64.S, we have:
>
> ENTRY(entry_SYSCALL_64)
>     /*
>      * Interrupts are off on entry.
>      * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON,
>      * it is too small to ever cause noticeable irq latency.
>      */
>     SWAPGS_UNSAFE_STACK
>     /*
>      * A hypervisor implementation might want to use a label
>      * after the swapgs, so that it can do the swapgs
>      * for the guest and jump here on syscall.
>      */
> GLOBAL(entry_SYSCALL_64_after_swapgs)
>
>     movq    %rsp, PER_CPU_VAR(rsp_scratch)
>     movq    PER_CPU_VAR(cpu_current_top_of_stack), %rsp
>
> It would be really, really nice if Xen entered the SYSCALL path
> *after* the mov to %rsp.
>
> Similarly, we have:
>
>     movq    RSP(%rsp), %rsp
>     /* big comment */
>     USERGS_SYSRET64
>
> It would be really nice if we could just mov to %rsp, swapgs, and
> sysret, without worrying that the sysret is actually a jump on Xen.
>
> I suspect that making Xen stop using these code paths would actually
> be a simplification.  On SYSCALL entry, Xen lands in
> xen_syscall_target (AFAICT) and clearly has rsp pointing somewhere
> valid.  Xen obligingly shoves the user RSP into the hardware RSP
> register and jumps into the entry code.
>
> Is that stuff running on the normal kernel stack?

Yes. The Xen ABI takes what is essentially tss->esp0 and uses that stack
for all "switch to kernel" actions, including syscall and sysenter.

>   If so, can we just
> enter later on:
>
>     pushq    %r11                /* pt_regs->flags */
>     pushq    $__USER_CS            /* pt_regs->cs */
>     pushq    %rcx                /* pt_regs->ip */
>
> <-- Xen enters here
>
>     pushq    %rax                /* pt_regs->orig_ax */
>     pushq    %rdi                /* pt_regs->di */
>     pushq    %rsi                /* pt_regs->si */
>     pushq    %rdx                /* pt_regs->dx */

This looks plausible, and indeed preferable to the current doublestep
with undo_xen_syscall.

One slight complication is that xen_enable_syscall() will want to
special case register_callback() to not set CALLBACKF_mask_events, as
the entry point is now after re-enabling interrupts.

>
> For SYSRET, I think the way to go is to force Xen to always use the
> syscall slow path.  Instead, Xen could hook into
> syscall_return_via_sysret or even right before the opportunistic
> sysret stuff.  Then we could remove the USERGS_SYSRET hooks entirely.
>
> Would this work?

None of the opportunistic sysret stuff makes sense under Xen.  The path
will inevitably end up in xen_iret making a hypercall.  Short circuiting
all of this seems like a good idea, especially if it allows for the
removal of the UERGS_SYSRET.

~Andrew