Re: [RFC PATCH v4 5/5] x86/vdso: Add __vdso_sgx_enter_enclave() to wrap SGX enclave transitions

[Sean Christopherson <sean.j.christopherson@intel.com>]

On Fri, Dec 14, 2018 at 10:20:39AM -0800, Josh Triplett wrote:
> On Fri, Dec 14, 2018 at 09:03:11AM -0800, Sean Christopherson wrote:
> > On Fri, Dec 14, 2018 at 07:38:30AM -0800, Sean Christopherson wrote:
> > > On Fri, Dec 14, 2018 at 07:12:04AM -0800, Sean Christopherson wrote:
> > > > On Fri, Dec 14, 2018 at 09:55:49AM +0000, Jethro Beekman wrote:
> > > > > On 2018-12-14 03:01, Sean Christopherson wrote:
> > > > > >+2:	pop	%rbx
> > > > > >+	pop	%r12
> > > > > >+	pop	%r13
> > > > > >+	pop	%r14
> > > > > >+	pop	%r15
> > > > > >+	pop	%rbp
> > > > > >+	ret
> > > > > 
> > > > > x86-64 ABI requires that you call CLD here (enclave may set it).
> > > > 
> > > > Ugh.  Technically MXCSR and the x87 CW also need to be preserved.
> > > > 
> > > > What if rather than treating the enclave as hostile we require it to be
> > > > compliant with the x86-64 ABI like any other function?  That would solve
> > > > the EFLAGS.DF, MXCSR and x87 issues without adding unnecessary overhead.
> > > > And we wouldn't have to save/restore R12-R15.  It'd mean we couldn't use
> > > > the stack's red zone to hold @regs and @e, but that's poor form anyways.
> > > 
> > > Grr, except the processor crushes R12-R15, FCW and MXCSR on asynchronous
> > > exits.  But not EFLAGS.DF, that's real helpful.
> > 
> > I can think of three options that are at least somewhat reasonable:
> > 
> >   1) Save/restore MXCSR and FCW
> > 
> >      + 100% compliant with the x86-64 ABI
> >      + Callable from any code
> >      + Minimal documentation required
> >      - Restoring MXCSR/FCW is likely unnecessary 99% of the time
> >      - Slow
> > 
> >   2) Clear EFLAGS.DF but not save/restore MXCSR and FCW
> > 
> >      + Mostly compliant with the x86-64 ABI
> >      + Callable from any code that doesn't use SIMD registers
> >      - Need to document deviations from x86-64 ABI
> > 
> >   3) Require the caller to save/restore everything.
> > 
> >      + Fast
> >      + Userspace can pass all GPRs to the enclave (minus EAX, RBX and RCX)
> >      - Completely custom ABI
> >      - For all intents and purposes must be called from an assembly wrapper
> > 
> > 
> > Option (3) actually isn't all that awful.  RCX can be used to pass an
> > optional pointer to a 'struct sgx_enclave_exception' and we can still
> > return standard error codes, e.g. -EFAULT.
> 
> Entering and exiting a syscall requires an assembly wrapper, and that
> doesn't seem completely unreasonable. It's an easy bit of inline
> assembly.

The code I posted had a few typos (stupid AT&T syntax), but with those
fixed the idea checks out.

My initial reaction to a barebones ABI was that it would be a
"documentation nightmare", but it's not too bad if it returns actual
error codes and fills in a struct on exceptions instead of stuffing
registers.  And with the MXCSR/FCW issues it might actually be less
documentation in the long run since we can simply say that all state
is the caller's responsibility.

I *really* like that we basically eliminate bikeshedding on which GPRs
to pass to/from the enclave.