* Re: RFC: userspace exception fixups
From: Jann Horn @ 2018-11-02 23:36 UTC (permalink / raw)
To: Andy Lutomirski
Cc: sean.j.christopherson, Dave Hansen, Linus Torvalds, dalias,
Dave Hansen, jethro, jarkko.sakkinen, Florian Weimer, Linux API,
the arch/x86 maintainers, linux-arch, kernel list, Peter Zijlstra,
nhorman, npmccallum, serge.ayoun, shay.katz-zamir, linux-sgx,
andriy.shevchenko, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
carlos, adhemerval.zane
In-Reply-To: <CALCETrXq1Kj=McxnPVjkscNq-b_7LeoFOdu7qKjHdvwcfFh9Og@mail.gmail.com>
On Sat, Nov 3, 2018 at 12:32 AM Andy Lutomirski <luto@kernel.org> wrote:
> On Fri, Nov 2, 2018 at 4:28 PM Jann Horn <jannh@google.com> wrote:
> > On Fri, Nov 2, 2018 at 11:04 PM Sean Christopherson
> > <sean.j.christopherson@intel.com> wrote:
> > > On Fri, Nov 02, 2018 at 08:02:23PM +0100, Jann Horn wrote:
> > > > On Fri, Nov 2, 2018 at 7:27 PM Sean Christopherson
> > > > <sean.j.christopherson@intel.com> wrote:
> > > > > On Fri, Nov 02, 2018 at 10:48:38AM -0700, Andy Lutomirski wrote:
> > > > > > This whole mechanism seems very complicated, and it's not clear
> > > > > > exactly what behavior user code wants.
> > > > >
> > > > > No argument there. That's why I like the approach of dumping the
> > > > > exception to userspace without trying to do anything intelligent in
> > > > > the kernel. Userspace can then do whatever it wants AND we don't
> > > > > have to worry about mucking with stacks.
> > > > >
> > > > > One of the hiccups with the VDSO approach is that the enclave may
> > > > > want to use the untrusted stack, i.e. the stack that has the VDSO's
> > > > > stack frame. For example, Intel's SDK uses the untrusted stack to
> > > > > pass parameters for EEXIT, which means an AEX might occur with what
> > > > > is effectively a bad stack from the VDSO's perspective.
> > > >
> > > > What exactly does "uses the untrusted stack to pass parameters for
> > > > EEXIT" mean? I guess you're saying that the enclave is writing to
> > > > RSP+[0...some_positive_offset], and the written data needs to be
> > > > visible to the code outside the enclave afterwards?
> > >
> > > As is, they actually do it the other way around, i.e. negative offsets
> > > relative to the untrusted %RSP. Going into the enclave there is no
> > > reserved space on the stack. The SDK uses EEXIT like a function call,
> > > i.e. pushing parameters on the stack and making an call outside of the
> > > enclave, hence the name out-call. This allows the SDK to handle any
> > > reasonable out-call without a priori knowledge of the application's
> > > maximum out-call "size".
> >
> > But presumably this is bounded to be at most 128 bytes (the red zone
> > size), right? Otherwise this would be incompatible with
> > non-sigaltstack signal delivery.
>
>
> I think Sean is saying that the enclave also updates RSP.
Ah, bleh, of course.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Andy Lutomirski @ 2018-11-02 23:32 UTC (permalink / raw)
To: Jann Horn
Cc: Christopherson, Sean J, Andrew Lutomirski, Dave Hansen,
Linus Torvalds, Rich Felker, Dave Hansen, Jethro Beekman,
Jarkko Sakkinen, Florian Weimer, Linux API, X86 ML, linux-arch,
LKML, Peter Zijlstra, nhorman, npmccallum, Ayoun, Serge,
shay.katz-zamir, linux-sgx, Andy Shevchenko, Thomas
In-Reply-To: <CAG48ez1+nGDbuBaKJVPAOzEwiBkXqHHs7vmyAqvbeM42r0nFHg@mail.gmail.com>
On Fri, Nov 2, 2018 at 4:28 PM Jann Horn <jannh@google.com> wrote:
>
> On Fri, Nov 2, 2018 at 11:04 PM Sean Christopherson
> <sean.j.christopherson@intel.com> wrote:
> > On Fri, Nov 02, 2018 at 08:02:23PM +0100, Jann Horn wrote:
> > > On Fri, Nov 2, 2018 at 7:27 PM Sean Christopherson
> > > <sean.j.christopherson@intel.com> wrote:
> > > > On Fri, Nov 02, 2018 at 10:48:38AM -0700, Andy Lutomirski wrote:
> > > > > This whole mechanism seems very complicated, and it's not clear
> > > > > exactly what behavior user code wants.
> > > >
> > > > No argument there. That's why I like the approach of dumping the
> > > > exception to userspace without trying to do anything intelligent in
> > > > the kernel. Userspace can then do whatever it wants AND we don't
> > > > have to worry about mucking with stacks.
> > > >
> > > > One of the hiccups with the VDSO approach is that the enclave may
> > > > want to use the untrusted stack, i.e. the stack that has the VDSO's
> > > > stack frame. For example, Intel's SDK uses the untrusted stack to
> > > > pass parameters for EEXIT, which means an AEX might occur with what
> > > > is effectively a bad stack from the VDSO's perspective.
> > >
> > > What exactly does "uses the untrusted stack to pass parameters for
> > > EEXIT" mean? I guess you're saying that the enclave is writing to
> > > RSP+[0...some_positive_offset], and the written data needs to be
> > > visible to the code outside the enclave afterwards?
> >
> > As is, they actually do it the other way around, i.e. negative offsets
> > relative to the untrusted %RSP. Going into the enclave there is no
> > reserved space on the stack. The SDK uses EEXIT like a function call,
> > i.e. pushing parameters on the stack and making an call outside of the
> > enclave, hence the name out-call. This allows the SDK to handle any
> > reasonable out-call without a priori knowledge of the application's
> > maximum out-call "size".
>
> But presumably this is bounded to be at most 128 bytes (the red zone
> size), right? Otherwise this would be incompatible with
> non-sigaltstack signal delivery.
I think Sean is saying that the enclave also updates RSP.
One might reasonably wonder how the SDX knows the offset from RSP to
the function ID. Presumably using RBP?
Anyway, it seems like this is a mess and it's going to be quite hard
to do this in the vDSO due to a lack of any sane way to store the
return address.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Jann Horn @ 2018-11-02 23:27 UTC (permalink / raw)
To: sean.j.christopherson
Cc: Andy Lutomirski, Dave Hansen, Linus Torvalds, dalias, Dave Hansen,
jethro, jarkko.sakkinen, Florian Weimer, Linux API,
the arch/x86 maintainers, linux-arch, kernel list, Peter Zijlstra,
nhorman, npmccallum, serge.ayoun, shay.katz-zamir, linux-sgx,
andriy.shevchenko, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
carlos, adhemerval.za
In-Reply-To: <20181102220437.GI7393@linux.intel.com>
On Fri, Nov 2, 2018 at 11:04 PM Sean Christopherson
<sean.j.christopherson@intel.com> wrote:
> On Fri, Nov 02, 2018 at 08:02:23PM +0100, Jann Horn wrote:
> > On Fri, Nov 2, 2018 at 7:27 PM Sean Christopherson
> > <sean.j.christopherson@intel.com> wrote:
> > > On Fri, Nov 02, 2018 at 10:48:38AM -0700, Andy Lutomirski wrote:
> > > > This whole mechanism seems very complicated, and it's not clear
> > > > exactly what behavior user code wants.
> > >
> > > No argument there. That's why I like the approach of dumping the
> > > exception to userspace without trying to do anything intelligent in
> > > the kernel. Userspace can then do whatever it wants AND we don't
> > > have to worry about mucking with stacks.
> > >
> > > One of the hiccups with the VDSO approach is that the enclave may
> > > want to use the untrusted stack, i.e. the stack that has the VDSO's
> > > stack frame. For example, Intel's SDK uses the untrusted stack to
> > > pass parameters for EEXIT, which means an AEX might occur with what
> > > is effectively a bad stack from the VDSO's perspective.
> >
> > What exactly does "uses the untrusted stack to pass parameters for
> > EEXIT" mean? I guess you're saying that the enclave is writing to
> > RSP+[0...some_positive_offset], and the written data needs to be
> > visible to the code outside the enclave afterwards?
>
> As is, they actually do it the other way around, i.e. negative offsets
> relative to the untrusted %RSP. Going into the enclave there is no
> reserved space on the stack. The SDK uses EEXIT like a function call,
> i.e. pushing parameters on the stack and making an call outside of the
> enclave, hence the name out-call. This allows the SDK to handle any
> reasonable out-call without a priori knowledge of the application's
> maximum out-call "size".
But presumably this is bounded to be at most 128 bytes (the red zone
size), right? Otherwise this would be incompatible with
non-sigaltstack signal delivery.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Jarkko Sakkinen @ 2018-11-02 22:42 UTC (permalink / raw)
To: Sean Christopherson
Cc: Jethro Beekman, Andy Lutomirski, Linus Torvalds, Rich Felker,
Jann Horn, Dave Hansen, Florian Weimer, Linux API, X86 ML,
linux-arch, LKML, Peter Zijlstra, nhorman@redhat.com,
npmccallum@redhat.com, Ayoun, Serge, shay.katz-zamir@intel.com,
linux-sgx@vger.kernel.org, Andy Shevchenko, Thomas Gleixner
In-Reply-To: <20181102165204.GC7393@linux.intel.com>
On Fri, Nov 02, 2018 at 09:52:04AM -0700, Sean Christopherson wrote:
> On Fri, Nov 02, 2018 at 04:37:10PM +0000, Jethro Beekman wrote:
> > On 2018-11-02 09:30, Sean Christopherson wrote:
> > >... The intended convention for EENTER is to have an ENCLU at the AEX target ...
> > >
> > >... to further enforce that the AEX target needs to be ENCLU.
> >
> > Some SGX runtimes may want to use a different AEX target.
>
> To what end? Userspace gets no indication as to why the AEX occurred.
> And if exceptions are getting transfered to userspace the trampoline
> would effectively be handling only INTR, NMI, #MC and EPC #PF.
I've understood that in some cases run-time implementation requires to
run a handler implemented inside the enclave i.e the sequence would be
1. #AEX
2. EENTER(in-enclave handler)
3. EEXIT(%rcx)
4. ERESUME
/Jarkko
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Jarkko Sakkinen @ 2018-11-02 22:37 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Linus Torvalds, Rich Felker, Jann Horn, Dave Hansen,
Christopherson, Sean J, Jethro Beekman, Florian Weimer, Linux API,
X86 ML, linux-arch, LKML, Peter Zijlstra, nhorman, npmccallum,
Ayoun, Serge, shay.katz-zamir, linux-sgx, Andy Shevchenko,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Carlos
In-Reply-To: <CALCETrWe4+apXJNswHAKVVqajGS3jTEKxdd2r3iu-MzGK1v0DA@mail.gmail.com>
On Thu, Nov 01, 2018 at 04:22:55PM -0700, Andy Lutomirski wrote:
> On Thu, Nov 1, 2018 at 2:24 PM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > On Thu, Nov 1, 2018 at 12:31 PM Rich Felker <dalias@libc.org> wrote:
> > >
> > > See my other emails in this thread. You would register the *address*
> > > (in TLS) of a function pointer object pointing to the handler, rather
> > > than the function address of the handler. Then switching handler is
> > > just a single store in userspace, no syscalls involved.
> >
> > Yes.
> >
> > And for just EENTER, maybe that's the right model.
> >
> > If we want to generalize it to other thread-synchronous faults, it
> > needs way more information and a list of handlers, but if we limit the
> > thing to _only_ EENTER getting an SGX fault, then a single "this is
> > the fault handler" address is probably the right thing to do.
>
> It sounds like you're saying that the kernel should know, *before*
> running any user fixup code, whether the fault in question is one that
> wants a fixup. Sounds reasonable.
>
> I think it would be nice, but not absolutely necessary, if user code
> didn't need to poke some value into TLS each time it ran a function
> that had a fixup. With the poke-into-TLS approach, it looks a lot
> like rseq, and rseq doesn't nest very nicely. I think we really want
> this mechanism to Just Work. So we could maybe have a syscall that
> associates a list of fixups with a given range of text addresses. We
> might want the kernel to automatically zap the fixups when the text in
> question is unmapped.
If we would have a syscall to specify a list fixups that would do the
job. Now essentially the only reason we require a vDSO is to implement
a single fixup for EENTER.
If this fixup stuff makes sense for other parts of the kernel,
introducing a vDSO for EENTER means essentially adding ABI to the kernel
that might possibly become legacy fast.
/Jarkko
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Jarkko Sakkinen @ 2018-11-02 22:07 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Dave Hansen, Christopherson, Sean J, Jethro Beekman,
Florian Weimer, Linux API, Jann Horn, Linus Torvalds, X86 ML,
linux-arch, LKML, Peter Zijlstra, Rich Felker, nhorman,
npmccallum, Ayoun, Serge, shay.katz-zamir, linux-sgx,
Andy Shevchenko, Thomas Gleixner, Ingo Molnar, Borislav Petkov
In-Reply-To: <CALCETrWdpoDkbZjkucKL91GWpDPG9p=VqYrULade2pFDR7S=GQ@mail.gmail.com>
On Thu, Nov 01, 2018 at 10:53:40AM -0700, Andy Lutomirski wrote:
> If a handler is registered, then, if a synchronous exception happens
> (page fault, etc), the kernel would set up an exception frame as usual
> but, rather than checking for signal handlers, it would just call the
> registered handler. That handler is expected to either handle the
> exception entirely on its own or to call one of two new syscalls to
> ask for normal signal delivery or to ask to retry the faulting
> instruction.
Why the syscalls are required? Couldn't the handler have just a return
value to indicate the appropriate action?
Another thing that I'm wondering is that what if a signal occurs inside
the exception handler?
/Jarkko
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Sean Christopherson @ 2018-11-02 22:04 UTC (permalink / raw)
To: Jann Horn
Cc: Andy Lutomirski, Dave Hansen, Linus Torvalds, dalias, Dave Hansen,
jethro, jarkko.sakkinen, Florian Weimer, Linux API,
the arch/x86 maintainers, linux-arch, kernel list, Peter Zijlstra,
nhorman, npmccallum, serge.ayoun, shay.katz-zamir, linux-sgx,
andriy.shevchenko, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
carlos, adhemerval.zane
In-Reply-To: <CAG48ez0x7ZPcKvLt01WS-VrLm59WfvYE3nE1xbnyn3Qsp5T2rA@mail.gmail.com>
On Fri, Nov 02, 2018 at 08:02:23PM +0100, Jann Horn wrote:
> On Fri, Nov 2, 2018 at 7:27 PM Sean Christopherson
> <sean.j.christopherson@intel.com> wrote:
> > On Fri, Nov 02, 2018 at 10:48:38AM -0700, Andy Lutomirski wrote:
> > > This whole mechanism seems very complicated, and it's not clear
> > > exactly what behavior user code wants.
> >
> > No argument there. That's why I like the approach of dumping the
> > exception to userspace without trying to do anything intelligent in
> > the kernel. Userspace can then do whatever it wants AND we don't
> > have to worry about mucking with stacks.
> >
> > One of the hiccups with the VDSO approach is that the enclave may
> > want to use the untrusted stack, i.e. the stack that has the VDSO's
> > stack frame. For example, Intel's SDK uses the untrusted stack to
> > pass parameters for EEXIT, which means an AEX might occur with what
> > is effectively a bad stack from the VDSO's perspective.
>
> What exactly does "uses the untrusted stack to pass parameters for
> EEXIT" mean? I guess you're saying that the enclave is writing to
> RSP+[0...some_positive_offset], and the written data needs to be
> visible to the code outside the enclave afterwards?
As is, they actually do it the other way around, i.e. negative offsets
relative to the untrusted %RSP. Going into the enclave there is no
reserved space on the stack. The SDK uses EEXIT like a function call,
i.e. pushing parameters on the stack and making an call outside of the
enclave, hence the name out-call. This allows the SDK to handle any
reasonable out-call without a priori knowledge of the application's
maximum out-call "size".
Rough outline of what happens in a non-faulting case.
1: Userspace executes EENTER
--------------------
| userspace stack |
-------------------- <-- %RSP at EENTER
2: Enclave does EEXIT to invoke out-call function
--------------------
| userspace stack |
-------------------- <-- %RSP at EENTER
| out-call func ID |
| param1 |
| ... |
| paramN |
-------------------- <-- %RSP at EEXIT
3: Userspace re-EENTERs enclave after handling EEXIT request
--------------------
| userspace stack |
-------------------- <-- %RSP at original EENTER
| out-call func ID |
| param1 |
| ... |
| paramN |
-------------------- <-- %RSP at post-EEXIT EENTER
4: Enclave cleans up the stack
--------------------
| userspace stack |
-------------------- <-- %RSP back at original EENTER
In the faulting case, an AEX can occur while the enclave is pushing
parameters onto the stack for EEXIT.
1: Userspace executes EENTER
--------------------
| userspace stack |
-------------------- <-- %RSP at EENTER
2: AEX occurs during enclave prep for EEXIT
--------------------
| userspace stack |
-------------------- <-- %RSP at EENTER
| out-call func ID |
| param1 |
| ... |
-------------------- <-- %RSP at AEX
3: Userspace re-EENTERs enclave to invoke enclave fault handler
--------------------
| userspace stack |
-------------------- <-- %RSP at original EENTER
| out-call func ID |
| param1 |
| ... |
-------------------- <-- %RSP at AEX
| userspace stack |
-------------------- <-- %RSP at EENTER to fault handler
4: Enclave handles the fault, EEXITs back to userspace
--------------------
| userspace stack |
-------------------- <-- %RSP at original EENTER
| out-call func ID |
| param1 |
| ... |
-------------------- <-- %RSP at AEX
| userspace stack |
-------------------- <-- %RSP at EEXIT from fault handler
5: Userspace pops its stack and ERESUMEs back to the enclave
--------------------
| userspace stack |
-------------------- <-- %RSP at original EENTER
| out-call func ID |
| param1 |
| ... |
-------------------- <-- %RSP at ERESUME
6: Enclave finishes its EEXIT to invoke out-call function
--------------------
| userspace stuff |
-------------------- <-- %RSP at original EENTER
| out-call func ID |
| param1 |
| ... |
| paramN |
-------------------- <-- %RSP at EEXIT
> In other words, the vDSO helper would have to not touch the stack
> pointer (only using the 128-byte redzone to store spilled data, at
> least across the enclave entry), and return by decrementing the stack
> pointer by 8 immediately before returning (storing the return pointer
> in the redzone)?
>
> So you'd call the vDSO helper with a normal "call
> vdso_helper_address", then the vDSO helper does "add rsp, 8", then the
> vDSO helper does its magic, and then it returns with "sub rsp, 8" and
> "ret"? That way you don't touch anything on the high-address side of
> RSP while still avoiding running into CET problems. (I'm assuming that
> you can use CET in a process that is hosting SGX enclaves?)
^ permalink raw reply
* Re: Supporting core-specific instruction sets (e.g. big.LITTLE) with restartable sequences
From: Andrew Pinski @ 2018-11-02 19:27 UTC (permalink / raw)
To: mathieu.desnoyers
Cc: Richard Henderson, Will Deacon, LKML, GNU C Library,
Carlos O'Donell, Florian Weimer, Joseph S. Myers,
Szabolcs Nagy, Thomas Gleixner, bmaurer, Peter Zijlstra,
Paul E. McKenney, boqun.feng, davejwatson, pjt, linux-api
In-Reply-To: <313542172.8.1541171544337.JavaMail.zimbra@efficios.com>
On Fri, Nov 2, 2018 at 8:12 AM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> Hi Richard,
>
> I stumbled on these articles:
>
> - https://medium.com/@jadr2ddude/a-big-little-problem-a-tale-of-big-little-gone-wrong-e7778ce744bb
> - https://www.mono-project.com/news/2016/09/12/arm64-icache/
>
> and discussed them with Will Deacon. He told me you were looking into gcc atomics and it might be
> worthwhile to discuss the possible use of the new rseq system call that has been added in Linux 4.18
> for those use-cases.
>
> Basically, the use-cases targeted are those where some cores on the system support a larger instruction
> set than others. So for instance, some cores could use a faster atomic add instruction than others, which
> should rely on a slower fallback. This is also the same story for reading the performance monitoring
> unit counters from user-space: it depends on the feature-set supported by the CPU on which the instruction
> is issued. Same applies to cores having different cache-line sizes.
>
> The main problem is that the kernel can migrate a thread at any point between user-space reading the
> current cpu number and issuing the instruction. This is where rseq can help.
>
> The core idea to solve the instruction set issue is to set a mask of cpus supporting the new instruction
> in a library constructor, and then load cpu_id, use it with the mask, and branch to either the new or
> old instruction, all with a rseq critical section. If the kernel needs to abort due to preemption or
> signal delivery, the abort behavior would be to issue the fallback (slow) atomic operation, which
> guarantees progress even if single-stepping.
>
> As long as the load, test and branch is faster than the performance delta between the old and new atomic
> instruction, it would be worth it.
>
> In the case of PMU read from user-space, using rseq to figure out how to issue the PMU read enables a
> use-case which is not otherwise possible to do on big.LITTLE. On rseq abort, it would fallback to a
> system call to read the PMU counter. This abort behavior guarantees forward progress.
>
> The second article is about cache line size discrepancy between CPUs. Here again, doing the cacheline
> flushing in a rseq critical section could allow tuning it to characteristics of the actual core it is
> running on. The fast-path would use a stride fitting the current core characteristics, and if rseq
> needs to abort, the slow-path would fall-back to a conservative value which would fit all cores (smaller
> cache line size on the overall system). Once again, this abort behavior guarantees forward progress.
> This would only work, of course, if cacheline invalidation done on a big core end up being propagated
> to other cores in a way that clears all the cache lines corresponding to the one targeted on the big
> core.
Cache flusing is only one thing that deals with cache line sizes
difference. Another thing which either needs to be emulated in the
software or disable is the "dc ZVA" instruction which is used in
memset.
There are most likely eithers too. For an example, dealing with dmb/dsb sizes.
Thanks,
Andrew
>
> Thoughts ?
>
> Thanks,
>
> Mathieu
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
^ permalink raw reply
* Re: Supporting core-specific instruction sets (e.g. big.LITTLE) with restartable sequences
From: Florian Weimer @ 2018-11-02 19:18 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: Richard Henderson, Will Deacon, linux-kernel, libc-alpha,
Carlos O'Donell, Joseph Myers, Szabolcs Nagy, Thomas Gleixner,
Ben Maurer, Peter Zijlstra, Paul E. McKenney, Boqun Feng,
Dave Watson, Paul Turner, linux-api
In-Reply-To: <313542172.8.1541171544337.JavaMail.zimbra@efficios.com>
* Mathieu Desnoyers:
> Basically, the use-cases targeted are those where some cores on the
> system support a larger instruction set than others. So for instance,
> some cores could use a faster atomic add instruction than others,
> which should rely on a slower fallback. This is also the same story
> for reading the performance monitoring unit counters from user-space:
> it depends on the feature-set supported by the CPU on which the
> instruction is issued. Same applies to cores having different
> cache-line sizes.
The kernel needs to present a consistent view to userspace, the common
denominator. I don't think there is any other way.
The situation is not new at all, by the way. It also arises with VM and
process migration. In glibc, we do not re-run CPU feature selection
upon resume (and how could we? function pointers would have to change),
and we have no plans to implement anything differently.
Thanks,
Florian
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Jann Horn @ 2018-11-02 19:02 UTC (permalink / raw)
To: sean.j.christopherson
Cc: Andy Lutomirski, Dave Hansen, Linus Torvalds, dalias, Dave Hansen,
jethro, jarkko.sakkinen, Florian Weimer, Linux API,
the arch/x86 maintainers, linux-arch, kernel list, Peter Zijlstra,
nhorman, npmccallum, serge.ayoun, shay.katz-zamir, linux-sgx,
andriy.shevchenko, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
carlos, adhemerval.za
In-Reply-To: <20181102182712.GG7393@linux.intel.com>
On Fri, Nov 2, 2018 at 7:27 PM Sean Christopherson
<sean.j.christopherson@intel.com> wrote:
> On Fri, Nov 02, 2018 at 10:48:38AM -0700, Andy Lutomirski wrote:
> > This whole mechanism seems very complicated, and it's not clear
> > exactly what behavior user code wants.
>
> No argument there. That's why I like the approach of dumping the
> exception to userspace without trying to do anything intelligent in
> the kernel. Userspace can then do whatever it wants AND we don't
> have to worry about mucking with stacks.
>
> One of the hiccups with the VDSO approach is that the enclave may
> want to use the untrusted stack, i.e. the stack that has the VDSO's
> stack frame. For example, Intel's SDK uses the untrusted stack to
> pass parameters for EEXIT, which means an AEX might occur with what
> is effectively a bad stack from the VDSO's perspective.
What exactly does "uses the untrusted stack to pass parameters for
EEXIT" mean? I guess you're saying that the enclave is writing to
RSP+[0...some_positive_offset], and the written data needs to be
visible to the code outside the enclave afterwards?
In other words, the vDSO helper would have to not touch the stack
pointer (only using the 128-byte redzone to store spilled data, at
least across the enclave entry), and return by decrementing the stack
pointer by 8 immediately before returning (storing the return pointer
in the redzone)?
So you'd call the vDSO helper with a normal "call
vdso_helper_address", then the vDSO helper does "add rsp, 8", then the
vDSO helper does its magic, and then it returns with "sub rsp, 8" and
"ret"? That way you don't touch anything on the high-address side of
RSP while still avoiding running into CET problems. (I'm assuming that
you can use CET in a process that is hosting SGX enclaves?)
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Sean Christopherson @ 2018-11-02 18:27 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Dave Hansen, Linus Torvalds, Rich Felker, Jann Horn, Dave Hansen,
Jethro Beekman, Jarkko Sakkinen, Florian Weimer, Linux API,
X86 ML, linux-arch, LKML, Peter Zijlstra, nhorman, npmccallum,
Ayoun, Serge, shay.katz-zamir, linux-sgx, Andy Shevchenko,
Thomas Gleixner, Ingo Molnar, Borislav Petkov <bp@
In-Reply-To: <CALCETrUQVPN0LSEz+5aNsbfFc=rQ33JHiOoRXVbvEEb9is9DDQ@mail.gmail.com>
On Fri, Nov 02, 2018 at 10:48:38AM -0700, Andy Lutomirski wrote:
> On Fri, Nov 2, 2018 at 10:33 AM Sean Christopherson
> <sean.j.christopherson@intel.com> wrote:
> >
> > On Fri, Nov 02, 2018 at 10:13:23AM -0700, Dave Hansen wrote:
> > > On 11/2/18 10:06 AM, Sean Christopherson wrote:
> > > > On Fri, Nov 02, 2018 at 09:56:44AM -0700, Dave Hansen wrote:
> > > >> On 11/2/18 9:30 AM, Sean Christopherson wrote:
> > > >>> What if rather than having userspace register an address for fixup, the
> > > >>> kernel instead unconditionally does fixup on the ENCLU opcode?
> > > >>
> > > >> The problem is knowing what to do for the fixup. If we have a simple
> > > >> action to take that's universal, like backing up %RIP, or setting some
> > > >> other register state, it's not bad.
> > > >
> > > > Isn't the EENTER/RESUME behavior universal? Or am I missing something?
> > >
> > > Could someone write down all the ways we get in and out of the enclave?
> > >
> > > I think we always get in from userspace calling EENTER or ERESUME. We
> > > can't ever enter directly from the kernel, like via an IRET from what I
> > > understand.
> >
> > Correct, the only way to get into the enclave is EENTER or ERESUME.
> > My understanding is that even SMIs bounce through the AEX target
> > before transitioning to SMM.
> >
> > > We get *out* from exceptions, hardware interrupts, or enclave-explicit
> > > EEXITs. Did I miss any? Remind me where the hardware lands the control
> > > flow in each of those exit cases.
> >
> > And VMExits. There are basically two cases: EEXIT and everything else.
> > EEXIT is a glorified indirect jump, e.g. %RBX holds the target %RIP.
> > Everything else is an Asynchronous Enclave Exit (AEX). On an AEX, %RIP
> > is set to a value specified by EENTER/ERESUME, %RBP and %RSP are
> > restored to pre-enclave values and all other registers are loaded with
> > synthetic state. The actual interrupt/exception/VMExit then triggers,
> > e.g. the %RIP on the stack for an exception is always the AEX target,
> > not the %RIP inside the enclave that actually faulted.
>
> So what exactly happens when an enclave accesses non-enclave memory
> and takes a page fault, for example? The SDM says that the #PF vector
> and error code are stored in the SSA frame where the kernel can't see
> them. Is a real #PF then delivered?
Yes. From there kernel's perspective a #PF occurred on the %RIP of the
AEX target. This holds true for all AEX types, e.g. GUEST_RIP on VMExit
also points at the AEX target. On an AEX, %RAX, %RBX and %RCX are set
to match the ERESUME parameter. The idea is for userspace to have an
ENCU at the AEX so that it automatically ERESUMEs the enclave after the
kernel handles the fault. And the trampoline approach means the ucode
flows for exceptions, interrupts, VMExit, VMEnter, IRET, RSM, etc...
generally don't need to be SGX-aware. The events themselves just need
to be redirected to the AEX target and then redo the event.
> I guess that, if the memory in question gets faulted in, then the
> kernel resumes exection at the AEP address, which does ERESUME, and
> the enclave resumes. But if the access is bad, then the kernel
> delivers a signal (or uses some other new mechanism), and then what
> happens? Is the enclave just considered dead? Is user code supposed
> to EENTER back into the enclave to tell it that it got an error?
Completely depends on the enclave and its runtime. A simple enclave
mayy never expect to encounter a bad access or #UD and so its runtime
would probably just kill it. A test/development enclave might have
its runtime call back into the enclave to dump state on a fatal fault.
Complex runtimes, e.g. libraries that wrap unmodified applications,
will call back into the enclave so that libraries in-enclave fault
handler can decode what went wrong and take action accordingly, e.g.
request CPUID information if unmodified code tried to do CPUID.
> This whole mechanism seems very complicated, and it's not clear
> exactly what behavior user code wants.
No argument there. That's why I like the approach of dumping the
exception to userspace without trying to do anything intelligent in
the kernel. Userspace can then do whatever it wants AND we don't
have to worry about mucking with stacks.
One of the hiccups with the VDSO approach is that the enclave may
want to use the untrusted stack, i.e. the stack that has the VDSO's
stack frame. For example, Intel's SDK uses the untrusted stack to
pass parameters for EEXIT, which means an AEX might occur with what
is effectively a bad stack from the VDSO's perspective.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Andy Lutomirski @ 2018-11-02 17:48 UTC (permalink / raw)
To: Christopherson, Sean J
Cc: Dave Hansen, Andrew Lutomirski, Linus Torvalds, Rich Felker,
Jann Horn, Dave Hansen, Jethro Beekman, Jarkko Sakkinen,
Florian Weimer, Linux API, X86 ML, linux-arch, LKML,
Peter Zijlstra, nhorman, npmccallum, Ayoun, Serge,
shay.katz-zamir, linux-sgx, Andy Shevchenko, Thomas Gleixner,
Ingo
In-Reply-To: <20181102173350.GF7393@linux.intel.com>
On Fri, Nov 2, 2018 at 10:33 AM Sean Christopherson
<sean.j.christopherson@intel.com> wrote:
>
> On Fri, Nov 02, 2018 at 10:13:23AM -0700, Dave Hansen wrote:
> > On 11/2/18 10:06 AM, Sean Christopherson wrote:
> > > On Fri, Nov 02, 2018 at 09:56:44AM -0700, Dave Hansen wrote:
> > >> On 11/2/18 9:30 AM, Sean Christopherson wrote:
> > >>> What if rather than having userspace register an address for fixup, the
> > >>> kernel instead unconditionally does fixup on the ENCLU opcode?
> > >>
> > >> The problem is knowing what to do for the fixup. If we have a simple
> > >> action to take that's universal, like backing up %RIP, or setting some
> > >> other register state, it's not bad.
> > >
> > > Isn't the EENTER/RESUME behavior universal? Or am I missing something?
> >
> > Could someone write down all the ways we get in and out of the enclave?
> >
> > I think we always get in from userspace calling EENTER or ERESUME. We
> > can't ever enter directly from the kernel, like via an IRET from what I
> > understand.
>
> Correct, the only way to get into the enclave is EENTER or ERESUME.
> My understanding is that even SMIs bounce through the AEX target
> before transitioning to SMM.
>
> > We get *out* from exceptions, hardware interrupts, or enclave-explicit
> > EEXITs. Did I miss any? Remind me where the hardware lands the control
> > flow in each of those exit cases.
>
> And VMExits. There are basically two cases: EEXIT and everything else.
> EEXIT is a glorified indirect jump, e.g. %RBX holds the target %RIP.
> Everything else is an Asynchronous Enclave Exit (AEX). On an AEX, %RIP
> is set to a value specified by EENTER/ERESUME, %RBP and %RSP are
> restored to pre-enclave values and all other registers are loaded with
> synthetic state. The actual interrupt/exception/VMExit then triggers,
> e.g. the %RIP on the stack for an exception is always the AEX target,
> not the %RIP inside the enclave that actually faulted.
So what exactly happens when an enclave accesses non-enclave memory
and takes a page fault, for example? The SDM says that the #PF vector
and error code are stored in the SSA frame where the kernel can't see
them. Is a real #PF then delivered?
I guess that, if the memory in question gets faulted in, then the
kernel resumes exection at the AEP address, which does ERESUME, and
the enclave resumes. But if the access is bad, then the kernel
delivers a signal (or uses some other new mechanism), and then what
happens? Is the enclave just considered dead? Is user code supposed
to EENTER back into the enclave to tell it that it got an error?
This whole mechanism seems very complicated, and it's not clear
exactly what behavior user code wants.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Sean Christopherson @ 2018-11-02 17:33 UTC (permalink / raw)
To: Dave Hansen
Cc: Andy Lutomirski, Linus Torvalds, Rich Felker, Jann Horn,
Dave Hansen, Jethro Beekman, Jarkko Sakkinen, Florian Weimer,
Linux API, X86 ML, linux-arch, LKML, Peter Zijlstra, nhorman,
npmccallum, Ayoun, Serge, shay.katz-zamir, linux-sgx,
Andy Shevchenko, Thomas Gleixner, Ingo Molnar, Borislav Petkov
In-Reply-To: <a4d2b3ec-43ec-f062-e180-c6e5a0d9fab8@intel.com>
On Fri, Nov 02, 2018 at 10:13:23AM -0700, Dave Hansen wrote:
> On 11/2/18 10:06 AM, Sean Christopherson wrote:
> > On Fri, Nov 02, 2018 at 09:56:44AM -0700, Dave Hansen wrote:
> >> On 11/2/18 9:30 AM, Sean Christopherson wrote:
> >>> What if rather than having userspace register an address for fixup, the
> >>> kernel instead unconditionally does fixup on the ENCLU opcode?
> >>
> >> The problem is knowing what to do for the fixup. If we have a simple
> >> action to take that's universal, like backing up %RIP, or setting some
> >> other register state, it's not bad.
> >
> > Isn't the EENTER/RESUME behavior universal? Or am I missing something?
>
> Could someone write down all the ways we get in and out of the enclave?
>
> I think we always get in from userspace calling EENTER or ERESUME. We
> can't ever enter directly from the kernel, like via an IRET from what I
> understand.
Correct, the only way to get into the enclave is EENTER or ERESUME.
My understanding is that even SMIs bounce through the AEX target
before transitioning to SMM.
> We get *out* from exceptions, hardware interrupts, or enclave-explicit
> EEXITs. Did I miss any? Remind me where the hardware lands the control
> flow in each of those exit cases.
And VMExits. There are basically two cases: EEXIT and everything else.
EEXIT is a glorified indirect jump, e.g. %RBX holds the target %RIP.
Everything else is an Asynchronous Enclave Exit (AEX). On an AEX, %RIP
is set to a value specified by EENTER/ERESUME, %RBP and %RSP are
restored to pre-enclave values and all other registers are loaded with
synthetic state. The actual interrupt/exception/VMExit then triggers,
e.g. the %RIP on the stack for an exception is always the AEX target,
not the %RIP inside the enclave that actually faulted.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Rich Felker @ 2018-11-02 17:32 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Jethro Beekman, Christopherson, Sean J, Linus Torvalds, Jann Horn,
Dave Hansen, Jarkko Sakkinen, Florian Weimer, Linux API, X86 ML,
linux-arch, LKML, Peter Zijlstra, nhorman, npmccallum,
Ayoun, Serge, shay.katz-zamir, linux-sgx, Andy Shevchenko,
Thomas Gleixner, Ingo Molnar, Borislav Petkov
In-Reply-To: <CALCETrVW23Hmwj9DeYJ95mGd-_6ZG4obZG+evc2NdXEGud6DBA@mail.gmail.com>
On Fri, Nov 02, 2018 at 10:16:02AM -0700, Andy Lutomirski wrote:
> On Fri, Nov 2, 2018 at 10:05 AM Jethro Beekman <jethro@fortanix.com> wrote:
> >
> > On 2018-11-02 10:01, Andy Lutomirski wrote:
> > > On Fri, Nov 2, 2018 at 9:56 AM Jethro Beekman <jethro@fortanix.com> wrote:
> > >>
> > >> On 2018-11-02 09:52, Sean Christopherson wrote:
> > >>> On Fri, Nov 02, 2018 at 04:37:10PM +0000, Jethro Beekman wrote:
> > >>>> On 2018-11-02 09:30, Sean Christopherson wrote:
> > >>>>> ... The intended convention for EENTER is to have an ENCLU at the AEX target ...
> > >>>>>
> > >>>>> ... to further enforce that the AEX target needs to be ENCLU.
> > >>>>
> > >>>> Some SGX runtimes may want to use a different AEX target.
> > >>>
> > >>> To what end? Userspace gets no indication as to why the AEX occurred.
> > >>> And if exceptions are getting transfered to userspace the trampoline
> > >>> would effectively be handling only INTR, NMI, #MC and EPC #PF.
> > >>>
> > >>
> > >> Various reasons...
> > >>
> > >> Userspace may have established an exception handling convention with the
> > >> enclave (by setting TCS.NSSA > 1) and may want to call EENTER instead of
> > >> ERESUME.
> > >>
> > >
> > > Ugh,
> > >
> > > I sincerely hope that a future ISA extension lets the kernel return
> > > directly back to enclave mode so that AEX events become entirely
> > > invisible to user code.
> >
> > Can you explain how this would work for things like #BR/#DE/#UD that
> > need to be fixed up by code running in the enclave before it can be resumed?
> >
>
> Sure. A better enclave entry function would complete in one of two ways:
>
> 1. The enclave exited normally. Some register output would indicate this.
>
> 2. The enclave existed due to an exception or interrupt. The kernel
> would be entered directly and notified of what happened. The kernel
> would fix it up if needed (#PF), handle an interrupt (for en enclave
> exit due to an interrupt) and reenter the enclave. If, of the error
> is not kernel-fixable-up, it would return back to userspace with some
> explanation of what happened. Kind of like normal user code.
>
> Alternatively, the CPU could directly distinguish between exceptions
> that need the enclave's attention (#BR) and those that don't.
>
> The fact that user code is involved in resuming an enclave when a
> hardware interrupt occurs is silly IMO.
Agreed absolutely. If this is necessary, it seems like there should be
an agreed-upon protocol such that the kernel can make it happen via
returning to code in the vdso that performs the actual resume, so that
the application never sees it.
Rich
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Andy Lutomirski @ 2018-11-02 17:16 UTC (permalink / raw)
To: Jethro Beekman
Cc: Andrew Lutomirski, Christopherson, Sean J, Linus Torvalds,
Rich Felker, Jann Horn, Dave Hansen, Jarkko Sakkinen,
Florian Weimer, Linux API, X86 ML, linux-arch, LKML,
Peter Zijlstra, nhorman, npmccallum, Ayoun, Serge,
shay.katz-zamir, linux-sgx, Andy Shevchenko, Thomas Gleixner,
Ingo Molnar
In-Reply-To: <1b87048e-7ed8-14a1-572f-3cd825319f8c@fortanix.com>
On Fri, Nov 2, 2018 at 10:05 AM Jethro Beekman <jethro@fortanix.com> wrote:
>
> On 2018-11-02 10:01, Andy Lutomirski wrote:
> > On Fri, Nov 2, 2018 at 9:56 AM Jethro Beekman <jethro@fortanix.com> wrote:
> >>
> >> On 2018-11-02 09:52, Sean Christopherson wrote:
> >>> On Fri, Nov 02, 2018 at 04:37:10PM +0000, Jethro Beekman wrote:
> >>>> On 2018-11-02 09:30, Sean Christopherson wrote:
> >>>>> ... The intended convention for EENTER is to have an ENCLU at the AEX target ...
> >>>>>
> >>>>> ... to further enforce that the AEX target needs to be ENCLU.
> >>>>
> >>>> Some SGX runtimes may want to use a different AEX target.
> >>>
> >>> To what end? Userspace gets no indication as to why the AEX occurred.
> >>> And if exceptions are getting transfered to userspace the trampoline
> >>> would effectively be handling only INTR, NMI, #MC and EPC #PF.
> >>>
> >>
> >> Various reasons...
> >>
> >> Userspace may have established an exception handling convention with the
> >> enclave (by setting TCS.NSSA > 1) and may want to call EENTER instead of
> >> ERESUME.
> >>
> >
> > Ugh,
> >
> > I sincerely hope that a future ISA extension lets the kernel return
> > directly back to enclave mode so that AEX events become entirely
> > invisible to user code.
>
> Can you explain how this would work for things like #BR/#DE/#UD that
> need to be fixed up by code running in the enclave before it can be resumed?
>
Sure. A better enclave entry function would complete in one of two ways:
1. The enclave exited normally. Some register output would indicate this.
2. The enclave existed due to an exception or interrupt. The kernel
would be entered directly and notified of what happened. The kernel
would fix it up if needed (#PF), handle an interrupt (for en enclave
exit due to an interrupt) and reenter the enclave. If, of the error
is not kernel-fixable-up, it would return back to userspace with some
explanation of what happened. Kind of like normal user code.
Alternatively, the CPU could directly distinguish between exceptions
that need the enclave's attention (#BR) and those that don't.
The fact that user code is involved in resuming an enclave when a
hardware interrupt occurs is silly IMO.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Dave Hansen @ 2018-11-02 17:13 UTC (permalink / raw)
To: Sean Christopherson
Cc: Andy Lutomirski, Linus Torvalds, Rich Felker, Jann Horn,
Dave Hansen, Jethro Beekman, Jarkko Sakkinen, Florian Weimer,
Linux API, X86 ML, linux-arch, LKML, Peter Zijlstra, nhorman,
npmccallum, Ayoun, Serge, shay.katz-zamir, linux-sgx,
Andy Shevchenko, Thomas Gleixner, Ingo Molnar, Borislav Petkov
In-Reply-To: <20181102170627.GD7393@linux.intel.com>
On 11/2/18 10:06 AM, Sean Christopherson wrote:
> On Fri, Nov 02, 2018 at 09:56:44AM -0700, Dave Hansen wrote:
>> On 11/2/18 9:30 AM, Sean Christopherson wrote:
>>> What if rather than having userspace register an address for fixup, the
>>> kernel instead unconditionally does fixup on the ENCLU opcode?
>>
>> The problem is knowing what to do for the fixup. If we have a simple
>> action to take that's universal, like backing up %RIP, or setting some
>> other register state, it's not bad.
>
> Isn't the EENTER/RESUME behavior universal? Or am I missing something?
Could someone write down all the ways we get in and out of the enclave?
I think we always get in from userspace calling EENTER or ERESUME. We
can't ever enter directly from the kernel, like via an IRET from what I
understand.
We get *out* from exceptions, hardware interrupts, or enclave-explicit
EEXITs. Did I miss any? Remind me where the hardware lands the control
flow in each of those exit cases.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Sean Christopherson @ 2018-11-02 17:12 UTC (permalink / raw)
To: Jethro Beekman
Cc: Andy Lutomirski, Linus Torvalds, Rich Felker, Jann Horn,
Dave Hansen, Jarkko Sakkinen, Florian Weimer, Linux API, X86 ML,
linux-arch, LKML, Peter Zijlstra, nhorman@redhat.com,
npmccallum@redhat.com, Ayoun, Serge, shay.katz-zamir@intel.com,
linux-sgx@vger.kernel.org, Andy Shevchenko, Thomas
In-Reply-To: <dd455b30-b586-bf5b-2195-cf20c1af9890@fortanix.com>
On Fri, Nov 02, 2018 at 04:56:36PM +0000, Jethro Beekman wrote:
> On 2018-11-02 09:52, Sean Christopherson wrote:
> >On Fri, Nov 02, 2018 at 04:37:10PM +0000, Jethro Beekman wrote:
> >>On 2018-11-02 09:30, Sean Christopherson wrote:
> >>>... The intended convention for EENTER is to have an ENCLU at the AEX target ...
> >>>
> >>>... to further enforce that the AEX target needs to be ENCLU.
> >>
> >>Some SGX runtimes may want to use a different AEX target.
> >
> >To what end? Userspace gets no indication as to why the AEX occurred.
> >And if exceptions are getting transfered to userspace the trampoline
> >would effectively be handling only INTR, NMI, #MC and EPC #PF.
> >
>
> Various reasons...
>
> Userspace may have established an exception handling convention with the
> enclave (by setting TCS.NSSA > 1) and may want to call EENTER instead of
> ERESUME.
The ERESUME trampoline would only be invoked for exceptions that aren't
transferred to userspace. On #BR, #UD, etc..., the kernel would fixup
%RIP to effectively point at @fault_handler. Userspace can then do
whatever it wants to handle the fault, e.g. do EENTER if the fault needs
to be serviced by the enclave.
> Userspace may want fine-grained control over enclave scheduling (e.g.
> SGX-Step)
Uh, isn't SGX-Step an attack on SGX? Preventing userspace from playing
games with enclave scheduling seems like a good thing.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Sean Christopherson @ 2018-11-02 17:06 UTC (permalink / raw)
To: Dave Hansen
Cc: Andy Lutomirski, Linus Torvalds, Rich Felker, Jann Horn,
Dave Hansen, Jethro Beekman, Jarkko Sakkinen, Florian Weimer,
Linux API, X86 ML, linux-arch, LKML, Peter Zijlstra, nhorman,
npmccallum, Ayoun, Serge, shay.katz-zamir, linux-sgx,
Andy Shevchenko, Thomas Gleixner, Ingo Molnar, Borislav Petkov
In-Reply-To: <7050972d-a874-dc08-3214-93e81181da60@intel.com>
On Fri, Nov 02, 2018 at 09:56:44AM -0700, Dave Hansen wrote:
> On 11/2/18 9:30 AM, Sean Christopherson wrote:
> > What if rather than having userspace register an address for fixup, the
> > kernel instead unconditionally does fixup on the ENCLU opcode?
>
> The problem is knowing what to do for the fixup. If we have a simple
> action to take that's universal, like backing up %RIP, or setting some
> other register state, it's not bad.
Isn't the EENTER/RESUME behavior universal? Or am I missing something?
> Think of our prefetch fixups in the page fault code. We do some
> instruction decoding to look for them, and then largely return from the
> fault and let the CPU retry. We know *exactly* what to do for these.
>
> But, if we need to call arbitrary code, or switch stacks, we need an
> explicit ABI around it *anyway*, because the action to take isn't clear.
>
> For an enclave exit that's because of a hardware interrupt or page
> fault, life is good. We really *could* just set %RIP to let ERESUME run
> again, kinda like we do for (some) syscall situations. But the
> situations for which we can't just call ERESUME, like the out-calls make
> this more challenging. I think we'd need some explicit new interfaces
> for those.
I don't see how out-calls are a problem. Once EEXIT completes we're
no longer in the enclave and EPCM faults are no longer a concern, i.e.
we don't need to do fixup. Every other enclave exit is either an
exception or an interrupt. And the only way to get back into the
enclave is via ENCLU (EENTER or ERESUME).
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Jethro Beekman @ 2018-11-02 17:05 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Christopherson, Sean J, Linus Torvalds, Rich Felker, Jann Horn,
Dave Hansen, Jarkko Sakkinen, Florian Weimer, Linux API, X86 ML,
linux-arch, LKML, Peter Zijlstra, nhorman@redhat.com,
npmccallum@redhat.com, Ayoun, Serge, shay.katz-zamir@intel.com,
linux-sgx@vger.kernel.org, Andy Shevchenko
In-Reply-To: <CALCETrUo8uwnu3dBVSHVtB_z4uQgW=40grVdZWmTt64Tei6WiQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1335 bytes --]
On 2018-11-02 10:01, Andy Lutomirski wrote:
> On Fri, Nov 2, 2018 at 9:56 AM Jethro Beekman <jethro@fortanix.com> wrote:
>>
>> On 2018-11-02 09:52, Sean Christopherson wrote:
>>> On Fri, Nov 02, 2018 at 04:37:10PM +0000, Jethro Beekman wrote:
>>>> On 2018-11-02 09:30, Sean Christopherson wrote:
>>>>> ... The intended convention for EENTER is to have an ENCLU at the AEX target ...
>>>>>
>>>>> ... to further enforce that the AEX target needs to be ENCLU.
>>>>
>>>> Some SGX runtimes may want to use a different AEX target.
>>>
>>> To what end? Userspace gets no indication as to why the AEX occurred.
>>> And if exceptions are getting transfered to userspace the trampoline
>>> would effectively be handling only INTR, NMI, #MC and EPC #PF.
>>>
>>
>> Various reasons...
>>
>> Userspace may have established an exception handling convention with the
>> enclave (by setting TCS.NSSA > 1) and may want to call EENTER instead of
>> ERESUME.
>>
>
> Ugh,
>
> I sincerely hope that a future ISA extension lets the kernel return
> directly back to enclave mode so that AEX events become entirely
> invisible to user code.
Can you explain how this would work for things like #BR/#DE/#UD that
need to be fixed up by code running in the enclave before it can be resumed?
--
Jethro Beekman | Fortanix
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3990 bytes --]
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Andy Lutomirski @ 2018-11-02 17:01 UTC (permalink / raw)
To: Jethro Beekman
Cc: Christopherson, Sean J, Andrew Lutomirski, Linus Torvalds,
Rich Felker, Jann Horn, Dave Hansen, Jarkko Sakkinen,
Florian Weimer, Linux API, X86 ML, linux-arch, LKML,
Peter Zijlstra, nhorman, npmccallum, Ayoun, Serge,
shay.katz-zamir, linux-sgx, Andy Shevchenko, Thomas Gleixner,
Ingo Molnar
In-Reply-To: <dd455b30-b586-bf5b-2195-cf20c1af9890@fortanix.com>
On Fri, Nov 2, 2018 at 9:56 AM Jethro Beekman <jethro@fortanix.com> wrote:
>
> On 2018-11-02 09:52, Sean Christopherson wrote:
> > On Fri, Nov 02, 2018 at 04:37:10PM +0000, Jethro Beekman wrote:
> >> On 2018-11-02 09:30, Sean Christopherson wrote:
> >>> ... The intended convention for EENTER is to have an ENCLU at the AEX target ...
> >>>
> >>> ... to further enforce that the AEX target needs to be ENCLU.
> >>
> >> Some SGX runtimes may want to use a different AEX target.
> >
> > To what end? Userspace gets no indication as to why the AEX occurred.
> > And if exceptions are getting transfered to userspace the trampoline
> > would effectively be handling only INTR, NMI, #MC and EPC #PF.
> >
>
> Various reasons...
>
> Userspace may have established an exception handling convention with the
> enclave (by setting TCS.NSSA > 1) and may want to call EENTER instead of
> ERESUME.
>
Ugh,
I sincerely hope that a future ISA extension lets the kernel return
directly back to enclave mode so that AEX events become entirely
invisible to user code. It would be nice if user developers didn't
start depending on the rather odd AEX semantics right now. But I
don't think the kernel can sanely do much about it.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Dave Hansen @ 2018-11-02 16:56 UTC (permalink / raw)
To: Sean Christopherson, Andy Lutomirski
Cc: Linus Torvalds, Rich Felker, Jann Horn, Dave Hansen,
Jethro Beekman, Jarkko Sakkinen, Florian Weimer, Linux API,
X86 ML, linux-arch, LKML, Peter Zijlstra, nhorman, npmccallum,
Ayoun, Serge, shay.katz-zamir, linux-sgx, Andy Shevchenko,
Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Carlos O'Donell
In-Reply-To: <20181102163034.GB7393@linux.intel.com>
On 11/2/18 9:30 AM, Sean Christopherson wrote:
> What if rather than having userspace register an address for fixup, the
> kernel instead unconditionally does fixup on the ENCLU opcode?
The problem is knowing what to do for the fixup. If we have a simple
action to take that's universal, like backing up %RIP, or setting some
other register state, it's not bad.
Think of our prefetch fixups in the page fault code. We do some
instruction decoding to look for them, and then largely return from the
fault and let the CPU retry. We know *exactly* what to do for these.
But, if we need to call arbitrary code, or switch stacks, we need an
explicit ABI around it *anyway*, because the action to take isn't clear.
For an enclave exit that's because of a hardware interrupt or page
fault, life is good. We really *could* just set %RIP to let ERESUME run
again, kinda like we do for (some) syscall situations. But the
situations for which we can't just call ERESUME, like the out-calls make
this more challenging. I think we'd need some explicit new interfaces
for those.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Jethro Beekman @ 2018-11-02 16:56 UTC (permalink / raw)
To: Sean Christopherson
Cc: Andy Lutomirski, Linus Torvalds, Rich Felker, Jann Horn,
Dave Hansen, Jarkko Sakkinen, Florian Weimer, Linux API, X86 ML,
linux-arch, LKML, Peter Zijlstra, nhorman@redhat.com,
npmccallum@redhat.com, Ayoun, Serge, shay.katz-zamir@intel.com,
linux-sgx@vger.kernel.org, Andy Shevchenko, Thomas
In-Reply-To: <20181102165204.GC7393@linux.intel.com>
[-- Attachment #1: Type: text/plain, Size: 916 bytes --]
On 2018-11-02 09:52, Sean Christopherson wrote:
> On Fri, Nov 02, 2018 at 04:37:10PM +0000, Jethro Beekman wrote:
>> On 2018-11-02 09:30, Sean Christopherson wrote:
>>> ... The intended convention for EENTER is to have an ENCLU at the AEX target ...
>>>
>>> ... to further enforce that the AEX target needs to be ENCLU.
>>
>> Some SGX runtimes may want to use a different AEX target.
>
> To what end? Userspace gets no indication as to why the AEX occurred.
> And if exceptions are getting transfered to userspace the trampoline
> would effectively be handling only INTR, NMI, #MC and EPC #PF.
>
Various reasons...
Userspace may have established an exception handling convention with the
enclave (by setting TCS.NSSA > 1) and may want to call EENTER instead of
ERESUME.
Userspace may want fine-grained control over enclave scheduling (e.g.
SGX-Step)
--
Jethro Beekman | Fortanix
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3990 bytes --]
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Sean Christopherson @ 2018-11-02 16:52 UTC (permalink / raw)
To: Jethro Beekman
Cc: Andy Lutomirski, Linus Torvalds, Rich Felker, Jann Horn,
Dave Hansen, Jarkko Sakkinen, Florian Weimer, Linux API, X86 ML,
linux-arch, LKML, Peter Zijlstra, nhorman@redhat.com,
npmccallum@redhat.com, Ayoun, Serge, shay.katz-zamir@intel.com,
linux-sgx@vger.kernel.org, Andy Shevchenko, Thomas
In-Reply-To: <7e14ee0e-ce15-1e88-7ae9-4d0f40cb3d84@fortanix.com>
On Fri, Nov 02, 2018 at 04:37:10PM +0000, Jethro Beekman wrote:
> On 2018-11-02 09:30, Sean Christopherson wrote:
> >... The intended convention for EENTER is to have an ENCLU at the AEX target ...
> >
> >... to further enforce that the AEX target needs to be ENCLU.
>
> Some SGX runtimes may want to use a different AEX target.
To what end? Userspace gets no indication as to why the AEX occurred.
And if exceptions are getting transfered to userspace the trampoline
would effectively be handling only INTR, NMI, #MC and EPC #PF.
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Jethro Beekman @ 2018-11-02 16:37 UTC (permalink / raw)
To: Sean Christopherson, Andy Lutomirski
Cc: Linus Torvalds, Rich Felker, Jann Horn, Dave Hansen,
Jarkko Sakkinen, Florian Weimer, Linux API, X86 ML, linux-arch,
LKML, Peter Zijlstra, nhorman@redhat.com, npmccallum@redhat.com,
Ayoun, Serge, shay.katz-zamir@intel.com,
linux-sgx@vger.kernel.org, Andy Shevchenko, Thomas Gleixner, Ingo
In-Reply-To: <20181102163034.GB7393@linux.intel.com>
[-- Attachment #1: Type: text/plain, Size: 298 bytes --]
On 2018-11-02 09:30, Sean Christopherson wrote:
> ... The intended convention for EENTER is to have an ENCLU at the AEX target ...
>
> ... to further enforce that the AEX target needs to be ENCLU.
Some SGX runtimes may want to use a different AEX target.
--
Jethro Beekman | Fortanix
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3990 bytes --]
^ permalink raw reply
* Re: RFC: userspace exception fixups
From: Sean Christopherson @ 2018-11-02 16:30 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Linus Torvalds, Rich Felker, Jann Horn, Dave Hansen,
Jethro Beekman, Jarkko Sakkinen, Florian Weimer, Linux API,
X86 ML, linux-arch, LKML, Peter Zijlstra, nhorman, npmccallum,
Ayoun, Serge, shay.katz-zamir, linux-sgx, Andy Shevchenko,
Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Carlos O'Donell
In-Reply-To: <CALCETrWe4+apXJNswHAKVVqajGS3jTEKxdd2r3iu-MzGK1v0DA@mail.gmail.com>
On Thu, Nov 01, 2018 at 04:22:55PM -0700, Andy Lutomirski wrote:
> On Thu, Nov 1, 2018 at 2:24 PM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > On Thu, Nov 1, 2018 at 12:31 PM Rich Felker <dalias@libc.org> wrote:
> > >
> > > See my other emails in this thread. You would register the *address*
> > > (in TLS) of a function pointer object pointing to the handler, rather
> > > than the function address of the handler. Then switching handler is
> > > just a single store in userspace, no syscalls involved.
> >
> > Yes.
> >
> > And for just EENTER, maybe that's the right model.
> >
> > If we want to generalize it to other thread-synchronous faults, it
> > needs way more information and a list of handlers, but if we limit the
> > thing to _only_ EENTER getting an SGX fault, then a single "this is
> > the fault handler" address is probably the right thing to do.
>
> It sounds like you're saying that the kernel should know, *before*
> running any user fixup code, whether the fault in question is one that
> wants a fixup. Sounds reasonable.
>
> I think it would be nice, but not absolutely necessary, if user code
> didn't need to poke some value into TLS each time it ran a function
> that had a fixup. With the poke-into-TLS approach, it looks a lot
> like rseq, and rseq doesn't nest very nicely. I think we really want
> this mechanism to Just Work. So we could maybe have a syscall that
> associates a list of fixups with a given range of text addresses. We
> might want the kernel to automatically zap the fixups when the text in
> question is unmapped.
If this is EENTER specific then nesting isn't an issue. But I don't
see a simple way to restrict the mechanism to EENTER.
What if rather than having userspace register an address for fixup the
kernel instead unconditionally does fixup on the ENCLU opcode? For
example, skip the instruction and put fault info into some combination
of RDX/RSI/RDI (they're cleared on asynchronous enclave exits).
The decode logic is straightforward since ENCLU doesn't have operands,
we'd just have to eat any ignored prefixes. The intended convention
for EENTER is to have an ENCLU at the AEX target (to automatically do
ERESUME after INTR, etc...), so this would work regardless of whether
the fault happened on EENTER or in the enclave. EENTER/ERESUME are
the only ENCLU functions that are allowed outside of an enclave so
there's no danger of accidentally crushing something else.
This way we wouldn't need a VDSO blob and we'd enforce the kernel's
ABI, e.g. a library that tried to use signal handling would go off the
rails when the kernel mucked with the registers. We could even have
the SGX EPC fault handler return VM_FAULT_SIGBUS if the faulting
instruction isn't ENCLU, e.g. to further enforce that the AEX target
needs to be ENCLU.
Userspace would look something like this:
mov tcs, %xbx /* Thread Control Structure address */
leaq async_exit(%rip), %rcx /* AEX target for EENTER/RESUME */
mov $SGX_EENTER, %rax /* EENTER leaf */
async_exit:
ENCLU
fault_handler:
<handle fault>
enclave_exit: /* EEXIT target */
<handle enclave request>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox