* [PATCH v2 1/2] x86, entry: Switch stacks on a paranoid entry from userspace
2014-11-13 22:31 [PATCH v2 0/2] Handle IST interrupts from userspace on the normal stack Andy Lutomirski
@ 2014-11-13 22:31 ` Andy Lutomirski
2014-11-13 22:31 ` [PATCH v2 2/2] uprobes, x86: Fix _TIF_UPROBE vs _TIF_NOTIFY_RESUME Andy Lutomirski
2014-11-13 23:19 ` [PATCH v2 0/2] Handle IST interrupts from userspace on the normal stack Andy Lutomirski
2 siblings, 0 replies; 7+ messages in thread
From: Andy Lutomirski @ 2014-11-13 22:31 UTC (permalink / raw)
To: Borislav Petkov, x86
Cc: linux-kernel, Peter Zijlstra, Oleg Nesterov, Tony Luck,
Andi Kleen, Andy Lutomirski
This causes all non-NMI kernel entries from userspace to run on the
normal kernel stack.
This is, suprisingly, simpler and shorter than the current code. I
removes the IMO rather frightening paranoid_userspace path, and it
make sync_regs much simpler.
There is no risk of stack overflow due to this change -- the kernel
stack that we switch to is empty.
This may enable the machine check code to be simplified at some
point, because a machine check from userspace will be able to safely
enable interrupts. It will also allow the upcoming fsgsbase code to
be simplified, because it doesn't need to worry about usergs when
scheduling in paranoid_exit, as that code no longer exists.
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Andi Kleen <andi@firstfloor.org>
Signed-off-by: Andy Lutomirski <luto@amacapital.net>
---
Documentation/x86/entry_64.txt | 18 +++++---
Documentation/x86/x86_64/kernel-stacks | 8 ++--
arch/x86/kernel/entry_64.S | 82 +++++++++++++++++-----------------
arch/x86/kernel/traps.c | 23 +++-------
4 files changed, 64 insertions(+), 67 deletions(-)
diff --git a/Documentation/x86/entry_64.txt b/Documentation/x86/entry_64.txt
index bc7226ef5055..8e53217e6d40 100644
--- a/Documentation/x86/entry_64.txt
+++ b/Documentation/x86/entry_64.txt
@@ -75,9 +75,6 @@ The expensive (paranoid) way is to read back the MSR_GS_BASE value
xorl %ebx,%ebx
1: ret
-and the whole paranoid non-paranoid macro complexity is about whether
-to suffer that RDMSR cost.
-
If we are at an interrupt or user-trap/gate-alike boundary then we can
use the faster check: the stack will be a reliable indicator of
whether SWAPGS was already done: if we see that we are a secondary
@@ -90,6 +87,15 @@ which might have triggered right after a normal entry wrote CS to the
stack but before we executed SWAPGS, then the only safe way to check
for GS is the slower method: the RDMSR.
-So we try only to mark those entry methods 'paranoid' that absolutely
-need the more expensive check for the GS base - and we generate all
-'normal' entry points with the regular (faster) entry macros.
+Therefore, super-atomic entries (except NMI, which is handled separately)
+must use idtentry with paranoid=1 to handle gsbase correctly. This
+triggers three main behavior changes:
+
+ - Interrupt entry will use the slower gsbase check.
+ - Interrupt entry from user mode will switch off the IST stack.
+ - Interrupt exit to kernel mode will not attempt to reschedule.
+
+We try to only use IST entries and the paranoid entry code for vectors
+that absolutely need the more expensive check for the GS base - and we
+generate all 'normal' entry points with the regular (faster) paranoid=0
+variant.
diff --git a/Documentation/x86/x86_64/kernel-stacks b/Documentation/x86/x86_64/kernel-stacks
index a01eec5d1d0b..e3c8a49d1a2f 100644
--- a/Documentation/x86/x86_64/kernel-stacks
+++ b/Documentation/x86/x86_64/kernel-stacks
@@ -40,9 +40,11 @@ An IST is selected by a non-zero value in the IST field of an
interrupt-gate descriptor. When an interrupt occurs and the hardware
loads such a descriptor, the hardware automatically sets the new stack
pointer based on the IST value, then invokes the interrupt handler. If
-software wants to allow nested IST interrupts then the handler must
-adjust the IST values on entry to and exit from the interrupt handler.
-(This is occasionally done, e.g. for debug exceptions.)
+the interrupt came from user mode, then the interrupt handler prologue
+will switch back to the per-thread stack. If software wants to allow
+nested IST interrupts then the handler must adjust the IST values on
+entry to and exit from the interrupt handler. (This is occasionally
+done, e.g. for debug exceptions.)
Events with different IST codes (i.e. with different stacks) can be
nested. For example, a debug interrupt can safely be interrupted by an
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index df088bb03fb3..77005c86c514 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -1064,6 +1064,9 @@ ENTRY(\sym)
CFI_ADJUST_CFA_OFFSET ORIG_RAX-R15
.if \paranoid
+ CFI_REMEMBER_STATE
+ testl $3, CS(%rsp) /* If coming from userspace, switch */
+ jnz 1f /* stacks. */
call save_paranoid
.else
call error_entry
@@ -1104,6 +1107,36 @@ ENTRY(\sym)
jmp error_exit /* %ebx: no swapgs flag */
.endif
+ .if \paranoid
+ CFI_RESTORE_STATE
+ /*
+ * Paranoid entry from userspace. Switch stacks and treat it
+ * as a normal entry. This means that paranoid handlers
+ * run in real process context if user_mode(regs).
+ */
+1:
+ call error_entry
+
+ DEFAULT_FRAME 0
+
+ movq %rsp,%rdi /* pt_regs pointer */
+ call sync_regs
+ movq %rax,%rsp /* switch stack */
+
+ movq %rsp,%rdi /* pt_regs pointer */
+
+ .if \has_error_code
+ movq ORIG_RAX(%rsp),%rsi /* get error code */
+ movq $-1,ORIG_RAX(%rsp) /* no syscall to restart */
+ .else
+ xorl %esi,%esi /* no error code */
+ .endif
+
+ call \do_sym
+
+ jmp error_exit /* %ebx: no swapgs flag */
+ .endif
+
CFI_ENDPROC
END(\sym)
.endm
@@ -1305,16 +1338,14 @@ idtentry machine_check has_error_code=0 paranoid=1 do_sym=*machine_check_vector(
#endif
/*
- * "Paranoid" exit path from exception stack.
- * Paranoid because this is used by NMIs and cannot take
- * any kernel state for granted.
- * We don't do kernel preemption checks here, because only
- * NMI should be common and it does not enable IRQs and
- * cannot get reschedule ticks.
+ * "Paranoid" exit path from exception stack. This is invoked
+ * only on return from non-NMI IST interrupts that came
+ * from kernel space.
*
- * "trace" is 0 for the NMI handler only, because irq-tracing
- * is fundamentally NMI-unsafe. (we cannot change the soft and
- * hard flags at once, atomically)
+ * We may be returning to very strange contexts (e.g. very early
+ * in syscall entry), so checking for preemption here would
+ * be complicated. Fortunately, we there's no good reason
+ * to try to handle preemption here.
*/
/* ebx: no swapgs flag */
@@ -1324,43 +1355,14 @@ ENTRY(paranoid_exit)
TRACE_IRQS_OFF_DEBUG
testl %ebx,%ebx /* swapgs needed? */
jnz paranoid_restore
- testl $3,CS(%rsp)
- jnz paranoid_userspace
-paranoid_swapgs:
TRACE_IRQS_IRETQ 0
SWAPGS_UNSAFE_STACK
RESTORE_ALL 8
- jmp irq_return
+ INTERRUPT_RETURN
paranoid_restore:
TRACE_IRQS_IRETQ_DEBUG 0
RESTORE_ALL 8
- jmp irq_return
-paranoid_userspace:
- GET_THREAD_INFO(%rcx)
- movl TI_flags(%rcx),%ebx
- andl $_TIF_WORK_MASK,%ebx
- jz paranoid_swapgs
- movq %rsp,%rdi /* &pt_regs */
- call sync_regs
- movq %rax,%rsp /* switch stack for scheduling */
- testl $_TIF_NEED_RESCHED,%ebx
- jnz paranoid_schedule
- movl %ebx,%edx /* arg3: thread flags */
- TRACE_IRQS_ON
- ENABLE_INTERRUPTS(CLBR_NONE)
- xorl %esi,%esi /* arg2: oldset */
- movq %rsp,%rdi /* arg1: &pt_regs */
- call do_notify_resume
- DISABLE_INTERRUPTS(CLBR_NONE)
- TRACE_IRQS_OFF
- jmp paranoid_userspace
-paranoid_schedule:
- TRACE_IRQS_ON
- ENABLE_INTERRUPTS(CLBR_ANY)
- SCHEDULE_USER
- DISABLE_INTERRUPTS(CLBR_ANY)
- TRACE_IRQS_OFF
- jmp paranoid_userspace
+ INTERRUPT_RETURN
CFI_ENDPROC
END(paranoid_exit)
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 0d0e922fafc1..837843513ac3 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -375,27 +375,14 @@ NOKPROBE_SYMBOL(do_int3);
#ifdef CONFIG_X86_64
/*
- * Help handler running on IST stack to switch back to user stack
- * for scheduling or signal handling. The actual stack switch is done in
- * entry.S
+ * Help handler running on IST stack to switch off the IST stack if the
+ * interrupted code was in user mode. The actual stack switch is done in
+ * entry_64.S
*/
asmlinkage __visible struct pt_regs *sync_regs(struct pt_regs *eregs)
{
- struct pt_regs *regs = eregs;
- /* Did already sync */
- if (eregs == (struct pt_regs *)eregs->sp)
- ;
- /* Exception from user space */
- else if (user_mode(eregs))
- regs = task_pt_regs(current);
- /*
- * Exception from kernel and interrupts are enabled. Move to
- * kernel process stack.
- */
- else if (eregs->flags & X86_EFLAGS_IF)
- regs = (struct pt_regs *)(eregs->sp -= sizeof(struct pt_regs));
- if (eregs != regs)
- *regs = *eregs;
+ struct pt_regs *regs = task_pt_regs(current);
+ *regs = *eregs;
return regs;
}
NOKPROBE_SYMBOL(sync_regs);
--
1.9.3
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH v2 2/2] uprobes, x86: Fix _TIF_UPROBE vs _TIF_NOTIFY_RESUME
2014-11-13 22:31 [PATCH v2 0/2] Handle IST interrupts from userspace on the normal stack Andy Lutomirski
2014-11-13 22:31 ` [PATCH v2 1/2] x86, entry: Switch stacks on a paranoid entry from userspace Andy Lutomirski
@ 2014-11-13 22:31 ` Andy Lutomirski
2014-11-14 6:08 ` Srikar Dronamraju
2014-11-13 23:19 ` [PATCH v2 0/2] Handle IST interrupts from userspace on the normal stack Andy Lutomirski
2 siblings, 1 reply; 7+ messages in thread
From: Andy Lutomirski @ 2014-11-13 22:31 UTC (permalink / raw)
To: Borislav Petkov, x86
Cc: linux-kernel, Peter Zijlstra, Oleg Nesterov, Tony Luck,
Andi Kleen, Andy Lutomirski, Srikar Dronamraju
x86 call do_notify_resume on paranoid returns if TIF_UPROBE is set
but not on non-paranoid returns. I suspect that this is a mistake
and that the code only works because int3 is paranoid.
Setting _TIF_NOTIFY_RESUME in the uprobe code was probably a
workaround for the x86 bug. With that bug fixed, we can remove
_TIF_NOTIFY_RESUME from the uprobes code.
Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Reported-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Andy Lutomirski <luto@amacapital.net>
---
arch/x86/include/asm/thread_info.h | 2 +-
kernel/events/uprobes.c | 1 -
2 files changed, 1 insertion(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thread_info.h
index 854053889d4d..547e344a6dc6 100644
--- a/arch/x86/include/asm/thread_info.h
+++ b/arch/x86/include/asm/thread_info.h
@@ -141,7 +141,7 @@ struct thread_info {
/* Only used for 64 bit */
#define _TIF_DO_NOTIFY_MASK \
(_TIF_SIGPENDING | _TIF_MCE_NOTIFY | _TIF_NOTIFY_RESUME | \
- _TIF_USER_RETURN_NOTIFY)
+ _TIF_USER_RETURN_NOTIFY | _TIF_UPROBE)
/* flags to check in __switch_to() */
#define _TIF_WORK_CTXSW \
diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
index 1d0af8a2c646..ed8f2cde34c5 100644
--- a/kernel/events/uprobes.c
+++ b/kernel/events/uprobes.c
@@ -1640,7 +1640,6 @@ bool uprobe_deny_signal(void)
if (__fatal_signal_pending(t) || arch_uprobe_xol_was_trapped(t)) {
utask->state = UTASK_SSTEP_TRAPPED;
set_tsk_thread_flag(t, TIF_UPROBE);
- set_tsk_thread_flag(t, TIF_NOTIFY_RESUME);
}
}
--
1.9.3
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH v2 0/2] Handle IST interrupts from userspace on the normal stack
2014-11-13 22:31 [PATCH v2 0/2] Handle IST interrupts from userspace on the normal stack Andy Lutomirski
2014-11-13 22:31 ` [PATCH v2 1/2] x86, entry: Switch stacks on a paranoid entry from userspace Andy Lutomirski
2014-11-13 22:31 ` [PATCH v2 2/2] uprobes, x86: Fix _TIF_UPROBE vs _TIF_NOTIFY_RESUME Andy Lutomirski
@ 2014-11-13 23:19 ` Andy Lutomirski
2 siblings, 0 replies; 7+ messages in thread
From: Andy Lutomirski @ 2014-11-13 23:19 UTC (permalink / raw)
To: Borislav Petkov, X86 ML
Cc: linux-kernel@vger.kernel.org, Peter Zijlstra, Oleg Nesterov,
Tony Luck, Andi Kleen, Andy Lutomirski
On Thu, Nov 13, 2014 at 2:31 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> We currently run IST interrupt handlers on the IST stack. Changing
> it may simplify a few things. See patch 2 for details.
>
> Patch 1 is a fix for a not-quite-bug in uprobes that Oleg noticed
> that would be exposed by patch 2.
>
> Andy Lutomirski (2):
> x86, entry: Switch stacks on a paranoid entry from userspace
> uprobes, x86: Fix _TIF_UPROBE vs _TIF_NOTIFY_RESUME
>
I reversed the order of the patches -- oops. If I send v3, I'll fix that.
--Andy
> Documentation/x86/entry_64.txt | 18 +++++---
> Documentation/x86/x86_64/kernel-stacks | 8 ++--
> arch/x86/include/asm/thread_info.h | 2 +-
> arch/x86/kernel/entry_64.S | 82 +++++++++++++++++-----------------
> arch/x86/kernel/traps.c | 23 +++-------
> kernel/events/uprobes.c | 1 -
> 6 files changed, 65 insertions(+), 69 deletions(-)
>
> --
> 1.9.3
>
--
Andy Lutomirski
AMA Capital Management, LLC
^ permalink raw reply [flat|nested] 7+ messages in thread