* [ 01/10] i387: math_state_restore() isnt called from asm
2012-02-24 0:05 [ 00/10] 3.2.8-stable review Greg KH
@ 2012-02-24 0:02 ` Greg KH
2012-02-24 0:02 ` [ 02/10] i387: make irq_fpu_usable() tests more robust Greg KH
` (8 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Greg KH @ 2012-02-24 0:02 UTC (permalink / raw)
To: linux-kernel, stable; +Cc: torvalds, akpm, alan
3.2-stable review patch. If anyone has any objections, please let me know.
------------------
From: Linus Torvalds <torvalds@linux-foundation.org>
commit be98c2cdb15ba26148cd2bd58a857d4f7759ed38 upstream.
It was marked asmlinkage for some really old and stale legacy reasons.
Fix that and the equally stale comment.
Noticed when debugging the irq_fpu_usable() bugs.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/include/asm/i387.h | 2 +-
arch/x86/kernel/traps.c | 6 +++---
2 files changed, 4 insertions(+), 4 deletions(-)
--- a/arch/x86/include/asm/i387.h
+++ b/arch/x86/include/asm/i387.h
@@ -29,7 +29,7 @@ extern unsigned int sig_xstate_size;
extern void fpu_init(void);
extern void mxcsr_feature_mask_init(void);
extern int init_fpu(struct task_struct *child);
-extern asmlinkage void math_state_restore(void);
+extern void math_state_restore(void);
extern void __math_state_restore(void);
extern int dump_fpu(struct pt_regs *, struct user_i387_struct *);
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -590,10 +590,10 @@ void __math_state_restore(void)
* Careful.. There are problems with IBM-designed IRQ13 behaviour.
* Don't touch unless you *really* know how it works.
*
- * Must be called with kernel preemption disabled (in this case,
- * local interrupts are disabled at the call-site in entry.S).
+ * Must be called with kernel preemption disabled (eg with local
+ * local interrupts as in the case of do_device_not_available).
*/
-asmlinkage void math_state_restore(void)
+void math_state_restore(void)
{
struct thread_info *thread = current_thread_info();
struct task_struct *tsk = thread->task;
^ permalink raw reply [flat|nested] 11+ messages in thread* [ 02/10] i387: make irq_fpu_usable() tests more robust
2012-02-24 0:05 [ 00/10] 3.2.8-stable review Greg KH
2012-02-24 0:02 ` [ 01/10] i387: math_state_restore() isnt called from asm Greg KH
@ 2012-02-24 0:02 ` Greg KH
2012-02-24 0:02 ` [ 03/10] i387: fix sense of sanity check Greg KH
` (7 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Greg KH @ 2012-02-24 0:02 UTC (permalink / raw)
To: linux-kernel, stable; +Cc: torvalds, akpm, alan
3.2-stable review patch. If anyone has any objections, please let me know.
------------------
From: Linus Torvalds <torvalds@linux-foundation.org>
commit 5b1cbac37798805c1fee18c8cebe5c0a13975b17 upstream.
Some code - especially the crypto layer - wants to use the x86
FP/MMX/AVX register set in what may be interrupt (typically softirq)
context.
That *can* be ok, but the tests for when it was ok were somewhat
suspect. We cannot touch the thread-specific status bits either, so
we'd better check that we're not going to try to save FP state or
anything like that.
Now, it may be that the TS bit is always cleared *before* we set the
USEDFPU bit (and only set when we had already cleared the USEDFP
before), so the TS bit test may actually have been sufficient, but it
certainly was not obviously so.
So this explicitly verifies that we will not touch the TS_USEDFPU bit,
and adds a few related sanity-checks. Because it seems that somehow
AES-NI is corrupting user FP state. The cause is not clear, and this
patch doesn't fix it, but while debugging it I really wanted the code to
be more obviously correct and robust.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/include/asm/i387.h | 54 +++++++++++++++++++++++++++++++++++++-------
arch/x86/kernel/traps.c | 1
2 files changed, 47 insertions(+), 8 deletions(-)
--- a/arch/x86/include/asm/i387.h
+++ b/arch/x86/include/asm/i387.h
@@ -307,9 +307,54 @@ static inline void __clear_fpu(struct ta
}
}
+/*
+ * Were we in an interrupt that interrupted kernel mode?
+ *
+ * We can do a kernel_fpu_begin/end() pair *ONLY* if that
+ * pair does nothing at all: TS_USEDFPU must be clear (so
+ * that we don't try to save the FPU state), and TS must
+ * be set (so that the clts/stts pair does nothing that is
+ * visible in the interrupted kernel thread).
+ */
+static inline bool interrupted_kernel_fpu_idle(void)
+{
+ return !(current_thread_info()->status & TS_USEDFPU) &&
+ (read_cr0() & X86_CR0_TS);
+}
+
+/*
+ * Were we in user mode (or vm86 mode) when we were
+ * interrupted?
+ *
+ * Doing kernel_fpu_begin/end() is ok if we are running
+ * in an interrupt context from user mode - we'll just
+ * save the FPU state as required.
+ */
+static inline bool interrupted_user_mode(void)
+{
+ struct pt_regs *regs = get_irq_regs();
+ return regs && user_mode_vm(regs);
+}
+
+/*
+ * Can we use the FPU in kernel mode with the
+ * whole "kernel_fpu_begin/end()" sequence?
+ *
+ * It's always ok in process context (ie "not interrupt")
+ * but it is sometimes ok even from an irq.
+ */
+static inline bool irq_fpu_usable(void)
+{
+ return !in_interrupt() ||
+ interrupted_user_mode() ||
+ interrupted_kernel_fpu_idle();
+}
+
static inline void kernel_fpu_begin(void)
{
struct thread_info *me = current_thread_info();
+
+ WARN_ON_ONCE(!irq_fpu_usable());
preempt_disable();
if (me->status & TS_USEDFPU)
__save_init_fpu(me->task);
@@ -323,14 +368,6 @@ static inline void kernel_fpu_end(void)
preempt_enable();
}
-static inline bool irq_fpu_usable(void)
-{
- struct pt_regs *regs;
-
- return !in_interrupt() || !(regs = get_irq_regs()) || \
- user_mode(regs) || (read_cr0() & X86_CR0_TS);
-}
-
/*
* Some instructions like VIA's padlock instructions generate a spurious
* DNA fault but don't modify SSE registers. And these instructions
@@ -367,6 +404,7 @@ static inline void irq_ts_restore(int TS
*/
static inline void save_init_fpu(struct task_struct *tsk)
{
+ WARN_ON_ONCE(task_thread_info(tsk)->status & TS_USEDFPU);
preempt_disable();
__save_init_fpu(tsk);
stts();
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -622,6 +622,7 @@ EXPORT_SYMBOL_GPL(math_state_restore);
dotraplinkage void __kprobes
do_device_not_available(struct pt_regs *regs, long error_code)
{
+ WARN_ON_ONCE(!user_mode_vm(regs));
#ifdef CONFIG_MATH_EMULATION
if (read_cr0() & X86_CR0_EM) {
struct math_emu_info info = { };
^ permalink raw reply [flat|nested] 11+ messages in thread* [ 03/10] i387: fix sense of sanity check
2012-02-24 0:05 [ 00/10] 3.2.8-stable review Greg KH
2012-02-24 0:02 ` [ 01/10] i387: math_state_restore() isnt called from asm Greg KH
2012-02-24 0:02 ` [ 02/10] i387: make irq_fpu_usable() tests more robust Greg KH
@ 2012-02-24 0:02 ` Greg KH
2012-02-24 0:02 ` [ 04/10] i387: fix x86-64 preemption-unsafe user stack save/restore Greg KH
` (6 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Greg KH @ 2012-02-24 0:02 UTC (permalink / raw)
To: linux-kernel, stable; +Cc: torvalds, akpm, alan
3.2-stable review patch. If anyone has any objections, please let me know.
------------------
From: Linus Torvalds <torvalds@linux-foundation.org>
commit c38e23456278e967f094b08247ffc3711b1029b2 upstream.
The check for save_init_fpu() (introduced in commit 5b1cbac37798: "i387:
make irq_fpu_usable() tests more robust") was the wrong way around, but
I hadn't noticed, because my "tests" were bogus: the FPU exceptions are
disabled by default, so even doing a divide by zero never actually
triggers this code at all unless you do extra work to enable them.
So if anybody did enable them, they'd get one spurious warning.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/include/asm/i387.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- a/arch/x86/include/asm/i387.h
+++ b/arch/x86/include/asm/i387.h
@@ -404,7 +404,7 @@ static inline void irq_ts_restore(int TS
*/
static inline void save_init_fpu(struct task_struct *tsk)
{
- WARN_ON_ONCE(task_thread_info(tsk)->status & TS_USEDFPU);
+ WARN_ON_ONCE(!(task_thread_info(tsk)->status & TS_USEDFPU));
preempt_disable();
__save_init_fpu(tsk);
stts();
^ permalink raw reply [flat|nested] 11+ messages in thread* [ 04/10] i387: fix x86-64 preemption-unsafe user stack save/restore
2012-02-24 0:05 [ 00/10] 3.2.8-stable review Greg KH
` (2 preceding siblings ...)
2012-02-24 0:02 ` [ 03/10] i387: fix sense of sanity check Greg KH
@ 2012-02-24 0:02 ` Greg KH
2012-02-24 0:02 ` [ 05/10] i387: move TS_USEDFPU clearing out of __save_init_fpu and into callers Greg KH
` (5 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Greg KH @ 2012-02-24 0:02 UTC (permalink / raw)
To: linux-kernel, stable; +Cc: torvalds, akpm, alan
3.2-stable review patch. If anyone has any objections, please let me know.
------------------
From: Linus Torvalds <torvalds@linux-foundation.org>
commit 15d8791cae75dca27bfda8ecfe87dca9379d6bb0 upstream.
Commit 5b1cbac37798 ("i387: make irq_fpu_usable() tests more robust")
added a sanity check to the #NM handler to verify that we never cause
the "Device Not Available" exception in kernel mode.
However, that check actually pinpointed a (fundamental) race where we do
cause that exception as part of the signal stack FPU state save/restore
code.
Because we use the floating point instructions themselves to save and
restore state directly from user mode, we cannot do that atomically with
testing the TS_USEDFPU bit: the user mode access itself may cause a page
fault, which causes a task switch, which saves and restores the FP/MMX
state from the kernel buffers.
This kind of "recursive" FP state save is fine per se, but it means that
when the signal stack save/restore gets restarted, it will now take the
'#NM' exception we originally tried to avoid. With preemption this can
happen even without the page fault - but because of the user access, we
cannot just disable preemption around the save/restore instruction.
There are various ways to solve this, including using the
"enable/disable_page_fault()" helpers to not allow page faults at all
during the sequence, and fall back to copying things by hand without the
use of the native FP state save/restore instructions.
However, the simplest thing to do is to just allow the #NM from kernel
space, but fix the race in setting and clearing CR0.TS that this all
exposed: the TS bit changes and the TS_USEDFPU bit absolutely have to be
atomic wrt scheduling, so while the actual state save/restore can be
interrupted and restarted, the act of actually clearing/setting CR0.TS
and the TS_USEDFPU bit together must not.
Instead of just adding random "preempt_disable/enable()" calls to what
is already excessively ugly code, this introduces some helper functions
that mostly mirror the "kernel_fpu_begin/end()" functionality, just for
the user state instead.
Those helper functions should probably eventually replace the other
ad-hoc CR0.TS and TS_USEDFPU tests too, but I'll need to think about it
some more: the task switching functionality in particular needs to
expose the difference between the 'prev' and 'next' threads, while the
new helper functions intentionally were written to only work with
'current'.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/include/asm/i387.h | 42 ++++++++++++++++++++++++++++++++++++++++++
arch/x86/kernel/traps.c | 1 -
arch/x86/kernel/xsave.c | 10 +++-------
3 files changed, 45 insertions(+), 8 deletions(-)
--- a/arch/x86/include/asm/i387.h
+++ b/arch/x86/include/asm/i387.h
@@ -400,6 +400,48 @@ static inline void irq_ts_restore(int TS
}
/*
+ * The question "does this thread have fpu access?"
+ * is slightly racy, since preemption could come in
+ * and revoke it immediately after the test.
+ *
+ * However, even in that very unlikely scenario,
+ * we can just assume we have FPU access - typically
+ * to save the FP state - we'll just take a #NM
+ * fault and get the FPU access back.
+ *
+ * The actual user_fpu_begin/end() functions
+ * need to be preemption-safe, though.
+ *
+ * NOTE! user_fpu_end() must be used only after you
+ * have saved the FP state, and user_fpu_begin() must
+ * be used only immediately before restoring it.
+ * These functions do not do any save/restore on
+ * their own.
+ */
+static inline int user_has_fpu(void)
+{
+ return current_thread_info()->status & TS_USEDFPU;
+}
+
+static inline void user_fpu_end(void)
+{
+ preempt_disable();
+ current_thread_info()->status &= ~TS_USEDFPU;
+ stts();
+ preempt_enable();
+}
+
+static inline void user_fpu_begin(void)
+{
+ preempt_disable();
+ if (!user_has_fpu()) {
+ clts();
+ current_thread_info()->status |= TS_USEDFPU;
+ }
+ preempt_enable();
+}
+
+/*
* These disable preemption on their own and are safe
*/
static inline void save_init_fpu(struct task_struct *tsk)
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -622,7 +622,6 @@ EXPORT_SYMBOL_GPL(math_state_restore);
dotraplinkage void __kprobes
do_device_not_available(struct pt_regs *regs, long error_code)
{
- WARN_ON_ONCE(!user_mode_vm(regs));
#ifdef CONFIG_MATH_EMULATION
if (read_cr0() & X86_CR0_EM) {
struct math_emu_info info = { };
--- a/arch/x86/kernel/xsave.c
+++ b/arch/x86/kernel/xsave.c
@@ -168,7 +168,7 @@ int save_i387_xstate(void __user *buf)
if (!used_math())
return 0;
- if (task_thread_info(tsk)->status & TS_USEDFPU) {
+ if (user_has_fpu()) {
if (use_xsave())
err = xsave_user(buf);
else
@@ -176,8 +176,7 @@ int save_i387_xstate(void __user *buf)
if (err)
return err;
- task_thread_info(tsk)->status &= ~TS_USEDFPU;
- stts();
+ user_fpu_end();
} else {
sanitize_i387_state(tsk);
if (__copy_to_user(buf, &tsk->thread.fpu.state->fxsave,
@@ -292,10 +291,7 @@ int restore_i387_xstate(void __user *buf
return err;
}
- if (!(task_thread_info(current)->status & TS_USEDFPU)) {
- clts();
- task_thread_info(current)->status |= TS_USEDFPU;
- }
+ user_fpu_begin();
if (use_xsave())
err = restore_user_xstate(buf);
else
^ permalink raw reply [flat|nested] 11+ messages in thread* [ 05/10] i387: move TS_USEDFPU clearing out of __save_init_fpu and into callers
2012-02-24 0:05 [ 00/10] 3.2.8-stable review Greg KH
` (3 preceding siblings ...)
2012-02-24 0:02 ` [ 04/10] i387: fix x86-64 preemption-unsafe user stack save/restore Greg KH
@ 2012-02-24 0:02 ` Greg KH
2012-02-24 0:02 ` [ 06/10] i387: dont ever touch TS_USEDFPU directly, use helper functions Greg KH
` (4 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Greg KH @ 2012-02-24 0:02 UTC (permalink / raw)
To: linux-kernel, stable; +Cc: torvalds, akpm, alan
3.2-stable review patch. If anyone has any objections, please let me know.
------------------
From: Linus Torvalds <torvalds@linux-foundation.org>
commit b6c66418dcad0fcf83cd1d0a39482db37bf4fc41 upstream.
Touching TS_USEDFPU without touching CR0.TS is confusing, so don't do
it. By moving it into the callers, we always do the TS_USEDFPU next to
the CR0.TS accesses in the source code, and it's much easier to see how
the two go hand in hand.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/include/asm/i387.h | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
--- a/arch/x86/include/asm/i387.h
+++ b/arch/x86/include/asm/i387.h
@@ -259,7 +259,6 @@ static inline void fpu_save_init(struct
static inline void __save_init_fpu(struct task_struct *tsk)
{
fpu_save_init(&tsk->thread.fpu);
- task_thread_info(tsk)->status &= ~TS_USEDFPU;
}
static inline int fpu_fxrstor_checking(struct fpu *fpu)
@@ -290,6 +289,7 @@ static inline void __unlazy_fpu(struct t
{
if (task_thread_info(tsk)->status & TS_USEDFPU) {
__save_init_fpu(tsk);
+ task_thread_info(tsk)->status &= ~TS_USEDFPU;
stts();
} else
tsk->fpu_counter = 0;
@@ -356,9 +356,11 @@ static inline void kernel_fpu_begin(void
WARN_ON_ONCE(!irq_fpu_usable());
preempt_disable();
- if (me->status & TS_USEDFPU)
+ if (me->status & TS_USEDFPU) {
__save_init_fpu(me->task);
- else
+ me->status &= ~TS_USEDFPU;
+ /* We do 'stts()' in kernel_fpu_end() */
+ } else
clts();
}
@@ -449,6 +451,7 @@ static inline void save_init_fpu(struct
WARN_ON_ONCE(!(task_thread_info(tsk)->status & TS_USEDFPU));
preempt_disable();
__save_init_fpu(tsk);
+ task_thread_info(tsk)->status &= ~TS_USEDFPU;
stts();
preempt_enable();
}
^ permalink raw reply [flat|nested] 11+ messages in thread* [ 06/10] i387: dont ever touch TS_USEDFPU directly, use helper functions
2012-02-24 0:05 [ 00/10] 3.2.8-stable review Greg KH
` (4 preceding siblings ...)
2012-02-24 0:02 ` [ 05/10] i387: move TS_USEDFPU clearing out of __save_init_fpu and into callers Greg KH
@ 2012-02-24 0:02 ` Greg KH
2012-02-24 0:02 ` [ 07/10] i387: do not preload FPU state at task switch time Greg KH
` (3 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Greg KH @ 2012-02-24 0:02 UTC (permalink / raw)
To: linux-kernel, stable; +Cc: torvalds, akpm, alan
3.2-stable review patch. If anyone has any objections, please let me know.
------------------
From: Linus Torvalds <torvalds@linux-foundation.org>
commit 6d59d7a9f5b723a7ac1925c136e93ec83c0c3043 upstream.
This creates three helper functions that do the TS_USEDFPU accesses, and
makes everybody that used to do it by hand use those helpers instead.
In addition, there's a couple of helper functions for the "change both
CR0.TS and TS_USEDFPU at the same time" case, and the places that do
that together have been changed to use those. That means that we have
fewer random places that open-code this situation.
The intent is partly to clarify the code without actually changing any
semantics yet (since we clearly still have some hard to reproduce bug in
this area), but also to make it much easier to use another approach
entirely to caching the CR0.TS bit for software accesses.
Right now we use a bit in the thread-info 'status' variable (this patch
does not change that), but we might want to make it a full field of its
own or even make it a per-cpu variable.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/include/asm/i387.h | 75 ++++++++++++++++++++++++++++++++------------
arch/x86/kernel/traps.c | 2 -
arch/x86/kernel/xsave.c | 2 -
arch/x86/kvm/vmx.c | 2 -
4 files changed, 58 insertions(+), 23 deletions(-)
--- a/arch/x86/include/asm/i387.h
+++ b/arch/x86/include/asm/i387.h
@@ -280,6 +280,47 @@ static inline int restore_fpu_checking(s
}
/*
+ * Software FPU state helpers. Careful: these need to
+ * be preemption protection *and* they need to be
+ * properly paired with the CR0.TS changes!
+ */
+static inline int __thread_has_fpu(struct thread_info *ti)
+{
+ return ti->status & TS_USEDFPU;
+}
+
+/* Must be paired with an 'stts' after! */
+static inline void __thread_clear_has_fpu(struct thread_info *ti)
+{
+ ti->status &= ~TS_USEDFPU;
+}
+
+/* Must be paired with a 'clts' before! */
+static inline void __thread_set_has_fpu(struct thread_info *ti)
+{
+ ti->status |= TS_USEDFPU;
+}
+
+/*
+ * Encapsulate the CR0.TS handling together with the
+ * software flag.
+ *
+ * These generally need preemption protection to work,
+ * do try to avoid using these on their own.
+ */
+static inline void __thread_fpu_end(struct thread_info *ti)
+{
+ __thread_clear_has_fpu(ti);
+ stts();
+}
+
+static inline void __thread_fpu_begin(struct thread_info *ti)
+{
+ clts();
+ __thread_set_has_fpu(ti);
+}
+
+/*
* Signal frame handlers...
*/
extern int save_i387_xstate(void __user *buf);
@@ -287,23 +328,21 @@ extern int restore_i387_xstate(void __us
static inline void __unlazy_fpu(struct task_struct *tsk)
{
- if (task_thread_info(tsk)->status & TS_USEDFPU) {
+ if (__thread_has_fpu(task_thread_info(tsk))) {
__save_init_fpu(tsk);
- task_thread_info(tsk)->status &= ~TS_USEDFPU;
- stts();
+ __thread_fpu_end(task_thread_info(tsk));
} else
tsk->fpu_counter = 0;
}
static inline void __clear_fpu(struct task_struct *tsk)
{
- if (task_thread_info(tsk)->status & TS_USEDFPU) {
+ if (__thread_has_fpu(task_thread_info(tsk))) {
/* Ignore delayed exceptions from user space */
asm volatile("1: fwait\n"
"2:\n"
_ASM_EXTABLE(1b, 2b));
- task_thread_info(tsk)->status &= ~TS_USEDFPU;
- stts();
+ __thread_fpu_end(task_thread_info(tsk));
}
}
@@ -311,14 +350,14 @@ static inline void __clear_fpu(struct ta
* Were we in an interrupt that interrupted kernel mode?
*
* We can do a kernel_fpu_begin/end() pair *ONLY* if that
- * pair does nothing at all: TS_USEDFPU must be clear (so
+ * pair does nothing at all: the thread must not have fpu (so
* that we don't try to save the FPU state), and TS must
* be set (so that the clts/stts pair does nothing that is
* visible in the interrupted kernel thread).
*/
static inline bool interrupted_kernel_fpu_idle(void)
{
- return !(current_thread_info()->status & TS_USEDFPU) &&
+ return !__thread_has_fpu(current_thread_info()) &&
(read_cr0() & X86_CR0_TS);
}
@@ -356,9 +395,9 @@ static inline void kernel_fpu_begin(void
WARN_ON_ONCE(!irq_fpu_usable());
preempt_disable();
- if (me->status & TS_USEDFPU) {
+ if (__thread_has_fpu(me)) {
__save_init_fpu(me->task);
- me->status &= ~TS_USEDFPU;
+ __thread_clear_has_fpu(me);
/* We do 'stts()' in kernel_fpu_end() */
} else
clts();
@@ -422,24 +461,21 @@ static inline void irq_ts_restore(int TS
*/
static inline int user_has_fpu(void)
{
- return current_thread_info()->status & TS_USEDFPU;
+ return __thread_has_fpu(current_thread_info());
}
static inline void user_fpu_end(void)
{
preempt_disable();
- current_thread_info()->status &= ~TS_USEDFPU;
- stts();
+ __thread_fpu_end(current_thread_info());
preempt_enable();
}
static inline void user_fpu_begin(void)
{
preempt_disable();
- if (!user_has_fpu()) {
- clts();
- current_thread_info()->status |= TS_USEDFPU;
- }
+ if (!user_has_fpu())
+ __thread_fpu_begin(current_thread_info());
preempt_enable();
}
@@ -448,11 +484,10 @@ static inline void user_fpu_begin(void)
*/
static inline void save_init_fpu(struct task_struct *tsk)
{
- WARN_ON_ONCE(!(task_thread_info(tsk)->status & TS_USEDFPU));
+ WARN_ON_ONCE(!__thread_has_fpu(task_thread_info(tsk)));
preempt_disable();
__save_init_fpu(tsk);
- task_thread_info(tsk)->status &= ~TS_USEDFPU;
- stts();
+ __thread_fpu_end(task_thread_info(tsk));
preempt_enable();
}
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -579,7 +579,7 @@ void __math_state_restore(void)
return;
}
- thread->status |= TS_USEDFPU; /* So we fnsave on switch_to() */
+ __thread_set_has_fpu(thread); /* clts in caller! */
tsk->fpu_counter++;
}
--- a/arch/x86/kernel/xsave.c
+++ b/arch/x86/kernel/xsave.c
@@ -47,7 +47,7 @@ void __sanitize_i387_state(struct task_s
if (!fx)
return;
- BUG_ON(task_thread_info(tsk)->status & TS_USEDFPU);
+ BUG_ON(__thread_has_fpu(task_thread_info(tsk)));
xstate_bv = tsk->thread.fpu.state->xsave.xsave_hdr.xstate_bv;
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -1456,7 +1456,7 @@ static void __vmx_load_host_state(struct
#ifdef CONFIG_X86_64
wrmsrl(MSR_KERNEL_GS_BASE, vmx->msr_host_kernel_gs_base);
#endif
- if (current_thread_info()->status & TS_USEDFPU)
+ if (__thread_has_fpu(current_thread_info()))
clts();
load_gdt(&__get_cpu_var(host_gdt));
}
^ permalink raw reply [flat|nested] 11+ messages in thread* [ 07/10] i387: do not preload FPU state at task switch time
2012-02-24 0:05 [ 00/10] 3.2.8-stable review Greg KH
` (5 preceding siblings ...)
2012-02-24 0:02 ` [ 06/10] i387: dont ever touch TS_USEDFPU directly, use helper functions Greg KH
@ 2012-02-24 0:02 ` Greg KH
2012-02-24 0:02 ` [ 08/10] i387: move AMD K7/K8 fpu fxsave/fxrstor workaround from save to restore Greg KH
` (2 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Greg KH @ 2012-02-24 0:02 UTC (permalink / raw)
To: linux-kernel, stable; +Cc: torvalds, akpm, alan
3.2-stable review patch. If anyone has any objections, please let me know.
------------------
From: Linus Torvalds <torvalds@linux-foundation.org>
commit b3b0870ef3ffed72b92415423da864f440f57ad6 upstream.
Yes, taking the trap to re-load the FPU/MMX state is expensive, but so
is spending several days looking for a bug in the state save/restore
code. And the preload code has some rather subtle interactions with
both paravirtualization support and segment state restore, so it's not
nearly as simple as it should be.
Also, now that we no longer necessarily depend on a single bit (ie
TS_USEDFPU) for keeping track of the state of the FPU, we migth be able
to do better. If we are really switching between two processes that
keep touching the FP state, save/restore is inevitable, but in the case
of having one process that does most of the FPU usage, we may actually
be able to do much better than the preloading.
In particular, we may be able to keep track of which CPU the process ran
on last, and also per CPU keep track of which process' FP state that CPU
has. For modern CPU's that don't destroy the FPU contents on save time,
that would allow us to do a lazy restore by just re-enabling the
existing FPU state - with no restore cost at all!
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/include/asm/i387.h | 1 -
arch/x86/kernel/process_32.c | 20 --------------------
arch/x86/kernel/process_64.c | 23 -----------------------
arch/x86/kernel/traps.c | 35 +++++++++++------------------------
4 files changed, 11 insertions(+), 68 deletions(-)
--- a/arch/x86/include/asm/i387.h
+++ b/arch/x86/include/asm/i387.h
@@ -30,7 +30,6 @@ extern void fpu_init(void);
extern void mxcsr_feature_mask_init(void);
extern int init_fpu(struct task_struct *child);
extern void math_state_restore(void);
-extern void __math_state_restore(void);
extern int dump_fpu(struct pt_regs *, struct user_i387_struct *);
extern user_regset_active_fn fpregs_active, xfpregs_active;
--- a/arch/x86/kernel/process_32.c
+++ b/arch/x86/kernel/process_32.c
@@ -297,23 +297,11 @@ __switch_to(struct task_struct *prev_p,
*next = &next_p->thread;
int cpu = smp_processor_id();
struct tss_struct *tss = &per_cpu(init_tss, cpu);
- bool preload_fpu;
/* never put a printk in __switch_to... printk() calls wake_up*() indirectly */
- /*
- * If the task has used fpu the last 5 timeslices, just do a full
- * restore of the math state immediately to avoid the trap; the
- * chances of needing FPU soon are obviously high now
- */
- preload_fpu = tsk_used_math(next_p) && next_p->fpu_counter > 5;
-
__unlazy_fpu(prev_p);
- /* we're going to use this soon, after a few expensive things */
- if (preload_fpu)
- prefetch(next->fpu.state);
-
/*
* Reload esp0.
*/
@@ -352,11 +340,6 @@ __switch_to(struct task_struct *prev_p,
task_thread_info(next_p)->flags & _TIF_WORK_CTXSW_NEXT))
__switch_to_xtra(prev_p, next_p, tss);
- /* If we're going to preload the fpu context, make sure clts
- is run while we're batching the cpu state updates. */
- if (preload_fpu)
- clts();
-
/*
* Leave lazy mode, flushing any hypercalls made here.
* This must be done before restoring TLS segments so
@@ -366,9 +349,6 @@ __switch_to(struct task_struct *prev_p,
*/
arch_end_context_switch(next_p);
- if (preload_fpu)
- __math_state_restore();
-
/*
* Restore %gs if needed (which is common)
*/
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -381,18 +381,6 @@ __switch_to(struct task_struct *prev_p,
int cpu = smp_processor_id();
struct tss_struct *tss = &per_cpu(init_tss, cpu);
unsigned fsindex, gsindex;
- bool preload_fpu;
-
- /*
- * If the task has used fpu the last 5 timeslices, just do a full
- * restore of the math state immediately to avoid the trap; the
- * chances of needing FPU soon are obviously high now
- */
- preload_fpu = tsk_used_math(next_p) && next_p->fpu_counter > 5;
-
- /* we're going to use this soon, after a few expensive things */
- if (preload_fpu)
- prefetch(next->fpu.state);
/*
* Reload esp0, LDT and the page table pointer:
@@ -425,10 +413,6 @@ __switch_to(struct task_struct *prev_p,
/* Must be after DS reload */
__unlazy_fpu(prev_p);
- /* Make sure cpu is ready for new context */
- if (preload_fpu)
- clts();
-
/*
* Leave lazy mode, flushing any hypercalls made here.
* This must be done before restoring TLS segments so
@@ -487,13 +471,6 @@ __switch_to(struct task_struct *prev_p,
task_thread_info(prev_p)->flags & _TIF_WORK_CTXSW_PREV))
__switch_to_xtra(prev_p, next_p, tss);
- /*
- * Preload the FPU context, now that we've determined that the
- * task is likely to be using it.
- */
- if (preload_fpu)
- __math_state_restore();
-
return prev_p;
}
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -562,28 +562,6 @@ asmlinkage void __attribute__((weak)) sm
}
/*
- * __math_state_restore assumes that cr0.TS is already clear and the
- * fpu state is all ready for use. Used during context switch.
- */
-void __math_state_restore(void)
-{
- struct thread_info *thread = current_thread_info();
- struct task_struct *tsk = thread->task;
-
- /*
- * Paranoid restore. send a SIGSEGV if we fail to restore the state.
- */
- if (unlikely(restore_fpu_checking(tsk))) {
- stts();
- force_sig(SIGSEGV, tsk);
- return;
- }
-
- __thread_set_has_fpu(thread); /* clts in caller! */
- tsk->fpu_counter++;
-}
-
-/*
* 'math_state_restore()' saves the current math information in the
* old math state array, and gets the new ones from the current task
*
@@ -613,9 +591,18 @@ void math_state_restore(void)
local_irq_disable();
}
- clts(); /* Allow maths ops (or we recurse) */
+ __thread_fpu_begin(thread);
- __math_state_restore();
+ /*
+ * Paranoid restore. send a SIGSEGV if we fail to restore the state.
+ */
+ if (unlikely(restore_fpu_checking(tsk))) {
+ __thread_fpu_end(thread);
+ force_sig(SIGSEGV, tsk);
+ return;
+ }
+
+ tsk->fpu_counter++;
}
EXPORT_SYMBOL_GPL(math_state_restore);
^ permalink raw reply [flat|nested] 11+ messages in thread* [ 08/10] i387: move AMD K7/K8 fpu fxsave/fxrstor workaround from save to restore
2012-02-24 0:05 [ 00/10] 3.2.8-stable review Greg KH
` (6 preceding siblings ...)
2012-02-24 0:02 ` [ 07/10] i387: do not preload FPU state at task switch time Greg KH
@ 2012-02-24 0:02 ` Greg KH
2012-02-24 0:02 ` [ 09/10] i387: move TS_USEDFPU flag from thread_info to task_struct Greg KH
2012-02-24 0:02 ` [ 10/10] i387: re-introduce FPU state preloading at context switch time Greg KH
9 siblings, 0 replies; 11+ messages in thread
From: Greg KH @ 2012-02-24 0:02 UTC (permalink / raw)
To: linux-kernel, stable; +Cc: torvalds, akpm, alan
3.2-stable review patch. If anyone has any objections, please let me know.
------------------
From: Linus Torvalds <torvalds@linux-foundation.org>
commit 4903062b5485f0e2c286a23b44c9b59d9b017d53 upstream.
The AMD K7/K8 CPUs don't save/restore FDP/FIP/FOP unless an exception is
pending. In order to not leak FIP state from one process to another, we
need to do a floating point load after the fxsave of the old process,
and before the fxrstor of the new FPU state. That resets the state to
the (uninteresting) kernel load, rather than some potentially sensitive
user information.
We used to do this directly after the FPU state save, but that is
actually very inconvenient, since it
(a) corrupts what is potentially perfectly good FPU state that we might
want to lazy avoid restoring later and
(b) on x86-64 it resulted in a very annoying ordering constraint, where
"__unlazy_fpu()" in the task switch needs to be delayed until after
the DS segment has been reloaded just to get the new DS value.
Coupling it to the fxrstor instead of the fxsave automatically avoids
both of these issues, and also ensures that we only do it when actually
necessary (the FP state after a save may never actually get used). It's
simply a much more natural place for the leaked state cleanup.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/include/asm/i387.h | 19 -------------------
arch/x86/kernel/process_64.c | 5 ++---
arch/x86/kernel/traps.c | 14 ++++++++++++++
3 files changed, 16 insertions(+), 22 deletions(-)
--- a/arch/x86/include/asm/i387.h
+++ b/arch/x86/include/asm/i387.h
@@ -211,15 +211,6 @@ static inline void fpu_fxsave(struct fpu
#endif /* CONFIG_X86_64 */
-/* We need a safe address that is cheap to find and that is already
- in L1 during context switch. The best choices are unfortunately
- different for UP and SMP */
-#ifdef CONFIG_SMP
-#define safe_address (__per_cpu_offset[0])
-#else
-#define safe_address (kstat_cpu(0).cpustat.user)
-#endif
-
/*
* These must be called with preempt disabled
*/
@@ -243,16 +234,6 @@ static inline void fpu_save_init(struct
if (unlikely(fpu->state->fxsave.swd & X87_FSW_ES))
asm volatile("fnclex");
-
- /* AMD K7/K8 CPUs don't save/restore FDP/FIP/FOP unless an exception
- is pending. Clear the x87 state here by setting it to fixed
- values. safe_address is a random variable that should be in L1 */
- alternative_input(
- ASM_NOP8 ASM_NOP2,
- "emms\n\t" /* clear stack tags */
- "fildl %P[addr]", /* set F?P to defined value */
- X86_FEATURE_FXSAVE_LEAK,
- [addr] "m" (safe_address));
}
static inline void __save_init_fpu(struct task_struct *tsk)
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -382,6 +382,8 @@ __switch_to(struct task_struct *prev_p,
struct tss_struct *tss = &per_cpu(init_tss, cpu);
unsigned fsindex, gsindex;
+ __unlazy_fpu(prev_p);
+
/*
* Reload esp0, LDT and the page table pointer:
*/
@@ -410,9 +412,6 @@ __switch_to(struct task_struct *prev_p,
load_TLS(next, cpu);
- /* Must be after DS reload */
- __unlazy_fpu(prev_p);
-
/*
* Leave lazy mode, flushing any hypercalls made here.
* This must be done before restoring TLS segments so
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -576,6 +576,10 @@ void math_state_restore(void)
struct thread_info *thread = current_thread_info();
struct task_struct *tsk = thread->task;
+ /* We need a safe address that is cheap to find and that is already
+ in L1. We just brought in "thread->task", so use that */
+#define safe_address (thread->task)
+
if (!tsk_used_math(tsk)) {
local_irq_enable();
/*
@@ -593,6 +597,16 @@ void math_state_restore(void)
__thread_fpu_begin(thread);
+ /* AMD K7/K8 CPUs don't save/restore FDP/FIP/FOP unless an exception
+ is pending. Clear the x87 state here by setting it to fixed
+ values. safe_address is a random variable that should be in L1 */
+ alternative_input(
+ ASM_NOP8 ASM_NOP2,
+ "emms\n\t" /* clear stack tags */
+ "fildl %P[addr]", /* set F?P to defined value */
+ X86_FEATURE_FXSAVE_LEAK,
+ [addr] "m" (safe_address));
+
/*
* Paranoid restore. send a SIGSEGV if we fail to restore the state.
*/
^ permalink raw reply [flat|nested] 11+ messages in thread* [ 09/10] i387: move TS_USEDFPU flag from thread_info to task_struct
2012-02-24 0:05 [ 00/10] 3.2.8-stable review Greg KH
` (7 preceding siblings ...)
2012-02-24 0:02 ` [ 08/10] i387: move AMD K7/K8 fpu fxsave/fxrstor workaround from save to restore Greg KH
@ 2012-02-24 0:02 ` Greg KH
2012-02-24 0:02 ` [ 10/10] i387: re-introduce FPU state preloading at context switch time Greg KH
9 siblings, 0 replies; 11+ messages in thread
From: Greg KH @ 2012-02-24 0:02 UTC (permalink / raw)
To: linux-kernel, stable; +Cc: torvalds, akpm, alan, Arjan van de Ven, Peter Anvin
3.2-stable review patch. If anyone has any objections, please let me know.
------------------
From: Linus Torvalds <torvalds@linux-foundation.org>
commit f94edacf998516ac9d849f7bc6949a703977a7f3 upstream.
This moves the bit that indicates whether a thread has ownership of the
FPU from the TS_USEDFPU bit in thread_info->status to a word of its own
(called 'has_fpu') in task_struct->thread.has_fpu.
This fixes two independent bugs at the same time:
- changing 'thread_info->status' from the scheduler causes nasty
problems for the other users of that variable, since it is defined to
be thread-synchronous (that's what the "TS_" part of the naming was
supposed to indicate).
So perfectly valid code could (and did) do
ti->status |= TS_RESTORE_SIGMASK;
and the compiler was free to do that as separate load, or and store
instructions. Which can cause problems with preemption, since a task
switch could happen in between, and change the TS_USEDFPU bit. The
change to TS_USEDFPU would be overwritten by the final store.
In practice, this seldom happened, though, because the 'status' field
was seldom used more than once, so gcc would generally tend to
generate code that used a read-modify-write instruction and thus
happened to avoid this problem - RMW instructions are naturally low
fat and preemption-safe.
- On x86-32, the current_thread_info() pointer would, during interrupts
and softirqs, point to a *copy* of the real thread_info, because
x86-32 uses %esp to calculate the thread_info address, and thus the
separate irq (and softirq) stacks would cause these kinds of odd
thread_info copy aliases.
This is normally not a problem, since interrupts aren't supposed to
look at thread information anyway (what thread is running at
interrupt time really isn't very well-defined), but it confused the
heck out of irq_fpu_usable() and the code that tried to squirrel
away the FPU state.
(It also caused untold confusion for us poor kernel developers).
It also turns out that using 'task_struct' is actually much more natural
for most of the call sites that care about the FPU state, since they
tend to work with the task struct for other reasons anyway (ie
scheduling). And the FPU data that we are going to save/restore is
found there too.
Thanks to Arjan Van De Ven <arjan@linux.intel.com> for pointing us to
the %esp issue.
Cc: Arjan van de Ven <arjan@linux.intel.com>
Reported-and-tested-by: Raphael Prevost <raphael@buro.asia>
Acked-and-tested-by: Suresh Siddha <suresh.b.siddha@intel.com>
Tested-by: Peter Anvin <hpa@zytor.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/include/asm/i387.h | 44 ++++++++++++++++++-------------------
arch/x86/include/asm/processor.h | 1
arch/x86/include/asm/thread_info.h | 2 -
arch/x86/kernel/traps.c | 11 ++++-----
arch/x86/kernel/xsave.c | 2 -
arch/x86/kvm/vmx.c | 2 -
6 files changed, 30 insertions(+), 32 deletions(-)
--- a/arch/x86/include/asm/i387.h
+++ b/arch/x86/include/asm/i387.h
@@ -264,21 +264,21 @@ static inline int restore_fpu_checking(s
* be preemption protection *and* they need to be
* properly paired with the CR0.TS changes!
*/
-static inline int __thread_has_fpu(struct thread_info *ti)
+static inline int __thread_has_fpu(struct task_struct *tsk)
{
- return ti->status & TS_USEDFPU;
+ return tsk->thread.has_fpu;
}
/* Must be paired with an 'stts' after! */
-static inline void __thread_clear_has_fpu(struct thread_info *ti)
+static inline void __thread_clear_has_fpu(struct task_struct *tsk)
{
- ti->status &= ~TS_USEDFPU;
+ tsk->thread.has_fpu = 0;
}
/* Must be paired with a 'clts' before! */
-static inline void __thread_set_has_fpu(struct thread_info *ti)
+static inline void __thread_set_has_fpu(struct task_struct *tsk)
{
- ti->status |= TS_USEDFPU;
+ tsk->thread.has_fpu = 1;
}
/*
@@ -288,16 +288,16 @@ static inline void __thread_set_has_fpu(
* These generally need preemption protection to work,
* do try to avoid using these on their own.
*/
-static inline void __thread_fpu_end(struct thread_info *ti)
+static inline void __thread_fpu_end(struct task_struct *tsk)
{
- __thread_clear_has_fpu(ti);
+ __thread_clear_has_fpu(tsk);
stts();
}
-static inline void __thread_fpu_begin(struct thread_info *ti)
+static inline void __thread_fpu_begin(struct task_struct *tsk)
{
clts();
- __thread_set_has_fpu(ti);
+ __thread_set_has_fpu(tsk);
}
/*
@@ -308,21 +308,21 @@ extern int restore_i387_xstate(void __us
static inline void __unlazy_fpu(struct task_struct *tsk)
{
- if (__thread_has_fpu(task_thread_info(tsk))) {
+ if (__thread_has_fpu(tsk)) {
__save_init_fpu(tsk);
- __thread_fpu_end(task_thread_info(tsk));
+ __thread_fpu_end(tsk);
} else
tsk->fpu_counter = 0;
}
static inline void __clear_fpu(struct task_struct *tsk)
{
- if (__thread_has_fpu(task_thread_info(tsk))) {
+ if (__thread_has_fpu(tsk)) {
/* Ignore delayed exceptions from user space */
asm volatile("1: fwait\n"
"2:\n"
_ASM_EXTABLE(1b, 2b));
- __thread_fpu_end(task_thread_info(tsk));
+ __thread_fpu_end(tsk);
}
}
@@ -337,7 +337,7 @@ static inline void __clear_fpu(struct ta
*/
static inline bool interrupted_kernel_fpu_idle(void)
{
- return !__thread_has_fpu(current_thread_info()) &&
+ return !__thread_has_fpu(current) &&
(read_cr0() & X86_CR0_TS);
}
@@ -371,12 +371,12 @@ static inline bool irq_fpu_usable(void)
static inline void kernel_fpu_begin(void)
{
- struct thread_info *me = current_thread_info();
+ struct task_struct *me = current;
WARN_ON_ONCE(!irq_fpu_usable());
preempt_disable();
if (__thread_has_fpu(me)) {
- __save_init_fpu(me->task);
+ __save_init_fpu(me);
__thread_clear_has_fpu(me);
/* We do 'stts()' in kernel_fpu_end() */
} else
@@ -441,13 +441,13 @@ static inline void irq_ts_restore(int TS
*/
static inline int user_has_fpu(void)
{
- return __thread_has_fpu(current_thread_info());
+ return __thread_has_fpu(current);
}
static inline void user_fpu_end(void)
{
preempt_disable();
- __thread_fpu_end(current_thread_info());
+ __thread_fpu_end(current);
preempt_enable();
}
@@ -455,7 +455,7 @@ static inline void user_fpu_begin(void)
{
preempt_disable();
if (!user_has_fpu())
- __thread_fpu_begin(current_thread_info());
+ __thread_fpu_begin(current);
preempt_enable();
}
@@ -464,10 +464,10 @@ static inline void user_fpu_begin(void)
*/
static inline void save_init_fpu(struct task_struct *tsk)
{
- WARN_ON_ONCE(!__thread_has_fpu(task_thread_info(tsk)));
+ WARN_ON_ONCE(!__thread_has_fpu(tsk));
preempt_disable();
__save_init_fpu(tsk);
- __thread_fpu_end(task_thread_info(tsk));
+ __thread_fpu_end(tsk);
preempt_enable();
}
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -456,6 +456,7 @@ struct thread_struct {
unsigned long trap_no;
unsigned long error_code;
/* floating point and extended processor state */
+ unsigned long has_fpu;
struct fpu fpu;
#ifdef CONFIG_X86_32
/* Virtual 86 mode info */
--- a/arch/x86/include/asm/thread_info.h
+++ b/arch/x86/include/asm/thread_info.h
@@ -242,8 +242,6 @@ static inline struct thread_info *curren
* ever touches our thread-synchronous status, so we don't
* have to worry about atomic accesses.
*/
-#define TS_USEDFPU 0x0001 /* FPU was used by this task
- this quantum (SMP) */
#define TS_COMPAT 0x0002 /* 32bit syscall active (64BIT)*/
#define TS_POLLING 0x0004 /* idle task polling need_resched,
skip sending interrupt */
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -573,12 +573,11 @@ asmlinkage void __attribute__((weak)) sm
*/
void math_state_restore(void)
{
- struct thread_info *thread = current_thread_info();
- struct task_struct *tsk = thread->task;
+ struct task_struct *tsk = current;
/* We need a safe address that is cheap to find and that is already
- in L1. We just brought in "thread->task", so use that */
-#define safe_address (thread->task)
+ in L1. We're just bringing in "tsk->thread.has_fpu", so use that */
+#define safe_address (tsk->thread.has_fpu)
if (!tsk_used_math(tsk)) {
local_irq_enable();
@@ -595,7 +594,7 @@ void math_state_restore(void)
local_irq_disable();
}
- __thread_fpu_begin(thread);
+ __thread_fpu_begin(tsk);
/* AMD K7/K8 CPUs don't save/restore FDP/FIP/FOP unless an exception
is pending. Clear the x87 state here by setting it to fixed
@@ -611,7 +610,7 @@ void math_state_restore(void)
* Paranoid restore. send a SIGSEGV if we fail to restore the state.
*/
if (unlikely(restore_fpu_checking(tsk))) {
- __thread_fpu_end(thread);
+ __thread_fpu_end(tsk);
force_sig(SIGSEGV, tsk);
return;
}
--- a/arch/x86/kernel/xsave.c
+++ b/arch/x86/kernel/xsave.c
@@ -47,7 +47,7 @@ void __sanitize_i387_state(struct task_s
if (!fx)
return;
- BUG_ON(__thread_has_fpu(task_thread_info(tsk)));
+ BUG_ON(__thread_has_fpu(tsk));
xstate_bv = tsk->thread.fpu.state->xsave.xsave_hdr.xstate_bv;
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -1456,7 +1456,7 @@ static void __vmx_load_host_state(struct
#ifdef CONFIG_X86_64
wrmsrl(MSR_KERNEL_GS_BASE, vmx->msr_host_kernel_gs_base);
#endif
- if (__thread_has_fpu(current_thread_info()))
+ if (__thread_has_fpu(current))
clts();
load_gdt(&__get_cpu_var(host_gdt));
}
^ permalink raw reply [flat|nested] 11+ messages in thread* [ 10/10] i387: re-introduce FPU state preloading at context switch time
2012-02-24 0:05 [ 00/10] 3.2.8-stable review Greg KH
` (8 preceding siblings ...)
2012-02-24 0:02 ` [ 09/10] i387: move TS_USEDFPU flag from thread_info to task_struct Greg KH
@ 2012-02-24 0:02 ` Greg KH
9 siblings, 0 replies; 11+ messages in thread
From: Greg KH @ 2012-02-24 0:02 UTC (permalink / raw)
To: linux-kernel, stable; +Cc: torvalds, akpm, alan
3.2-stable review patch. If anyone has any objections, please let me know.
------------------
From: Linus Torvalds <torvalds@linux-foundation.org>
commit 34ddc81a230b15c0e345b6b253049db731499f7e upstream.
After all the FPU state cleanups and finally finding the problem that
caused all our FPU save/restore problems, this re-introduces the
preloading of FPU state that was removed in commit b3b0870ef3ff ("i387:
do not preload FPU state at task switch time").
However, instead of simply reverting the removal, this reimplements
preloading with several fixes, most notably
- properly abstracted as a true FPU state switch, rather than as
open-coded save and restore with various hacks.
In particular, implementing it as a proper FPU state switch allows us
to optimize the CR0.TS flag accesses: there is no reason to set the
TS bit only to then almost immediately clear it again. CR0 accesses
are quite slow and expensive, don't flip the bit back and forth for
no good reason.
- Make sure that the same model works for both x86-32 and x86-64, so
that there are no gratuitous differences between the two due to the
way they save and restore segment state differently due to
architectural differences that really don't matter to the FPU state.
- Avoid exposing the "preload" state to the context switch routines,
and in particular allow the concept of lazy state restore: if nothing
else has used the FPU in the meantime, and the process is still on
the same CPU, we can avoid restoring state from memory entirely, just
re-expose the state that is still in the FPU unit.
That optimized lazy restore isn't actually implemented here, but the
infrastructure is set up for it. Of course, older CPU's that use
'fnsave' to save the state cannot take advantage of this, since the
state saving also trashes the state.
In other words, there is now an actual _design_ to the FPU state saving,
rather than just random historical baggage. Hopefully it's easier to
follow as a result.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/include/asm/i387.h | 110 ++++++++++++++++++++++++++++++++++++-------
arch/x86/kernel/process_32.c | 5 +
arch/x86/kernel/process_64.c | 5 +
arch/x86/kernel/traps.c | 55 ++++++++++++---------
4 files changed, 133 insertions(+), 42 deletions(-)
--- a/arch/x86/include/asm/i387.h
+++ b/arch/x86/include/asm/i387.h
@@ -29,6 +29,7 @@ extern unsigned int sig_xstate_size;
extern void fpu_init(void);
extern void mxcsr_feature_mask_init(void);
extern int init_fpu(struct task_struct *child);
+extern void __math_state_restore(struct task_struct *);
extern void math_state_restore(void);
extern int dump_fpu(struct pt_regs *, struct user_i387_struct *);
@@ -212,9 +213,10 @@ static inline void fpu_fxsave(struct fpu
#endif /* CONFIG_X86_64 */
/*
- * These must be called with preempt disabled
+ * These must be called with preempt disabled. Returns
+ * 'true' if the FPU state is still intact.
*/
-static inline void fpu_save_init(struct fpu *fpu)
+static inline int fpu_save_init(struct fpu *fpu)
{
if (use_xsave()) {
fpu_xsave(fpu);
@@ -223,22 +225,33 @@ static inline void fpu_save_init(struct
* xsave header may indicate the init state of the FP.
*/
if (!(fpu->state->xsave.xsave_hdr.xstate_bv & XSTATE_FP))
- return;
+ return 1;
} else if (use_fxsr()) {
fpu_fxsave(fpu);
} else {
asm volatile("fnsave %[fx]; fwait"
: [fx] "=m" (fpu->state->fsave));
- return;
+ return 0;
}
- if (unlikely(fpu->state->fxsave.swd & X87_FSW_ES))
+ /*
+ * If exceptions are pending, we need to clear them so
+ * that we don't randomly get exceptions later.
+ *
+ * FIXME! Is this perhaps only true for the old-style
+ * irq13 case? Maybe we could leave the x87 state
+ * intact otherwise?
+ */
+ if (unlikely(fpu->state->fxsave.swd & X87_FSW_ES)) {
asm volatile("fnclex");
+ return 0;
+ }
+ return 1;
}
-static inline void __save_init_fpu(struct task_struct *tsk)
+static inline int __save_init_fpu(struct task_struct *tsk)
{
- fpu_save_init(&tsk->thread.fpu);
+ return fpu_save_init(&tsk->thread.fpu);
}
static inline int fpu_fxrstor_checking(struct fpu *fpu)
@@ -301,20 +314,79 @@ static inline void __thread_fpu_begin(st
}
/*
- * Signal frame handlers...
+ * FPU state switching for scheduling.
+ *
+ * This is a two-stage process:
+ *
+ * - switch_fpu_prepare() saves the old state and
+ * sets the new state of the CR0.TS bit. This is
+ * done within the context of the old process.
+ *
+ * - switch_fpu_finish() restores the new state as
+ * necessary.
*/
-extern int save_i387_xstate(void __user *buf);
-extern int restore_i387_xstate(void __user *buf);
+typedef struct { int preload; } fpu_switch_t;
+
+/*
+ * FIXME! We could do a totally lazy restore, but we need to
+ * add a per-cpu "this was the task that last touched the FPU
+ * on this CPU" variable, and the task needs to have a "I last
+ * touched the FPU on this CPU" and check them.
+ *
+ * We don't do that yet, so "fpu_lazy_restore()" always returns
+ * false, but some day..
+ */
+#define fpu_lazy_restore(tsk) (0)
+#define fpu_lazy_state_intact(tsk) do { } while (0)
-static inline void __unlazy_fpu(struct task_struct *tsk)
+static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct task_struct *new)
{
- if (__thread_has_fpu(tsk)) {
- __save_init_fpu(tsk);
- __thread_fpu_end(tsk);
- } else
- tsk->fpu_counter = 0;
+ fpu_switch_t fpu;
+
+ fpu.preload = tsk_used_math(new) && new->fpu_counter > 5;
+ if (__thread_has_fpu(old)) {
+ if (__save_init_fpu(old))
+ fpu_lazy_state_intact(old);
+ __thread_clear_has_fpu(old);
+ old->fpu_counter++;
+
+ /* Don't change CR0.TS if we just switch! */
+ if (fpu.preload) {
+ __thread_set_has_fpu(new);
+ prefetch(new->thread.fpu.state);
+ } else
+ stts();
+ } else {
+ old->fpu_counter = 0;
+ if (fpu.preload) {
+ if (fpu_lazy_restore(new))
+ fpu.preload = 0;
+ else
+ prefetch(new->thread.fpu.state);
+ __thread_fpu_begin(new);
+ }
+ }
+ return fpu;
+}
+
+/*
+ * By the time this gets called, we've already cleared CR0.TS and
+ * given the process the FPU if we are going to preload the FPU
+ * state - all we need to do is to conditionally restore the register
+ * state itself.
+ */
+static inline void switch_fpu_finish(struct task_struct *new, fpu_switch_t fpu)
+{
+ if (fpu.preload)
+ __math_state_restore(new);
}
+/*
+ * Signal frame handlers...
+ */
+extern int save_i387_xstate(void __user *buf);
+extern int restore_i387_xstate(void __user *buf);
+
static inline void __clear_fpu(struct task_struct *tsk)
{
if (__thread_has_fpu(tsk)) {
@@ -474,7 +546,11 @@ static inline void save_init_fpu(struct
static inline void unlazy_fpu(struct task_struct *tsk)
{
preempt_disable();
- __unlazy_fpu(tsk);
+ if (__thread_has_fpu(tsk)) {
+ __save_init_fpu(tsk);
+ __thread_fpu_end(tsk);
+ } else
+ tsk->fpu_counter = 0;
preempt_enable();
}
--- a/arch/x86/kernel/process_32.c
+++ b/arch/x86/kernel/process_32.c
@@ -297,10 +297,11 @@ __switch_to(struct task_struct *prev_p,
*next = &next_p->thread;
int cpu = smp_processor_id();
struct tss_struct *tss = &per_cpu(init_tss, cpu);
+ fpu_switch_t fpu;
/* never put a printk in __switch_to... printk() calls wake_up*() indirectly */
- __unlazy_fpu(prev_p);
+ fpu = switch_fpu_prepare(prev_p, next_p);
/*
* Reload esp0.
@@ -355,6 +356,8 @@ __switch_to(struct task_struct *prev_p,
if (prev->gs | next->gs)
lazy_load_gs(next->gs);
+ switch_fpu_finish(next_p, fpu);
+
percpu_write(current_task, next_p);
return prev_p;
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -381,8 +381,9 @@ __switch_to(struct task_struct *prev_p,
int cpu = smp_processor_id();
struct tss_struct *tss = &per_cpu(init_tss, cpu);
unsigned fsindex, gsindex;
+ fpu_switch_t fpu;
- __unlazy_fpu(prev_p);
+ fpu = switch_fpu_prepare(prev_p, next_p);
/*
* Reload esp0, LDT and the page table pointer:
@@ -452,6 +453,8 @@ __switch_to(struct task_struct *prev_p,
wrmsrl(MSR_KERNEL_GS_BASE, next->gs);
prev->gsindex = gsindex;
+ switch_fpu_finish(next_p, fpu);
+
/*
* Switch the PDA and FPU contexts.
*/
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -562,6 +562,37 @@ asmlinkage void __attribute__((weak)) sm
}
/*
+ * This gets called with the process already owning the
+ * FPU state, and with CR0.TS cleared. It just needs to
+ * restore the FPU register state.
+ */
+void __math_state_restore(struct task_struct *tsk)
+{
+ /* We need a safe address that is cheap to find and that is already
+ in L1. We've just brought in "tsk->thread.has_fpu", so use that */
+#define safe_address (tsk->thread.has_fpu)
+
+ /* AMD K7/K8 CPUs don't save/restore FDP/FIP/FOP unless an exception
+ is pending. Clear the x87 state here by setting it to fixed
+ values. safe_address is a random variable that should be in L1 */
+ alternative_input(
+ ASM_NOP8 ASM_NOP2,
+ "emms\n\t" /* clear stack tags */
+ "fildl %P[addr]", /* set F?P to defined value */
+ X86_FEATURE_FXSAVE_LEAK,
+ [addr] "m" (safe_address));
+
+ /*
+ * Paranoid restore. send a SIGSEGV if we fail to restore the state.
+ */
+ if (unlikely(restore_fpu_checking(tsk))) {
+ __thread_fpu_end(tsk);
+ force_sig(SIGSEGV, tsk);
+ return;
+ }
+}
+
+/*
* 'math_state_restore()' saves the current math information in the
* old math state array, and gets the new ones from the current task
*
@@ -575,10 +606,6 @@ void math_state_restore(void)
{
struct task_struct *tsk = current;
- /* We need a safe address that is cheap to find and that is already
- in L1. We're just bringing in "tsk->thread.has_fpu", so use that */
-#define safe_address (tsk->thread.has_fpu)
-
if (!tsk_used_math(tsk)) {
local_irq_enable();
/*
@@ -595,25 +622,7 @@ void math_state_restore(void)
}
__thread_fpu_begin(tsk);
-
- /* AMD K7/K8 CPUs don't save/restore FDP/FIP/FOP unless an exception
- is pending. Clear the x87 state here by setting it to fixed
- values. safe_address is a random variable that should be in L1 */
- alternative_input(
- ASM_NOP8 ASM_NOP2,
- "emms\n\t" /* clear stack tags */
- "fildl %P[addr]", /* set F?P to defined value */
- X86_FEATURE_FXSAVE_LEAK,
- [addr] "m" (safe_address));
-
- /*
- * Paranoid restore. send a SIGSEGV if we fail to restore the state.
- */
- if (unlikely(restore_fpu_checking(tsk))) {
- __thread_fpu_end(tsk);
- force_sig(SIGSEGV, tsk);
- return;
- }
+ __math_state_restore(tsk);
tsk->fpu_counter++;
}
^ permalink raw reply [flat|nested] 11+ messages in thread