From: Suraj Jitindar Singh <surajjs@amazon.com>
To: <stable@vger.kernel.org>
Cc: <surajjs@amazon.com>, <sjitindarsingh@gmail.com>,
<cascardo@canonical.com>, <kvm@vger.kernel.org>,
<pbonzini@redhat.com>, <jpoimboe@kernel.org>,
<peterz@infradead.org>, <x86@kernel.org>
Subject: [PATCH 4.14 12/34] x86/entry: Add kernel IBRS implementation
Date: Thu, 27 Oct 2022 13:54:52 -0700 [thread overview]
Message-ID: <20221027205452.17271-4-surajjs@amazon.com> (raw)
In-Reply-To: <20221027205452.17271-1-surajjs@amazon.com>
From: Thadeu Lima de Souza Cascardo <cascardo@canonical.com>
commit 2dbb887e875b1de3ca8f40ddf26bcfe55798c609 upstream.
Implement Kernel IBRS - currently the only known option to mitigate RSB
underflow speculation issues on Skylake hardware.
Note: since IBRS_ENTER requires fuller context established than
UNTRAIN_RET, it must be placed after it. However, since UNTRAIN_RET
itself implies a RET, it must come after IBRS_ENTER. This means
IBRS_ENTER needs to also move UNTRAIN_RET.
Note 2: KERNEL_IBRS is sub-optimal for XenPV.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
[cascardo: conflict at arch/x86/entry/entry_64_compat.S]
[cascardo: conflict fixups, no ANNOTATE_NOENDBR]
Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@canonical.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
[ bp: Adjust context ]
Signed-off-by: Suraj Jitindar Singh <surajjs@amazon.com>
---
arch/x86/entry/calling.h | 58 ++++++++++++++++++++++++++++++
arch/x86/entry/entry_64.S | 33 +++++++++++++++++
arch/x86/entry/entry_64_compat.S | 12 ++++++-
arch/x86/include/asm/cpufeatures.h | 2 +-
4 files changed, 103 insertions(+), 2 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 273b562af8fa..ef759951fd0f 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -6,6 +6,8 @@
#include <asm/percpu.h>
#include <asm/asm-offsets.h>
#include <asm/processor-flags.h>
+#include <asm/msr.h>
+#include <asm/nospec-branch.h>
/*
@@ -328,6 +330,62 @@ For 32-bit we have the following conventions - kernel is built with
#endif
+/*
+ * IBRS kernel mitigation for Spectre_v2.
+ *
+ * Assumes full context is established (PUSH_REGS, CR3 and GS) and it clobbers
+ * the regs it uses (AX, CX, DX). Must be called before the first RET
+ * instruction (NOTE! UNTRAIN_RET includes a RET instruction)
+ *
+ * The optional argument is used to save/restore the current value,
+ * which is used on the paranoid paths.
+ *
+ * Assumes x86_spec_ctrl_{base,current} to have SPEC_CTRL_IBRS set.
+ */
+.macro IBRS_ENTER save_reg
+ ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_KERNEL_IBRS
+ movl $MSR_IA32_SPEC_CTRL, %ecx
+
+.ifnb \save_reg
+ rdmsr
+ shl $32, %rdx
+ or %rdx, %rax
+ mov %rax, \save_reg
+ test $SPEC_CTRL_IBRS, %eax
+ jz .Ldo_wrmsr_\@
+ lfence
+ jmp .Lend_\@
+.Ldo_wrmsr_\@:
+.endif
+
+ movq PER_CPU_VAR(x86_spec_ctrl_current), %rdx
+ movl %edx, %eax
+ shr $32, %rdx
+ wrmsr
+.Lend_\@:
+.endm
+
+/*
+ * Similar to IBRS_ENTER, requires KERNEL GS,CR3 and clobbers (AX, CX, DX)
+ * regs. Must be called after the last RET.
+ */
+.macro IBRS_EXIT save_reg
+ ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_KERNEL_IBRS
+ movl $MSR_IA32_SPEC_CTRL, %ecx
+
+.ifnb \save_reg
+ mov \save_reg, %rdx
+.else
+ movq PER_CPU_VAR(x86_spec_ctrl_current), %rdx
+ andl $(~SPEC_CTRL_IBRS), %edx
+.endif
+
+ movl %edx, %eax
+ shr $32, %rdx
+ wrmsr
+.Lend_\@:
+.endm
+
/*
* Mitigate Spectre v1 for conditional swapgs code paths.
*
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index d3472e773b5a..fee102f5a7d5 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -230,6 +230,10 @@ GLOBAL(entry_SYSCALL_64_after_hwframe)
/* IRQs are off. */
movq %rsp, %rdi
+
+ /* clobbers %rax, make sure it is after saving the syscall nr */
+ IBRS_ENTER
+
call do_syscall_64 /* returns with IRQs disabled */
TRACE_IRQS_IRETQ /* we're about to change IF */
@@ -301,6 +305,7 @@ GLOBAL(entry_SYSCALL_64_after_hwframe)
* perf profiles. Nothing jumps here.
*/
syscall_return_via_sysret:
+ IBRS_EXIT
POP_REGS pop_rdi=0
/*
@@ -590,6 +595,7 @@ GLOBAL(retint_user)
TRACE_IRQS_IRETQ
GLOBAL(swapgs_restore_regs_and_return_to_usermode)
+ IBRS_EXIT
#ifdef CONFIG_DEBUG_ENTRY
/* Assert that pt_regs indicates user mode. */
testb $3, CS(%rsp)
@@ -1133,6 +1139,9 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
* Save all registers in pt_regs, and switch gs if needed.
* Use slow, but surefire "are we in kernel?" check.
* Return: ebx=0: need swapgs on exit, ebx=1: otherwise
+ *
+ * R14 - old CR3
+ * R15 - old SPEC_CTRL
*/
ENTRY(paranoid_entry)
UNWIND_HINT_FUNC
@@ -1156,6 +1165,12 @@ ENTRY(paranoid_entry)
*/
FENCE_SWAPGS_KERNEL_ENTRY
+ /*
+ * Once we have CR3 and %GS setup save and set SPEC_CTRL. Just like
+ * CR3 above, keep the old value in a callee saved register.
+ */
+ IBRS_ENTER save_reg=%r15
+
ret
END(paranoid_entry)
@@ -1170,9 +1185,19 @@ END(paranoid_entry)
* to try to handle preemption here.
*
* On entry, ebx is "no swapgs" flag (1: don't need swapgs, 0: need it)
+ *
+ * R14 - old CR3
+ * R15 - old SPEC_CTRL
*/
ENTRY(paranoid_exit)
UNWIND_HINT_REGS
+
+ /*
+ * Must restore IBRS state before both CR3 and %GS since we need access
+ * to the per-CPU x86_spec_ctrl_shadow variable.
+ */
+ IBRS_EXIT save_reg=%r15
+
DISABLE_INTERRUPTS(CLBR_ANY)
TRACE_IRQS_OFF_DEBUG
testl %ebx, %ebx /* swapgs needed? */
@@ -1207,8 +1232,10 @@ ENTRY(error_entry)
FENCE_SWAPGS_USER_ENTRY
/* We have user CR3. Change to kernel CR3. */
SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
+ IBRS_ENTER
.Lerror_entry_from_usermode_after_swapgs:
+
/* Put us onto the real thread stack. */
popq %r12 /* save return addr in %12 */
movq %rsp, %rdi /* arg0 = pt_regs pointer */
@@ -1271,6 +1298,7 @@ ENTRY(error_entry)
SWAPGS
FENCE_SWAPGS_USER_ENTRY
SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
+ IBRS_ENTER
/*
* Pretend that the exception came from user mode: set up pt_regs
@@ -1376,6 +1404,8 @@ ENTRY(nmi)
PUSH_AND_CLEAR_REGS rdx=(%rdx)
ENCODE_FRAME_POINTER
+ IBRS_ENTER
+
/*
* At this point we no longer need to worry about stack damage
* due to nesting -- we're on the normal thread stack and we're
@@ -1599,6 +1629,9 @@ end_repeat_nmi:
movq $-1, %rsi
call do_nmi
+ /* Always restore stashed SPEC_CTRL value (see paranoid_entry) */
+ IBRS_EXIT save_reg=%r15
+
RESTORE_CR3 scratch_reg=%r15 save_reg=%r14
testl %ebx, %ebx /* swapgs needed? */
diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_compat.S
index 304e3daf82dd..8cdc3af68c53 100644
--- a/arch/x86/entry/entry_64_compat.S
+++ b/arch/x86/entry/entry_64_compat.S
@@ -4,7 +4,6 @@
*
* Copyright 2000-2002 Andi Kleen, SuSE Labs.
*/
-#include "calling.h"
#include <asm/asm-offsets.h>
#include <asm/current.h>
#include <asm/errno.h>
@@ -17,6 +16,8 @@
#include <linux/linkage.h>
#include <linux/err.h>
+#include "calling.h"
+
.section .entry.text, "ax"
/*
@@ -106,6 +107,8 @@ ENTRY(entry_SYSENTER_compat)
xorl %r15d, %r15d /* nospec r15 */
cld
+ IBRS_ENTER
+
/*
* SYSENTER doesn't filter flags, so we need to clear NT and AC
* ourselves. To save a few cycles, we can check whether
@@ -250,6 +253,8 @@ GLOBAL(entry_SYSCALL_compat_after_hwframe)
*/
TRACE_IRQS_OFF
+ IBRS_ENTER
+
movq %rsp, %rdi
call do_fast_syscall_32
/* XEN PV guests always use IRET path */
@@ -259,6 +264,9 @@ GLOBAL(entry_SYSCALL_compat_after_hwframe)
/* Opportunistic SYSRET */
sysret32_from_system_call:
TRACE_IRQS_ON /* User mode traces as IRQs on. */
+
+ IBRS_EXIT
+
movq RBX(%rsp), %rbx /* pt_regs->rbx */
movq RBP(%rsp), %rbp /* pt_regs->rbp */
movq EFLAGS(%rsp), %r11 /* pt_regs->flags (in r11) */
@@ -385,6 +393,8 @@ ENTRY(entry_INT80_compat)
*/
TRACE_IRQS_OFF
+ IBRS_ENTER
+
movq %rsp, %rdi
call do_int80_syscall_32
.Lsyscall_32_done:
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 2e96129ed68b..b78acf1e70b6 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -202,7 +202,7 @@
#define X86_FEATURE_PROC_FEEDBACK ( 7*32+ 9) /* AMD ProcFeedbackInterface */
#define X86_FEATURE_SME ( 7*32+10) /* AMD Secure Memory Encryption */
#define X86_FEATURE_PTI ( 7*32+11) /* Kernel Page Table Isolation enabled */
-/* FREE! ( 7*32+12) */
+#define X86_FEATURE_KERNEL_IBRS ( 7*32+12) /* "" Set/clear IBRS on kernel entry/exit */
/* FREE! ( 7*32+13) */
#define X86_FEATURE_INTEL_PPIN ( 7*32+14) /* Intel Processor Inventory Number */
#define X86_FEATURE_CDP_L2 ( 7*32+15) /* Code and Data Prioritization L2 */
--
2.17.1
next prev parent reply other threads:[~2022-10-27 21:03 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-27 20:48 [PATCH 4.14 00/34] Retbleed & PBRSB Mitigations Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 01/34] Revert "x86/cpu: Add a steppings field to struct x86_cpu_id" Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 02/34] x86/cpufeature: Add facility to check for min microcode revisions Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 03/34] x86/cpufeature: Fix various quality problems in the <asm/cpu_device_hd.h> header Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 04/34] x86/devicetable: Move x86 specific macro out of generic code Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 05/34] x86/cpu: Add consistent CPU match macros Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 06/34] x86/cpu: Add a steppings field to struct x86_cpu_id Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 07/34] x86/entry: Remove skip_r11rcx Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 08/34] x86/cpufeatures: Move RETPOLINE flags to word 11 Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 09/34] x86/bugs: Report AMD retbleed vulnerability Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 10/34] x86/bugs: Add AMD retbleed= boot parameter Suraj Jitindar Singh
2022-10-27 20:54 ` [PATCH 4.14 11/34] x86/bugs: Keep a per-CPU IA32_SPEC_CTRL value Suraj Jitindar Singh
2022-10-27 20:54 ` Suraj Jitindar Singh [this message]
2022-10-27 20:54 ` [PATCH 4.14 13/34] x86/bugs: Optimize SPEC_CTRL MSR writes Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 14/34] x86/speculation: Add spectre_v2=ibrs option to support Kernel IBRS Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 15/34] x86/bugs: Split spectre_v2_select_mitigation() and spectre_v2_user_select_mitigation() Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 16/34] x86/bugs: Report Intel retbleed vulnerability Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 17/34] entel_idle: Disable IBRS during long idle Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 18/34] x86/speculation: Change FILL_RETURN_BUFFER to work with objtool Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 19/34] x86/speculation: Add LFENCE to RSB fill sequence Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 20/34] x86/speculation: Fix RSB filling with CONFIG_RETPOLINE=n Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 21/34] x86/speculation: Fix firmware entry SPEC_CTRL handling Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 22/34] x86/speculation: Fix SPEC_CTRL write on SMT state change Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 23/34] x86/speculation: Use cached host SPEC_CTRL value for guest entry/exit Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 24/34] x86/speculation: Remove x86_spec_ctrl_mask Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 25/34] KVM: VMX: Prevent guest RSB poisoning attacks with eIBRS Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 26/34] KVM: VMX: Fix IBRS handling after vmexit Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 27/34] x86/speculation: Fill RSB on vmexit for IBRS Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 28/34] x86/common: Stamp out the stepping madness Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 29/34] x86/cpu/amd: Enumerate BTC_NO Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 30/34] x86/bugs: Add Cannon lake to RETBleed affected CPU list Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 31/34] x86/speculation: Disable RRSBA behavior Suraj Jitindar Singh
2022-10-27 20:55 ` [PATCH 4.14 32/34] x86/speculation: Use DECLARE_PER_CPU for x86_spec_ctrl_current Suraj Jitindar Singh
2022-10-27 20:56 ` [PATCH 4.14 33/34] x86/bugs: Warn when "ibrs" mitigation is selected on Enhanced IBRS parts Suraj Jitindar Singh
2022-10-27 20:56 ` [PATCH 4.14 34/34] x86/speculation: Add RSB VM Exit protections Suraj Jitindar Singh
2022-10-31 7:00 ` [PATCH 4.14 00/34] Retbleed & PBRSB Mitigations Greg KH
-- strict thread matches above, loose matches on Subject: below --
2022-10-31 7:02 [PATCH 4.14 00/34] 4.14.297-rc1 review Greg Kroah-Hartman
2022-10-31 7:02 ` [PATCH 4.14 12/34] x86/entry: Add kernel IBRS implementation Greg Kroah-Hartman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221027205452.17271-4-surajjs@amazon.com \
--to=surajjs@amazon.com \
--cc=cascardo@canonical.com \
--cc=jpoimboe@kernel.org \
--cc=kvm@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=peterz@infradead.org \
--cc=sjitindarsingh@gmail.com \
--cc=stable@vger.kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.