[PATCH v3 0/6] x86/bugs: RSB mitigation fixes and documentation

[PATCH v3 0/6] x86/bugs: RSB mitigation fixes and documentation

[Josh Poimboeuf]

v3:
- fancy new doc
- more fixes

v2: https://lore.kernel.org/cover.1732219175.git.jpoimboe@kernel.org

Josh Poimboeuf (6):
  x86/bugs: Rename entry_ibpb()
  x86/bugs: Use SBPB in __write_ibpb() if applicable
  x86/bugs: Fix RSB clearing in indirect_branch_prediction_barrier()
  x86/bugs: Don't fill RSB on VMEXIT with eIBRS+retpoline
  x86/bugs: Don't fill RSB on context switch with eIBRS
  x86/bugs: Add RSB mitigation document

 Documentation/admin-guide/hw-vuln/index.rst |   1 +
 Documentation/admin-guide/hw-vuln/rsb.rst   | 241 ++++++++++++++++++++
 arch/x86/entry/entry.S                      |   9 +-
 arch/x86/include/asm/nospec-branch.h        |  12 +-
 arch/x86/kernel/cpu/bugs.c                  | 109 +++------
 arch/x86/mm/tlb.c                           |   6 +-
 6 files changed, 291 insertions(+), 87 deletions(-)
 create mode 100644 Documentation/admin-guide/hw-vuln/rsb.rst

-- 
2.48.1

[PATCH v3 1/6] x86/bugs: Rename entry_ibpb()

[Josh Poimboeuf]

There's nothing entry-specific about entry_ibpb().  In preparation for
calling it from elsewhere, rename it to __write_ibpb().

Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 arch/x86/entry/entry.S               | 7 ++++---
 arch/x86/include/asm/nospec-branch.h | 6 +++---
 arch/x86/kernel/cpu/bugs.c           | 6 +++---
 3 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/arch/x86/entry/entry.S b/arch/x86/entry/entry.S
index d3caa31240ed..3a53319988b9 100644
--- a/arch/x86/entry/entry.S
+++ b/arch/x86/entry/entry.S
@@ -17,7 +17,8 @@
 
 .pushsection .noinstr.text, "ax"
 
-SYM_FUNC_START(entry_ibpb)
+// Clobbers AX, CX, DX
+SYM_FUNC_START(__write_ibpb)
 	ANNOTATE_NOENDBR
 	movl	$MSR_IA32_PRED_CMD, %ecx
 	movl	$PRED_CMD_IBPB, %eax
@@ -27,9 +28,9 @@ SYM_FUNC_START(entry_ibpb)
 	/* Make sure IBPB clears return stack preductions too. */
 	FILL_RETURN_BUFFER %rax, RSB_CLEAR_LOOPS, X86_BUG_IBPB_NO_RET
 	RET
-SYM_FUNC_END(entry_ibpb)
+SYM_FUNC_END(__write_ibpb)
 /* For KVM */
-EXPORT_SYMBOL_GPL(entry_ibpb);
+EXPORT_SYMBOL_GPL(__write_ibpb);
 
 .popsection
 
diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
index 8a5cc8e70439..bbac79cad04c 100644
--- a/arch/x86/include/asm/nospec-branch.h
+++ b/arch/x86/include/asm/nospec-branch.h
@@ -269,7 +269,7 @@
  * typically has NO_MELTDOWN).
  *
  * While retbleed_untrain_ret() doesn't clobber anything but requires stack,
- * entry_ibpb() will clobber AX, CX, DX.
+ * __write_ibpb() will clobber AX, CX, DX.
  *
  * As such, this must be placed after every *SWITCH_TO_KERNEL_CR3 at a point
  * where we have a stack but before any RET instruction.
@@ -279,7 +279,7 @@
 	VALIDATE_UNRET_END
 	CALL_UNTRAIN_RET
 	ALTERNATIVE_2 "",						\
-		      "call entry_ibpb", \ibpb_feature,			\
+		      "call __write_ibpb", \ibpb_feature,			\
 		     __stringify(\call_depth_insns), X86_FEATURE_CALL_DEPTH
 #endif
 .endm
@@ -368,7 +368,7 @@ extern void srso_return_thunk(void);
 extern void srso_alias_return_thunk(void);
 
 extern void entry_untrain_ret(void);
-extern void entry_ibpb(void);
+extern void __write_ibpb(void);
 
 #ifdef CONFIG_X86_64
 extern void clear_bhb_loop(void);
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 4386aa6c69e1..310cb3f7139c 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1142,7 +1142,7 @@ static void __init retbleed_select_mitigation(void)
 		setup_clear_cpu_cap(X86_FEATURE_RETHUNK);
 
 		/*
-		 * There is no need for RSB filling: entry_ibpb() ensures
+		 * There is no need for RSB filling: __write_ibpb() ensures
 		 * all predictions, including the RSB, are invalidated,
 		 * regardless of IBPB implementation.
 		 */
@@ -2676,7 +2676,7 @@ static void __init srso_select_mitigation(void)
 				setup_clear_cpu_cap(X86_FEATURE_RETHUNK);
 
 				/*
-				 * There is no need for RSB filling: entry_ibpb() ensures
+				 * There is no need for RSB filling: __write_ibpb() ensures
 				 * all predictions, including the RSB, are invalidated,
 				 * regardless of IBPB implementation.
 				 */
@@ -2701,7 +2701,7 @@ static void __init srso_select_mitigation(void)
 				srso_mitigation = SRSO_MITIGATION_IBPB_ON_VMEXIT;
 
 				/*
-				 * There is no need for RSB filling: entry_ibpb() ensures
+				 * There is no need for RSB filling: __write_ibpb() ensures
 				 * all predictions, including the RSB, are invalidated,
 				 * regardless of IBPB implementation.
 				 */
-- 
2.48.1

Re: [PATCH v3 1/6] x86/bugs: Rename entry_ibpb()

[Borislav Petkov]

On Wed, Apr 02, 2025 at 11:19:18AM -0700, Josh Poimboeuf wrote:
> There's nothing entry-specific about entry_ibpb().  In preparation for

Not anymore - it was done on entry back then AFAIR.

> calling it from elsewhere, rename it to __write_ibpb().
> 
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
> ---
>  arch/x86/entry/entry.S               | 7 ++++---
>  arch/x86/include/asm/nospec-branch.h | 6 +++---
>  arch/x86/kernel/cpu/bugs.c           | 6 +++---
>  3 files changed, 10 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/x86/entry/entry.S b/arch/x86/entry/entry.S
> index d3caa31240ed..3a53319988b9 100644
> --- a/arch/x86/entry/entry.S
> +++ b/arch/x86/entry/entry.S
> @@ -17,7 +17,8 @@
>  
>  .pushsection .noinstr.text, "ax"
>  
> -SYM_FUNC_START(entry_ibpb)
> +// Clobbers AX, CX, DX

Why the ugly comment style if the rest of the file is already using the
multiline one...

> +SYM_FUNC_START(__write_ibpb)

... and why the __ ?

>  	ANNOTATE_NOENDBR
>  	movl	$MSR_IA32_PRED_CMD, %ecx
>  	movl	$PRED_CMD_IBPB, %eax

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH v3 1/6] x86/bugs: Rename entry_ibpb()

[Josh Poimboeuf]

On Wed, Apr 02, 2025 at 08:29:28PM +0200, Borislav Petkov wrote:
> On Wed, Apr 02, 2025 at 11:19:18AM -0700, Josh Poimboeuf wrote:
> > There's nothing entry-specific about entry_ibpb().  In preparation for
> 
> Not anymore - it was done on entry back then AFAIR.
> 
> > calling it from elsewhere, rename it to __write_ibpb().
> > 
> > Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
> > ---
> >  arch/x86/entry/entry.S               | 7 ++++---
> >  arch/x86/include/asm/nospec-branch.h | 6 +++---
> >  arch/x86/kernel/cpu/bugs.c           | 6 +++---
> >  3 files changed, 10 insertions(+), 9 deletions(-)
> > 
> > diff --git a/arch/x86/entry/entry.S b/arch/x86/entry/entry.S
> > index d3caa31240ed..3a53319988b9 100644
> > --- a/arch/x86/entry/entry.S
> > +++ b/arch/x86/entry/entry.S
> > @@ -17,7 +17,8 @@
> >  
> >  .pushsection .noinstr.text, "ax"
> >  
> > -SYM_FUNC_START(entry_ibpb)
> > +// Clobbers AX, CX, DX
> 
> Why the ugly comment style if the rest of the file is already using the
> multiline one...

It helps it stand out more? :-)

> > +SYM_FUNC_START(__write_ibpb)
> 
> ... and why the __ ?

I was thinking the calling interface is a bit nonstandard.  But actually
it's fine to call from C as those registers are already caller-saved
anyway.  So yeah, let's drop the '__'.

-- 
Josh

Re: [PATCH v3 1/6] x86/bugs: Rename entry_ibpb()

[Borislav Petkov]

On Wed, Apr 02, 2025 at 11:44:15AM -0700, Josh Poimboeuf wrote:
> It helps it stand out more? :-)

Please don't.

Someone thought that it is a good idea to start using that // ugliness all of
a sudden.

So we decided we should limit it in tip:

Documentation/process/maintainer-tip.rst

The paragrapn that starts with "Use C++ style, tail comments when documenting
..."

> I was thinking the calling interface is a bit nonstandard.  But actually
> it's fine to call from C as those registers are already caller-saved
> anyway.  So yeah, let's drop the '__'.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH v3 1/6] x86/bugs: Rename entry_ibpb()

[Ingo Molnar]

* Josh Poimboeuf <jpoimboe@kernel.org> wrote:

> There's nothing entry-specific about entry_ibpb().  In preparation for
> calling it from elsewhere, rename it to __write_ibpb().

Minor Git-log hygienic request, could you please mention in such 
patches what the *new* name is?

This title is annoyingly passive-aggressive:

  x86/bugs: Rename entry_ibpb()

Let's make it:

  x86/bugs: Rename entry_ibpb() to write_ibpb()

... or so. Because the new name is infinitely more important going 
forward than the old name is ever going to be. :-)

Thanks,

	Ingo

Re: [PATCH v3 1/6] x86/bugs: Rename entry_ibpb()

[Josh Poimboeuf]

On Wed, Apr 02, 2025 at 09:37:07PM +0200, Ingo Molnar wrote:
> 
> * Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> 
> > There's nothing entry-specific about entry_ibpb().  In preparation for
> > calling it from elsewhere, rename it to __write_ibpb().
> 
> Minor Git-log hygienic request, could you please mention in such 
> patches what the *new* name is?
> 
> This title is annoyingly passive-aggressive:

LOL

>   x86/bugs: Rename entry_ibpb()
> 
> Let's make it:
> 
>   x86/bugs: Rename entry_ibpb() to write_ibpb()
> 
> ... or so. Because the new name is infinitely more important going 
> forward than the old name is ever going to be. :-)

Indeed, thanks.

-- 
Josh

[PATCH v3 2/6] x86/bugs: Use SBPB in __write_ibpb() if applicable

[Josh Poimboeuf]

__write_ibpb() does IBPB, which (among other things) flushes branch type
predictions on AMD.  If the CPU has SRSO_NO, or if the SRSO mitigation
has been disabled, branch type flushing isn't needed, in which case the
lighter-weight SBPB can be used.

Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 arch/x86/entry/entry.S     | 2 +-
 arch/x86/kernel/cpu/bugs.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/entry/entry.S b/arch/x86/entry/entry.S
index 3a53319988b9..a5b421ec19c0 100644
--- a/arch/x86/entry/entry.S
+++ b/arch/x86/entry/entry.S
@@ -21,7 +21,7 @@
 SYM_FUNC_START(__write_ibpb)
 	ANNOTATE_NOENDBR
 	movl	$MSR_IA32_PRED_CMD, %ecx
-	movl	$PRED_CMD_IBPB, %eax
+	movl	_ASM_RIP(x86_pred_cmd), %eax
 	xorl	%edx, %edx
 	wrmsr
 
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 310cb3f7139c..c8b8dc829046 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -58,7 +58,7 @@ EXPORT_SYMBOL_GPL(x86_spec_ctrl_base);
 DEFINE_PER_CPU(u64, x86_spec_ctrl_current);
 EXPORT_PER_CPU_SYMBOL_GPL(x86_spec_ctrl_current);
 
-u64 x86_pred_cmd __ro_after_init = PRED_CMD_IBPB;
+u32 x86_pred_cmd __ro_after_init = PRED_CMD_IBPB;
 EXPORT_SYMBOL_GPL(x86_pred_cmd);
 
 static u64 __ro_after_init x86_arch_cap_msr;
-- 
2.48.1

Re: [PATCH v3 2/6] x86/bugs: Use SBPB in __write_ibpb() if applicable

[Tom Lendacky]

On 4/2/25 13:19, Josh Poimboeuf wrote:
> __write_ibpb() does IBPB, which (among other things) flushes branch type
> predictions on AMD.  If the CPU has SRSO_NO, or if the SRSO mitigation
> has been disabled, branch type flushing isn't needed, in which case the
> lighter-weight SBPB can be used.

Maybe add something here that indicates the x86_pred_cmd variable tracks
this optimization so switch to using that variable vs the hardcoded IBPB?

Thanks,
Tom

> 
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
> ---
>  arch/x86/entry/entry.S     | 2 +-
>  arch/x86/kernel/cpu/bugs.c | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/entry/entry.S b/arch/x86/entry/entry.S
> index 3a53319988b9..a5b421ec19c0 100644
> --- a/arch/x86/entry/entry.S
> +++ b/arch/x86/entry/entry.S
> @@ -21,7 +21,7 @@
>  SYM_FUNC_START(__write_ibpb)
>  	ANNOTATE_NOENDBR
>  	movl	$MSR_IA32_PRED_CMD, %ecx
> -	movl	$PRED_CMD_IBPB, %eax
> +	movl	_ASM_RIP(x86_pred_cmd), %eax
>  	xorl	%edx, %edx
>  	wrmsr
>  
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 310cb3f7139c..c8b8dc829046 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -58,7 +58,7 @@ EXPORT_SYMBOL_GPL(x86_spec_ctrl_base);
>  DEFINE_PER_CPU(u64, x86_spec_ctrl_current);
>  EXPORT_PER_CPU_SYMBOL_GPL(x86_spec_ctrl_current);
>  
> -u64 x86_pred_cmd __ro_after_init = PRED_CMD_IBPB;
> +u32 x86_pred_cmd __ro_after_init = PRED_CMD_IBPB;
>  EXPORT_SYMBOL_GPL(x86_pred_cmd);
>  
>  static u64 __ro_after_init x86_arch_cap_msr;

Re: [PATCH v3 2/6] x86/bugs: Use SBPB in __write_ibpb() if applicable

[Josh Poimboeuf]

On Wed, Apr 02, 2025 at 03:41:25PM -0500, Tom Lendacky wrote:
> On 4/2/25 13:19, Josh Poimboeuf wrote:
> > __write_ibpb() does IBPB, which (among other things) flushes branch type
> > predictions on AMD.  If the CPU has SRSO_NO, or if the SRSO mitigation
> > has been disabled, branch type flushing isn't needed, in which case the
> > lighter-weight SBPB can be used.
> 
> Maybe add something here that indicates the x86_pred_cmd variable tracks
> this optimization so switch to using that variable vs the hardcoded IBPB?

Indeed, adding a second paragraph to clarify that:

  x86/bugs: Use SBPB in write_ibpb() if applicable

  write_ibpb() does IBPB, which (among other things) flushes branch type
  predictions on AMD.  If the CPU has SRSO_NO, or if the SRSO mitigation
  has been disabled, branch type flushing isn't needed, in which case the
  lighter-weight SBPB can be used.

  The 'x86_pred_cmd' variable already keeps track of whether IBPB or SBPB
  should be used.  Use that instead of hardcoding IBPB.

-- 
Josh

Re: [PATCH v3 2/6] x86/bugs: Use SBPB in __write_ibpb() if applicable

[Jim Mattson]

On Wed, Apr 2, 2025 at 11:20 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> __write_ibpb() does IBPB, which (among other things) flushes branch type
> predictions on AMD.  If the CPU has SRSO_NO, or if the SRSO mitigation
> has been disabled, branch type flushing isn't needed, in which case the
> lighter-weight SBPB can be used.

When nested SVM is not supported, should KVM "promote"
SRSO_USER_KERNEL_NO on the host to SRSO_NO in KVM_GET_SUPPORTED_CPUID?
Or is a Linux guest clever enough to do the promotion itself if
CPUID.80000001H:ECX.SVM[bit 2] is clear?

Re: [PATCH v3 2/6] x86/bugs: Use SBPB in __write_ibpb() if applicable

[Josh Poimboeuf]

On Wed, Apr 02, 2025 at 02:04:04PM -0700, Jim Mattson wrote:
> On Wed, Apr 2, 2025 at 11:20 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> >
> > __write_ibpb() does IBPB, which (among other things) flushes branch type
> > predictions on AMD.  If the CPU has SRSO_NO, or if the SRSO mitigation
> > has been disabled, branch type flushing isn't needed, in which case the
> > lighter-weight SBPB can be used.
> 
> When nested SVM is not supported, should KVM "promote"
> SRSO_USER_KERNEL_NO on the host to SRSO_NO in KVM_GET_SUPPORTED_CPUID?
> Or is a Linux guest clever enough to do the promotion itself if
> CPUID.80000001H:ECX.SVM[bit 2] is clear?

I'm afraid that question is beyond my pay grade, maybe some AMD or virt
folks can chime in.

-- 
Josh

Re: [PATCH v3 2/6] x86/bugs: Use SBPB in __write_ibpb() if applicable

[Jim Mattson]

On Wed, Apr 2, 2025 at 7:18 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Wed, Apr 02, 2025 at 02:04:04PM -0700, Jim Mattson wrote:
> > On Wed, Apr 2, 2025 at 11:20 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> > >
> > > __write_ibpb() does IBPB, which (among other things) flushes branch type
> > > predictions on AMD.  If the CPU has SRSO_NO, or if the SRSO mitigation
> > > has been disabled, branch type flushing isn't needed, in which case the
> > > lighter-weight SBPB can be used.
> >
> > When nested SVM is not supported, should KVM "promote"
> > SRSO_USER_KERNEL_NO on the host to SRSO_NO in KVM_GET_SUPPORTED_CPUID?
> > Or is a Linux guest clever enough to do the promotion itself if
> > CPUID.80000001H:ECX.SVM[bit 2] is clear?
>
> I'm afraid that question is beyond my pay grade, maybe some AMD or virt
> folks can chime in.

That question aside, I'm not sure that this series is safe with
respect to nested virtualization.

If the CPU has SRSO_NO, then KVM will report SRSO_NO in
KVM_GET_SUPPORTED_CPUID. However, in nested virtualization, the L1
guest and the L2 guest share a prediction domain. KVM currently
ensures isolation between L1 and L2 with a call to
indirect_branch_prediction_barrier() in svm_vcpu_load(). I think that
particular barrier should *always* be a full IBPB--even if the host
has SRSO_NO.

RE: [PATCH v3 2/6] x86/bugs: Use SBPB in __write_ibpb() if applicable

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Jim Mattson <jmattson@google.com>
> Sent: Wednesday, April 9, 2025 11:07 AM
> To: Josh Poimboeuf <jpoimboe@kernel.org>
> Cc: x86@kernel.org; linux-kernel@vger.kernel.org; amit@kernel.org;
> kvm@vger.kernel.org; Shah, Amit <Amit.Shah@amd.com>; Lendacky, Thomas
> <Thomas.Lendacky@amd.com>; bp@alien8.de; tglx@linutronix.de;
> peterz@infradead.org; pawan.kumar.gupta@linux.intel.com; corbet@lwn.net;
> mingo@redhat.com; dave.hansen@linux.intel.com; hpa@zytor.com;
> seanjc@google.com; pbonzini@redhat.com; daniel.sneddon@linux.intel.com;
> kai.huang@intel.com; Das1, Sandipan <Sandipan.Das@amd.com>;
> boris.ostrovsky@oracle.com; Moger, Babu <Babu.Moger@amd.com>; Kaplan,
> David <David.Kaplan@amd.com>; dwmw@amazon.co.uk;
> andrew.cooper3@citrix.com
> Subject: Re: [PATCH v3 2/6] x86/bugs: Use SBPB in __write_ibpb() if applicable
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Wed, Apr 2, 2025 at 7:18 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> >
> > On Wed, Apr 02, 2025 at 02:04:04PM -0700, Jim Mattson wrote:
> > > On Wed, Apr 2, 2025 at 11:20 AM Josh Poimboeuf <jpoimboe@kernel.org>
> wrote:
> > > >
> > > > __write_ibpb() does IBPB, which (among other things) flushes
> > > > branch type predictions on AMD.  If the CPU has SRSO_NO, or if the
> > > > SRSO mitigation has been disabled, branch type flushing isn't
> > > > needed, in which case the lighter-weight SBPB can be used.
> > >
> > > When nested SVM is not supported, should KVM "promote"
> > > SRSO_USER_KERNEL_NO on the host to SRSO_NO in
> KVM_GET_SUPPORTED_CPUID?
> > > Or is a Linux guest clever enough to do the promotion itself if
> > > CPUID.80000001H:ECX.SVM[bit 2] is clear?
> >
> > I'm afraid that question is beyond my pay grade, maybe some AMD or
> > virt folks can chime in.
>
> That question aside, I'm not sure that this series is safe with respect to nested
> virtualization.
>
> If the CPU has SRSO_NO, then KVM will report SRSO_NO in
> KVM_GET_SUPPORTED_CPUID. However, in nested virtualization, the L1 guest
> and the L2 guest share a prediction domain. KVM currently ensures isolation
> between L1 and L2 with a call to
> indirect_branch_prediction_barrier() in svm_vcpu_load(). I think that particular
> barrier should *always* be a full IBPB--even if the host has SRSO_NO.

I don't think that's true.

If SRSO_NO=1, the indirect_branch_prediction_barrier() in svm_vcpu_load() I believe only needs to prevent indirect predictions from leaking from one VM to another, which is what SBPB provides.  Keep in mind that before SRSO came out, IBPB on these parts was only flushing indirect predictions.  SBPB become the 'legacy' IBPB functionality while IBPB turned into a full flush (on certain parts).  If the CPU is immune to SRSO, you don't need the full flush.

I also don't think promoting SRSO_USER_KERNEL_NO to SRSO_NO should ever be done.  When SRSO_NO=1, it tells the OS that it can use SBPB on context switches because the only process->process BTB concern is with indirect predictions.  The OS has to use IBPB if SRSO_NO=0 (regardless of SRSO_USER_KERNEL_NO) to prevent SRSO attacks from process->process.

--David Kaplan

Re: [PATCH v3 2/6] x86/bugs: Use SBPB in __write_ibpb() if applicable

[Jim Mattson]

On Wed, Apr 9, 2025 at 11:29 AM Kaplan, David <David.Kaplan@amd.com> wrote:
>
> [AMD Official Use Only - AMD Internal Distribution Only]
>
> > -----Original Message-----
> > From: Jim Mattson <jmattson@google.com>
> > Sent: Wednesday, April 9, 2025 11:07 AM
> > To: Josh Poimboeuf <jpoimboe@kernel.org>
> > Cc: x86@kernel.org; linux-kernel@vger.kernel.org; amit@kernel.org;
> > kvm@vger.kernel.org; Shah, Amit <Amit.Shah@amd.com>; Lendacky, Thomas
> > <Thomas.Lendacky@amd.com>; bp@alien8.de; tglx@linutronix.de;
> > peterz@infradead.org; pawan.kumar.gupta@linux.intel.com; corbet@lwn.net;
> > mingo@redhat.com; dave.hansen@linux.intel.com; hpa@zytor.com;
> > seanjc@google.com; pbonzini@redhat.com; daniel.sneddon@linux.intel.com;
> > kai.huang@intel.com; Das1, Sandipan <Sandipan.Das@amd.com>;
> > boris.ostrovsky@oracle.com; Moger, Babu <Babu.Moger@amd.com>; Kaplan,
> > David <David.Kaplan@amd.com>; dwmw@amazon.co.uk;
> > andrew.cooper3@citrix.com
> > Subject: Re: [PATCH v3 2/6] x86/bugs: Use SBPB in __write_ibpb() if applicable
> >
> > Caution: This message originated from an External Source. Use proper caution
> > when opening attachments, clicking links, or responding.
> >
> >
> > On Wed, Apr 2, 2025 at 7:18 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> > >
> > > On Wed, Apr 02, 2025 at 02:04:04PM -0700, Jim Mattson wrote:
> > > > On Wed, Apr 2, 2025 at 11:20 AM Josh Poimboeuf <jpoimboe@kernel.org>
> > wrote:
> > > > >
> > > > > __write_ibpb() does IBPB, which (among other things) flushes
> > > > > branch type predictions on AMD.  If the CPU has SRSO_NO, or if the
> > > > > SRSO mitigation has been disabled, branch type flushing isn't
> > > > > needed, in which case the lighter-weight SBPB can be used.
> > > >
> > > > When nested SVM is not supported, should KVM "promote"
> > > > SRSO_USER_KERNEL_NO on the host to SRSO_NO in
> > KVM_GET_SUPPORTED_CPUID?
> > > > Or is a Linux guest clever enough to do the promotion itself if
> > > > CPUID.80000001H:ECX.SVM[bit 2] is clear?
> > >
> > > I'm afraid that question is beyond my pay grade, maybe some AMD or
> > > virt folks can chime in.
> >
> > That question aside, I'm not sure that this series is safe with respect to nested
> > virtualization.
> >
> > If the CPU has SRSO_NO, then KVM will report SRSO_NO in
> > KVM_GET_SUPPORTED_CPUID. However, in nested virtualization, the L1 guest
> > and the L2 guest share a prediction domain. KVM currently ensures isolation
> > between L1 and L2 with a call to
> > indirect_branch_prediction_barrier() in svm_vcpu_load(). I think that particular
> > barrier should *always* be a full IBPB--even if the host has SRSO_NO.
>
> I don't think that's true.
>
> If SRSO_NO=1, the indirect_branch_prediction_barrier() in svm_vcpu_load() I believe only needs to prevent indirect predictions from leaking from one VM to another, which is what SBPB provides.  Keep in mind that before SRSO came out, IBPB on these parts was only flushing indirect predictions.  SBPB become the 'legacy' IBPB functionality while IBPB turned into a full flush (on certain parts).  If the CPU is immune to SRSO, you don't need the full flush.
>
> I also don't think promoting SRSO_USER_KERNEL_NO to SRSO_NO should ever be done.  When SRSO_NO=1, it tells the OS that it can use SBPB on context switches because the only process->process BTB concern is with indirect predictions.  The OS has to use IBPB if SRSO_NO=0 (regardless of SRSO_USER_KERNEL_NO) to prevent SRSO attacks from process->process.

Thanks, David!

Conceptually, it sounds like SRSO_NO = (SRSO_USER_KERNEL_NO +
SRSO_SAME_MODE_NO). You don't need an IBPB between L1 and L2 on
SRSO_NO parts, because SRSO_SAME_MODE_NO is implied. Similarly, you
can't "promote" SRSO_USER_KERNEL_NO to SRSO_NO, even absent SVM,
because SRSO_SAME_MODE_NO is missing.

[PATCH v3 3/6] x86/bugs: Fix RSB clearing in indirect_branch_prediction_barrier()

[Josh Poimboeuf]

IBPB is expected to clear the RSB.  However, if X86_BUG_IBPB_NO_RET is
set, that doesn't happen.  Make indirect_branch_prediction_barrier()
take that into account by calling __write_ibpb() which already does the
right thing.

Fixes: 50e4b3b94090 ("x86/entry: Have entry_ibpb() invalidate return predictions")
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 arch/x86/include/asm/nospec-branch.h | 6 +++---
 arch/x86/kernel/cpu/bugs.c           | 1 -
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
index bbac79cad04c..f99b32f014ec 100644
--- a/arch/x86/include/asm/nospec-branch.h
+++ b/arch/x86/include/asm/nospec-branch.h
@@ -514,11 +514,11 @@ void alternative_msr_write(unsigned int msr, u64 val, unsigned int feature)
 		: "memory");
 }
 
-extern u64 x86_pred_cmd;
-
 static inline void indirect_branch_prediction_barrier(void)
 {
-	alternative_msr_write(MSR_IA32_PRED_CMD, x86_pred_cmd, X86_FEATURE_IBPB);
+	asm_inline volatile(ALTERNATIVE("", "call __write_ibpb", X86_FEATURE_IBPB)
+			    : ASM_CALL_CONSTRAINT
+			    :: "rax", "rcx", "rdx", "memory");
 }
 
 /* The Intel SPEC CTRL MSR base value cache */
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index c8b8dc829046..9f9637cff7a3 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -59,7 +59,6 @@ DEFINE_PER_CPU(u64, x86_spec_ctrl_current);
 EXPORT_PER_CPU_SYMBOL_GPL(x86_spec_ctrl_current);
 
 u32 x86_pred_cmd __ro_after_init = PRED_CMD_IBPB;
-EXPORT_SYMBOL_GPL(x86_pred_cmd);
 
 static u64 __ro_after_init x86_arch_cap_msr;
 
-- 
2.48.1

Re: [PATCH v3 3/6] x86/bugs: Fix RSB clearing in indirect_branch_prediction_barrier()

[Nikolay Borisov]

On 2.04.25 г. 21:19 ч., Josh Poimboeuf wrote:
> IBPB is expected to clear the RSB.  However, if X86_BUG_IBPB_NO_RET is
> set, that doesn't happen.  Make indirect_branch_prediction_barrier()
> take that into account by calling __write_ibpb() which already does the
> right thing.

I find this changelog somewhat dubious. So zen < 4 basically have 
IBPB_NO_RET, your patch 2 in this series makes using SBPB for cores 
which have SRSO_NO or if the mitigation is disabled. So if you have a 
core which is zen <4 and doesn't use SBPB then what happens?

> 
> Fixes: 50e4b3b94090 ("x86/entry: Have entry_ibpb() invalidate return predictions")
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
> ---
>   arch/x86/include/asm/nospec-branch.h | 6 +++---
>   arch/x86/kernel/cpu/bugs.c           | 1 -
>   2 files changed, 3 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
> index bbac79cad04c..f99b32f014ec 100644
> --- a/arch/x86/include/asm/nospec-branch.h
> +++ b/arch/x86/include/asm/nospec-branch.h
> @@ -514,11 +514,11 @@ void alternative_msr_write(unsigned int msr, u64 val, unsigned int feature)
>   		: "memory");
>   }
>   
> -extern u64 x86_pred_cmd;
> -
>   static inline void indirect_branch_prediction_barrier(void)
>   {
> -	alternative_msr_write(MSR_IA32_PRED_CMD, x86_pred_cmd, X86_FEATURE_IBPB);
> +	asm_inline volatile(ALTERNATIVE("", "call __write_ibpb", X86_FEATURE_IBPB)
> +			    : ASM_CALL_CONSTRAINT
> +			    :: "rax", "rcx", "rdx", "memory");
>   }
>   
>   /* The Intel SPEC CTRL MSR base value cache */
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index c8b8dc829046..9f9637cff7a3 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -59,7 +59,6 @@ DEFINE_PER_CPU(u64, x86_spec_ctrl_current);
>   EXPORT_PER_CPU_SYMBOL_GPL(x86_spec_ctrl_current);
>   
>   u32 x86_pred_cmd __ro_after_init = PRED_CMD_IBPB;
> -EXPORT_SYMBOL_GPL(x86_pred_cmd);
>   
>   static u64 __ro_after_init x86_arch_cap_msr;
>   

Re: [PATCH v3 3/6] x86/bugs: Fix RSB clearing in indirect_branch_prediction_barrier()

[Josh Poimboeuf]

On Fri, Apr 04, 2025 at 05:45:37PM +0300, Nikolay Borisov wrote:
> 
> 
> On 2.04.25 г. 21:19 ч., Josh Poimboeuf wrote:
> > IBPB is expected to clear the RSB.  However, if X86_BUG_IBPB_NO_RET is
> > set, that doesn't happen.  Make indirect_branch_prediction_barrier()
> > take that into account by calling __write_ibpb() which already does the
> > right thing.
> 
> I find this changelog somewhat dubious. So zen < 4 basically have
> IBPB_NO_RET, your patch 2 in this series makes using SBPB for cores which
> have SRSO_NO or if the mitigation is disabled. So if you have a core which
> is zen <4 and doesn't use SBPB then what happens?

I'm afraid I don't understand the question.  In that case write_ibpb()
uses IBPB and manually clears the RSB.

-- 
Josh

Re: [PATCH v3 3/6] x86/bugs: Fix RSB clearing in indirect_branch_prediction_barrier()

[Nikolay Borisov]

On 4.04.25 г. 18:17 ч., Josh Poimboeuf wrote:
> On Fri, Apr 04, 2025 at 05:45:37PM +0300, Nikolay Borisov wrote:
>>
>>
>> On 2.04.25 г. 21:19 ч., Josh Poimboeuf wrote:
>>> IBPB is expected to clear the RSB.  However, if X86_BUG_IBPB_NO_RET is
>>> set, that doesn't happen.  Make indirect_branch_prediction_barrier()
>>> take that into account by calling __write_ibpb() which already does the
>>> right thing.
>>
>> I find this changelog somewhat dubious. So zen < 4 basically have
>> IBPB_NO_RET, your patch 2 in this series makes using SBPB for cores which
>> have SRSO_NO or if the mitigation is disabled. So if you have a core which
>> is zen <4 and doesn't use SBPB then what happens?
> 
> I'm afraid I don't understand the question.  In that case write_ibpb()
> uses IBPB and manually clears the RSB.
> 

Actually isn't this patch a noop. The old code simply wrote the value of 
x86_pred_cmd to the IA32-PRED_CMD register iff FEATURE_IBPB was set. So 
x86_pred_cmd might contain either PRED_CMD_IBPB or PRED_CMD_SBPB, 
meaning the correct value was written.

With your change you now call __write_ibpb() which does effectively the 
same thing.

Re: [PATCH v3 3/6] x86/bugs: Fix RSB clearing in indirect_branch_prediction_barrier()

[Josh Poimboeuf]

On Sat, Apr 05, 2025 at 01:56:59AM +0300, Nikolay Borisov wrote:
> On 4.04.25 г. 18:17 ч., Josh Poimboeuf wrote:
> > On Fri, Apr 04, 2025 at 05:45:37PM +0300, Nikolay Borisov wrote:
> > > 
> > > 
> > > On 2.04.25 г. 21:19 ч., Josh Poimboeuf wrote:
> > > > IBPB is expected to clear the RSB.  However, if X86_BUG_IBPB_NO_RET is
> > > > set, that doesn't happen.  Make indirect_branch_prediction_barrier()
> > > > take that into account by calling __write_ibpb() which already does the
> > > > right thing.
> > > 
> > > I find this changelog somewhat dubious. So zen < 4 basically have
> > > IBPB_NO_RET, your patch 2 in this series makes using SBPB for cores which
> > > have SRSO_NO or if the mitigation is disabled. So if you have a core which
> > > is zen <4 and doesn't use SBPB then what happens?
> > 
> > I'm afraid I don't understand the question.  In that case write_ibpb()
> > uses IBPB and manually clears the RSB.
> > 
> 
> Actually isn't this patch a noop. The old code simply wrote the value of
> x86_pred_cmd to the IA32-PRED_CMD register iff FEATURE_IBPB was set. So
> x86_pred_cmd might contain either PRED_CMD_IBPB or PRED_CMD_SBPB, meaning
> the correct value was written.
> 
> With your change you now call __write_ibpb() which does effectively the same
> thing.

Hm, are you getting SBPB and IBPB_NO_RET mixed up?  They're completely
separate and distinct:

  - SBPB is an AMD feature which is just like IBPB, except it doesn't
    flush branch type predictions.  It can be used when the SRSO
    mitigation isn't needed.  That was fixed by the previous patch.

  - AMD has a bug on older CPUs where IBPB doesn't flush the RSB.  Such
    CPUs have X86_BUG_IBPB_NO_RET set.  That's fixed with this patch due
    to the fact that write_ibpb() has this:

	/* Make sure IBPB clears return stack preductions too. */
	FILL_RETURN_BUFFER %rax, RSB_CLEAR_LOOPS, X86_BUG_IBPB_NO_RET

So you're right in that this patch doesn't change SBPB behavior.  But
that's not what it intends to do :-)

-- 
Josh

Re: [PATCH v3 3/6] x86/bugs: Fix RSB clearing in indirect_branch_prediction_barrier()

[Nikolay Borisov]

On 5.04.25 г. 3:56 ч., Josh Poimboeuf wrote:
> On Sat, Apr 05, 2025 at 01:56:59AM +0300, Nikolay Borisov wrote:
>> On 4.04.25 г. 18:17 ч., Josh Poimboeuf wrote:
>>> On Fri, Apr 04, 2025 at 05:45:37PM +0300, Nikolay Borisov wrote:
>>>>
>>>>
>>>> On 2.04.25 г. 21:19 ч., Josh Poimboeuf wrote:
>>>>> IBPB is expected to clear the RSB.  However, if X86_BUG_IBPB_NO_RET is
>>>>> set, that doesn't happen.  Make indirect_branch_prediction_barrier()
>>>>> take that into account by calling __write_ibpb() which already does the
>>>>> right thing.
>>>>
>>>> I find this changelog somewhat dubious. So zen < 4 basically have
>>>> IBPB_NO_RET, your patch 2 in this series makes using SBPB for cores which
>>>> have SRSO_NO or if the mitigation is disabled. So if you have a core which
>>>> is zen <4 and doesn't use SBPB then what happens?
>>>
>>> I'm afraid I don't understand the question.  In that case write_ibpb()
>>> uses IBPB and manually clears the RSB.
>>>
>>
>> Actually isn't this patch a noop. The old code simply wrote the value of
>> x86_pred_cmd to the IA32-PRED_CMD register iff FEATURE_IBPB was set. So
>> x86_pred_cmd might contain either PRED_CMD_IBPB or PRED_CMD_SBPB, meaning
>> the correct value was written.
>>
>> With your change you now call __write_ibpb() which does effectively the same
>> thing.
> 
> Hm, are you getting SBPB and IBPB_NO_RET mixed up?  They're completely
> separate and distinct:
> 
>    - SBPB is an AMD feature which is just like IBPB, except it doesn't
>      flush branch type predictions.  It can be used when the SRSO
>      mitigation isn't needed.  That was fixed by the previous patch.
> 
>    - AMD has a bug on older CPUs where IBPB doesn't flush the RSB.  Such
>      CPUs have X86_BUG_IBPB_NO_RET set.  That's fixed with this patch due
>      to the fact that write_ibpb() has this:
> 
> 	/* Make sure IBPB clears return stack preductions too. */
> 	FILL_RETURN_BUFFER %rax, RSB_CLEAR_LOOPS, X86_BUG_IBPB_NO_RET
> 
> So you're right in that this patch doesn't change SBPB behavior.  But
> that's not what it intends to do :-)

Fair enough, it just wasn't obvious from the diff context that there is 
the FILL_RETURN_BUFFER , my bad for not looking at __write_ibpb.

Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>

[PATCH v3 4/6] x86/bugs: Don't fill RSB on VMEXIT with eIBRS+retpoline

[Josh Poimboeuf]

eIBRS protects against guest->host RSB underflow/poisoning attacks.
Adding retpoline to the mix doesn't change that.  Retpoline has a
balanced CALL/RET anyway.

So the current full RSB filling on VMEXIT with eIBRS+retpoline is
overkill.  Disable it or do the VMEXIT_LITE mitigation if needed.

Suggested-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Reviewed-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Reviewed-by: Amit Shah <amit.shah@amd.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 arch/x86/kernel/cpu/bugs.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 9f9637cff7a3..354411fd4800 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1617,20 +1617,20 @@ static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_
 	case SPECTRE_V2_NONE:
 		return;
 
-	case SPECTRE_V2_EIBRS_LFENCE:
 	case SPECTRE_V2_EIBRS:
+	case SPECTRE_V2_EIBRS_LFENCE:
+	case SPECTRE_V2_EIBRS_RETPOLINE:
 		if (boot_cpu_has_bug(X86_BUG_EIBRS_PBRSB)) {
-			setup_force_cpu_cap(X86_FEATURE_RSB_VMEXIT_LITE);
 			pr_info("Spectre v2 / PBRSB-eIBRS: Retire a single CALL on VMEXIT\n");
+			setup_force_cpu_cap(X86_FEATURE_RSB_VMEXIT_LITE);
 		}
 		return;
 
-	case SPECTRE_V2_EIBRS_RETPOLINE:
 	case SPECTRE_V2_RETPOLINE:
 	case SPECTRE_V2_LFENCE:
 	case SPECTRE_V2_IBRS:
-		setup_force_cpu_cap(X86_FEATURE_RSB_VMEXIT);
 		pr_info("Spectre v2 / SpectreRSB : Filling RSB on VMEXIT\n");
+		setup_force_cpu_cap(X86_FEATURE_RSB_VMEXIT);
 		return;
 	}
 
-- 
2.48.1

Re: [PATCH v3 4/6] x86/bugs: Don't fill RSB on VMEXIT with eIBRS+retpoline

[Nikolay Borisov]

On 2.04.25 г. 21:19 ч., Josh Poimboeuf wrote:
> eIBRS protects against guest->host RSB underflow/poisoning attacks.
> Adding retpoline to the mix doesn't change that.  Retpoline has a
> balanced CALL/RET anyway.
> 
> So the current full RSB filling on VMEXIT with eIBRS+retpoline is
> overkill.  Disable it or do the VMEXIT_LITE mitigation if needed.
> 
> Suggested-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> Reviewed-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> Reviewed-by: Amit Shah <amit.shah@amd.com>
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>

Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>

[PATCH v3 5/6] x86/bugs: Don't fill RSB on context switch with eIBRS

[Josh Poimboeuf]

User->user Spectre v2 attacks (including RSB) across context switches
are already mitigated by IBPB in cond_mitigation(), if enabled globally
or if either the prev or the next task has opted in to protection.  RSB
filling without IBPB serves no purpose for protecting user space, as
indirect branches are still vulnerable.

User->kernel RSB attacks are mitigated by eIBRS.  In which case the RSB
filling on context switch isn't needed, so remove it.

Suggested-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Reviewed-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Reviewed-by: Amit Shah <amit.shah@amd.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 arch/x86/kernel/cpu/bugs.c | 24 ++++++++++++------------
 arch/x86/mm/tlb.c          |  6 +++---
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 354411fd4800..680c779e9711 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1591,7 +1591,7 @@ static void __init spec_ctrl_disable_kernel_rrsba(void)
 	rrsba_disabled = true;
 }
 
-static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_mitigation mode)
+static void __init spectre_v2_select_rsb_mitigation(enum spectre_v2_mitigation mode)
 {
 	/*
 	 * Similar to context switches, there are two types of RSB attacks
@@ -1615,7 +1615,7 @@ static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_
 	 */
 	switch (mode) {
 	case SPECTRE_V2_NONE:
-		return;
+		break;
 
 	case SPECTRE_V2_EIBRS:
 	case SPECTRE_V2_EIBRS_LFENCE:
@@ -1624,18 +1624,21 @@ static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_
 			pr_info("Spectre v2 / PBRSB-eIBRS: Retire a single CALL on VMEXIT\n");
 			setup_force_cpu_cap(X86_FEATURE_RSB_VMEXIT_LITE);
 		}
-		return;
+		break;
 
 	case SPECTRE_V2_RETPOLINE:
 	case SPECTRE_V2_LFENCE:
 	case SPECTRE_V2_IBRS:
-		pr_info("Spectre v2 / SpectreRSB : Filling RSB on VMEXIT\n");
+		pr_info("Spectre v2 / SpectreRSB: Filling RSB on context switch and VMEXIT\n");
+		setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW);
 		setup_force_cpu_cap(X86_FEATURE_RSB_VMEXIT);
-		return;
-	}
+		break;
 
-	pr_warn_once("Unknown Spectre v2 mode, disabling RSB mitigation at VM exit");
-	dump_stack();
+	default:
+		pr_warn_once("Unknown Spectre v2 mode, disabling RSB mitigation\n");
+		dump_stack();
+		break;
+	}
 }
 
 /*
@@ -1867,10 +1870,7 @@ static void __init spectre_v2_select_mitigation(void)
 	 *
 	 * FIXME: Is this pointless for retbleed-affected AMD?
 	 */
-	setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW);
-	pr_info("Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch\n");
-
-	spectre_v2_determine_rsb_fill_type_at_vmexit(mode);
+	spectre_v2_select_rsb_mitigation(mode);
 
 	/*
 	 * Retpoline protects the kernel, but doesn't protect firmware.  IBRS
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index e459d97ef397..eb83348f9305 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -667,9 +667,9 @@ static void cond_mitigation(struct task_struct *next)
 	prev_mm = this_cpu_read(cpu_tlbstate.last_user_mm_spec);
 
 	/*
-	 * Avoid user/user BTB poisoning by flushing the branch predictor
-	 * when switching between processes. This stops one process from
-	 * doing Spectre-v2 attacks on another.
+	 * Avoid user->user BTB/RSB poisoning by flushing them when switching
+	 * between processes. This stops one process from doing Spectre-v2
+	 * attacks on another.
 	 *
 	 * Both, the conditional and the always IBPB mode use the mm
 	 * pointer to avoid the IBPB when switching between tasks of the
-- 
2.48.1

Re: [PATCH v3 5/6] x86/bugs: Don't fill RSB on context switch with eIBRS

[Nikolay Borisov]

On 2.04.25 г. 21:19 ч., Josh Poimboeuf wrote:
> User->user Spectre v2 attacks (including RSB) across context switches
> are already mitigated by IBPB in cond_mitigation(), if enabled globally
> or if either the prev or the next task has opted in to protection.  RSB
> filling without IBPB serves no purpose for protecting user space, as
> indirect branches are still vulnerable.
> 
> User->kernel RSB attacks are mitigated by eIBRS.  In which case the RSB
> filling on context switch isn't needed, so remove it.
> 
> Suggested-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> Reviewed-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> Reviewed-by: Amit Shah <amit.shah@amd.com>
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>

Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>

[PATCH v3 6/6] x86/bugs: Add RSB mitigation document

[Josh Poimboeuf]

Create a document to summarize hard-earned knowledge about RSB-related
mitigations, with references, and replace the overly verbose yet
incomplete comments with a reference to the document.

Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 Documentation/admin-guide/hw-vuln/index.rst |   1 +
 Documentation/admin-guide/hw-vuln/rsb.rst   | 241 ++++++++++++++++++++
 arch/x86/kernel/cpu/bugs.c                  |  64 ++----
 3 files changed, 255 insertions(+), 51 deletions(-)
 create mode 100644 Documentation/admin-guide/hw-vuln/rsb.rst

diff --git a/Documentation/admin-guide/hw-vuln/index.rst b/Documentation/admin-guide/hw-vuln/index.rst
index ff0b440ef2dc..451874b8135d 100644
--- a/Documentation/admin-guide/hw-vuln/index.rst
+++ b/Documentation/admin-guide/hw-vuln/index.rst
@@ -22,3 +22,4 @@ are configurable at compile, boot or run time.
    srso
    gather_data_sampling
    reg-file-data-sampling
+   rsb
diff --git a/Documentation/admin-guide/hw-vuln/rsb.rst b/Documentation/admin-guide/hw-vuln/rsb.rst
new file mode 100644
index 000000000000..97bf75993d5d
--- /dev/null
+++ b/Documentation/admin-guide/hw-vuln/rsb.rst
@@ -0,0 +1,241 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=======================
+RSB-related mitigations
+=======================
+
+.. warning::
+   Please keep this document up-to-date, otherwise you will be
+   volunteered to update it and convert it to a very long comment in
+   bugs.c!
+
+Since 2018 there have been many Spectre CVEs related to the Return Stack
+Buffer (RSB).  Information about these CVEs and how to mitigate them is
+scattered amongst a myriad of microarchitecture-specific documents.
+
+This document attempts to consolidate all the relevant information in
+once place and clarify the reasoning behind the current RSB-related
+mitigations.
+
+At a high level, there are two classes of RSB attacks: RSB poisoning
+(Intel and AMD) and RSB underflow (Intel only).  They must each be
+considered individually for each attack vector (and microarchitecture
+where applicable).
+
+----
+
+RSB poisoning (Intel and AMD)
+=============================
+
+SpectreRSB
+~~~~~~~~~~
+
+RSB poisoning is a technique used by Spectre-RSB [#spectre-rsb]_ where
+an attacker poisons an RSB entry to cause a victim's return instruction
+to speculate to an attacker-controlled address.  This can happen when
+there are unbalanced CALLs/RETs after a context switch or VMEXIT.
+
+* All attack vectors can potentially be mitigated by flushing out any
+  poisoned RSB entries using an RSB filling sequence
+  [#intel-rsb-filling]_ [#amd-rsb-filling]_ when transitioning between
+  untrusted and trusted domains.  But this has a performance impact and
+  should be avoided whenever possible.
+
+* On context switch, the user->user mitigation requires ensuring the
+  RSB gets filled or cleared whenever IBPB gets written [#cond-ibpb]_
+  during a context switch:
+
+  * AMD:
+	IBPB (or SBPB [#amd-sbpb]_ if used) automatically clears the RSB
+	if IBPB_RET is set in CPUID [#amd-ibpb-rsb]_.  Otherwise the RSB
+	filling sequence [#amd-rsb-filling]_ must be always be done in
+	addition to IBPB.
+
+  * Intel:
+	IBPB automatically clears the RSB:
+
+	"Software that executed before the IBPB command cannot control
+	the predicted targets of indirect branches executed after the
+	command on the same logical processor. The term indirect branch
+	in this context includes near return instructions, so these
+	predicted targets may come from the RSB." [#intel-ibpb-rsb]_
+
+* On context switch, user->kernel attacks are mitigated by SMEP, as user
+  space can only insert its own return addresses into the RSB:
+
+  * AMD:
+	"Finally, branches that are predicted as 'ret' instructions get
+	their predicted targets from the Return Address Predictor (RAP).
+	AMD recommends software use a RAP stuffing sequence (mitigation
+	V2-3 in [2]) and/or Supervisor Mode Execution Protection (SMEP)
+	to ensure that the addresses in the RAP are safe for
+	speculation. Collectively, we refer to these mitigations as "RAP
+	Protection"." [#amd-smep-rsb]_
+
+  * Intel:
+	"On processors with enhanced IBRS, an RSB overwrite sequence may
+	not suffice to prevent the predicted target of a near return
+	from using an RSB entry created in a less privileged predictor
+	mode.  Software can prevent this by enabling SMEP (for
+	transitions from user mode to supervisor mode) and by having
+	IA32_SPEC_CTRL.IBRS set during VM exits." [#intel-smep-rsb]_
+
+* On VMEXIT, guest->host attacks are mitigated by eIBRS (and PBRSB
+  mitigation if needed):
+
+  * AMD:
+	"When Automatic IBRS is enabled, the internal return address
+	stack used for return address predictions is cleared on VMEXIT."
+	[#amd-eibrs-vmexit]_
+
+  * Intel:
+	"On processors with enhanced IBRS, an RSB overwrite sequence may
+	not suffice to prevent the predicted target of a near return
+	from using an RSB entry created in a less privileged predictor
+	mode.  Software can prevent this by enabling SMEP (for
+	transitions from user mode to supervisor mode) and by having
+	IA32_SPEC_CTRL.IBRS set during VM exits. Processors with
+	enhanced IBRS still support the usage model where IBRS is set
+	only in the OS/VMM for OSes that enable SMEP. To do this, such
+	processors will ensure that guest behavior cannot control the
+	RSB after a VM exit once IBRS is set, even if IBRS was not set
+	at the time of the VM exit." [#intel-eibrs-vmexit]_
+
+    Note that some Intel CPUs are susceptible to Post-barrier Return
+    Stack Buffer Predictions (PBRSB)[#intel-pbrsb]_, where the last CALL
+    from the guest can be used to predict the first unbalanced RET.  In
+    this case the PBRSB mitigation is needed in addition to eIBRS.
+
+AMD Retbleed / SRSO / Branch Type Confusion
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+On AMD, poisoned RSB entries can also be created by the AMD Retbleed
+variant [#retbleed-paper]_ and/or Speculative Return Stack Overflow
+[#amd-srso]_ (Inception [#inception-paper]_).  These attacks are made
+possible by Branch Type Confusion [#amd-btc]_.  The kernel protects
+itself by replacing every RET in the kernel with a branch to a single
+safe RET.
+
+----
+
+RSB underflow (Intel only)
+==========================
+
+Intel Retbleed
+~~~~~~~~~~~~~~
+
+Some Intel Skylake-generation CPUs are susceptible to the Intel variant
+of Retbleed [#retbleed-paper]_ (Return Stack Buffer Underflow
+[#intel-rsbu]_).  If a RET is executed when the RSB buffer is empty due
+to mismatched CALLs/RETs or returning from a deep call stack, the branch
+predictor can fall back to using the Branch Target Buffer (BTB).  If a
+user forces a BTB collision then the RET can speculatively branch to a
+user-controlled address.
+
+* Note that RSB filling doesn't fully mitigate this issue.  If there
+  are enough unbalanced RETs, the RSB may still underflow and fall back
+  to using a poisoned BTB entry.
+
+* On context switch, user->user underflow attacks are mitigated by the
+  conditional IBPB [#cond-ibpb]_ on context switch which clears the BTB:
+
+  * "The indirect branch predictor barrier (IBPB) is an indirect branch
+    control mechanism that establishes a barrier, preventing software
+    that executed before the barrier from controlling the predicted
+    targets of indirect branches executed after the barrier on the same
+    logical processor." [#intel-ibpb-btb]_
+
+    .. note::
+       I wasn't able to find any offical documentation from Intel
+       explicitly stating that IBPB clears the BTB.  However, it's
+       broadly known to be true and relied upon in several mitigations.
+
+* On context switch and VMEXIT, user->kernel and guest->host underflows
+  are mitigated by IBRS or eIBRS:
+
+  * "Enabling IBRS (including enhanced IBRS) will mitigate the "RSBU"
+    attack demonstrated by the researchers. As previously documented,
+    Intel recommends the use of enhanced IBRS, where supported. This
+    includes any processor that enumerates RRSBA but not RRSBA_DIS_S."
+    [#intel-rsbu]_
+
+  As an alternative to classic IBRS, call depth tracking can be used to
+  track kernel returns and fill the RSB when it gets close to being
+  empty.
+
+Restricted RSB Alternate (RRSBA)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Some newer Intel CPUs have Restricted RSB Alternate (RRSBA) behavior,
+which, similar to the Intel variant of Retbleed described above, also
+falls back to using the BTB on RSB underflow.  The only difference is
+that the predicted targets are restricted to the current domain.
+
+* "Restricted RSB Alternate (RRSBA) behavior allows alternate branch
+  predictors to be used by near RET instructions when the RSB is
+  empty.  When eIBRS is enabled, the predicted targets of these
+  alternate predictors are restricted to those belonging to the
+  indirect branch predictor entries of the current prediction domain.
+  [#intel-eibrs-rrsba]_
+
+When a CPU with RRSBA is vulnerable to Branch History Injection
+[#bhi-paper]_ [#intel-bhi]_, an RSB underflow could be used for an
+intra-mode BTI attack.  This is mitigated by clearing the BHB on
+kernel entry.
+
+However if the kernel uses retpolines instead of eIBRS, it needs to
+disable RRSBA:
+
+* "Where software is using retpoline as a mitigation for BHI or
+  intra-mode BTI, and the processor both enumerates RRSBA and
+  enumerates RRSBA_DIS controls, it should disable this behavior. "
+  [#intel-retpoline-rrsba]_
+
+----
+
+References
+==========
+
+.. [#spectre-rsb] `Spectre Returns! Speculation Attacks using the Return Stack Buffer <https://arxiv.org/pdf/1807.07940.pdf>`_
+
+.. [#intel-rsb-filling] "Empty RSB Mitigation on Skylake-generation" in `Retpoline: A Branch Target Injection Mitigation <https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/retpoline-branch-target-injection-mitigation.html#inpage-nav-5-1>`_
+
+.. [#amd-rsb-filling] "Mitigation V2-3" in `Software Techniques for Managing Speculation <https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/software-techniques-for-managing-speculation.pdf>`_
+
+.. [#cond-ibpb] Whether IBPB is written depends on whether the prev and/or next task is protected from Spectre attacks.  It typically requires opting in per task or system-wide.  For more details see the documentation for the ``spectre_v2_user`` cmdline option in Documentation/admin-guide/kernel-parameters.txt.
+
+.. [#amd-sbpb] IBPB without flushing of branch type predictions.  Only exists for AMD.
+
+.. [#amd-ibpb-rsb] "Function 8000_0008h -- Processor Capacity Parameters and Extended Feature Identification" in `AMD64 Architecture Programmer's Manual Volume 3: General-Purpose and System Instructions <https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/24594.pdf>`_.  SBPB behaves the same way according to `this email <https://lore.kernel.org/5175b163a3736ca5fd01cedf406735636c99a>`_.
+
+.. [#intel-ibpb-rsb] "Introduction" in `Post-barrier Return Stack Buffer Predictions / CVE-2022-26373 / INTEL-SA-00706 <https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/post-barrier-return-stack-buffer-predictions.html>`_
+
+.. [#amd-smep-rsb] "Existing Mitigations" in `Technical Guidance for Mitigating Branch Type Confusion <https://www.amd.com/content/dam/amd/en/documents/resources/technical-guidance-for-mitigating-branch-type-confusion.pdf>`_
+
+.. [#intel-smep-rsb] "Enhanced IBRS" in `Indirect Branch Restricted Speculation <https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/indirect-branch-restricted-speculation.html>`_
+
+.. [#amd-eibrs-vmexit] "Extended Feature Enable Register (EFER)" in `AMD64 Architecture Programmer's Manual Volume 2: System Programming <https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/24593.pdf>`_
+
+.. [#intel-eibrs-vmexit] "Enhanced IBRS" in `Indirect Branch Restricted Speculation <https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/indirect-branch-restricted-speculation.html>`_
+
+.. [#intel-pbrsb] `Post-barrier Return Stack Buffer Predictions / CVE-2022-26373 / INTEL-SA-00706 <https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/post-barrier-return-stack-buffer-predictions.html>`_
+
+.. [#retbleed-paper] `Retbleed: Arbitrary Speculative Code Execution with Return Instruction <https://comsec.ethz.ch/wp-content/files/retbleed_sec22.pdf>`_
+
+.. [#amd-btc] `Technical Guidance for Mitigating Branch Type Confusion <https://www.amd.com/content/dam/amd/en/documents/resources/technical-guidance-for-mitigating-branch-type-confusion.pdf>`_
+
+.. [#amd-srso] `Technical Update Regarding Speculative Return Stack Overflow <https://www.amd.com/content/dam/amd/en/documents/corporate/cr/speculative-return-stack-overflow-whitepaper.pdf>`_
+
+.. [#inception-paper] `Inception: Exposing New Attack Surfaces with Training in Transient Execution <https://comsec.ethz.ch/wp-content/files/inception_sec23.pdf>`_
+
+.. [#intel-rsbu] `Return Stack Buffer Underflow / Return Stack Buffer Underflow / CVE-2022-29901, CVE-2022-28693 / INTEL-SA-00702 <https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/return-stack-buffer-underflow.html>`_
+
+.. [#intel-ibpb-btb] `Indirect Branch Predictor Barrier' <https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/indirect-branch-predictor-barrier.html>`_
+
+.. [#intel-eibrs-rrsba] "Guidance for RSBU" in `Return Stack Buffer Underflow / Return Stack Buffer Underflow / CVE-2022-29901, CVE-2022-28693 / INTEL-SA-00702 <https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/return-stack-buffer-underflow.html>`_
+
+.. [#bhi-paper] `Branch History Injection: On the Effectiveness of Hardware Mitigations Against Cross-Privilege Spectre-v2 Attacks <http://download.vusec.net/papers/bhi-spectre-bhb_sec22.pdf>`_
+
+.. [#intel-bhi] `Branch History Injection and Intra-mode Branch Target Injection / CVE-2022-0001, CVE-2022-0002 / INTEL-SA-00598 <https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html>`_
+
+.. [#intel-retpoline-rrsba] "Retpoline" in `Branch History Injection and Intra-mode Branch Target Injection / CVE-2022-0001, CVE-2022-0002 / INTEL-SA-00598 <https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html>`_
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 680c779e9711..e78bb781c091 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1594,25 +1594,25 @@ static void __init spec_ctrl_disable_kernel_rrsba(void)
 static void __init spectre_v2_select_rsb_mitigation(enum spectre_v2_mitigation mode)
 {
 	/*
-	 * Similar to context switches, there are two types of RSB attacks
-	 * after VM exit:
+	 * WARNING! There are many subtleties to consider when changing *any*
+	 * code related to RSB-related mitigations.  Before doing so, carefully
+	 * read the following document, and update if necessary:
 	 *
-	 * 1) RSB underflow
+	 *   Documentation/admin-guide/hw-vuln/rsb.rst
 	 *
-	 * 2) Poisoned RSB entry
+	 * In an overly simplified nutshell:
 	 *
-	 * When retpoline is enabled, both are mitigated by filling/clearing
-	 * the RSB.
+	 *   - User->user RSB attacks are conditionally mitigated during
+	 *     context switch by cond_mitigation -> __write_ibpb().
 	 *
-	 * When IBRS is enabled, while #1 would be mitigated by the IBRS branch
-	 * prediction isolation protections, RSB still needs to be cleared
-	 * because of #2.  Note that SMEP provides no protection here, unlike
-	 * user-space-poisoned RSB entries.
+	 *   - User->kernel and guest->host attacks are mitigated by eIBRS or
+	 *     RSB filling.
 	 *
-	 * eIBRS should protect against RSB poisoning, but if the EIBRS_PBRSB
-	 * bug is present then a LITE version of RSB protection is required,
-	 * just a single call needs to retire before a RET is executed.
+	 *     Though, depending on config, note that other alternative
+	 *     mitigations may end up getting used instead, e.g., IBPB on
+	 *     entry/vmexit, call depth tracking, or return thunks.
 	 */
+
 	switch (mode) {
 	case SPECTRE_V2_NONE:
 		break;
@@ -1832,44 +1832,6 @@ static void __init spectre_v2_select_mitigation(void)
 	spectre_v2_enabled = mode;
 	pr_info("%s\n", spectre_v2_strings[mode]);
 
-	/*
-	 * If Spectre v2 protection has been enabled, fill the RSB during a
-	 * context switch.  In general there are two types of RSB attacks
-	 * across context switches, for which the CALLs/RETs may be unbalanced.
-	 *
-	 * 1) RSB underflow
-	 *
-	 *    Some Intel parts have "bottomless RSB".  When the RSB is empty,
-	 *    speculated return targets may come from the branch predictor,
-	 *    which could have a user-poisoned BTB or BHB entry.
-	 *
-	 *    AMD has it even worse: *all* returns are speculated from the BTB,
-	 *    regardless of the state of the RSB.
-	 *
-	 *    When IBRS or eIBRS is enabled, the "user -> kernel" attack
-	 *    scenario is mitigated by the IBRS branch prediction isolation
-	 *    properties, so the RSB buffer filling wouldn't be necessary to
-	 *    protect against this type of attack.
-	 *
-	 *    The "user -> user" attack scenario is mitigated by RSB filling.
-	 *
-	 * 2) Poisoned RSB entry
-	 *
-	 *    If the 'next' in-kernel return stack is shorter than 'prev',
-	 *    'next' could be tricked into speculating with a user-poisoned RSB
-	 *    entry.
-	 *
-	 *    The "user -> kernel" attack scenario is mitigated by SMEP and
-	 *    eIBRS.
-	 *
-	 *    The "user -> user" scenario, also known as SpectreBHB, requires
-	 *    RSB clearing.
-	 *
-	 * So to mitigate all cases, unconditionally fill RSB on context
-	 * switches.
-	 *
-	 * FIXME: Is this pointless for retbleed-affected AMD?
-	 */
 	spectre_v2_select_rsb_mitigation(mode);
 
 	/*
-- 
2.48.1

Re: [PATCH v3 6/6] x86/bugs: Add RSB mitigation document

[Ingo Molnar]

* Josh Poimboeuf <jpoimboe@kernel.org> wrote:

> Create a document to summarize hard-earned knowledge about RSB-related
> mitigations, with references, and replace the overly verbose yet
> incomplete comments with a reference to the document.

Just a few nits:

> +RSB poisoning (Intel and AMD)
> +=============================
> +
> +SpectreRSB
> +~~~~~~~~~~
>
>+
>+RSB poisoning is a technique used by Spectre-RSB [#spectre-rsb]_ where
>+an attacker poisons an RSB entry to cause a victim's return instruction
>+to speculate to an attacker-controlled address.  This can happen when
>+there are unbalanced CALLs/RETs after a context switch or VMEXIT.

s/Spectre-RSB
 /SpectreRSB

Which is the name the title just a few lines above uses, and which 
appears to be broadly the in-tree consensus spelling as well.

> +
> +AMD Retbleed / SRSO / Branch Type Confusion
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Nit: the underline is one character too long. :-)

> +On AMD, poisoned RSB entries can also be created by the AMD Retbleed
> +variant [#retbleed-paper]_ and/or Speculative Return Stack Overflow
> +[#amd-srso]_ (Inception [#inception-paper]_).  These attacks are made
> +possible by Branch Type Confusion [#amd-btc]_.  The kernel protects
> +itself by replacing every RET in the kernel with a branch to a single
> +safe RET.

s/Retbleed
 /RETBleed

Seems to be the consensus spelling in-tree. (There's a few more cases 
in this document as well.)

> +	 * WARNING! There are many subtleties to consider when changing *any*
> +	 * code related to RSB-related mitigations.  Before doing so, carefully
> +	 * read the following document, and update if necessary:
>  	 *
> +	 *   Documentation/admin-guide/hw-vuln/rsb.rst
>  	 *
> +	 * In an overly simplified nutshell:
>  	 *
> +	 *   - User->user RSB attacks are conditionally mitigated during
> +	 *     context switch by cond_mitigation -> __write_ibpb().

s/during context switch
 /during context switches

>  	 *
> +	 *   - User->kernel and guest->host attacks are mitigated by eIBRS or
> +	 *     RSB filling.
>  	 *
> +	 *     Though, depending on config, note that other alternative
> +	 *     mitigations may end up getting used instead, e.g., IBPB on
> +	 *     entry/vmexit, call depth tracking, or return thunks.
>  	 */

s/__write_ibpb()
 /write_ibpb()

as per the discussion under patch #1.

Thanks,

	Ingo

Re: [PATCH v3 6/6] x86/bugs: Add RSB mitigation document

[Josh Poimboeuf]

On Wed, Apr 02, 2025 at 09:58:59PM +0200, Ingo Molnar wrote:
> 
> * Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> 
> > Create a document to summarize hard-earned knowledge about RSB-related
> > mitigations, with references, and replace the overly verbose yet
> > incomplete comments with a reference to the document.
> 
> Just a few nits:

Ack, thanks!

-- 
Josh

Re: [PATCH v3 6/6] x86/bugs: Add RSB mitigation document

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 1407 bytes --]

On Wed, Apr 02, 2025 at 11:19:23AM -0700, Josh Poimboeuf wrote:
> +    Note that some Intel CPUs are susceptible to Post-barrier Return
> +    Stack Buffer Predictions (PBRSB)[#intel-pbrsb]_, where the last CALL
> +    from the guest can be used to predict the first unbalanced RET.  In
> +    this case the PBRSB mitigation is needed in addition to eIBRS.

I get Sphinx unreferenced footnote warning:

Documentation/admin-guide/hw-vuln/rsb.rst:221: WARNING: Footnote [#] is not referenced. [ref.footnote]

To fix that, I have to separate the footnote:

---- >8 ----
diff --git a/Documentation/admin-guide/hw-vuln/rsb.rst b/Documentation/admin-guide/hw-vuln/rsb.rst
index 97bf75993d5d43..dd727048b00204 100644
--- a/Documentation/admin-guide/hw-vuln/rsb.rst
+++ b/Documentation/admin-guide/hw-vuln/rsb.rst
@@ -102,7 +102,7 @@ there are unbalanced CALLs/RETs after a context switch or VMEXIT.
 	at the time of the VM exit." [#intel-eibrs-vmexit]_
 
     Note that some Intel CPUs are susceptible to Post-barrier Return
-    Stack Buffer Predictions (PBRSB)[#intel-pbrsb]_, where the last CALL
+    Stack Buffer Predictions (PBRSB) [#intel-pbrsb]_, where the last CALL
     from the guest can be used to predict the first unbalanced RET.  In
     this case the PBRSB mitigation is needed in addition to eIBRS.
 
Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v3 6/6] x86/bugs: Add RSB mitigation document

[Josh Poimboeuf]

On Thu, Apr 03, 2025 at 09:57:29AM +0700, Bagas Sanjaya wrote:
> On Wed, Apr 02, 2025 at 11:19:23AM -0700, Josh Poimboeuf wrote:
> > +    Note that some Intel CPUs are susceptible to Post-barrier Return
> > +    Stack Buffer Predictions (PBRSB)[#intel-pbrsb]_, where the last CALL
> > +    from the guest can be used to predict the first unbalanced RET.  In
> > +    this case the PBRSB mitigation is needed in addition to eIBRS.
> 
> I get Sphinx unreferenced footnote warning:
> 
> Documentation/admin-guide/hw-vuln/rsb.rst:221: WARNING: Footnote [#] is not referenced. [ref.footnote]
> 
> To fix that, I have to separate the footnote:
> 
> ---- >8 ----
> diff --git a/Documentation/admin-guide/hw-vuln/rsb.rst b/Documentation/admin-guide/hw-vuln/rsb.rst
> index 97bf75993d5d43..dd727048b00204 100644
> --- a/Documentation/admin-guide/hw-vuln/rsb.rst
> +++ b/Documentation/admin-guide/hw-vuln/rsb.rst
> @@ -102,7 +102,7 @@ there are unbalanced CALLs/RETs after a context switch or VMEXIT.
>  	at the time of the VM exit." [#intel-eibrs-vmexit]_
>  
>      Note that some Intel CPUs are susceptible to Post-barrier Return
> -    Stack Buffer Predictions (PBRSB)[#intel-pbrsb]_, where the last CALL
> +    Stack Buffer Predictions (PBRSB) [#intel-pbrsb]_, where the last CALL
>      from the guest can be used to predict the first unbalanced RET.  In
>      this case the PBRSB mitigation is needed in addition to eIBRS.

Fixed, thanks!

-- 
Josh

Re: [PATCH v3 6/6] x86/bugs: Add RSB mitigation document

[Andrew Cooper]

On 02/04/2025 7:19 pm, Josh Poimboeuf wrote:
> Create a document to summarize hard-earned knowledge about RSB-related
> mitigations, with references, and replace the overly verbose yet
> incomplete comments with a reference to the document.
>
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>

Excellent.  I've been trying to do something like this for a while, and
not managing to get this far.

> diff --git a/Documentation/admin-guide/hw-vuln/rsb.rst b/Documentation/admin-guide/hw-vuln/rsb.rst
> new file mode 100644
> index 000000000000..97bf75993d5d
> --- /dev/null
> +++ b/Documentation/admin-guide/hw-vuln/rsb.rst
> @@ -0,0 +1,241 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +=======================
> +RSB-related mitigations
> +=======================
> +
> +.. warning::
> +   Please keep this document up-to-date, otherwise you will be
> +   volunteered to update it and convert it to a very long comment in
> +   bugs.c!
> +
> +Since 2018 there have been many Spectre CVEs related to the Return Stack
> +Buffer (RSB).

2017.  I, and many others, spent the whole second half of 2017 tied up
in all of this.

I'd drop the Spectre, or swap it for Speculation-related.  Simply
"Spectre" CVEs tend to focus on the conditional and indirect predictors.

>   Information about these CVEs and how to mitigate them is
> +scattered amongst a myriad of microarchitecture-specific documents.

You should note that the AMD terms RAS and RAP are the same thing,
considering that you link to the documents.

> +
> +This document attempts to consolidate all the relevant information in
> +once place and clarify the reasoning behind the current RSB-related
> +mitigations.
> +
> +At a high level, there are two classes of RSB attacks: RSB poisoning
> +(Intel and AMD) and RSB underflow (Intel only).  They must each be
> +considered individually for each attack vector (and microarchitecture
> +where applicable).
> +

One question before getting further.  This seems to be focused on
(mis)prediction of RETs ?

That's fine, but it wants spelling out, because it is distinct from the
other class of issues when RET instructions execute in bad ways.

When lecturing, my go-to example is Spectre-v1.1 / BCBS (store which
clobbers or aliases the return address), because an astounding number of
things can go wrong in different ways from there.


Next, before diving into the specifics, it's incredibly relevant to have
a section briefly describing how the RSB typically works.  It's key to
understanding the rest of the documents, and there will definitely be
people reading the document who don't know it.

The salient points are (on CPUs since ~Nehalem era.  Confirm with Intel
and AMD.  You can spot it in the optimisation guides, because it's where
the phrase such as "only taken branches consume prediction resource"
began appearing):

* Branch prediction is **prior** to instruction decode, and guesses at
the location, type, and target of all near branches.
* The RSB is a prediction structure used by branches predicted as CALLs
or RETs.
* When a CALL is predicted, the predicted return-address is pushed on
the RSB
* When a RET is predicted, the RSB is popped
* Later, decode will cross-check the prediction with the instruction
stream.  It can issue corrections to the predictor state, and restart
prediction/fetch from the point that things appeared to go wrong.  This
can include editing transient state in the RSB.

For the observant reader, yes, the RSB is filled using predicted
targets.  This is why the SRSO vuln is so evil.

So, with the behaviour summarised, next some properties (disclaimer:
varies by vendor)
* It is logically a stack, but has finite capacity.  Executing more RET
instructions than CALLs will underflow it.
** AMD reuses the -1'th entry and doesn't move the pointer
** Intel may fall back to a prediction from a different predictor
* It is a structure shared across all security domains in Core/Thread. 
Guest & Host is most relevant to the doc, but SMM/ACM/SEAM/XuCode are
all included.
** Some AMD CPUs dynamically re-partition the RSB(RAS) when a sibling
thread goes idle
** Some Intel CPUs only have a 32-bit wide RSB, and reuse the upper bits
of the location for the predicted target
** Some Intel CPUs hardwire bit 47(?) which causes the kernel to follow
userspace predictions.

> +----
> +
> +RSB poisoning (Intel and AMD)
> +=============================
> +
> +SpectreRSB
> +~~~~~~~~~~
> +
> +RSB poisoning is a technique used by Spectre-RSB [#spectre-rsb]_ where
> +an attacker poisons an RSB entry to cause a victim's return instruction
> +to speculate to an attacker-controlled address.  This can happen when
> +there are unbalanced CALLs/RETs after a context switch or VMEXIT.
> +
> +* All attack vectors can potentially be mitigated by flushing out any
> +  poisoned RSB entries using an RSB filling sequence
> +  [#intel-rsb-filling]_ [#amd-rsb-filling]_ when transitioning between
> +  untrusted and trusted domains.  But this has a performance impact and
> +  should be avoided whenever possible.

More importantly, 32-entry RSB stuffing loops are only applicable to
pre-eIBRS and pre-ERAPS hardware.

They are known unsafe to use on newer microarchitectures, inc Gracemont
(128 entries) and Zen5 (64 entries).

> +
> +* On context switch, the user->user mitigation requires ensuring the
> +  RSB gets filled or cleared whenever IBPB gets written [#cond-ibpb]_
> +  during a context switch:
> +
> +  * AMD:
> +	IBPB (or SBPB [#amd-sbpb]_ if used) automatically clears the RSB
> +	if IBPB_RET is set in CPUID [#amd-ibpb-rsb]_.  Otherwise the RSB
> +	filling sequence [#amd-rsb-filling]_ must be always be done in
> +	addition to IBPB.

Honestly, I dislike this way of characterising it.   IBPB was very
clearly spec'd to flush all indirect prediction structures, and some AMD
CPUs have an errata where this isn't true and has to be filled in by the OS.

> +
> +  * Intel:
> +	IBPB automatically clears the RSB:
> +
> +	"Software that executed before the IBPB command cannot control
> +	the predicted targets of indirect branches executed after the
> +	command on the same logical processor. The term indirect branch
> +	in this context includes near return instructions, so these
> +	predicted targets may come from the RSB." [#intel-ibpb-rsb]_
> +
> +* On context switch, user->kernel attacks are mitigated by SMEP, as user
> +  space can only insert its own return addresses into the RSB:

It's more subtle than this (see the 32-bit wide prediction).

A user/supervisor split address space limits the ranges of addresses
that userspace can insert into the predictor.

There is a corner case at the canonical boundary.  Userspace can insert
the first non-canonincal address, and on some CPUs, this is interpreted
as the first high canonical address.  Guard pages on both sides of the
canonical boundary mitigate this.

In the unbalanced case for user->kernel, a bad prediction really is made
(coming out of the RSB), and it's only the SMEP #PF at instruction fetch
which prevents you speculatively executing in userspace.

In the 32-bit width case, the kernel predicts to {high_kern:low_user}
target.


> +
> +  * AMD:
> +	"Finally, branches that are predicted as 'ret' instructions get
> +	their predicted targets from the Return Address Predictor (RAP).
> +	AMD recommends software use a RAP stuffing sequence (mitigation
> +	V2-3 in [2]) and/or Supervisor Mode Execution Protection (SMEP)
> +	to ensure that the addresses in the RAP are safe for
> +	speculation. Collectively, we refer to these mitigations as "RAP
> +	Protection"." [#amd-smep-rsb]_
> +
> +  * Intel:
> +	"On processors with enhanced IBRS, an RSB overwrite sequence may
> +	not suffice to prevent the predicted target of a near return
> +	from using an RSB entry created in a less privileged predictor
> +	mode.  Software can prevent this by enabling SMEP (for
> +	transitions from user mode to supervisor mode) and by having
> +	IA32_SPEC_CTRL.IBRS set during VM exits." [#intel-smep-rsb]_
> +
> +* On VMEXIT, guest->host attacks are mitigated by eIBRS (and PBRSB
> +  mitigation if needed):
> +
> +  * AMD:
> +	"When Automatic IBRS is enabled, the internal return address
> +	stack used for return address predictions is cleared on VMEXIT."
> +	[#amd-eibrs-vmexit]_
> +
> +  * Intel:
> +	"On processors with enhanced IBRS, an RSB overwrite sequence may
> +	not suffice to prevent the predicted target of a near return
> +	from using an RSB entry created in a less privileged predictor
> +	mode.  Software can prevent this by enabling SMEP (for
> +	transitions from user mode to supervisor mode) and by having
> +	IA32_SPEC_CTRL.IBRS set during VM exits. Processors with
> +	enhanced IBRS still support the usage model where IBRS is set
> +	only in the OS/VMM for OSes that enable SMEP. To do this, such
> +	processors will ensure that guest behavior cannot control the
> +	RSB after a VM exit once IBRS is set, even if IBRS was not set
> +	at the time of the VM exit." [#intel-eibrs-vmexit]_
> +
> +    Note that some Intel CPUs are susceptible to Post-barrier Return
> +    Stack Buffer Predictions (PBRSB)[#intel-pbrsb]_, where the last CALL
> +    from the guest can be used to predict the first unbalanced RET.  In
> +    this case the PBRSB mitigation is needed in addition to eIBRS.
> +
> +AMD Retbleed / SRSO / Branch Type Confusion
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +On AMD, poisoned RSB entries can also be created by the AMD Retbleed
> +variant [#retbleed-paper]_ and/or Speculative Return Stack Overflow
> +[#amd-srso]_ (Inception [#inception-paper]_).  These attacks are made
> +possible by Branch Type Confusion [#amd-btc]_.  The kernel protects
> +itself by replacing every RET in the kernel with a branch to a single
> +safe RET.

BTC and SRSO are unrelated things.

"predicted branch types" is an inherent property of the predictor.  BTC
is specifically the case where decode doesn't halt, and still issues the
wrong uops into the pipeline to execute.

SRSO goes entirely wrong at the BP/IF stages (BP racing ahead while IF
is stalled unable to fetch anything), so the damage is done by the time
Decode sees the instruction stream.


You also need to cover the AMD going-idle issue where the other thread's
RSB entries magically appear in my RSB, and my head/tail pointer is
reset to a random position.

> +
> +----
> +
> +RSB underflow (Intel only)
> +==========================

Well, not really.  AMD can underflow too.  It just picks a fixed entry
and keeps on reusing that.  (Great for the alarming number of
programming languages which consider recursion a virtue.)

> +
> +Intel Retbleed
> +~~~~~~~~~~~~~~
> +
> +Some Intel Skylake-generation CPUs are susceptible to the Intel variant
> +of Retbleed [#retbleed-paper]_ (Return Stack Buffer Underflow
> +[#intel-rsbu]_).  If a RET is executed when the RSB buffer is empty due
> +to mismatched CALLs/RETs or returning from a deep call stack, the branch
> +predictor can fall back to using the Branch Target Buffer (BTB).  If a
> +user forces a BTB collision then the RET can speculatively branch to a
> +user-controlled address.
> +
> +* Note that RSB filling doesn't fully mitigate this issue.  If there
> +  are enough unbalanced RETs, the RSB may still underflow and fall back
> +  to using a poisoned BTB entry.
> +
> +* On context switch, user->user underflow attacks are mitigated by the
> +  conditional IBPB [#cond-ibpb]_ on context switch which clears the BTB:
> +
> +  * "The indirect branch predictor barrier (IBPB) is an indirect branch
> +    control mechanism that establishes a barrier, preventing software
> +    that executed before the barrier from controlling the predicted
> +    targets of indirect branches executed after the barrier on the same
> +    logical processor." [#intel-ibpb-btb]_
> +
> +    .. note::
> +       I wasn't able to find any offical documentation from Intel
> +       explicitly stating that IBPB clears the BTB.  However, it's
> +       broadly known to be true and relied upon in several mitigations.

Part of this is because when the vendors say the BTB, they're
translating their internal names into what academia calls them.

"Flush the BTB" isn't a helpful statement anyway.  See AMD's IBPB vs
SBPB which controls whether the branch types predictions remain intact.

Given how many rounds of Intel microcode there have been making IBPB
scrub more, it clearly wasn't scrubbing everything in the first place.

> +
> +* On context switch and VMEXIT, user->kernel and guest->host underflows
> +  are mitigated by IBRS or eIBRS:
> +
> +  * "Enabling IBRS (including enhanced IBRS) will mitigate the "RSBU"
> +    attack demonstrated by the researchers.

Yeah, except it doesn't.  Intra-mode BTI is a thing, and that will leak
your secrets too.

>  As previously documented,
> +    Intel recommends the use of enhanced IBRS, where supported. This
> +    includes any processor that enumerates RRSBA but not RRSBA_DIS_S."
> +    [#intel-rsbu]_
> +
> +  As an alternative to classic IBRS, call depth tracking can be used to

legacy IBRS.  Please don't invent yet another term for it :), and
"classic" further implies there might be a time when anyone looks back
fondly on it.

> +  track kernel returns and fill the RSB when it gets close to being
> +  empty.
> +
> +Restricted RSB Alternate (RRSBA)
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This needs to discuss RSBA and RRSBA.  They're distinct.  To a first
approximation, RRSBA == RSBA + eIBRS.  The "restricted" nature is
"mode-tagged predictions".

Except there's a narrow range of CPUs around Icelake which have eIBRS
and do not suffer (R)RSBA.

> +
> +Some newer Intel CPUs have Restricted RSB Alternate (RRSBA) behavior,
> +which, similar to the Intel variant of Retbleed described above, also
> +falls back to using the BTB on RSB underflow.  The only difference is
> +that the predicted targets are restricted to the current domain.
> +
> +* "Restricted RSB Alternate (RRSBA) behavior allows alternate branch
> +  predictors to be used by near RET instructions when the RSB is
> +  empty.  When eIBRS is enabled, the predicted targets of these
> +  alternate predictors are restricted to those belonging to the
> +  indirect branch predictor entries of the current prediction domain.
> +  [#intel-eibrs-rrsba]_
> +
> +When a CPU with RRSBA is vulnerable to Branch History Injection
> +[#bhi-paper]_ [#intel-bhi]_, an RSB underflow could be used for an
> +intra-mode BTI attack.  This is mitigated by clearing the BHB on
> +kernel entry.
> +
> +However if the kernel uses retpolines instead of eIBRS, it needs to
> +disable RRSBA:
> +
> +* "Where software is using retpoline as a mitigation for BHI or
> +  intra-mode BTI, and the processor both enumerates RRSBA and
> +  enumerates RRSBA_DIS controls, it should disable this behavior. "
> +  [#intel-retpoline-rrsba]_

IIRC, not all CPUs which suffer RRSBA have the RRSBA_DIS_* controls.

But, I think that's enough nitpicking for now.  I really think some
Intel and AMD architects ought to weigh in too.

~Andrew

Re: [PATCH v3 6/6] x86/bugs: Add RSB mitigation document

[Josh Poimboeuf]

On Fri, Apr 04, 2025 at 03:39:32AM +0100, Andrew Cooper wrote:
> On 02/04/2025 7:19 pm, Josh Poimboeuf wrote:
> > +Since 2018 there have been many Spectre CVEs related to the Return Stack
> > +Buffer (RSB).
> 
> 2017.  I, and many others, spent the whole second half of 2017 tied up
> in all of this.

Right.  Personally I got pulled in in October 2017.  Even Skylake RSB
underflow was already well understood, which led to the whole distro
IBRS vs upstream retpoline divergence debacle.

But with respect to the above sentence, the first Spectre CVEs were
disclosed in 2018.

> I'd drop the Spectre, or swap it for Speculation-related.  Simply
> "Spectre" CVEs tend to focus on the conditional and indirect predictors.

As you are well aware, a RET is basically an indirect branch.
SpectreRSB literally has Spectre in the name.  And rightfully so IMO,
it's just another way to exploit CPU branch prediction.  And the same
CPU data structures are involved.

So from my perspective, Spectre refers to the entire class of branch
prediction exploits.

But I can change it if you think that's incorrect.

> >   Information about these CVEs and how to mitigate them is
> > +scattered amongst a myriad of microarchitecture-specific documents.
> 
> You should note that the AMD terms RAS and RAP are the same thing,
> considering that you link to the documents.

Ack.  What does RAS stand for?

> One question before getting further.  This seems to be focused on
> (mis)prediction of RETs ?
> 
> That's fine, but it wants spelling out, because it is distinct from the
> other class of issues when RET instructions execute in bad ways.
> 
> When lecturing, my go-to example is Spectre-v1.1 / BCBS (store which
> clobbers or aliases the return address), because an astounding number of
> things can go wrong in different ways from there.

The document is strictly related to the RSB mitigations.

> Next, before diving into the specifics, it's incredibly relevant to have
> a section briefly describing how the RSB typically works.  It's key to
> understanding the rest of the documents, and there will definitely be
> people reading the document who don't know it.
> 
> The salient points are (on CPUs since ~Nehalem era.  Confirm with Intel
> and AMD.  You can spot it in the optimisation guides, because it's where
> the phrase such as "only taken branches consume prediction resource"
> began appearing):
> 
> * Branch prediction is **prior** to instruction decode, and guesses at
> the location, type, and target of all near branches.
> * The RSB is a prediction structure used by branches predicted as CALLs
> or RETs.
> * When a CALL is predicted, the predicted return-address is pushed on
> the RSB
> * When a RET is predicted, the RSB is popped
> * Later, decode will cross-check the prediction with the instruction
> stream.  It can issue corrections to the predictor state, and restart
> prediction/fetch from the point that things appeared to go wrong.  This
> can include editing transient state in the RSB.
> 
> For the observant reader, yes, the RSB is filled using predicted
> targets.  This is why the SRSO vuln is so evil.
> 
> So, with the behaviour summarised, next some properties (disclaimer:
> varies by vendor)
> * It is logically a stack, but has finite capacity.  Executing more RET
> instructions than CALLs will underflow it.
> ** AMD reuses the -1'th entry and doesn't move the pointer
> ** Intel may fall back to a prediction from a different predictor
> * It is a structure shared across all security domains in Core/Thread. 
> Guest & Host is most relevant to the doc, but SMM/ACM/SEAM/XuCode are
> all included.
> ** Some AMD CPUs dynamically re-partition the RSB(RAS) when a sibling
> thread goes idle
> ** Some Intel CPUs only have a 32-bit wide RSB, and reuse the upper bits
> of the location for the predicted target
> ** Some Intel CPUs hardwire bit 47(?) which causes the kernel to follow
> userspace predictions.

That's some really good information.  However, I'm not sure gritty
details about how RSBs work are within the scope for this document.

My goal is for it to be as concise as possible, focused only on the
current kernel mitigations: what are the RSB-related attack vectors for
each uarch and how are we currently mitigating those?

It's basically a glorified comment, but too long to actually be one.  So
that when the next CVE comes along, one can quickly refer to this as a
refresher to see what we're already doing and why.

If the reader wants to know details about the RSB itself, or the related
exploits, there are plenty of references for them to dive into.

> > +* All attack vectors can potentially be mitigated by flushing out any
> > +  poisoned RSB entries using an RSB filling sequence
> > +  [#intel-rsb-filling]_ [#amd-rsb-filling]_ when transitioning between
> > +  untrusted and trusted domains.  But this has a performance impact and
> > +  should be avoided whenever possible.
> 
> More importantly, 32-entry RSB stuffing loops are only applicable to
> pre-eIBRS and pre-ERAPS hardware.
> 
> They are known unsafe to use on newer microarchitectures, inc Gracemont
> (128 entries) and Zen5 (64 entries).

Right now we're doing 32 entries regardless of CPU model.  Do we need to
increase the loop count for newer HW?  Have any references handy?

> > +  * AMD:
> > +	IBPB (or SBPB [#amd-sbpb]_ if used) automatically clears the RSB
> > +	if IBPB_RET is set in CPUID [#amd-ibpb-rsb]_.  Otherwise the RSB
> > +	filling sequence [#amd-rsb-filling]_ must be always be done in
> > +	addition to IBPB.
> 
> Honestly, I dislike this way of characterising it.   IBPB was very
> clearly spec'd to flush all indirect prediction structures, and some AMD
> CPUs have an errata where this isn't true and has to be filled in by the OS.

I don't see how that statement characterizes anything, it just describes
the relevant facts as concisely as possible.

> > +* On context switch, user->kernel attacks are mitigated by SMEP, as user
> > +  space can only insert its own return addresses into the RSB:
> 
> It's more subtle than this (see the 32-bit wide prediction).
> 
> A user/supervisor split address space limits the ranges of addresses
> that userspace can insert into the predictor.
> 
> There is a corner case at the canonical boundary.  Userspace can insert
> the first non-canonincal address, and on some CPUs, this is interpreted
> as the first high canonical address.  Guard pages on both sides of the
> canonical boundary mitigate this.

Ack.  Have a reference handy?

> In the unbalanced case for user->kernel, a bad prediction really is made
> (coming out of the RSB), and it's only the SMEP #PF at instruction fetch
> which prevents you speculatively executing in userspace.

Right, I should probably clarify that a bit more.

> In the 32-bit width case, the kernel predicts to {high_kern:low_user}
> target.

Ok.  But blocked by SMEP regardless.

> > +On AMD, poisoned RSB entries can also be created by the AMD Retbleed
> > +variant [#retbleed-paper]_ and/or Speculative Return Stack Overflow
> > +[#amd-srso]_ (Inception [#inception-paper]_).  These attacks are made
> > +possible by Branch Type Confusion [#amd-btc]_.  The kernel protects
> > +itself by replacing every RET in the kernel with a branch to a single
> > +safe RET.
> 
> BTC and SRSO are unrelated things.
> 
> "predicted branch types" is an inherent property of the predictor.  BTC
> is specifically the case where decode doesn't halt, and still issues the
> wrong uops into the pipeline to execute.
> 
> SRSO goes entirely wrong at the BP/IF stages (BP racing ahead while IF
> is stalled unable to fetch anything), so the damage is done by the time
> Decode sees the instruction stream.

As a simple software engineer, I was under the impression that SRSO is a
special case of BTC.  Apparently I was wrong :-)

> You also need to cover the AMD going-idle issue where the other thread's
> RSB entries magically appear in my RSB, and my head/tail pointer is
> reset to a random position.

I'm not familiar with that one, do you have a reference?

> > +RSB underflow (Intel only)
> > +==========================
> 
> Well, not really.  AMD can underflow too.  It just picks a fixed entry
> and keeps on reusing that.  (Great for the alarming number of
> programming languages which consider recursion a virtue.)

I'm not sure that's relevant here since there are no mitigations needed
for that?

> > +* On context switch, user->user underflow attacks are mitigated by the
> > +  conditional IBPB [#cond-ibpb]_ on context switch which clears the BTB:
> > +
> > +  * "The indirect branch predictor barrier (IBPB) is an indirect branch
> > +    control mechanism that establishes a barrier, preventing software
> > +    that executed before the barrier from controlling the predicted
> > +    targets of indirect branches executed after the barrier on the same
> > +    logical processor." [#intel-ibpb-btb]_
> > +
> > +    .. note::
> > +       I wasn't able to find any offical documentation from Intel
> > +       explicitly stating that IBPB clears the BTB.  However, it's
> > +       broadly known to be true and relied upon in several mitigations.
> 
> Part of this is because when the vendors say the BTB, they're
> translating their internal names into what academia calls them.
> 
> "Flush the BTB" isn't a helpful statement anyway.  See AMD's IBPB vs
> SBPB which controls whether the branch types predictions remain intact.
> 
> Given how many rounds of Intel microcode there have been making IBPB
> scrub more, it clearly wasn't scrubbing everything in the first place.

Any suggestions for how I can improve the wording here?  I want to
explain why IBPB protects against RSB underflow and how we know that.

I should probably also mention IBPB_RET here as well.

> > +* On context switch and VMEXIT, user->kernel and guest->host underflows
> > +  are mitigated by IBRS or eIBRS:
> > +
> > +  * "Enabling IBRS (including enhanced IBRS) will mitigate the "RSBU"
> > +    attack demonstrated by the researchers.
> 
> Yeah, except it doesn't.  Intra-mode BTI is a thing, and that will leak
> your secrets too.

Ok, I'll add a mention of that.

> >  As previously documented,
> > +    Intel recommends the use of enhanced IBRS, where supported. This
> > +    includes any processor that enumerates RRSBA but not RRSBA_DIS_S."
> > +    [#intel-rsbu]_
> > +
> > +  As an alternative to classic IBRS, call depth tracking can be used to
> 
> legacy IBRS.  Please don't invent yet another term for it :), and
> "classic" further implies there might be a time when anyone looks back
> fondly on it.

:-)

> > +Restricted RSB Alternate (RRSBA)
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> This needs to discuss RSBA and RRSBA.  They're distinct.  To a first
> approximation, RRSBA == RSBA + eIBRS.  The "restricted" nature is
> "mode-tagged predictions".

RSBA is the original Intel Retbleed, right?  That was discussed
extensively in the preceding section.  I can mention that Intel Retbleed
may also be referred to as RSBA.

> > +However if the kernel uses retpolines instead of eIBRS, it needs to
> > +disable RRSBA:
> > +
> > +* "Where software is using retpoline as a mitigation for BHI or
> > +  intra-mode BTI, and the processor both enumerates RRSBA and
> > +  enumerates RRSBA_DIS controls, it should disable this behavior. "
> > +  [#intel-retpoline-rrsba]_
> 
> IIRC, not all CPUs which suffer RRSBA have the RRSBA_DIS_* controls.

Hm, have a reference for that?

-- 
Josh