[PATCH] x86/speculation, KVM: respect user IBPB configuration

[Jon Kohler <jon@nutanix.com>]

On vmx_vcpu_load_vmcs and svm_vcpu_load, respect user IBPB config and only
attempt IBPB MSR if either always_ibpb or cond_ibpb and the vcpu thread
has TIF_SPEC_IB.

A vcpu thread will have TIF_SPEC_IB on qemu-kvm using -sandbox on if
kernel cmdline spectre_v2_user=seccomp, which would indicate that the user
is looking for a higher security environment and has workloads that need
to be secured from each other.

Note: The behavior of spectre_v2_user recently changed in 5.16 on
commit 2f46993d83ff ("x86: change default to
spec_store_bypass_disable=prctl spectre_v2_user=prctl")

Prior to that, qemu-kvm with -sandbox on would also have TIF_SPEC_IB 
if spectre_v2_user=auto.

Signed-off-by: Jon Kohler <jon@nutanix.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Waiman Long <longman@redhat.com>
---
 arch/x86/include/asm/spec-ctrl.h | 12 ++++++++++++
 arch/x86/kernel/cpu/bugs.c       |  6 ++++--
 arch/x86/kvm/svm/svm.c           |  2 +-
 arch/x86/kvm/vmx/vmx.c           |  2 +-
 4 files changed, 18 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/spec-ctrl.h b/arch/x86/include/asm/spec-ctrl.h
index 5393babc0598..552757847d5b 100644
--- a/arch/x86/include/asm/spec-ctrl.h
+++ b/arch/x86/include/asm/spec-ctrl.h
@@ -85,4 +85,16 @@ static inline void speculative_store_bypass_ht_init(void) { }
 extern void speculation_ctrl_update(unsigned long tif);
 extern void speculation_ctrl_update_current(void);
 
+/*
+ * Always issue IBPB if switch_mm_always_ibpb and respect conditional
+ * IBPB if this thread does not have !TIF_SPEC_IB.
+ */
+static inline void maybe_indirect_branch_prediction_barrier(void)
+{
+	if (static_key_enabled(&switch_mm_always_ibpb) ||
+	    (static_key_enabled(&switch_mm_cond_ibpb) &&
+	     test_thread_flag(TIF_SPEC_IB)))
+		indirect_branch_prediction_barrier();
+}
+
 #endif
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 6296e1ebed1d..737826bf974c 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -66,10 +66,12 @@ u64 __ro_after_init x86_amd_ls_cfg_ssbd_mask;
 
 /* Control conditional STIBP in switch_to() */
 DEFINE_STATIC_KEY_FALSE(switch_to_cond_stibp);
-/* Control conditional IBPB in switch_mm() */
+/* Control conditional IBPB in switch_mm() and vmcs/vmcb load */
 DEFINE_STATIC_KEY_FALSE(switch_mm_cond_ibpb);
-/* Control unconditional IBPB in switch_mm() */
+EXPORT_SYMBOL_GPL(switch_mm_cond_ibpb);
+/* Control unconditional IBPB in switch_mm() and vmcs/vmcb load */
 DEFINE_STATIC_KEY_FALSE(switch_mm_always_ibpb);
+EXPORT_SYMBOL_GPL(switch_mm_always_ibpb);
 
 /* Control MDS CPU buffer clear before returning to user space */
 DEFINE_STATIC_KEY_FALSE(mds_user_clear);
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index bd4c64b362d2..7762ca1197b5 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -1302,7 +1302,7 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 
 	if (sd->current_vmcb != svm->vmcb) {
 		sd->current_vmcb = svm->vmcb;
-		indirect_branch_prediction_barrier();
+		maybe_indirect_branch_prediction_barrier();
 	}
 	if (kvm_vcpu_apicv_active(vcpu))
 		__avic_vcpu_load(vcpu, cpu);
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 04d170c4b61e..baaf658263b5 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -1270,7 +1270,7 @@ void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu,
 		 * The L1 VMM can protect itself with retpolines, IBPB or IBRS.
 		 */
 		if (!buddy || WARN_ON_ONCE(buddy->vmcs != prev))
-			indirect_branch_prediction_barrier();
+			maybe_indirect_branch_prediction_barrier();
 	}
 
 	if (!already_loaded) {
-- 
2.30.1 (Apple Git-130)